Signal recognition method and signal recognition device

Abstract

In this signal recognition method: an image in front of a vehicle 1 is acquired by a sensor mounted on the vehicle (S1); an indication state of a traffic light contained in the image is determined (S5); a determination result of the indication state of the traffic light is stored (S6); when the stored indication state has changed in an appropriate order, a weighting for the indication state after the change is configured to be larger than a weighting configured for the indication state before the change (S11 to S16); and the indication state of the traffic light is recognised on the basis of weighted values obtained by multiplying the number of times each stored indication state has been determined by the weighting set for each indication state (S17, S18).

Description

Signal recognition method and signal recognition device

[0001] The present invention relates to a signal recognition method and a signal recognition device.

[0002] The traffic light recognition device described in Patent Document 1 below accumulates the results of determining the light color of multiple frames of captured images of a traffic light, and recognizes the light color of the traffic light by majority voting of the accumulated multiple frames.

[0003] International Publication No. 2019 / 177019

[0004] For example, when a traffic light's indication state (lighting state) changes, it is necessary to recognize the traffic light early. The majority voting process described in Patent Document 1 may result in delayed recognition of the traffic light's indication state. The present invention aims to shorten the time required to recognize a change in the indication state when determining the indication state of a traffic light based on an image of the traffic light and recognizing the indication state of the traffic light based on the accumulated results of the indication state determination.

[0005] In one aspect of the signal recognition method of the present invention, an image in front of the vehicle 1 is acquired using a sensor mounted on the vehicle, the indication state of the traffic light contained in the image is determined, the determination results of the indication state of the traffic light are accumulated, and if the accumulated indication states change in an appropriate order, a weight for the indication state after the change is set higher than the weight set for the indication state before the change, and the indication state of the traffic light is recognized based on a weighted value obtained by multiplying the accumulated number of determinations for each indication state by the weight set for each indication state.

[0006] According to the present invention, when determining the indication state of a traffic light based on an image of the traffic light and recognizing the indication state of the traffic light based on the accumulated results of the determination of the indication state, the time required to recognize a change in the indication state can be reduced. The objects and advantages of the present invention are realized and attained by using the elements and combinations set forth in the claims. It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit the invention as defined by the claims.

[0007] FIG. 1 is a schematic configuration diagram of an example of a driving assistance device according to an embodiment; FIG. 2 is a block diagram of an example of the functional configuration of a controller; FIG. 3 is an explanatory diagram (part 4) of a method for recognizing the indication state of a traffic light; FIG. 4 is an explanatory diagram (part 5) of a method for recognizing the indication state of a traffic light; FIG. 5 is an explanatory diagram of a situation in which a traffic light included in a camera image switches; and FIG. 6 is a flowchart of an example of a signal recognition method according to an embodiment.

[0008] (Configuration) Fig. 1 is a schematic diagram of an example of a driving assistance device according to an embodiment. A vehicle 1 is equipped with a driving assistance device 10 that assists in driving the vehicle 1. The driving assistance control performed by the driving assistance device 10 includes autonomous driving control that automatically drives the vehicle 1 without the involvement of a driver based on the driving environment around the vehicle 1, and driving assistance control that assists in driving the vehicle 1 by controlling at least one of driving, braking, and steering of the vehicle 1. The driving assistance control may be, for example, automatic steering, automatic braking, preceding vehicle following control, constant speed driving control, lane keeping control, merging assistance control, etc.

[0009] The driving assistance device 10 may recognize the indication state (i.e., the lighting state) of a traffic signal present ahead of the vehicle 1 in autonomous driving control or driving assistance control. The driving assistance device 10 stops or starts the vehicle 1, or causes the vehicle 1 to travel, according to the recognition result of the indication state. The driving assistance control by the driving assistance device 10 may include outputting a visual or auditory signal to notify an occupant of the vehicle 1 (e.g., the driver) of the recognition result of the indication state. The driving assistance device 10 of the embodiment is an example of a "signal recognition device" as defined in the claims.

[0010] The driving assistance device 10 includes an external environment sensor 11, a vehicle sensor 12, a positioning device 13, a map database (map DB) 14, a human machine interface (HMI) 15, a steering actuator (steering ACTR) 17a, an accelerator actuator (accelerator ACTR) 17b, a brake actuator (brake ACTR) 17c, and a controller 18. The external environment sensor 11 includes a plurality of different types of object detection sensors that detect objects around the vehicle 1, such as a camera mounted on the vehicle 1, a laser radar, a millimeter-wave radar, and a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging). The external environment sensor 11 outputs surrounding environment information, which is information about the detected surrounding environment of the vehicle 1, to the controller 18. For example, the external environment sensor 11 includes a front camera that captures an image ahead of the vehicle 1. The front camera captures an image of the area in front of the vehicle 1 and outputs the captured image (hereinafter sometimes referred to as a “camera image”) to the controller 18 .

[0011] The vehicle sensors 12 are mounted on the vehicle 1 and detect various information (vehicle signals) obtained from the vehicle 1. The vehicle sensors 12 include, for example, a vehicle speed sensor that detects the vehicle speed of the vehicle 1, a wheel speed sensor that detects the rotational speed of the tires of the vehicle 1, a three-axis acceleration sensor that detects the acceleration and deceleration in three axial directions of the vehicle 1, a steering angle sensor that detects the steering angle of the steering wheel, a turning angle sensor that detects the turning angle of the steered wheels, a gyro sensor that detects the angular velocity of the vehicle 1, a yaw rate sensor that detects the yaw rate, an accelerator sensor that detects the accelerator opening of the vehicle 1, and a brake sensor that detects the amount of brake operation by the driver.

[0012] The positioning device 13 includes a Global Navigation Satellite System (GNSS) receiver and receives radio waves from multiple navigation satellites to measure the current position of the vehicle 1. The GNSS receiver may be, for example, a Global Positioning System (GPS) receiver. The positioning device 13 may be, for example, an Inertial Measurement Unit (IMU). The map database 14 stores road map data. For example, the map database 14 may store high-precision map data (hereinafter simply referred to as a "high-definition map") suitable as map information for autonomous driving. The map database 14 may also store map data for navigation (hereinafter simply referred to as a "navigation map").

[0013] The HMI 15 is an interface device that exchanges information between the driving assistance device 10 and the occupant of the vehicle 1. The HMI 15 includes a display device (e.g., a display screen of a navigation system) that can be seen by the occupant, and a speaker and a buzzer for outputting warning sounds, notification sounds, and audio information. The controller 18 is an electronic control unit (ECU) that performs driving assistance control of the vehicle 1 based on surrounding environment information (including camera images) from the external sensors 11, vehicle signals from the vehicle sensors 12, measurement results of the current position of the vehicle 1 by the positioning device 13, and road map data in the map database 14.

[0014] The controller 18 includes a processor 18a and peripheral components such as a storage device 18b. The processor 18a may be, for example, a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit). The storage device 18b may include a semiconductor storage device, a magnetic storage device, an optical storage device, or the like. The storage device 18b may include a register, a cache memory, and memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory) used as main storage devices. The functions of the controller 18 described below are realized, for example, by the processor 18a executing a computer program stored in the storage device 18b.

[0015] The controller 18 may be formed by dedicated hardware for executing each information processing described below. For example, the controller 18 may include a functional logic circuit configured in a general-purpose semiconductor integrated circuit. For example, the controller 18 may include a programmable logic device (PLD) such as a field-programmable gate array (FPGA). The steering actuator 17a controls the steering direction and steering amount of the steering mechanism of the vehicle 1 in response to a control signal from the controller 18. The accelerator actuator 17b adjusts the accelerator opening of the drive device, such as the engine or drive motor, in response to a control signal from the controller 18. The brake actuator 17c activates a braking device in response to a control signal from the controller 18. For example, in autonomous driving control or driving assistance control by the driving assistance device 10, the controller 18 may stop or start the vehicle 1, or cause the vehicle 1 to move, by controlling the steering actuator 17a, accelerator actuator 17b and / or brake actuator 17c according to the indication status of a traffic signal ahead of the vehicle 1.

[0016] 2 is a block diagram showing an example of the functional configuration of the controller 18. The controller 18 includes an image acquisition unit 20, a signal position acquisition unit 21, a self-position acquisition unit 22, a region of interest (ROI) setting unit 23, an instruction state determination unit 24, a determination result storage unit 25, an instruction state recognition unit 26, a recognition result output unit 27, and a vehicle motion control unit 28. The image acquisition unit 20 acquires a camera image from the front camera of the external environment sensor 11.

[0017] The traffic light position acquisition unit 21 acquires position information of a traffic light (sometimes referred to as a "target traffic light" in the following description) that regulates (controls) the travel of the vehicle 1 based on the road map data in the map database 14. For example, the traffic light position acquisition unit 21 may acquire, as the position information of the target traffic light, the position information of the traffic light that is in front of and closest to the vehicle 1, among the traffic lights that regulate road traffic in the lane in which the vehicle 1 is traveling. The self-position acquisition unit 22 acquires position information of the current position (including attitude) of the vehicle 1 from the positioning device 13. The self-position acquisition unit 22 may calculate the current position of the vehicle 1 based on surrounding environment information from the external sensor 11.

[0018] The ROI setting unit 23 sets an ROI, which is an area in the camera image where the image of the target traffic light is located, based on the position information of the target traffic light acquired by the signal position acquisition unit 21, the position information of the current position of the vehicle 1 acquired by the self-position acquisition unit 22, and the shooting conditions of the front camera (the viewpoint position, optical axis direction, and angle of view of the front camera). The indication state determination unit 24 determines the indication state of the target traffic light by performing image recognition processing on the image of the target traffic light located within the ROI. For example, the indication state determination unit 24 determines the indication state of the target traffic light for each processing cycle (hereinafter, sometimes simply referred to as "processing cycle") of the recognition processing of the traffic light indication state by the driving assistance device 10. For example, the indication state determination unit 24 may determine the indication state using pattern matching, machine learning, or deep learning.

[0019] The determination result storage unit 25 stores (accumulates) in a buffer a time series of the determination results of the indication state by the indication state determination unit 24. For example, the determination result storage unit 25 stores the determination results output from the indication state determination unit 24 in each processing cycle in the order in which they were output in the buffer. For example, the determination result storage unit 25 may accumulate the determination results in the buffer using a first-in, first-out (FIFO) method. That is, the determination result storage unit 25 may store a finite number of determination results in the buffer, and when the number of determination results accumulated in the buffer reaches an upper limit, the oldest determination result (i.e., the determination result stored first) may be discarded, and the latest determination result may be stored in the freed storage area.

[0020] 3A is a schematic diagram of a target traffic signal St1. The target traffic signal St1 includes lamps Pb1, Py, Pr, and Pb2. For example, the light colors of lamps Pb1, Py, and Pr may be blue (or green), yellow, and red, respectively. Lamp Pb2 may also have a blue arrow light. The indication states of the target traffic signal St1 may include an indication state "B1" in which only lamp Pb1 is lit, an indication state "Y" in which only lamp Py is lit, an indication state "B2" in which lamps Pr and Pb2 are lit simultaneously, and an indication state "R" in which only lamp Pr is lit.

[0021] For example, the indication state "B1" may be a state in which the traffic signal allows traffic users to proceed, the indication state "R" may be a state in which the traffic signal instructs traffic users to stop, and the indication state "B2" may be a state in which the traffic signal allows traffic users to proceed in the direction of the arrow. Also, the indication state Y indicates a transition state from the indication state "B1" to the indication state "B2" or a transition state from the indication state "B2" to the indication state "R", and may be a state in which the traffic signal instructs traffic users to stop. The target traffic signal St1 changes its indication state in the order of "B1" → "Y" → "B2" → "Y" → "R", and returns to the initial indication state "B1" after the indication state "R". Now, at time t ｉ Assume that the indication state of the target traffic light St1 changes from "B1" to "Y" as shown in FIG. 3A.

[0022] FIG. 3B shows the time t ｉ ~t ｉ＋６ 1 is a schematic diagram showing the state of the buffer BU of the determination result storage unit 25 at time t ｉ＋１ ~t ｉ＋６ are at time t ｉ The time is one to six processing cycles later than the current time. Reference symbol m9 indicates a storage area in which the determination result of the instruction state determination unit 24 in the current processing cycle is stored, and reference symbols m8 to m1 indicate storage areas in which determination results at times one to eight cycles before the current processing cycle are stored. Note that the areas m1 to m9 do not represent specific memory addresses, and a circular buffer may be used as the buffer BU. That is, a fixed array in memory may be prepared as the buffer BU, and a pointer indicating the address where the oldest determination result is stored and a pointer indicating the address where the latest determination result is stored may be circulated in the array. The example of the buffer BU in FIG. 3B stores a total of nine determination results. At time t ｉ Since the indication state is switched from indication state "B1" to indication state "Y" at time t ｉ In this example, the indication state after the change "Y" is stored in the area m9 that stores the latest determination area, and the indication state before the change "B1" is stored in areas m1 to m8.

[0023] At a later time t ｉ＋１ At time t ｉ＋２ At time t, the instruction state "Y" is stored in areas m7 to m9, and the instruction state "B1" is stored in areas m1 to m6. ｉ＋３ At time t, the instruction state "Y" is stored in areas m6 to m9, and the instruction state "B1" is stored in areas m1 to m5. ｉ＋４ At time t, the instruction state "Y" is stored in the areas m5 to m9, and the instruction state "B1" is stored in the areas m1 to m4. ｉ＋５ At time t, the instruction state "Y" is stored in the areas m4 to m9, and the instruction state "B1" is stored in the areas m1 to m3. ｉ＋６ In this example, the indication status "Y" is stored in the areas m3 to m9, and the indication status "B1" is stored in the areas m1 and m2.

[0024] The determination result storage unit 25 sets a weight for each of the indication states to be stored in the buffer BU, and stores the set weight in association with each of the indication states. At this time, when storing the determination result output from the indication state determination unit 24 in the current processing cycle in area m9 of the buffer BU, the determination result storage unit 25 determines whether the determination result output in the current processing cycle (i.e., the determination result stored in area m9) has changed from the determination result output from the indication state determination unit 24 in the immediately preceding processing cycle (i.e., the determination result stored in area m8). If the determination result has not changed, the determination result storage unit 25 sets the same weight as the weight set for the determination result in the immediately preceding processing cycle for the latest determination result. In the example of FIG. 3B , at time t ｉ The determination results from 1 to 8 cycles before the current state remain unchanged at the indication state "B1." Therefore, the same weight "1" is assigned to the determination results of the indication state "B1."

[0025] When the determination results change, the determination result storage unit 25 determines whether the order of changes in the determination results (i.e., the order of changes in the determination results in time series) is appropriate (correct) as the order of changes in the indication states of the traffic signal. If the order of changes in the determination results is appropriate, the determination result storage unit 25 sets a weight for the indication state after the change that is greater than the weight set for the indication state before the change. In the example of FIG. 3B , since the order of changes from the indication state "B1" to the indication state "Y" is appropriate, the determination result storage unit 25 sets a weight of "1.5" for the indication state after the change that is greater than the weight of "1" set for the indication state "B1" before the change.

[0026] For example, the determination result storage unit 25 may set the weight for the post-change indication state "Y" to the product (1 x k1) of a predetermined first weighting coefficient k1 set to a value greater than "1" and the weight "1" for the pre-change indication state "B1." For example, in the example of FIG. 3B, the first weighting coefficient k1 is set to "1.5." See FIG. 2. The indication state recognition unit 26 calculates a weighted value by multiplying the number of determinations for each indication state stored in the buffer BU by the weight set for each indication state. In the example of FIG. 3B,ｉ At time t, the number of judgments for "B1" stored in the buffer BU is "8" and the weight is "1". Therefore, the weighted value for the instruction state "B1" is calculated as 8 x 1 = 8. ｉ In this example, the number of determinations for the indication state "Y" stored in the buffer BU is "1" and the weight is "1.5." Therefore, the weighted value for the indication state "Y" is calculated as 1 x 1.5 = 1.5.

[0027] Similarly, at time t ｉ＋１ The weighted values ​​of the instruction states “B1” and “Y” at time t ｉ＋２ The weighted values ​​of the instruction states “B1” and “Y” at time t ｉ＋３ The weighted values ​​of the instruction states “B1” and “Y” at time t ｉ＋４ The weighted values ​​of the instruction states “B1” and “Y” at time t ｉ＋５ The weighted values ​​of the instruction states “B1” and “Y” at time t ｉ＋６ The weighted values ​​for the indication states "B1" and "Y" in the example are "2" and "10.5", respectively.

[0028] The indication state recognizing unit 26 recognizes the indication state of the target traffic light St1 based on the weighted value. For example, the indication state recognizing unit 26 may recognize the indication state for which the largest weighted value is calculated, out of the indication states "B1" and "Y" stored in the buffer BU, as the indication state of the target traffic light St1. For this reason, the indication state recognizing unit 26 may recognize the indication state for which the largest weighted value is calculated, out of the indication states "B1" and "Y" stored in the buffer BU, as the indication state of the target traffic light St1. ｉ ~t ｉ＋２ At time t ｉ＋３ ~t ｉ＋６ Therefore, the indication state recognizing unit 26 recognizes that the indication state of the target traffic light St1 has changed to "Y" at time t ｉ＋３ It can be recognized by.

[0029] On the other hand, if the instruction state is simply recognized based on the majority decision process of the instruction state determination counts stored in the buffer BU, the instruction state "Y" will reach the majority "5" at time t ｉ＋４ It is not possible to recognize that the indication state of the target traffic signal St1 has changed to "Y" until the indication state of the target traffic signal St1 is changed to "Y." In this way, according to the present invention, the change in the indication state of a traffic signal can be recognized more quickly than with a simple majority voting process. Note that the indication state recognition unit 26 may recognize, from among the indication states stored in the buffer BU, an indication state for which a weighted value equal to or greater than a predetermined threshold is calculated as the indication state of the target traffic signal St1.

[0030] Next, as shown in FIG. 4A, at time t ｉ＋６ Time t later than ｊ Assume that the indication state of the target traffic light St1 changes from "Y" to "B2" in the above example. The indication state "B2" after the change is an indication state in which the two lamps Pr and Pb2 are simultaneously lit. The determination result storage unit 25 determines that the order of the indication state change from "Y" to "B2" is appropriate. Therefore, the determination result storage unit 25 assigns a weight "3" to the indication state "B2" after the change, which is larger than the weight "1.5" assigned to the indication state "Y" before the change.

[0031] In this way, when the number of simultaneously lit lamps in the post-change indication state is two or more, the determination result storage unit 25 may set the weight for the post-change indication state "B2" to the product (1 × k2) of a second weighting coefficient k2, which is greater than the first weighting coefficient k1, and the weight "1.5" for the pre-change indication state "Y." For example, in the example of FIG. 4B , the second weighting coefficient k2 is set to "2." In this way, the determination result storage unit 25 may set a larger weighting coefficient the greater the number of simultaneously lit lamps in the post-change indication state. As a result, the weight set when the number of simultaneously lit lamps is greater is greater than the weight set when the number of simultaneously lit lamps is smaller. For example, if the weight of the pre-change indication state is "W," the weight set (k2 × W) when two or more lamps are simultaneously lit in the post-change indication state is greater than the weight set (k1 × W) when one or more lamps are lit.

[0032] As a result, when two or more lamps are lit simultaneously in the changed indication state, the change in indication state can be recognized more quickly than when only one lamp is lit in the changed indication state. For example, in the example of FIG. 4B, at time t ｊ The weighted values ​​of the instruction states “Y” and “B2” at time t ｊ＋１ The weighted values ​​of the instruction states “Y” and “B2” at time t ｊ＋２ The weighted values ​​of the instruction states “Y” and “B2” at time t ｊ＋３ The weighted values ​​of the instruction states “Y” and “B2” at time t ｊ＋４ The weighted values ​​of the instruction states "Y" and "B2" in this example are "6" and "15", respectively.

[0033] Therefore, the instruction state recognition unit 26 ｊ , t ｉ＋１ At time t ｊ＋２ ~t ｊ＋４ 3B, the indicator state recognition unit 26 recognizes that the indicator state of the target traffic light St1 is "B2." In this way, the indicator state recognition unit 26 can recognize the change in indicator state one processing cycle earlier than in the case of FIG. 3B, where one lamp is lit in the changed indicator state. Note that if the probability of erroneously recognizing that one lamp is lit is Pr, the probability of erroneously recognizing that N lamps are lit at the same time is Pr Ｎ Therefore, the probability that the indication state determination unit 24 will erroneously determine an indication state in which two or more lamps are lit simultaneously is smaller than the probability that it will erroneously determine an indication state in which one lamp is lit. Therefore, even if the second weighting coefficient k2 is set to be larger than the first weighting coefficient k1, the probability that the indication state recognition unit 26 will erroneously recognize an indication state in which two or more lamps are lit simultaneously can be reduced.

[0034] 5, in this example, time t ｉ At time t ｉ＋４Assume that the instruction state determination unit 24 erroneously determines that the instruction state is "B1" at time t ｉ＋４ In this case, the determination result storage unit 25 determines that the order of the change in the indication state from "Y" to "B1" is inappropriate. In this case, the determination result storage unit 25 sets the weight for the indication state after the change "B1" to the product (1.5 x k3) of a third weighting coefficient k3 smaller than the first weighting coefficient k1 and the weight "1.5" for the indication state before the change "Y." For example, in the example of FIG. 5, the third weighting coefficient k3 is set to "0.5."

[0035] As a result, the time t ｉ＋４ Of the indication states "B1" and "Y" stored in the buffer BU, the weighted value of the indication state "B1" (weight "1" x number of judgments "4" + weight "0.75" x number of judgments "1" = 4.57) is smaller than the weighted value of the indication state "Y" (weight "1.5" x number of judgments "4" = 6). As a result, even if the indication state judgment unit 24 makes an erroneous judgment, the indication state recognition unit 26 continues to recognize the indication state "Y" before the erroneous judgment occurred as the indication state of the target traffic light St1, thereby preventing erroneous recognition from occurring.

[0036] The third weighting coefficient k3 is preferably set so that the weight calculated by multiplying the weight for the pre-change indication state by the third weighting coefficient k3 is smaller than the weights set for the other accumulated indication states. In the example of FIG. 5 , the weight calculated by multiplying the weight "1.5" for the pre-change indication state "Y" by the third weighting coefficient k3 (1.5 x k3 = 0.75) is preferably set so that the weight calculated is smaller than the weights "1" and "1.5" set for the other accumulated indication states. This allows the weight "4.75" for the erroneously determined indication state "B1" to be smaller than the weight "6" for the correct indication state "Y." For example, the third weighting coefficient k3 may be set to the reciprocal of the second weighting coefficient k2 (1 / w2) or a value smaller than 1 / w2.

[0037] See Figure 6. When vehicle 1 travels from point P1 to point P2, the target traffic light included in the camera image changes from traffic light St1 to traffic light St2. The determination result storage unit 25 determines whether the target traffic light has changed. For example, the determination result storage unit 25 may determine whether the target traffic light has changed by determining whether an intersection has been passed, or by determining whether a stop line, intersection, or traffic light that is a stopping target for vehicle 1 has changed. When the target traffic light has changed, the indication state recognition unit 26 initializes the weight set for the indication state determination results stored in buffer BU to an initial value.

[0038] For example, the indication state recognition unit 26 initializes the weight value set for the determination result of the indication state of traffic light St2 that is first accumulated in the buffer BU after the target traffic light switches to traffic light St2 to an initial value (e.g., "1") that is smaller than the weight set for the determination result of the indication state of traffic light St1 that was last accumulated before the target traffic light switched to traffic light St2.

[0039] 2 , the recognition result output unit 27 outputs a visual or auditory signal to notify the occupants of the vehicle 1 of the recognition result of the indication state of the target traffic light St1 by the indication state recognition unit 26 via the HMI 15. In autonomous driving control or driving assistance control, the vehicle motion control unit 28 stops or starts the vehicle 1, or causes the vehicle 1 to travel, by controlling the steering actuator 17 a, accelerator actuator 17 b, and / or brake actuator 17 c in accordance with the indication state of the target traffic light based on the recognition result by the indication state recognition unit 26.

[0040] 7 is a flowchart of an example of a traffic light recognition method according to an embodiment. In step S1, the image acquisition unit 20 acquires a camera image from the front camera of the external sensor 11. In step S2, the self-position acquisition unit 22 acquires position information on the current position of the vehicle 1. In step S3, the traffic light position acquisition unit 21 acquires position information on the target traffic light. In step S4, the ROI setting unit 23 sets an ROI, which is an area in the camera image where the image of the target traffic light exists. In step S5, the indication state determination unit 24 determines the indication state of the target traffic light based on the image of the target traffic light that exists within the ROI.

[0041] In step S6, the determination result storage unit 25 deletes the history of the oldest determination result stored in the buffer BU, and stores the latest determination result determined in the current processing cycle in the buffer BU. In step S7, the determination result storage unit 25 determines whether the target traffic light in the camera image has changed to another traffic light. If the target traffic light has not changed (step S7: N), the process proceeds to step S9. If the target traffic light has changed (step S7: Y), the process proceeds to step S8.

[0042] In step S8, the judgment result storage unit 25 initializes the weight to be set for the newly stored indication state judgment result in the buffer to an initial value. Then, the process proceeds to step S17. In step S9, the judgment result storage unit 25 determines whether the indication state of the target traffic light has changed. If the indication state has not changed (step S9: N), the process proceeds to step S10. If the indication state has changed (step S9: Y), the process proceeds to step S11. In step S10, the judgment result storage unit 25 sets the weight to be set for the newly stored indication state judgment result in the buffer to the same value as the previous time. Then, the process proceeds to step S17.

[0043] In step S11, the determination result storage unit 25 determines whether the order of changes in the indication status is appropriate as the order of changes in the indication status of a traffic signal. If the order is not appropriate (step S11: N), the process proceeds to step S15. If the order is appropriate (step S11: Y), the process proceeds to step S12. In step S12, the determination result storage unit 25 determines whether the number of lit lamps in the indication status after the change is one. If the number of lit lamps is not one (step S12: N), the process proceeds to step S14. If the number of lit lamps is one (step S12: Y), the process proceeds to step S13.

[0044] In step S13, the determination result storage unit 25 sets a first weighting factor k1 as the weighting factor to be used in calculating the weight of the indication state newly stored in the buffer. Then, the process proceeds to step S16. In step S14, the determination result storage unit 25 sets a second weighting factor k2 as the weighting factor to be used in calculating the weight of the indication state newly stored in the buffer. Then, the process proceeds to step S16. In step S15, the determination result storage unit 25 sets a third weighting factor k3 as the weighting factor to be used in calculating the weight of the indication state newly stored in the buffer. Then, the process proceeds to step S16.

[0045] In step S16, the weight of the indication state newly stored in the buffer is calculated by multiplying the weight set for the indication state most recently stored in the buffer by the weighting coefficient set in one of steps S13 to S15. Then, the process proceeds to step S17. In step S17, the indication state recognition unit 26 calculates weighted values, and recognizes the indication state for which the maximum weighted value is calculated as the indication state of the target traffic light. In step S18, the recognition result output unit 27 outputs the recognition result by the indication state recognition unit 26. Then, the process ends.

[0046] (Effects of the Embodiment) (1) In the traffic light recognition method, an image ahead of the vehicle 1 is acquired using a sensor mounted on the vehicle, the traffic light status included in the image is determined, and the determination results of the traffic light status are accumulated. If the accumulated traffic light status changes in an appropriate order, a weight for the traffic light status after the change is set to be greater than the weight for the traffic light status before the change. The traffic light status is recognized based on a weighted value obtained by multiplying the accumulated number of determinations for each traffic light status by the weight set for each traffic light status. For example, the traffic light status may be recognized as the traffic light status if the weighted value is equal to or greater than a predetermined threshold value or if the traffic light status has the largest weighted value. This reduces the time required to recognize a change in the traffic light status.

[0047] (2) The weight of a second indication state, in which a greater number of lamps are simultaneously lit than in the first indication state, may be set to be greater than the weight of the first indication state. This reduces the time required to recognize that an indication state has been switched to in which two or more lamps are simultaneously lit. (3) If the previously accumulated indication state and the newly accumulated indication state are equal, the weight of the newly accumulated indication state may be set to be the same as the weight set for the previously accumulated indication state. This allows the same weighting to be applied when the indication state does not change.

[0048] (4) The weight for the indication state of the second traffic light that is initially accumulated after the traffic light included in the image switches from the first traffic light to the second traffic light may be initialized to an initial value that is smaller than the weight for the indication state of the first traffic light that was last accumulated before the traffic light included in the image switched from the first traffic light to the second traffic light. This makes it possible to recognize the indication state of the traffic light after the switch, regardless of the magnitude of the weight that was set before the traffic light to be recognized switched.

[0049] (5) If the accumulated indication states change in an appropriate order and the number of lights that are simultaneously lit in the indication state after the change is one, the weight for the indication state after the change may be set to a value obtained by multiplying the weight for the indication state before the change by a first weighting coefficient greater than 1. This reduces the time required to recognize that the indication state of the traffic signal has changed.

[0050] (6) If the accumulated indication states change in an appropriate order and the number of lamps that are simultaneously lit in the indication state after the change is two or more, the weight for the indication state after the change may be set to a value obtained by multiplying the weight for the indication state before the change by a second weighting factor that is greater than the first weighting factor. This can further reduce the time required to recognize an indication state with a short duration, such as an indication state in which two or more lamps are simultaneously lit.

[0051] (7) If the accumulated indication states change in an improper order, the weight for the indication state after the change may be set to a value obtained by multiplying the weight for the indication state before the change by a third weighting factor smaller than the first weighting factor. For example, the third weighting factor may be set so that the weight calculated by multiplying the weight for the indication state before the change by the third weighting factor is smaller than the weights set for the other accumulated indication states. This reduces the weight for the erroneously determined indication state, thereby preventing erroneous recognition.

[0052] All examples and conditional terms described herein are intended for educational purposes to aid the reader in understanding the present invention and the concepts provided by the inventor for the advancement of technology, and should be construed without limitation to the specifically described examples and conditions above, and the configuration of examples herein for illustrating the advantages and disadvantages of the present invention. Although the embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions, and alterations can be made thereto without departing from the spirit and scope of the present invention.

[0053] 1...vehicle, 10...driving assistance device, 11...external sensor, 12...vehicle sensor, 13...positioning device, 14...map database, 15...human-machine interface, 17a...steering actuator, 17b...accelerator actuator, 17c...brake actuator, 18...controller, 18a...processor, 18b...storage device, 20...image acquisition unit, 21...signal position acquisition unit, 22...self-position acquisition unit, 23...region of interest setting unit, 24...instruction state determination unit, 25...determination result storage unit, 26...instruction state recognition unit, 27...recognition result output unit, 28...vehicle motion control unit

Claims

1. A signal recognition method comprising: acquiring an image of an area ahead of the vehicle using a sensor mounted on the vehicle; determining the indication status of a traffic light included in the image; storing the determination results of the indication status of the traffic light; when the stored indication status changes in an appropriate order, setting a weight for the indication status after the change that is greater than the weight set for the indication status before the change; and recognizing the indication status of the traffic light based on a weighted value obtained by multiplying the stored number of determinations for each indication status by the weight set for each indication status.

2. A signal recognition method as described in claim 1, characterized in that the indication state in which the weighted value is equal to or greater than a predetermined threshold value or the indication state in which the weighted value is the largest is recognized as the indication state of the traffic light.

3. A signal recognition method as described in claim 1 or 2, characterized in that the weight of a second indication state in which a greater number of lamps are lit simultaneously than in the first indication state is set to be greater than the weight of the first indication state.

4. A signal recognition method as described in any one of claims 1 to 3, characterized in that if the previously accumulated indication state and the newly accumulated indication state are equal, a weight equal to the weight set for the previously accumulated indication state is set for the newly accumulated indication state.

5. A signal recognition method as described in any one of claims 1 to 4, characterized in that the weight for the indication state of the second traffic light that is first accumulated after the traffic light included in the image switches from a first traffic light to a second traffic light is initialized to an initial value that is smaller than the weight for the indication state of the first traffic light that was last accumulated before the traffic light included in the image switched from the first traffic light to the second traffic light.

6. A signal recognition method as described in claim 5, characterized in that if the accumulated indication states change in an appropriate order and the number of lamps that are simultaneously lit in the indication state after the change is one, the weight for the indication state after the change is set to a value obtained by multiplying the weight for the indication state before the change by a first weighting coefficient that is greater than 1.

7. A signal recognition method as described in claim 6, characterized in that if the accumulated indication states change in an appropriate order and the number of lamps that are lit simultaneously in the indication state after the change is two or more, the weight for the indication state after the change is set to a value obtained by multiplying the weight for the indication state before the change by a second weighting coefficient that is larger than the first weighting coefficient.

8. A signal recognition method as described in claim 6 or 7, characterized in that if the accumulated indication states change in an inappropriate order, the weight for the indication state after the change is set to a value obtained by multiplying the weight for the indication state before the change by a third weighting coefficient smaller than the first weighting coefficient.

9. A signal recognition method as described in claim 8, characterized in that the third weighting coefficient is set so that the weight calculated by multiplying the weight for the indication state before the change by the third weighting coefficient is smaller than the weights set for the other accumulated indication states.

10. A traffic light recognition device comprising: a sensor mounted on a vehicle; and a controller that acquires an image of the area in front of the vehicle captured by the sensor, determines the indication state of a traffic light contained in the image, stores the determination results of the indication state of the traffic light, and, if the stored indication states change in an appropriate order, sets a weight for the indication state after the change that is greater than the weight set for the indication state before the change, and recognizes the indication state of the traffic light based on a weighted value obtained by multiplying the stored number of determinations for each indication state by the weight set for each indication state.

Images