Traffic signal recognition method and traffic signal recognition device

Abstract

To provide a traffic signal recognition method capable of outputting more appropriate information as an indication of a traffic signal in front of a vehicle.SOLUTION: When a stopping position (predicted stopping position) when the car is decelerated at a predetermined deceleration is on the near side in the traveling direction of the car from a reference position (S107 "No"), an area corresponding to a first traffic signal on an image acquired from an imaging part is set as a gaze area (S108). On the other hand, when the predicted stopping position is behind the reference position in the vehicle traveling direction (S107 "Yes"), the area corresponding to the second traffic signal is set as the attention area (S109). Then, image processing is performed on the focus area to determine the indication of the traffic light ahead of the automobile (S110 to S115).SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a traffic light recognition method and a traffic light recognition device. [Background technology]

[0002] Conventionally, a driving assistance device has been proposed that, for example, when it is recognized that there are two or more traffic lights ahead of the vehicle, the nearest traffic light closest to the vehicle is red (proceed prohibited), and the next traffic light ahead of the nearest traffic light is green (proceed permitted), determines whether the vehicle exceeds the upper limit of the vehicle speed at which the vehicle can be stopped at the stop line of the nearest traffic light, and outputs a warning to the driver of the vehicle if it determines that the upper limit is exceeded (see, for example, Patent Document 1).The driving assistance device described in Patent Document 1 issues a warning, so that, for example, if the driver mistakenly recognizes the light color of the next traffic light as the light color of the next traffic light, the vehicle is prevented from entering an intersection where the nearest traffic light is installed despite the light color (red, proceed prohibited) of the nearest traffic light. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-92371 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the driving assistance device described in Patent Document 1 does not take into consideration the case where the light color of the nearest traffic light is green (proceeding permitted). Therefore, for example, if the light color of the nearest traffic light is green, no warning is issued even if the light color of the traffic light ahead is red (proceeding prohibited). Therefore, for example, if the distance between the nearest traffic light and the next traffic light is short and a stop line or the like of the next traffic light is located near the nearest traffic light, the timing of the warning may be too late relative to the distance between the stop line or the like and the vehicle, making it difficult for the vehicle to decelerate and stop smoothly (safely). The present disclosure aims to provide a traffic light recognition method and a traffic light recognition device that can output more appropriate information as the indication of a traffic light ahead of a vehicle. [Means for solving the problem]

[0005] A traffic light recognition method according to one aspect of the present disclosure uses an imaging unit mounted on a vehicle to capture an image in the direction of travel of the vehicle, acquires map information including the vehicle's current position and speed, and position information of traffic lights, acquires, based on the vehicle's current position and map information, the position of the stop line corresponding to a first traffic light that is located in the vehicle's direction of travel and is closest to the vehicle, or a reference position that is a position just before the intersection that corresponds to the first traffic light, calculates an expected stopping position that is the stopping position where the vehicle will stop if it is decelerated at a predetermined deceleration based on the vehicle's current position, compares the expected stopping position with the reference position, and sets it as a gaze area on the image, performs image processing on the gaze area to determine the current state of the traffic light ahead of the vehicle, and if the expected stopping position is closer to the vehicle than the reference position in the direction of travel of the vehicle, sets the gaze area to an area on the image that corresponds to the first traffic light, and if the expected stopping position is further back from the reference position in the direction of travel of the vehicle, sets the gaze area to an area on the image that corresponds to a second traffic light that is located in the vehicle's direction of travel and is the next closest to the vehicle after the first traffic light.

[0006] a reference position acquisition unit that acquires, based on the self-position acquired by the self-position acquisition unit and the map information acquired by the map information acquisition unit, a reference position that is the position of a stop line corresponding to a first traffic light that is located in the traveling direction of the vehicle and is the closest traffic light to the vehicle, or a position just before an intersection corresponding to the first traffic light, and a reference position acquisition unit that acquires, based on the self-position acquired by the self-position acquisition unit, the vehicle speed acquired by the vehicle speed acquisition unit, and the map information acquired by the map information acquisition unit, a reference position acquisition unit that acquires, based on the self-position acquired by the self-position acquisition unit, the vehicle speed acquired by the vehicle speed acquisition unit, and the map information acquired by the map information acquisition unit, a reference position that is the position of a stop line corresponding to a first traffic light that is located in the traveling direction of the vehicle and is the closest to the vehicle, or a position just before an intersection corresponding to the first traffic light; a gaze area setting unit that compares the predicted stopping position calculated by the limit position calculation unit with the reference position acquired by the reference position acquisition unit to set a gaze area on the image; and an indication determination unit that performs image processing on the gaze area set by the gaze area setting unit to determine the indication of a traffic light ahead of the vehicle. If the predicted stopping position is closer to the front of the reference position in the direction of travel of the vehicle, the gaze area setting unit sets an area on the image corresponding to the first traffic light as the gaze area, and if the predicted stopping position is further back than the reference position in the direction of travel of the vehicle, it sets an area on the image corresponding to a second traffic light that is located in the direction of travel of the vehicle and is the second closest to the vehicle after the first traffic light as the gaze area. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to provide a traffic light recognition method and a traffic light recognition device that can output more appropriate information as the indication of a traffic light ahead of a vehicle. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing a schematic configuration of a vehicle control system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a functional configuration of a controller. [Figure 3] 10 is a flowchart showing the operation of a data acquisition unit and the like. [Figure 4A]FIG. 10 is a diagram showing a case where a gaze area is set in the area of ​​a first traffic light on an image. [Figure 4B] FIG. 10 is a diagram showing a case where a gaze area is set in the area of ​​a second traffic light on an image. [Figure 5] FIG. 1 is a diagram illustrating an operation of a vehicle control system. [Figure 6] FIG. 1 is a diagram illustrating an operation of a vehicle control system. [Figure 7] 10 is a flowchart showing the operation of a fixation area setting unit and the like of the second embodiment. [Figure 8] FIG. 10 is a diagram illustrating a functional configuration of a controller according to a third embodiment. [Figure 9] 10 is a flowchart showing the operation of a route acquisition unit and the like. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the drawings are schematic and may differ from the actual embodiments. Furthermore, the embodiments of the present disclosure shown below are examples of devices and methods for embodying the technical ideas of the present disclosure, and the technical ideas of the present disclosure do not limit the structure, arrangement, etc. of the components to those described below. Various modifications can be made to the technical ideas of the present disclosure within the technical scope defined by the claims.

[0010] (Vehicle control system configuration) FIG. 1 is a diagram showing a schematic configuration of a vehicle control system 1 according to a first embodiment. In the first embodiment, as shown in FIG. 1, a traffic light recognition device 2 is applied to a vehicle control system 1 that controls a vehicle Ce based on the aspect of a traffic light ahead of the vehicle Ce. Examples of control of the vehicle Ce include driving control that causes the vehicle Ce to autonomously drive according to a predetermined planned driving route, assistance control that assists the driving operation of an occupant of the vehicle Ce, and warning control that issues a warning to the occupant of the vehicle Ce. The first embodiment illustrates a case where warning control is performed to prompt the occupant to brake. The traffic light aspect is the content of the instruction indicated by the traffic light color, and indicates "proceed" when the traffic light color is green and "proceed not" when the traffic light color is red or yellow. The vehicle control system 1 includes a traffic light recognition device 2 and a vehicle control device 3. The traffic light recognition device 2 is mounted on a vehicle Ce, and includes an imaging unit 4, a vehicle position acquisition unit 5, a vehicle speed acquisition unit 6, a map information acquisition unit 7, and a controller 8.

[0011] The imaging unit 4 captures an image in the traveling direction of the vehicle Ce. As the imaging unit 4, for example, a camera having a solid-state imaging element such as a CCD image sensor or a CMOS image sensor can be used. The number of imaging units 4 (cameras) may be one or more. The imaging range of the imaging unit 4 is set by the focal length of the lens, the angle of view, and the vertical and horizontal angles of the optical axis of the lens. Data of the captured image (image data) is output to the controller 8. The self-position acquisition unit 5 acquires the self-position of the vehicle Ce. As the self-position acquisition unit 5, for example, a GPS receiver that receives radio waves from multiple navigation satellites to measure the current position of the vehicle Ce can be used. Also, for example, a method of calculating the self-position using output signals from surrounding sensors such as laser radar, millimeter-wave radar, cameras, and LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) mounted on the vehicle Ce and map information can be used. The acquired data of the self-position (position data) is output to the controller 8. The vehicle speed acquisition unit 6 acquires the vehicle speed of the vehicle Ce. For example, a CAN (Controller Area Network) interface that acquires the vehicle speed from a CAN bus of the vehicle Ce can be used as the vehicle speed acquisition unit 6. The acquired vehicle speed data (vehicle speed data) is output to the controller 8.

[0012] The map information acquisition unit 7 acquires map information indicating the structure of the road on which the vehicle Ce is traveling. Examples of map information that can be used include road structure information (such as lane position, lane connection, and lane relative position), traffic light position information, traffic light type, position of stop lines corresponding to traffic lights, position of intersections corresponding to traffic lights, and position of pedestrian crossings corresponding to traffic lights. Examples of map information acquisition unit 7 that can be used include an interface device that acquires map information from a map database (not shown) that is pre-installed in the vehicle Ce and stores map information, or a communication device that acquires map information from an external map data server using wireless communication (such as vehicle-to-vehicle communication, road-to-vehicle communication, or public mobile communication network). The acquired map information is output to controller 8.

[0013] The controller 8 is a computer that determines the current state of a traffic light ahead of the vehicle Ce based on the outputs of the imaging unit 4, the vehicle position acquisition unit 5, the vehicle speed acquisition unit 6, and the map information acquisition unit 7. The controller 8 includes a processor 8a and peripheral components such as a storage device 8b. The processor 8a may be, for example, a central processing unit (CPU) or a micro processing unit (MPU). The storage device 8b may be, for example, a semiconductor storage device, a magnetic storage device, or an optical storage device. The storage device 8b may include memory such as a register, a cache memory, or a ROM or RAM used as a main storage device. The storage device 8b is used to store computer programs and the like, and to implement a first buffer 8c for storing the light color determination results of the first traffic light T1 (described later), and a second buffer 8d for storing the light color determination results of the second traffic light T2 (described later). The first buffer 8c and the second buffer 8d may be, for example, ring buffers having a storage area capable of storing a predetermined number of data ML (e.g., nine data items) and deleting the oldest data as the newest data is stored. Each function of the controller 8 described below (see FIG. 2) is realized, for example, by the processor 8a executing a computer program stored in the storage device 8b. The controller 8 may be formed of dedicated hardware for executing each of the information processes described below. For example, the controller 8 may be configured to include a functional logic circuit set in a general-purpose semiconductor integrated circuit. For example, the controller 8 may include a PLD (Programmable Logic Device) such as an FPGA.

[0014] The vehicle control device 3 controls the vehicle Ce based on the indication of the traffic light ahead of the vehicle Ce output from the traffic light recognition device 2. In the first embodiment, an example is shown in which warning control is performed as the control of the vehicle Ce. In the warning control, if the indication output from the traffic light recognition device 2 is "no proceeding", a warning is issued to prompt the driver of the vehicle Ce to apply the brakes. For example, a voice message output from a speaker installed inside the vehicle can be used as the warning. On the other hand, if the indication output from the traffic light recognition device 2 is "permitted to proceed", no warning is issued.

[0015] Next, each function of the controller 8 will be described in detail. As shown in Fig. 2, the controller 8 includes a data acquisition unit 9, a reference position acquisition unit 10, a limit position calculation unit 11, a gaze area setting unit 12, an aspect determination unit 13, a light color determination unit 14, a determination result storage unit 15, and a light aspect output unit 16. Fig. 2 is a diagram showing the functional configuration of the controller 8. Fig. 3 is a flowchart showing the operation of the data acquisition unit 9 and the like. The operation shown in Fig. 3 is repeatedly executed every time a predetermined time (for example, 10 ms) has elapsed. The data acquisition unit 9 acquires image data, vehicle speed data, and position information output from the imaging unit 4, vehicle speed acquisition unit 6, and self-position acquisition unit 5 (S101, S102, S103 in FIG. 3). The reference position acquisition unit 10 acquires a reference position Pth to be compared with an expected stop position Pe, which will be described later, based on the position data and vehicle speed data acquired by the data acquisition unit 9 and the map information output from the map information acquisition unit 7. As an example, the reference position acquisition unit 10 acquires the position of a traffic light (hereinafter also referred to as a "first traffic light T1") located in the traveling direction of the vehicle Ce and closest to the vehicle Ce from the map information based on the vehicle speed data and the position data (S104 in FIG. 3), and acquires a reference position Pth from the map information based on the acquired position of the first traffic light T1 (S105 in FIG. 3). The first traffic light T1 can be, for example, a traffic light located above a lane traveling in the same direction as the traveling direction of the vehicle Ce. In the first embodiment, as shown in FIG. 5, a case where the first traffic light T1 is located near an intersection is illustrated. Furthermore, the reference position Pth can be, for example, the position of the stop line corresponding to the first traffic light T1 or a position just before the intersection corresponding to the first traffic light T1. The stop line corresponding to the first traffic light T1 may be, for example, a stop line drawn closer to the vehicle Ce in the direction of travel than the first traffic light T1, indicating the position where the leading vehicle must stop when the light color of the first traffic light T1 is red (progression is prohibited).The position before the intersection corresponding to the first traffic light T1 may be, for example, a boundary between the lane of the vehicle Ce and the intersection, closer to the intersection in the direction of travel of the vehicle Ce.

[0016] The limit position calculation unit 11 calculates a stopping position (hereinafter also referred to as "expected stopping position Pe") when the vehicle Ce is decelerated at a predetermined deceleration, based on the position data and vehicle speed data acquired by the data acquisition unit 9 and the map information output from the map information acquisition unit 7. As an example, the limit position calculation unit 11 calculates a stopping distance until the vehicle Ce stops based on the vehicle speed indicated by the vehicle speed data and the predetermined deceleration, and determines the position moved by the stopping distance along the lane indicated by the map information from the current position of the vehicle Ce indicated by the position data as the expected stopping position Pe (S106 in FIG. 3). As the predetermined deceleration, for example, a maximum deceleration at which the occupants of the vehicle Ce and occupants of surrounding vehicles do not feel uncomfortable due to deceleration, or a maximum deceleration that the vehicle Ce can physically generate, can be used.

[0017] The gaze area setting unit 12 compares the predicted stop position Pe calculated by the limit position calculation unit 11 with the reference position Pth acquired by the reference position acquisition unit 10, and sets a gaze area ROI (Region of Interest) on the image captured by the image capture unit 4. As an example, as shown in FIG. 4A, if the predicted stop position Pe is closer to the reference position Pth in the traveling direction of the vehicle Ce ("No" in S107 of FIG. 3), the gaze area setting unit 12 sets the gaze area ROI in an area on the image corresponding to the first traffic light T1 (S108 of FIG. 3). For example, based on the vehicle Ce's own position, map information (the position of the first traffic light T1), and the installation state (angle, etc.) of the image capture unit 4, the gaze area setting unit 12 estimates the area on the image where the first traffic light T1 is captured, and sets the gaze area ROI in the estimated area. On the other hand, as shown in FIG. 4B, if the predicted stop position Pe is located further back in the traveling direction of the vehicle Ce than the reference position Pth ("Yes" in S107 of FIG. 3), the gaze area setting unit 12 sets the region corresponding to the second traffic light T2 on the image captured by the image capture unit 4 as the gaze area ROI (S109 of FIG. 3). For example, the region in the image in which the second traffic light T2 is captured is estimated based on the vehicle Ce's current position, map information (the position of the second traffic light T2), and the installation state of the image capture unit 4, and the gaze area ROI is set in the estimated region. As the second traffic light T2, for example, a traffic light located in the traveling direction of the vehicle Ce and next closest to the vehicle Ce after the first traffic light T1 can be used. As the second traffic light T2, for example, a traffic light located above the lane traveling in the same direction as the traveling direction of the vehicle Ce can be used. In the first embodiment, as shown in FIG. 5, a case is illustrated in which the second traffic light T2 is located near the intersection that appears next to the intersection of the first traffic light T1.

[0018] The aspect determination unit 13 determines the aspect of the traffic light ahead of the vehicle Ce by performing image processing on the fixation area ROI set by the fixation area setting unit 12. The aspect determination unit 13 includes, for example, a light color determination unit 14, a determination result storage unit 15, and a light aspect output unit 16. The light color determination unit 14 performs image processing on the gaze area ROI set by the gaze area setting unit 12 to determine the light color of a traffic light within the gaze area ROI. As an example, the light color determination unit 14 searches for a traffic light within the gaze area ROI and detects the light color of the searched traffic light (S110 in FIG. 3). The method for detecting the light color of a traffic light is not limited as long as it is a method capable of detecting light color, such as pattern matching, machine learning, deep learning, etc. The determination result storage unit 15 stores the light color determination result obtained by the light color determination unit 14 in the first buffer 8c or the second buffer 8d (S111 in FIG. 3). At that time, if the gaze region ROI set by the gaze region setting unit 12 is the region corresponding to the first traffic light T1, the determination result storage unit 15 stores the determination result obtained by the light color determination unit 14 (the determination result of the light color of the traffic light within the gaze region ROI) in the first buffer 8c and deletes the oldest data from the second buffer 8d. As a result, the light color determination result for the first traffic light T1 is stored in the first buffer 8c. On the other hand, if the gaze region ROI set by the gaze region setting unit 12 is the region corresponding to the second traffic light T2, the determination result storage unit 15 stores the determination result obtained by the light color determination unit 14 in the second buffer 8d and deletes the oldest data from the first buffer 8c. As a result, the light color determination result for the second traffic light T2 is stored in the second buffer 8d.

[0019] The light aspect output unit 16 determines the aspect of the traffic light ahead of the vehicle Ce based on the light color determination results stored in the first buffer 8c and the second buffer 8d. As an example, the light aspect output unit 16 determines whether the fixation area ROI set by the fixation area setting unit 12 corresponds to the second traffic light T2 (S112 in FIG. 3). If it is determined that the fixation area ROI does not correspond to the second traffic light T2 ("No" in S112 in FIG. 3), the light aspect output unit 16 performs a majority vote on the light color determination results stored in the first buffer 8c and determines that the indication indicated by the light color determination result most frequently stored in the first buffer 8c is the aspect of the traffic light ahead of the vehicle Ce (S113 in FIG. 3). For example, if "green" is most frequently stored in the first buffer 8c as the light color determination result, the light aspect output unit 16 determines that the aspect of the traffic light ahead of the vehicle Ce is "proceed." Similarly, if the first buffer 8c stores the most "red" or most "yellow" light color as the light color determination result, the system determines that the traffic light ahead of vehicle Ce indicates "progress is not permitted." The determination result of the traffic light color is output to the vehicle control device 3.

[0020] On the other hand, if the light aspect output unit 16 determines that the gaze area ROI set by the gaze area setting unit 12 is an area corresponding to the second traffic light T2 ("Yes" in S112 of FIG. 3), it determines whether the number of light color determination results stored in the second buffer 8d is equal to or greater than half of the predetermined number ML (S114 of FIG. 3). If it determines that the number is less than half of the predetermined number ML ("No" in S114 of FIG. 3), it determines the aspect of the traffic light ahead of the vehicle Ce based on the light color determination results stored in the first buffer 8c (S113 of FIG. 3). For example, it performs a majority vote process on the light color determination results stored in the first buffer 8c, and determines that the indication indicated by the light color determination result that is most frequently stored in the first buffer 8c is the aspect of the traffic light ahead of the vehicle Ce. On the other hand, if it is determined that the number is equal to or greater than half of the predetermined number ML ("Yes" in S114 of FIG. 3), the state of the traffic light ahead of the vehicle Ce is determined based on the light color determination results stored in the second buffer 8d (S115 of FIG. 3). For example, a majority vote process is performed on the light color determination results stored in the second buffer 8d, and the indication indicated by the light color determination result that is most frequently stored in the second buffer 8d is determined to be the state of the traffic light ahead of the vehicle Ce.

[0021] (Vehicle control system operation) Next, we will explain the operation of the vehicle control system 1. Figures 5 and 6 are diagrams showing the operation of the vehicle control system 1. The operation of the vehicle control system 1 shown in Figures 5 and 6 is realized by repeating the operation shown in the flowchart of Figure 3 at predetermined time intervals. First, as shown at time t1 in FIG. 5, assume that while vehicle Ce is traveling, a first traffic light T1 and a second traffic light T2 appear ahead of vehicle Ce, with the first traffic light T1 having a green light and the second traffic light T2 having a red light. At time t1 in FIG. 5, the predicted stop position Pe is located closer to the reference position Pth in the traveling direction of vehicle Ce ("No" in S107 in FIG. 3), so traffic light recognition device 2 sets the region on the image corresponding to first traffic light T1 as the gaze region ROI (S108 in FIG. 3; FIG. 4A). As a result, the light color determination results (blue) are sequentially stored in first buffer 8c, and data is sequentially deleted from second buffer 8d. Furthermore, by setting the gaze area ROI in the area corresponding to the first traffic light T1, the traffic light recognition device 2 performs majority voting on the light color determination results stored in the first buffer 8c, and determines that the indication indicated by the light color determination result (blue) most frequently stored in the first buffer 8c is the current indication of the traffic light ahead of the vehicle Ce (S112 "No", S113 in FIG. 3). As a result, the vehicle control device 3 follows the current indication (proceeding permission) determined by the traffic light recognition device 2, and does not issue a warning to the driver of the vehicle Ce.

[0022] As the above flow is repeated, when the predicted stop position Pe becomes further inward in the traveling direction of the vehicle Ce than the reference position Pth, as shown at time t2 in Fig. 5 (S107 "Yes" in Fig. 3), the traffic light recognition device 2 determines that the vehicle Ce cannot stop at the reference position Pth (the position of the stop line of the first traffic light T1 in Fig. 5), and sets the area on the image corresponding to the second traffic light T2 as the gaze area ROI (S109 in Fig. 3; Fig. 4B). As a result, storage of the light color determination results in the first buffer 8c is stopped, old data is sequentially deleted from the first buffer 8c, and the light color determination results (red) are sequentially stored in the second buffer 8d. 5, the number of determination results stored in the second buffer 8d has not reached half of the predetermined number ML ("No" in S114 in FIG. 3), so the traffic light recognition device 2 performs majority voting on the light color determination results (green and no data) stored in the first buffer 8c, and determines whether the traffic light ahead of vehicle Ce is indicating (proceeding) (S113 in FIG. 3). As a result, the vehicle control device 3 follows the indication (proceeding) determined by the traffic light recognition device 2 and does not issue a warning to the driver of vehicle Ce.

[0023] As the above flow continues, as shown at time t3 in FIG. 5, it is assumed that the number of determination results for the second traffic light T2 stored in the second buffer 8d reaches half or more of the predetermined number ML ("Yes" in S114 in FIG. 3). Then, the traffic light recognition device 2 performs a majority vote on the light color determination results (red and no data) stored in the second buffer 8d, and determines that the indication indicated by the light color determination result (red) most frequently stored in the second buffer 8d is the aspect of the traffic light ahead of the vehicle Ce (S115 in FIG. 3). As a result, the vehicle control device 3 starts issuing a warning to the driver of the vehicle Ce in accordance with the aspect (progression prohibited) determined by the traffic light recognition device 2. This causes the vehicle control device 3 to switch the traffic light to be followed before the vehicle Ce passes through the intersection just before the first traffic light T1, enabling the vehicle Ce to decelerate earlier in accordance with the aspect of the second traffic light T2.

[0024] Meanwhile, as shown at time t1 in FIG. 6, suppose that while vehicle Ce is traveling, a first traffic light T1 and a second traffic light T2 appear ahead of vehicle Ce, and the light colors of the first traffic light T1 and the second traffic light T2 are green. In this case, similar to the case shown in FIG. 5 where the light colors of the first traffic light T1 and the second traffic light T2 are green and red, respectively, the traffic light to be recognized changes before vehicle Ce passes the intersection just before first traffic light T1. Here, for example, if a configuration is adopted in which a majority vote process is performed on the light color determination results stored in second buffer 8d and the aspect determination is performed based on the majority vote process immediately after the traffic light to be recognized is switched to second traffic light T2, there is a possibility that a non-recognition period will occur during which the aspect cannot be determined. Therefore, for example, if the configuration is such that the aspect of the traffic light ahead of vehicle Ce is determined to be "progress prohibited" during the non-recognition period, a warning will be issued to the driver of vehicle Ce, which could cause the driver to unnecessarily decelerate. In contrast, in the vehicle control system 1 of the first embodiment, after the traffic light to be recognized is switched to the second traffic light T2, majority voting processing etc. is performed on the light color determination results stored in the second buffer 8d once the number of determination results for the second traffic light T2 stored in the second buffer 8d becomes more than half of the predetermined number ML, thereby reducing the non-recognition time and preventing unnecessary deceleration of the vehicle Ce.

[0025] For example, near Sakuragicho Station in Yokohama, (1) Suzukake Nishi → (2) Minatomirai 4-chome → (3) Icho-dori Nishi → (4) Keyaki-dori Nishi → (5) Sakura-dori Nishi → (6) Nippon Maru → (7) City Hall → (8) Honmachi 5-chome → (9) Honmachi 4-chome → (10) Honmachi 3-chome → (11) Honmachi 2-chome → (12) Honmachi 1-chome → (13) Yokohama District Court → (14) Aioi 1-chome → (15) Hamasta Entrance → (16) Yokohama Stadium → (17) Chinatown West Gate → (18) Central Hospital → (19) Nishinobashi → (20) Ichiba-dori → (21) Maedabashi → (22) Daikanbashi → (23) Yokohama Cathedral Ruins → (24) Consider a route that passes through the following routes in order: Chinatown East Gate → (25) Yamashitacho → (26) Prefectural Hall Entrance → (27) Yamashitacho Fire Station → (28) Osanbashi Entrance → (29) Minato Post Office → (30) Prefectural Office → (31) Honmachi 1-chome → (32) Honmachi 2-chome → (33) Honmachi 3-chome → (34) Honmachi 4-chome → (35) Honmachi 5-chome → (36) City Hall → (37) Nippon Maru → (38) Sakura Dori West → (39) Keyaki Dori West → (40) Icho Dori West → (41) Minatomirai 4-chome → (42) Suzukake Dori West → (43) Tochinoki Dori West → (44) Just before Kinkocho → (45) East Exit Rotary → (46) Anonymous Traffic Light. On this course, there are 45 intersections where the vehicle must go straight through, and 12 intersections where the distance from the intersection to the stop line is 86 m or less. Therefore, by using the vehicle control system 1 of the first embodiment, the vehicle Ce can be smoothly decelerated and stopped at 12 / 45 (=26.6%) of the intersections, and the number of intersections where the vehicle can be smoothly decelerated and stopped increases by 26.6%. The above "86 m" is calculated based on a vehicle speed of 50 km / h and a predetermined deceleration of 0.15 m / s. 2 The calculated free running distance was 20.9 m.

[0026] (Effects of the first embodiment) (1) In the first embodiment, if the stopping position (anticipated stopping position Pe) of the vehicle Ce when decelerated at a predetermined deceleration is closer to the reference position Pth in the traveling direction of the vehicle Ce, the area corresponding to the first traffic light T1 on the image acquired from the imaging unit 4 is set as the gaze area ROI. On the other hand, if the anticipated stopping position Pe is further back from the reference position Pth in the traveling direction of the vehicle Ce, the area corresponding to the second traffic light T2 is set as the gaze area ROI. Then, image processing is performed on the gaze area ROI to determine the current state of the traffic light ahead of the vehicle Ce. As a result, when the predicted stopping position Pe of the vehicle Ce when decelerated at a predetermined deceleration exceeds the reference position Pth, the traffic light to be recognized switches from the first traffic light T1 to the second traffic light T2. Therefore, compared to a method in which the traffic light to be recognized switches from the first traffic light T1 to the second traffic light T2 when the vehicle Ce passes through the first traffic light T1, the timing of the switch to the second traffic light T2 can be made earlier. Therefore, for example, when the light color of the first traffic light T1 is green (progress permitted) and the light color of the second traffic light T2 is red (progress prohibited), the vehicle Ce can be encouraged to decelerate at an earlier timing, thereby facilitating a smooth deceleration and stop of the vehicle Ce. Therefore, more appropriate information can be output as the indication of the traffic light ahead of the vehicle Ce. Furthermore, unlike a method in which two areas on the image, one corresponding to the first traffic light T1 and the other corresponding to the second traffic light T2, are used as the gaze area ROI, there is always only one gaze area ROI (i.e., the area in which image processing is performed), which reduces the computational load due to image processing.

[0027] (2) In the first embodiment, image processing is performed on the gaze region ROI to determine the light color of the traffic light in the gaze region ROI, and the determination results are stored in a first buffer 8c or a second buffer 8d, which has a storage area capable of storing a predetermined number of ML of data and deletes the oldest data as the newest data is stored. The state of the traffic light ahead of the vehicle Ce is determined based on the determination results stored in the first buffer 8c or the second buffer 8d. Furthermore, if the gaze region ROI for which image processing is performed corresponds to the first traffic light T1, the determination results of the light color of the traffic light in the gaze region ROI are stored in the first buffer 8c, and the oldest data is deleted from the second buffer 8d. Furthermore, if the gaze region ROI corresponds to the second traffic light T2, the determination results are stored in the second buffer 8d, and the oldest data is deleted from the first buffer 8c. As a result, even if a temporary error occurs in the determination of the traffic light color using image processing due to, for example, light reflection, the past determination results stored in the first buffer 8c and the second buffer 8d can be taken into consideration, making it possible to more appropriately determine the current state of the traffic light ahead of vehicle Ce.

[0028] (3) In the first embodiment, when the number of light color determination results stored in the second buffer 8d is less than half (ML / 2) of the predetermined number ML, the aspect of the traffic light ahead of the vehicle Ce is determined based on the light color determination results stored in the first buffer 8c. On the other hand, when the number of light color determination results stored in the second buffer 8d is equal to or greater than half (ML / 2) of the predetermined number ML, the aspect of the traffic light ahead of the vehicle Ce is determined based on the light color determination results stored in the second buffer 8d. As a result, when the output information switches from the aspect of the first traffic light T1 to the aspect of the second traffic light T2, the second buffer 8d stores more than half of the predetermined number ML of light color determination results for the second traffic light T2. Therefore, immediately after the switch, the aspect of the traffic light ahead of the vehicle Ce can be determined based on the light color determination results stored in the second buffer 8d, thereby reducing the time during which the aspect of the traffic light ahead of the vehicle Ce is not recognized.

[0029] (Second embodiment) Next, a vehicle control system 1 according to a second embodiment will be described. The schematic configuration of the vehicle control system 1 according to the second embodiment is the same as that shown in FIG. 1, and therefore will not be illustrated. FIG. 7 is a diagram in which a part of the flowchart of FIG. 3 has been modified, and is a flowchart showing the operation of the gaze area setting unit 12 and the like. In FIG. 7, parts corresponding to those in FIG. 3 are assigned the same reference numerals, and duplicate explanations will be omitted.

[0030] In the second embodiment, the gaze area setting unit 12 determines whether the light color of the second traffic light T2 cannot be determined from the image acquired from the image capture unit 4 (S201 in FIG. 7). For example, based on the current position of the vehicle Ce and map information, it determines whether the distance from the vehicle Ce to the second traffic light T2 is equal to or greater than a predetermined distance. If it is determined that the distance is equal to or greater than the predetermined distance, it determines that the light color of the second traffic light T2 cannot be determined from the image. Alternatively, based on the current position of the vehicle Ce, map information, and the installation state (angle, etc.) of the image capture unit 4, it determines whether the second traffic light T2 is outside the imaging range of the image capture unit 4. If it is determined that the second traffic light T2 is outside the imaging range, it determines that the light color of the second traffic light T2 cannot be determined from the image. Then, if it determines that the light color of the second traffic light T2 cannot be determined ("Yes" in S201 in FIG. 7), the gaze area setting unit 12 sets the region on the image corresponding to the first traffic light T1 as the gaze area ROI (S108 in FIG. 7). On the other hand, if the gaze area setting unit 12 determines that the light color of the second traffic light T2 can be determined ("No" in S201 of FIG. 7), and if the predicted stop position Pe is closer to the reference position Pth in the traveling direction of the vehicle Ce ("No" in S107 of FIG. 7), it sets the gaze area ROI to the area on the image corresponding to the first traffic light T1 (S108 of FIG. 7). On the other hand, if the predicted stop position Pe is farther from the reference position Pth in the traveling direction of the vehicle Ce ("Yes" in S107 of FIG. 7), it sets the gaze area ROI to the area on the image corresponding to the second traffic light T2 (S109 of FIG. 7).

[0031] (Effects of the second embodiment) In the second embodiment, if the light color of the second traffic light T2 cannot be determined from the image acquired from the imaging unit 4, the area on the image corresponding to the first traffic light T1 is set as the gaze area ROI, even if the predicted stop position Pe is further back in the traveling direction of the vehicle Ce than the reference position Pth. This eliminates the need to determine whether to switch the gaze area ROI when the light color of the second traffic light T2 cannot be determined, thereby reducing the computational load associated with the switching decision. Furthermore, because the gaze area ROI is not switched, it is possible to prevent the second traffic light T2 from being unrecognized.

[0032] (Third embodiment) Next, a vehicle control system 1 according to a third embodiment will be described. The schematic configuration of the vehicle control system 1 according to the third embodiment is the same as that shown in FIG. 1, and therefore will not be illustrated. FIG. 8 is a diagram showing the functional configuration of the controller 8. FIG. 9 is a diagram in which a part of the flowchart of FIG. 3 has been modified, and is a flowchart showing the operation of the route acquisition unit 17 and the like. In FIGS. 8 and 9, parts corresponding to those in FIGS. 2 and 3 are assigned the same reference numerals, and duplicate explanations will be omitted.

[0033] 8, the traffic light recognition device 2 further includes a route acquisition unit 17 that acquires a planned driving route for the vehicle Ce. The route acquisition unit 17 may be, for example, an interface device that acquires the planned driving route from a car navigation device that sets a planned driving route from a departure point to a destination of the vehicle Ce and performs navigation using the planned driving route. Data on the acquired planned driving route is output to the controller 8. The controller 8 further includes a route setting unit 18. The route setting unit 18 acquires data on the planned travel route of the vehicle Ce from the route acquisition unit 17 (S301 in FIG. 9). The gaze area setting unit 12 determines whether the vehicle Ce will pass the position of the first traffic light T1 in a manner other than going straight, based on the planned travel route of the vehicle Ce acquired by the route setting unit 18 (S302 in FIG. 8). If it is determined that the vehicle Ce will pass in a manner other than going straight ("Yes" in S302 in FIG. 8), it sets a gaze area ROI in an area on the image corresponding to the first traffic light T1 (S108 in FIG. 8). On the other hand, if it is determined that the vehicle Ce will pass in a manner other than going straight ("No" in S302 in FIG. 8), and the predicted stopping position Pe is closer to the reference position Pth in the traveling direction of the vehicle Ce ("No" in S107 in FIG. 8), it sets a gaze area ROI in an area on the image corresponding to the first traffic light T1 (S108 in FIG. 8). On the other hand, if the predicted stopping position Pe is further back in the direction of travel of the vehicle Ce than the reference position Pth ("Yes" in S107 of Figure 8), the gaze area setting unit 12 sets the area corresponding to the second traffic light T2 on the image as the gaze area ROI (S109 of Figure 8).

[0034] (Effects of the third embodiment) In the third embodiment, a determination is made based on the planned travel route of the vehicle Ce as to whether the vehicle Ce will pass the position of the first traffic light T1 in a manner other than going straight. If it is determined that the vehicle Ce will pass in a manner other than going straight, the region corresponding to the first traffic light T1 on the image acquired from the imaging unit 4 is set as the gaze region ROI, even if the predicted stop position Pe is further back in the traveling direction of the vehicle Ce than the reference position Pth. This eliminates the need to determine whether to switch the gaze region ROI when the vehicle Ce will pass the position of the first traffic light T1 in a manner other than going straight, thereby reducing the computational load associated with the switching determination. Furthermore, because the gaze region ROI is not switched, it is possible to prevent the second traffic light T2 from being unrecognized.

[0035] (Variation) In the above embodiment, an example was shown in which the controller 8 mounted on the vehicle Ce implements each function of the data acquisition unit 9, reference position acquisition unit 10, limit position calculation unit 11, gaze area setting unit 12, aspect determination unit 13, light color determination unit 14, determination result storage unit 15, light aspect output unit 16, route setting unit 18, etc., but other configurations may also be adopted. For example, at least some of the above functions may be implemented by a device outside the vehicle, such as a server device, capable of transmitting and receiving data to and from the vehicle Ce. In this case, the controller 21 and the device outside the vehicle constitute the traffic light recognition device 2. [Explanation of symbols]

[0036] 1...vehicle control system, 2...traffic light recognition device, 3...vehicle control device, 4...imaging unit, 5...self-position acquisition unit, 6...vehicle speed acquisition unit, 7...map information acquisition unit, 8...controller, 8a...processor, 8b...storage device, 8c...first buffer, 8d...second buffer, 9...data acquisition unit, 10...reference position acquisition unit, 11...limit position calculation unit, 12...gazing area setting unit, 13...aspect determination unit, 14...light color determination unit, 15...determination result storage unit, 16...light aspect output unit, 17...route acquisition unit, 18...route setting unit

Claims

1. Using an imaging unit mounted on a vehicle, an image in the traveling direction of the vehicle is captured; acquiring map information including the vehicle's current position and speed, and position information of traffic lights; Based on the vehicle's own position and the map information, acquire a reference position which is the position of a stop line corresponding to a first traffic light which is located in the traveling direction of the vehicle and is closest to the vehicle, or a position just before an intersection which corresponds to the first traffic light; calculating an expected stopping position, which is a stopping position when the vehicle is decelerated at a predetermined deceleration, based on the vehicle position, the vehicle speed, and the map information; comparing the predicted stop position with the reference position and setting a fixation area on the image; performing image processing on the gaze area to determine the current state of a traffic light ahead of the vehicle; If the predicted stop position is closer to the front of the reference position in the traveling direction of the vehicle, an area on the image corresponding to the first traffic light is set as the gaze area. If the predicted stop position is farther from the reference position in the traveling direction of the vehicle, an area on the image corresponding to a second traffic light that is located in the traveling direction of the vehicle and is the next closest to the vehicle after the first traffic light is set as the gaze area. Traffic light recognition method.

2. performing image processing on the gaze area to determine the light color of a traffic light in the gaze area, storing the determination result in a first buffer or a second buffer having a storage area capable of storing a predetermined number of data, and determining the state of a traffic light ahead of the vehicle based on the determination result stored in the first buffer or the second buffer; If the gaze area corresponds to the first traffic light, the determination result of the light color of the traffic light in the gaze area is stored in the first buffer, and the oldest data is deleted from each of the first buffer and the second buffer. If the gaze area corresponds to the second traffic light, the determination result is stored in the second buffer, and the oldest data is deleted from each of the first buffer and the second buffer. The traffic light recognition method according to claim 1 .

3. When the area corresponding to the second traffic light is set as the gaze area, if the number of light color determination results stored in the second buffer is less than half of the predetermined number, the state of the traffic light ahead of the vehicle is determined based on the light color determination results stored in the first buffer, and if the number of light color determination results stored in the second buffer is equal to or greater than half of the predetermined number, the state of the traffic light ahead of the vehicle is determined based on the light color determination results stored in the second buffer. The traffic light recognition method according to claim 2.

4. When the light color of the second traffic light cannot be determined from the image, even if the predicted stop position is located further back in the traveling direction of the vehicle than the reference position, the area on the image corresponding to the first traffic light is set as the gaze area. The traffic light recognition method according to claim 1 .

5. Based on a planned travel route of the vehicle, it is determined whether the vehicle will pass the position of the first traffic light in a manner other than going straight, and when it is determined that the vehicle will pass in a manner other than going straight, even if the predicted stop position is located further back in the traveling direction of the vehicle than the reference position, an area on the image corresponding to the first traffic light is set as the gaze area. The traffic light recognition method according to claim 1 .

6. an imaging unit mounted on a vehicle and configured to capture an image in the traveling direction of the vehicle; a self-position acquisition unit that acquires a self-position of the vehicle; a vehicle speed acquisition unit that acquires a vehicle speed of the vehicle; a map information acquisition unit that acquires map information including position information of traffic lights; a reference position acquisition unit that acquires, based on the self-position acquired by the self-position acquisition unit and the map information acquired by the map information acquisition unit, a reference position that is the position of a stop line corresponding to a first traffic light that is located in the traveling direction of the vehicle and is closest to the vehicle, or a position just before an intersection that corresponds to the first traffic light; a limit position calculation unit that calculates an expected stopping position, which is a stopping position when the vehicle is decelerated at a predetermined deceleration, based on the self-position acquired by the self-position acquisition unit, the vehicle speed acquired by the vehicle speed acquisition unit, and the map information acquired by the map information acquisition unit; and a gaze area setting unit that compares the predicted stop position calculated by the limit position calculation unit with the reference position acquired by the reference position acquisition unit and sets a gaze area on the image; an aspect determination unit that performs image processing on the gaze area set by the gaze area setting unit to determine the aspect of a traffic light ahead of the vehicle, When the expected stop position is closer to the reference position in the traveling direction of the vehicle, the gaze area setting unit sets an area on the image corresponding to the first traffic light as the gaze area, and when the expected stop position is farther from the reference position in the traveling direction of the vehicle, sets an area on the image corresponding to a second traffic light that is located in the traveling direction of the vehicle and is the next closest to the vehicle after the first traffic light as the gaze area. Traffic light recognition device.

Images