Traffic light recognition method and traffic light recognition device

Abstract

To provide a traffic light recognition method and a traffic light recognition device that can distinguish between an object resembling a traffic light and a traffic light among traffic light candidates included in an image that captures ahead of a vehicle.SOLUTION: A traffic light recognition method and a traffic light recognition device include the steps of: extracting traffic light candidates from an image that captures ahead of a vehicle in a traveling direction; calculating, as a first change history, a model for a direction history corresponding to a traffic light candidate for the vehicle in a static mode based on travel information including the amount of movement and the amount of change in attitude angle of the vehicle; calculating, as a second change history, a direction history of the vehicle in the static mode based on positional trajectories of the traffic light candidates on the image; and identifying, as an identified traffic light candidate, only the traffic light candidates for which a difference between the first change history and the second change history is equal to or less than a prescribed amount.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a traffic light recognition method and a traffic light recognition device. [Background technology]

[0002] A traffic signal recognition system is known that selects candidate areas from an image frame that have a relatively high prior probability of including traffic signals based on location information from a navigation map, acquires traffic signal candidates from the candidate areas, and tracks the acquired traffic signal candidates (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-164853 Summary of the Invention [Problem to be solved by the invention]

[0004] According to the invention described in Patent Document 1, a candidate area is set based on the distance from the onboard camera to the intersection and the detection results in the image frame corresponding to the previous time step, and traffic light candidates are acquired and tracked. However, because traffic light candidates within the candidate area are tracked as they are, there is a problem in that it is not possible to distinguish between traffic lights and objects that resemble traffic lights (e.g., the taillights of a leading vehicle, signs indicating a construction site, etc.) among the traffic light candidates.

[0005] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a traffic light recognition method and a traffic light recognition device that can distinguish between a traffic light and an object that resembles a traffic light among traffic light candidate images captured in front of a vehicle. [Means for solving the problem]

[0006] A traffic light recognition method and a traffic light recognition device according to one aspect of the present invention extract traffic light candidates from an image captured ahead in the vehicle's traveling direction, and calculate a model of the direction history corresponding to the traffic light candidate in the vehicle's stationary system as a first change history based on driving information including the vehicle's movement amount and attitude angle change amount. A history of the vehicle's direction in the stationary system is calculated as a second change history based on the position trajectory of the traffic light candidate on the image. Only traffic light candidates for which the deviation between the first change history and the second change history is equal to or less than a predetermined amount are identified as specific traffic light candidates. [Effects of the Invention]

[0007] According to the present invention, it is possible to distinguish between traffic lights and objects that resemble traffic lights among traffic light candidates included in an image captured ahead of a vehicle. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the configuration of a traffic light recognition device according to one embodiment of the present invention. [Figure 2] FIG. 2 is a flowchart showing the processing of the traffic light recognition device according to one embodiment of the present invention. [Figure 3A] FIG. 3A is a diagram showing an example of a layout of traffic signals in the height direction. [Figure 3B] FIG. 3B is a diagram showing an example of a horizontal layout of traffic lights. [Figure 4A] FIG. 4A is a diagram showing an example of a "traffic light model" that represents the relationship between the distance between a traffic light and a vehicle and the pitch angle of the direction in which the traffic light is visible. [Figure 4B] FIG. 4B is a diagram showing an example of a "traffic light model" that represents the relationship between the distance between the traffic light and the vehicle and the yaw angle of the direction in which the traffic light is visible. [Figure 5A] FIG. 5A is a diagram showing an example of a plot generated with respect to pitch angles in directions corresponding to traffic light candidates. [Figure 5B] FIG. 5B is a diagram showing an example of a plot generated with respect to the yaw angle in the direction corresponding to the traffic light candidate. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are designated by the same reference numerals and the description thereof will be omitted.

[0010] [Configuration of traffic light recognition device] Fig. 1 is a block diagram showing the configuration of a traffic light recognition device according to this embodiment. As shown in Fig. 1, the traffic light recognition device according to this embodiment includes an imaging unit 71, an on-board sensor 73, and a controller 100. The traffic light recognition device is mounted on a vehicle (not shown). The controller 100 is connected to the imaging unit 71 and the on-board sensor 73 via a wired or wireless communication path.

[0011] The traffic light recognition device may include a map information acquisition unit 75 and a vehicle control device 400. In this case, the controller 100 is connected to the map information acquisition unit 75 and the vehicle control device 400 via a wired or wireless communication path. Note that the map information acquisition unit 75 and the vehicle control device 400 are not essential elements for the configuration of the traffic light recognition device, and may be omitted.

[0012] The imaging unit 71 is mounted on the vehicle and acquires an image of the area ahead in the direction of travel of the vehicle. For example, the imaging unit 71 is a digital camera equipped with a solid-state imaging element such as a CCD or CMOS, and captures an image of the area around the vehicle to acquire a digital image of the surrounding area. The imaging unit 71 captures an image of a predetermined range around the vehicle by setting the focal length, the lens angle of view, the imaging direction angle (the elevation angle and lateral angle of the imaging direction in the stationary system of the vehicle), etc.

[0013] The imaging unit 71 may be attached to the front of the vehicle so as to be able to capture an image of the area ahead of the vehicle.

[0014] The captured images captured by the imaging unit 71 are output to the controller 100 and stored in a storage unit (not shown) for a predetermined period of time. For example, the imaging unit 71 acquires captured images at predetermined time intervals, and the captured images acquired at the predetermined time intervals are stored in the storage unit as past images. The past images may be deleted after a predetermined period of time has passed since the capture of the past images.

[0015] Alternatively, the imaging unit 71 may be composed of multiple cameras with different angles of view. For example, the imaging unit 71 may include a narrow-angle camera (first imaging unit) and a wide-angle camera (second imaging unit). Here, the wide-angle camera may capture an image with a wider angle of view than the narrow-angle camera. The narrow-angle camera may be capable of capturing an area farther away than the area that the wide-angle camera can capture. More specifically, the wide-angle camera may have a short focal length and be capable of capturing an image at a wide angle so that it can capture an image near the vehicle. The narrow-angle camera may have a long focal length and be capable of capturing an image at a narrow angle so that it can capture an image farther away than the area that the wide-angle camera can capture.

[0016] The on-vehicle sensor 73 detects the state of the vehicle. For example, the on-vehicle sensor 73 detects the vehicle's moving speed (forward / backward moving speed, left / right moving speed, turning speed), the steering angle of the wheels of the vehicle, and the rate of change of the steering angle. The on-vehicle sensor 73 may also acquire the amount of movement and the amount of change in attitude angle of the vehicle based on the vehicle's moving speed, the steering angle of the wheels, and the rate of change of the steering angle.

[0017] Alternatively, the on-board sensor 73 may include a sensor that measures the absolute position of the vehicle, that is, the position, attitude, and speed of the vehicle relative to a predetermined reference point, using a position detection sensor that measures the absolute position of the vehicle, such as a GPS (Global Positioning System) or odometry. The on-board sensor 73 may acquire the amount of movement and the amount of change in attitude angle of the vehicle based on changes in the position and attitude of the vehicle relative to the predetermined reference point.

[0018] The map information acquisition unit 75 acquires map information indicating the structure of the road on which the vehicle is traveling. The map information acquired by the map information acquisition unit 75 includes road structure information such as absolute lane positions, lane connection relationships, and relative position relationships. The map information acquired by the map information acquisition unit 75 may also include location information to intersections, traffic light location information, traffic light types, traffic light installation positions (height), and the locations of stop lines corresponding to traffic lights. The map information acquisition unit 75 may own a map database that stores map information, or may acquire map information from an external map data server using cloud computing. The map information acquisition unit 75 may also acquire map information using vehicle-to-vehicle communication or road-to-vehicle communication.

[0019] The vehicle control device 400 controls the vehicle based on the results obtained by the controller 100. For example, the vehicle control device 400 may be a device that automatically drives the vehicle along a predetermined driving route, or may be a device that assists the driving operation of the vehicle occupant.

[0020] The controller 100 (an example of a control unit or processing unit) is a general-purpose microcomputer equipped with a CPU (Central Processing Unit), memory, and input / output units. A computer program (traffic light recognition program) for causing the controller 100 to function as part of a traffic light recognition device is installed in the controller 100. By executing the computer program, the controller 100 functions as multiple information processing circuits (110, 120, 130, 140, 150, 160, 170) equipped in the traffic light recognition device.

[0021] Here, an example is shown in which the multiple information processing circuits (110, 120, 130, 140, 150, 160, 170) provided in the traffic light recognition device are realized by software. However, it is also possible to configure the information processing circuits (110, 120, 130, 140, 150, 160, 170) by preparing dedicated hardware for executing each of the information processes described below. Also, the multiple information processing circuits (110, 120, 130, 140, 150, 160, 170) may be configured by individual hardware. Furthermore, the information processing circuits (110, 120, 130, 140, 150, 160, 170) may also be used as electronic control units (ECUs) used for other vehicle-related controls.

[0022] The controller 100 includes a plurality of information processing circuits (110, 120, 130, 140, 150, 160, 170), including a traffic light candidate extraction unit 110, a driving information generation unit 120, a first change history generation unit 130, a second change history generation unit 140, a driving route setting unit 150, a fitting execution unit 160, and an identification unit 170. Note that the driving route setting unit 150 is not an essential element and may be omitted.

[0023] The traffic light candidate extraction unit 110 extracts traffic light candidates from the image acquired by the imaging unit 71. For example, the traffic light candidate extraction unit 110 extracts traffic light candidates by template matching. Template matching uses an image of a standard traffic light as a template, scans the detection area on the image while shifting it by a predetermined number of pixels, and calculates the correlation of brightness distribution, for example. When the correlation reaches the highest value, it is estimated that a traffic light candidate exists at the position on the image where the template is located. Alternatively, the traffic light candidate extraction unit 110 may extract traffic light candidates from an image using machine learning such as a support vector machine or a neural network. The method of extracting traffic light candidates from an image is not limited to the examples given here.

[0024] When detecting potential traffic lights, it is possible to improve the recognition rate by preparing a training database that stores templates of traffic lights of different sizes and using different training databases depending on the distance to the traffic light.

[0025] The driving information generation unit 120 generates driving information including the amount of movement and the amount of change in the attitude angle of the vehicle. For example, the driving information generation unit 120 may acquire the vehicle's moving speed, the wheel steering angle, and the rate of change of the steering angle from the on-board sensor 73, and generate driving information based on the vehicle's moving speed, the wheel steering angle, and the rate of change of the steering angle. Alternatively, the driving information generation unit 120 may acquire the vehicle's position and attitude with respect to a predetermined reference point from the on-board sensor 73, and generate driving information based on the change in the vehicle's position and attitude.

[0026] Alternatively, when the map information acquisition unit 75 provides position information of an intersection, the traveling information generation unit 120 may acquire position information of a traffic light from the map information acquisition unit 75 and acquire the distance from the vehicle to the intersection ahead in the traveling direction of the vehicle. The traveling information generation unit 120 may generate traveling information including the distance to the intersection in addition to the amount of movement and the amount of change in attitude angle of the vehicle.

[0027] The first change history generation unit 130 calculates, based on the travel information, a model of the history of directions corresponding to traffic light candidates in the vehicle's stationary system as a first change history (traffic light model). More specifically, the first change history generation unit 130 calculates a traffic light model that models the relationship between the vehicle's movement amount, the attitude angle change amount, and the directions corresponding to traffic light candidates in the vehicle's stationary system.

[0028] The "traffic light model" will be described with reference to Figures 3A and 3B. Figure 3A is a diagram showing an example of a vertical layout of a traffic light. Figure 3B is a diagram showing an example of a horizontal layout of a traffic light.

[0029] When processing by the traffic light recognition device of this embodiment is started, a coordinate system with the origin at the position of the vehicle (particularly the position of the imaging unit 71) at a certain timing is represented by an orthogonal coordinate system with X, Y, and Z axes, as shown in Figures 3A and 3B. Here, the X axis is an axis whose positive axis is the direction of travel of the vehicle at that timing. The Y axis is an axis parallel to the road surface and perpendicular to the X axis at that timing. The Z axis is an axis whose positive axis is the height direction from the XY plane. In other words, at that timing, the vehicle is located at coordinates (X, Y, Z) = (0, 0, 0).

[0030] The traffic light is assumed to be located at coordinates (X, Y, Z) = (XS, YS, ZS). For simplicity, we will assume that the vehicle travels along the X axis (i.e., Y = 0, Z = 0 with respect to the vehicle's movement). We will also assume that the vehicle's posture does not deviate from the horizontal direction while traveling.

[0031] When the vehicle moves from the position at the above timing and is now located at coordinates (X, 0, 0), the apparent position of the traffic light as seen from the vehicle changes due to the movement of the vehicle.

[0032] According to the positional relationship between the traffic light and the vehicle described above, the pitch angle θ (the elevation angle in the vehicle's stationary system) in the direction in which the traffic light is visible is expressed as follows: θ=arctan(ZS / (XS-X)) ···(1) (where -π / 2<θ<π / 2)

[0033] The yaw angle ψ (the left-right angle in the vehicle's stationary system) in the direction in which the traffic light is visible is expressed as follows: ψ=arctan(YS / (XS-X)) ···(2) (where -π / 2<ψ<π / 2)

[0034] The direction corresponding to the traffic light candidate in the stationary system of the vehicle is expressed by at least one of the pitch angle θ and the yaw angle ψ described above.

[0035] Therefore, it can be seen that the traffic light model models the relationship between the vehicle's position coordinate X and the pitch angle θ and yaw angle ψ in the direction in which the traffic light is visible. Considering cases in which the vehicle does not travel along the X axis, or cases in which the vehicle's attitude changes while traveling, it can be seen that the traffic light model models the relationship between the vehicle's coordinates (X, Y, Z) corresponding to the amount of movement of the vehicle, the amount of change in the vehicle's attitude angle, the pitch angle θ, and the yaw angle ψ.

[0036] An example of a traffic light model will be described using Figures 4A and 4B. Figure 4A is a diagram showing an example of a "traffic light model" that represents the relationship between the distance between a traffic light and a vehicle and the pitch angle in the direction in which the traffic light is visible. Figure 4B is a diagram showing an example of a "traffic light model" that represents the relationship between the distance between a traffic light and a vehicle and the yaw angle in the direction in which the traffic light is visible.

[0037] FIG. 4A shows two types of traffic light models M1 and M2 with different parameter ZS (corresponding to the height of the traffic light). Compared to parameter ZS in traffic light model M1, parameter ZS in traffic light model M is larger. In other words, the height of the traffic light in traffic light model M2 is larger than the height of the traffic light in traffic light model M1. Therefore, it can be seen that when a vehicle approaches a traffic light (when approaching X = XS), the pitch angle θ in traffic light model M2 rises faster than the pitch angle θ in traffic light model M1.

[0038] Figure 4A shows three types of traffic light models M3, M4, and M5, each with a different parameter YS (corresponding to the horizontal position of the traffic light). The parameter YS in traffic light model M3 and the parameter YS in traffic light model M5 have opposite signs. In other words, the traffic light in traffic light model M3 is located on the left side of the vehicle's direction of travel, while the traffic light in traffic light model M5 is located on the right side of the vehicle's direction of travel. Therefore, as the vehicle approaches the traffic light (approaching X = XS), the yaw angle ψ of traffic light model M3 increases, while the yaw angle ψ of traffic light model M5 decreases.

[0039] Furthermore, the parameter YS in the traffic light model M4 is 0. That is, the traffic light in the traffic light model M4 is located ahead of the vehicle in the traveling direction. Therefore, the yaw angle ψ of the traffic light model M4 remains 0.

[0040] As described above, the traffic light model changes depending on the coordinates (XS, YS, ZS) where the traffic light is located. Therefore, the coordinates (XS, YS, ZS) can be estimated by performing fitting using the traffic light model on the actual history of the direction corresponding to the traffic light candidate in the vehicle's stationary system. In other words, it is possible to treat XS, YS, and ZS as fitting parameters and determine the values ​​of XS, YS, and ZS.

[0041] The second change history generating unit 140 calculates, as the second change history, a history of the direction corresponding to the traffic light candidate in the stationary system of the vehicle, based on the position trajectory of the traffic light candidate on the image.

[0042] More specifically, the second change history generation unit 140 calculates the direction corresponding to the traffic light candidate in the stationary frame of the vehicle based on the position of the traffic light candidate on the image. For example, the second change history generation unit 140 calculates the apparent direction (pitch angle θ and yaw angle ψ) of an object reflected in the image as a traffic light candidate in the stationary frame of the vehicle using the angle of the imaging direction of the camera in the imaging unit 71 (the elevation angle and lateral angle of the imaging direction in the stationary frame of the vehicle).

[0043] Then, the second change history generating unit 140 plots the direction corresponding to the traffic light candidate in the vehicle's stationary system in association with the vehicle's coordinates (X, Y, Z) and the vehicle's attitude. The relationship between the direction corresponding to the traffic light candidate in the vehicle's stationary system, the vehicle's coordinates (X, Y, Z), and the vehicle's attitude is treated as the second change history.

[0044] The plots generated by the second change history generating unit 140 are shown, for example, in Figures 5A and 5B. Figure 5A is a diagram showing an example of a plot generated for the pitch angle in the direction corresponding to the traffic light candidate. Figure 5B is a diagram showing an example of a plot generated for the yaw angle in the direction corresponding to the traffic light candidate.

[0045] Alternatively, the second change history generation unit 140 may classify the plotted points into several clusters. The number of traffic light candidates reflected in the image is not limited to one, but may be multiple. Therefore, the second change history generation unit 140 performs classification into clusters to improve the accuracy of fitting in the fitting execution unit 160 described below. Various methods can be used for classification into clusters, such as Ward's method, group average method, shortest distance method, longest distance method, and k-means method. The method for classification into clusters is not limited to the examples given here. Note that a cluster containing multiple points may itself be treated as the second change history.

[0046] The fitting execution unit 160 classifies points that are close to each other on the graph into the same cluster, and performs fitting using only multiple points included in the same cluster. The fitting execution unit 160 may also remove points that cannot be classified into a cluster.

[0047] For example, in Figure 5A, the points may be classified into a cluster along curve C1, a cluster along curve C2, and clusters E1 and E2, while in Figure 5B, the points may be classified into a cluster along curve C3, a cluster along curve C4, a cluster along curve C5, and clusters E3 and E4.

[0048] The driving route setting unit 150 sets a driving route for the vehicle and outputs information about the set driving route. When the map information acquisition unit 75 provides information about the location of intersections, the driving route setting unit 150 may output information about traffic lights that exist on the set driving route for the vehicle.

[0049] The travel route of the vehicle set by the travel route setting unit 150 may be output to the vehicle control device 400, and the travel of the vehicle may be controlled based on the travel route.

[0050] The fitting execution unit 160 determines whether the deviation between the first change history and the second change history is equal to or less than a predetermined amount. More specifically, the fitting execution unit 160 executes fitting using the first change history (traffic light model), which is a model of the history of the direction corresponding to the traffic light candidate in the stationary system of the vehicle, for the second change history, which is the actual history of the direction corresponding to the traffic light candidate in the stationary system of the vehicle. Then, when the deviation during fitting (i.e., the deviation between the first change history and the second change history) is equal to or less than a predetermined amount, the fitting execution unit 160 estimates the values ​​of the fitting parameters XS, YS, and ZS to be the position coordinates of the traffic light.

[0051] For example, the fitting execution unit 160 estimates the coordinates (XS, YS, ZS) of the object's location by performing fitting using a traffic light model on the multiple plotted points. The fitting execution unit 160 may determine fitting parameters so as to minimize the root mean square error between the first change history and the second change history. Then, the fitting execution unit 160 may calculate the minimized root mean square error as the deviation during fitting.

[0052] An example of fitting using a traffic light model will be described with reference to Figures 5A and 5B. For example, in a graph plotting points (X, θ) as shown in Figure 5A, the fitting execution unit 160 performs fitting based on the traffic light model for each cluster using multiple points classified into a cluster along curve C1 and a cluster along curve C2. By performing fitting using the traffic light model related to the pitch angle, it is possible to estimate XS and ZS, among the fitting parameters.

[0053] On the other hand, the fitting execution unit 160 may not perform fitting based on the clusters E1 and E2. There are several reasons for not performing fitting based on the clusters E1 and E2. For example, as shown in FIG. 5A, the clusters E1 and E2 are located near a cluster along the curve C2 that can be fitted by a traffic light model, and are therefore likely not to correspond to traffic lights. Also, the width of the clusters E1 and E2 in the X-axis direction is narrower than the width of the cluster along the curve C1 and the cluster along the curve C2 in the X-axis direction. In other words, it is likely that fitting cannot be performed with sufficient accuracy for the clusters E1 and E2. Another reason is that the narrow width of the clusters E1 and E2 in the X-axis direction makes it likely that the clusters E1 and E2 are collections of points caused by temporary flashing near the traffic light (for example, the taillights of a leading vehicle, a construction site sign, etc.).

[0054] 5B, in a graph plotting points (X, ψ), the fitting execution unit 160 performs fitting based on a traffic light model for each cluster using multiple points classified into a cluster along curve C3, a cluster along curve C4, and a cluster along curve C5. By performing fitting using the traffic light model related to the yaw angle, it is possible to estimate XS and YS, among the fitting parameters.

[0055] On the other hand, the fitting execution section 160 may not execute fitting based on clusters E3 and E4 for the same reason as the reason for not executing fitting based on clusters E1 and E2 in FIG.

[0056] The identification unit 170 identifies only traffic light candidates for which the deviation between the first change history and the second change history is equal to or less than a predetermined amount as specific traffic light candidates. More specifically, if the deviation during fitting performed to estimate the position coordinates of the traffic light is equal to or less than a predetermined amount, the identification unit 170 identifies the traffic light candidate corresponding to the second change history for which the position coordinates of the traffic light could be estimated as the specific traffic light candidate.

[0057] Furthermore, when information related to the driving route is output from the driving route setting unit 150, the identification unit 170 may identify, as the specific traffic light candidate, a traffic light candidate corresponding to the second change history that can be estimated to indicate that a traffic light is located at position coordinates on the driving route.When information identifying traffic lights present on the driving route is output from the driving route setting unit 150, the identification unit 170 may identify, as the specific traffic light candidate, a traffic light candidate corresponding to the second change history that can be estimated to indicate that a traffic light is located at position coordinates that are the same as or close to the position of a traffic light present on the driving route.

[0058] The specific traffic light candidate identified by the identification unit 170 may be output to a determination unit (not shown) and used to determine the current signal status of the traffic light. In this case, the determination unit (not shown) may prioritize specific traffic light candidates closer to the front of the vehicle's direction of travel (specific traffic light candidates with small absolute values ​​of the corresponding pitch angle θ and yaw angle ψ) over specific traffic light candidates farther ahead in the vehicle's direction of travel (specific traffic light candidates with large absolute values ​​of the corresponding pitch angle θ or yaw angle ψ) in determining the current signal status of the traffic light.

[0059] Alternatively, the identification unit 170 may output a control signal to the imaging unit 71 to switch the type of camera used when the imaging unit 71 acquires an image. More specifically, when the position of the identified specific traffic light candidate on the image moves from the center of the image toward the outside as the vehicle travels, the identification unit 170 may switch the camera to acquire an image of the area ahead in the vehicle's traveling direction using a camera with a wider angle of view than the camera currently in use. For example, consider a case where, while a first image acquired by a narrow-angle camera (first imaging unit) is being acquired as an image, the specific traffic light candidate moves outside the angle of view of the narrow-angle camera. In this case, the identification unit 170 may output a control signal to the imaging unit 71 so that the imaging unit 71 acquires, instead of the first image, a second image acquired by a wide-angle camera with a wider angle of view than the narrow-angle camera.

[0060] [Traffic signal recognition device processing procedure] Next, the processing procedure of the traffic light recognition device according to this embodiment will be described with reference to the flowchart of Fig. 2. The processing of the traffic light recognition device shown in Fig. 2 may be started based on a user instruction, or may be repeatedly executed at a predetermined interval.

[0061] First, in step S101, the image capturing unit 71 captures an image of the area ahead in the traveling direction of the vehicle.

[0062] In step S103 , the traffic light candidate extraction unit 110 extracts traffic light candidates from the image acquired by the imaging unit 71 .

[0063] In step S105, the driving information generating unit 120 generates driving information including the amount of movement and the amount of change in attitude angle of the vehicle.

[0064] In step S107, the first change history generating unit 130 calculates, based on the travel information, a model of the history of the direction corresponding to the traffic light candidate in the stationary system of the vehicle as the first change history (traffic light model).

[0065] In step S109, the second change history generating unit 140 calculates, as the second change history, a history of the direction corresponding to the traffic light candidate in the stationary system of the vehicle, based on the position trajectory of the traffic light candidate on the image.

[0066] In step S111, the second change history generation unit 140 classifies a plurality of points included in the second change history into clusters.

[0067] In step S113, the fitting execution section 160 selects one unprocessed cluster (a cluster for which fitting has not been executed) from among the clusters to be subjected to fitting.

[0068] In step S115, the fitting execution unit 160 executes fitting for the selected cluster. Then, if the deviation between the first change history and the second change history is equal to or less than a predetermined amount, the fitting execution unit 160 estimates that the position coordinates are those of a traffic light.

[0069] In step S117, the fitting execution section 160 determines whether or not there are any unprocessed clusters. If there are any unprocessed clusters (YES in step S117), the process returns to step S113.

[0070] If there are no unprocessed clusters (NO in step S117), in step S119, the identification unit 170 identifies only traffic light candidates whose deviation between the first change history and the second change history is less than a predetermined amount as identified traffic light candidates.

[0071] [Effects of the embodiment] As described above in detail, the traffic light recognition method and traffic light recognition device according to this embodiment extract traffic light candidates from an image captured ahead of the vehicle in the direction of travel, and calculate a model of the direction history corresponding to the traffic light candidate in the vehicle's stationary system as a first change history based on driving information including the vehicle's movement amount and attitude angle change amount. Then, the device calculates a model of the direction history of the vehicle in the stationary system as a second change history based on the position trajectory of the traffic light candidate on the image. Then, only traffic light candidates for which the deviation between the first change history and the second change history is equal to or less than a predetermined amount are identified as specific traffic light candidates.

[0072] This makes it possible to distinguish between traffic lights and objects that resemble traffic lights among traffic light candidate images captured in front of the vehicle. In particular, it is possible to prevent objects that resemble traffic lights, such as the taillights of a leading vehicle or signs at a construction site, from being mistakenly detected as traffic lights.

[0073] Furthermore, it is no longer necessary to track objects that resemble traffic lights in captured images, and it is possible to track only objects that are likely to be traffic lights. This also reduces the calculation load. Furthermore, by tracking only objects that are likely to be traffic lights, it is possible to reduce the delay in determining the current status of the traffic lights. As a result, convenience for vehicle users is improved.

[0074] In the traffic light recognition method and traffic light recognition device according to this embodiment, the direction corresponding to the traffic light candidate in the stationary system of the vehicle may be expressed by at least one of the elevation angle and the lateral angle in the stationary system of the vehicle, thereby making it possible to model how the apparent position of the traffic light as seen from the vehicle changes due to the vehicle's movement.

[0075] Furthermore, the direction corresponding to the traffic light candidate in the vehicle's stationary frame may be expressed by both the elevation angle and the lateral angle in the vehicle's stationary frame. This allows for a more detailed model of how the apparent position of the traffic light as seen from the vehicle changes due to the vehicle's movement. Furthermore, the accuracy of fitting can be improved when fitting the second change history using the first change history. As a result, it is possible to more accurately distinguish between traffic lights and objects that resemble traffic lights among traffic light candidate images captured in front of the vehicle.

[0076] Furthermore, the traffic light recognition method and traffic light recognition device according to this embodiment may acquire the distance from the vehicle to the intersection ahead in the direction of travel and generate driving information including the distance. This makes it possible to limit the range of fitting parameters when fitting the second change history using the first change history. As a result, it is possible to reduce the calculation load. Furthermore, it is possible to improve the fitting accuracy when fitting the second change history using the first change history, and it is possible to more accurately distinguish between objects similar to traffic lights and traffic lights.

[0077] In addition, the camera used to capture images of the area ahead of the vehicle's direction of travel can be switched depending on the distance. Furthermore, by using an appropriate camera, the accuracy of extracting traffic light candidates from images can be improved. As a result, among the traffic light candidates included in the image, it is possible to distinguish between objects that resemble traffic lights and traffic lights themselves.

[0078] Furthermore, the traffic light recognition method and traffic light recognition device according to this embodiment may set a vehicle travel route and, when the vehicle travels along the travel route, identify only traffic light candidates on the travel route as specific traffic light candidates. This makes it possible to more accurately distinguish between objects that resemble traffic lights and traffic lights that the vehicle should obey when traveling. This also makes it possible to accurately recognize traffic lights that the vehicle should obey when traveling.

[0079] Furthermore, the traffic light recognition method and traffic light recognition device according to this embodiment may be configured to acquire, instead of the first image, a second image acquired by a second imaging unit having a wider angle of view than the first imaging unit if the specific traffic light candidate moves outside the angle of view of the first imaging unit while the first image is being acquired as an image. This allows the image to be selected for processing by switching between multiple cameras with different angles of view. As a result, stable traffic light recognition can be achieved from far away to close to the intersection.

[0080] Each function described in the above embodiments may be implemented by one or more processing circuits, including programmed processors, electrical circuits, and even devices such as application specific integrated circuits (ASICs), circuit components arranged to perform the described functions.

[0081] Although the present invention has been described above based on the embodiments, it will be apparent to those skilled in the art that the present invention is not limited to these descriptions and that various modifications and improvements are possible. The descriptions and drawings that form part of this disclosure should not be understood as limiting the present invention. Various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art from this disclosure.

[0082] The present invention naturally includes various embodiments not described herein. Therefore, the technical scope of the present invention is defined only by the invention-specifying matters according to the scope of the claims that are appropriate from the above description. [Explanation of symbols]

[0083] 71 Imaging unit 73 In-vehicle sensors 75 Map information acquisition unit 100 Controllers 110 Traffic light candidate extraction unit 120 Driving information generation unit 130 First change history generation unit 140 Second change history generation unit 150 Travel route setting unit 160 Fitting Execution Department 170 Specific section 400 Vehicle control device

Claims

1. A traffic light recognition method for controlling a controller connected to an imaging unit mounted on a vehicle, comprising: the imaging unit acquires an image of a scene ahead in a traveling direction of the vehicle, The controller extracting traffic light candidates from the image; generating travel information including a movement amount and an attitude angle change amount of the vehicle; calculating a first change history that models a relationship between a movement amount of the vehicle, an attitude angle change amount, and a direction corresponding to the traffic light candidate in a stationary system of the vehicle based on the travel information; calculating an actual history of the direction in a stationary system of the vehicle as a second change history based on a position trajectory of the traffic light candidate on the image; Identifying only the traffic light candidate for which the difference between the first change history and the second change history is equal to or less than a predetermined amount as a specific traffic light candidate. A traffic light recognition method comprising:

2. 2. The traffic light recognition method according to claim 1, The direction is expressed by at least one of an elevation angle and a lateral angle in a stationary system of the vehicle. A traffic light recognition method comprising:

3. 3. The traffic light recognition method according to claim 1 or 2, The controller Acquire a distance from the vehicle to an intersection ahead in the traveling direction; generating the travel information including the distance; A traffic light recognition method comprising:

4. The traffic light recognition method according to any one of claims 1 to 3, The controller setting a travel route for the vehicle; When the vehicle is traveling along the travel route, only the traffic light candidate on the travel route is identified as the specific traffic light candidate. A traffic light recognition method comprising:

5. The traffic light recognition method according to any one of claims 1 to 4, When the specific traffic light candidate moves out of an angle of view of the first imaging unit while a first image acquired by a first imaging unit is being acquired as the image, the controller acquires a second image acquired by a second imaging unit having an angle of view wider than that of the first imaging unit as the image instead of the first image. A traffic light recognition method comprising:

6. A traffic light recognition device including an imaging unit mounted on a vehicle and a controller, the imaging unit acquires an image of a scene ahead in a traveling direction of the vehicle, The controller extracting traffic light candidates from the image; generating travel information including a movement amount and an attitude angle change amount of the vehicle; calculating a first change history that models a relationship between a movement amount of the vehicle, an attitude angle change amount, and a direction corresponding to the traffic light candidate in a stationary system of the vehicle based on the travel information; calculating an actual history of the direction in a stationary system of the vehicle as a second change history based on a position trajectory of the traffic light candidate on the image; Identifying only the traffic light candidate for which the difference between the first change history and the second change history is equal to or less than a predetermined amount as a specific traffic light candidate. A traffic light recognition device characterized by the above.

Images