Detection device, detection method, and detection program

Abstract

To improve accuracy in detecting a traffic light.SOLUTION: A detection device includes a controller. The controller performs: detecting an emitter corresponding to a lighting instrument of a traffic light from a camera image photographed in a vehicle; calculating a movement amount of the emitter from a change of the position of the emitter; excluding the emitter whose movement amount is equal to or less than a threshold from a detection target of the lighting instrument; and detecting the emitter whose movement amount of the emitter exceeds the threshold as the lighting instrument.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The disclosed embodiments relate to a detection device, a detection method, and a detection program.

Background Art

[0002] Conventionally, a technique for detecting the signal color of a traffic signal from a camera image captured by a camera mounted on a vehicle has been disclosed. Patent Document 1 discloses detecting traffic signal violations by a vehicle based on the signal color detected from the camera image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, there is room for improvement in improving the detection accuracy of traffic signals. Specifically, when detecting the signal color from a camera image, there is a risk of misdetecting the lights of streetlights or buildings located farther away than the traffic signal as the traffic signal lights.

[0005] The present invention has been made in view of the above, and an object thereof is to provide a detection device, a detection method, and a detection program capable of improving the detection accuracy of traffic signals.

Means for Solving the Problems

[0006] In order to solve the above-described problems and achieve the object, the detection device according to the present application has a controller. The controller detects a light-emitting body corresponding to a traffic signal lamp from a camera image taken by a vehicle, calculates a movement amount of the light-emitting body from a change in the position of the light-emitting body, excludes the light-emitting body whose movement amount is equal to or less than a threshold value from a detection target of the lamp, and detects the light-emitting body whose movement amount exceeds the threshold value as the lamp.

Effect of the Invention

[0007] According to the present invention, since the movement amount of the light-emitting body can be calculated and the light-emitting body whose movement amount is equal to or less than the threshold value can be excluded from the detection target of the traffic signal lamp, the detection accuracy of the traffic signal can be improved.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the detection device, detection method, and detection program disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited by the embodiments shown below.

[0010] First, the preprocessing of the detection method according to the embodiment will be described. FIG. 1 is a diagram showing the overall outline of the detection method. The detection method shown in FIG. 1 is executed by an in-vehicle device 1 described later. The in-vehicle device 1 is an example of the detection device according to the embodiment.

[0011] The detection method shown in FIG. 1 detects the lighting state of a traffic signal from a camera image captured by an in-vehicle camera (e.g., a drive recorder) that captures the front of the vehicle V (see FIG. 2). In the present disclosure, a case will be described in which, based on the detected lighting state of the traffic signal, a signal violation determination is further made as to whether the vehicle V passes through a red signal.

[0012] As shown in FIG. 1, the detection method according to the embodiment first extracts a rectangular region from the camera image (step S1). For example, the detection method according to the embodiment extracts a bounding box with a traffic signal score equal to or higher than a certain value obtained by a deep learning method such as YOLO as a rectangular region.

[0013] Further, the detection method according to the embodiment detects a light-emitting body corresponding to the traffic signal lamp in the rectangular region extracted from the camera image. The traffic signal lamps are a red signal, a blue signal, and a yellow signal, but may include arrow signals. The detection method according to the embodiment uses a model learned by a machine learning method such as deep learning to detect a light-emitting body corresponding to the lamp in the rectangular region. Hereinafter, the "traffic signal lamp" may be referred to as the "signal lamp" in some cases.

[0014] Subsequently, the detection method according to the embodiment calculates a locus on the virtual plane for such a rectangular region based on the rectangular region extracted in step S1 (step S2). Here, referring to FIG. 2, the virtual plane according to the embodiment will be described. FIG. 2 is an explanatory diagram of the virtual plane.

[0015] As shown in FIG. 2, the virtual plane is a coordinate plane set based on the height of the traffic signal 300 set in advance from the road surface. For example, the height of the traffic signal is set in the range of, for example, 5 m to 5.5 m.

[0016] The virtual plane is a virtual plane obtained by fixing the coordinate value of the vertical axis (the coordinate of the Z-axis) to the height of the traffic signal 300 in a coordinate system represented by three mutually perpendicular axes including the vertical axis.

[0017] In the detection method according to the embodiment, the position of the vehicle V is projected onto such a virtual plane as a virtual vehicle position, and the behavior of the vehicle V is estimated. Thereby, the detection method according to the embodiment can convert the change on the two-dimensional camera image into the change on the virtual plane conforming to the three-dimensional real space.

[0018] Then, the detection method according to the embodiment calculates the locus of the traffic signal 300 (rectangular area) from the transition of the positional relationship between the virtual vehicle position and the traffic signal 300 on the virtual plane with respect to the virtual plane.

[0019] Thereafter, in the detection method according to the embodiment, as shown in FIG. 1, target signal determination is performed (step S3). Note that the target signal is a traffic signal that is the target of the signal ignoring determination described later. In other words, the traffic signals excluded from the target signal will be excluded from the target of the signal ignoring determination.

[0020] Here, with reference to FIG. 3, a specific example of the target signal determination will be described. FIG. 3 is an explanatory diagram of the target signal determination. In FIG. 3, the vehicle V and the traffic signal are projected with respect to the virtual plane.

[0021] As shown in FIG. 3, in the detection method according to the embodiment, a detection range A is set on the virtual plane, and it is determined whether it is a target signal based on the locus of the traffic signal 300 on the virtual plane. In the example shown in FIG. 3, the case where the detection range A is a region of 20 m to 15 m set in the forward direction of the vehicle V and having a width of 10.5 m centered on the vehicle V is shown.

[0022] As shown in FIG. 3, in the target signal determination, based on the trajectory of the traffic signal, traffic signals that pass horizontally within the detection range A and traffic signals that turn and approach after exceeding the fixed line set at the end on the vehicle V side of the detection range A are excluded.

[0023] Also, as shown in FIG. 3, in the target signal determination, a traffic signal approaching the vehicle V straight ahead is determined as the target signal. In this way, the detection method according to the embodiment can exclude, from the target of the signal ignoring determination, traffic signals that the vehicle V does not pass through, for example, traffic signals where the vehicle V turns in an alley in front of the traffic signal.

[0024] Thereafter, as shown in FIG. 1, in the detection method according to the embodiment, a signal ignoring determination is performed (step S4), and the determination result is output. Note that the signal ignoring determination is made according to whether the traffic signal detected from the camera image is a red signal and whether the vehicle V has passed through the red signal.

[0025] By the way, when detecting a traffic signal based on the virtual plane set at the height of the traffic signal, there is a risk of misdetecting a light emitter different from the traffic signal as the traffic signal light. Specifically, for example, there is a risk of detecting a light emitter such as a street lamp or a building light as the traffic signal light.

[0026] Therefore, in the detection method according to the embodiment, an exclusion process for excluding light emitters other than the traffic signal from the detection target of the traffic signal is executed. Here, the outline of the exclusion process will be described with reference to FIG. 4.

[0027] FIG. 4 is an explanatory diagram showing the outline of the exclusion process. As shown in FIG. 4, when detecting the traffic signal light using camera image processing by AI or the like, there may be a case where a light emitter such as a street lamp or a building light is misdetected as the traffic signal light when projected onto the virtual plane.

[0028] Specifically, when viewed from the vehicle V, when a light emitter that is actually located farther and higher than the traffic signal is projected onto the virtual plane, there is a risk of misdetecting that light emitter as the traffic signal light.

[0029] In the example shown in FIG. 4, it shows that there may be a case where street lights located higher than the traffic signal or the lights of buildings are erroneously detected as traffic signal lights. In such a case, as shown in FIG. 4, since the actual distance from the vehicle V to the street light or the building light is greater than the actual distance from the vehicle V to the traffic signal, when projected onto the virtual plane, the change in coordinates is less than that of the traffic signal.

[0030] Specifically, in the case of FIG. 4, when comparing the movement amount of the traffic signal and the movement amount of the street light or the like in the virtual plane, the movement amount of the traffic signal corresponds to the traveling speed of the vehicle V, and the movement amount of the street light or the like becomes smaller compared to the traveling speed of the vehicle V.

[0031] Therefore, in the detection method according to the embodiment, paying attention to such a point, the movement amount of the light emitter detected from the camera image is calculated, and when the calculated movement amount is equal to or less than the threshold value, it is excluded from the detection target of the traffic signal light.

[0032] Here, in the present embodiment, the movement amount is a concept including both the movement distance and the movement speed. That is, the detection method according to the embodiment calculates the movement distance or the movement speed of the light emitter in the virtual plane, and when at least one of the movement distance or the movement speed is equal to or less than the threshold value, the light emitter is excluded from the detection target of the traffic signal.

[0033] In the detection method according to the embodiment, a light emitter corresponding to the traffic signal light is detected from the camera image, and based on the movement amount of the light emitter, it is determined whether the light emitter is a traffic signal light. Therefore, according to the detection method according to the embodiment, a light emitter with a small movement amount such as a street light can be excluded from the detection target of the traffic signal light, so that the detection accuracy of the traffic signal can be improved.

[0034] Next, with reference to FIG. 5, a configuration example of the in-vehicle device 1 according to the embodiment will be described. FIG. 5 is a block diagram of the in-vehicle device 1. As shown in FIG. 5, the in-vehicle device 1 is connected to a camera 50, a sensor 51, and a notification unit 52.

[0035] The camera 50 is, for example, a drive recorder and is a camera that captures the front of the vehicle V. For example, the camera 50 assigns a time stamp to each captured camera image and outputs it to the in-vehicle device 1. Note that the assignment of the time stamp may be executed on the side of the in-vehicle device 1.

[0036] The sensor 51 is a sensor mounted on the vehicle V. For example, the sensor 51 is a vehicle speed sensor that measures a vehicle speed pulse and outputs a vehicle speed pulse signal corresponding to the measured vehicle speed pulse to the in-vehicle device 1. Note that the sensor 51 may include other sensors such as an acceleration sensor.

[0037] The notification unit 52 is composed of a display and a speaker, and notifies the driver of the vehicle V of the information notified from the in-vehicle device 1. For example, the notification unit 52 notifies the driver of warning information and the like notified from the in-vehicle device 1 when signal ignoring of the vehicle V is detected by the in-vehicle device 1.

[0038] As shown in FIG. 5, the in-vehicle device 1 includes a controller 3 and a storage unit 4. The storage unit 4 is realized by a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), or a flash memory. Various types of information and various programs necessary for the detection of traffic lights are stored in advance in the storage unit 4.

[0039] The controller 3 corresponds to a so-called processor. The controller 3 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), or the like. The controller 3 executes various programs stored in the storage unit 4 using the RAM as a work area. Further, the controller 3 can be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0040] The controller 3 detects a light-emitting body corresponding to the signal lamp from the camera image taken by the vehicle V, and determines the light-emitting state of the signal lamp based on the detected light-emitting body. First, the controller 3 extracts a rectangular area corresponding to the signal lamp from the camera image taken by the vehicle V.

[0041] The controller 3 extracts, as the rectangular area, a bounding box obtained by a deep learning method from the camera image, where the score of the signal lamp is equal to or greater than a certain value.

[0042] Further, the controller 3 detects a light-emitting body corresponding to the signal lamp from the extracted rectangular area. The controller 3 uses a pre-trained model by a machine learning method such as deep learning to detect the light-emitting body corresponding to the signal lamp from the rectangular area. At this time, the controller 3 detects the light-emitting bodies corresponding to the red signal, blue signal, and yellow signal from the rectangular area, respectively.

[0043] Subsequently, the controller 3 calculates the movement amount of the light-emitting body from the change in the position of the detected light-emitting body. The controller 3 projects the position of the vehicle and the light-emitting body onto a virtual plane (see FIG. 2) set based on the height of the signal lamp, and calculates the movement amount of the light-emitting body according to the positional relationship between the light-emitting body and the vehicle on the virtual plane.

[0044] The controller 3 calculates, as the movement amount of the light-emitting body, the movement speed or movement distance of the light-emitting body on the virtual plane. The controller 3 tracks the light-emitting body on the virtual plane and calculates the movement distance of the light-emitting body by obtaining the amount of change in the coordinates of the light-emitting body between frames.

[0045] Also, the controller 3 calculates the movement speed of the light-emitting body based on the movement distance of the light-emitting body on the virtual plane and the time stamp attached to each camera image. The movement speed of the light-emitting body can be obtained by dividing the "movement distance of the light-emitting body" by the "difference in time at the time stamp".

[0046] Thereafter, based on the calculated movement amount, the controller 3 determines the target signal. The controller 3 excludes a light emitter that does not satisfy any of the conditions described below from the target signal, and determines a light emitter that satisfies all of the conditions described below as the target signal.

[0047] First, the controller 3 determines whether a light emitter exists within the detection range A shown in FIG. 3. That is, the controller 3 determines whether a light emitter exists on the planned path of the vehicle V, excludes a light emitter outside the detection range A from the target signal at this time, and continues the subsequent processing for the light emitter existing within the detection range A.

[0048] Next, the controller 3 determines whether a condition regarding the movement amount of the light emitter is satisfied. The controller 3 excludes from the target signal a light emitter whose movement speed obtained from the movement amount of the light emitter is equal to or less than the threshold value.

[0049] As described above, when performing the signal ignoring determination, when the vehicle V performs signal ignoring, it can be assumed that the vehicle is traveling at a running speed of a certain level or more. Therefore, in this case, the controller 3 excludes a light emitter from the target signal if the movement speed or movement distance of the light emitter is equal to or less than the threshold value on the premise that the vehicle V is at a certain speed or more.

[0050] In this case, since the controller 3 can exclude a light emitter actually existing farther than the detection range A from the target signal, false detection of the traffic signal can be suppressed. At this time, the controller 3 may exclude a light emitter from the target signal if the movement speed or movement distance of the light emitter is equal to or less than the threshold value on the condition that the vehicle speed of the vehicle V is the threshold value.

[0051] The controller 3 obtains the running speed of the vehicle V based on the vehicle speed pulse input from the sensor 51 (see FIG. 5), and if the running speed is faster than the threshold value and the movement speed or movement distance of the light emitter is equal to or less than the threshold value, the controller 3 excludes the light emitter from the target signal.

[0052] In this way, based on the actual traveling speed of the vehicle V, the controller 3 activates the above conditions, thereby avoiding problems such as excluding the actual traffic signal from the target signal, for example, when the vehicle V is stopped. Note that the controller 3 may calculate and use information regarding the actual traveling speed of the vehicle V from, for example, the transition of position information or the like, acceleration, etc., in addition to the vehicle speed pulse.

[0053] As a next condition, the controller 3 determines whether or not the difference between the actual movement amount of the vehicle V and the movement amount of the light emitter is equal to or less than a threshold value. If the light emitter is the projector of the traffic signal, the difference between the movement amount of the vehicle V and the movement amount of the light emitter is small. However, when the light emitter is a light of a building or the like located farther away than the traffic signal, the difference between the movement amount of the vehicle V and the movement amount of the light emitter becomes large. Therefore, if the difference between the traveling speed of the vehicle V and the movement speed of the light emitter exceeds the threshold value, the controller 3 excludes the light emitter from the target signal.

[0054] That is, the controller 3 excludes a light emitter having a movement speed sufficiently lower than the traveling speed of the vehicle V from the target signal. A light emitter excluded from the target signal under such conditions is a light emitter that is actually located farther away than the detection range A and is at a position higher than the traffic signal.

[0055] In this way, based on the difference between the movement amount of the vehicle V and the movement amount of the light emitter, the controller 3 performs target signal determination, thereby being able to exclude a light emitter other than the traffic signal from the target signal.

[0056] Next, the controller 3 performs target signal determination by paying attention to the light emission time of the light emitter. The controller 3 excludes a light emitter having a light emission period in which continuous light emission in a specific color exceeds a threshold value from the target signal.

[0057] Specifically, assume a case where the traffic signal is a yellow signal. In such a case, since the traffic signal changes from a yellow signal to a red signal in about several seconds, the light emission period of continuous light emission is limited.

[0058] Therefore, the controller 3 excludes a light emitter whose continuously emitted light period is equal to or less than the threshold value from the target signals, thereby excluding a traffic signal that is lit with a yellow signal from the target signals.

[0059] Here, actually, although it is a yellow signal, when it is detected as a red signal as a result of camera image processing, it is difficult to distinguish in the target signal determination based on the above-described movement amount. In contrast, the controller 3 can exclude a yellow signal from the target signals by performing target signal determination based on the light emission period, so that the detection accuracy of the red signal can be improved.

[0060] In addition to these conditions, the controller 3 determines a light emitter that approaches the vehicle V straight ahead as a target signal, and excludes from the target signals a traffic signal (light emitter) that passes horizontally through the detection range A with respect to the vehicle V and a traffic signal that approaches by turning from the detection range A.

[0061] After these target signal determinations, the controller 3 determines a light emitter that satisfies all the conditions as a target signal. Then, the controller 3 executes a signal ignoring determination. The controller 3 performs a color determination of the light emitter determined as a target signal, and when the color is red and the vehicle V has passed the target signal, determines that the vehicle V has ignored the signal.

[0062] When the controller 3 determines that the vehicle V has ignored the signal, it warns the driver through the notification unit 52. In this way, by warning the driver, it is possible to improve the awareness of safe driving.

[0063] Next, with reference to FIG. 6, the processing procedure executed by the in-vehicle device 1 according to the embodiment will be described. FIG. 6 is a flowchart showing the processing procedure executed by the in-vehicle device 1. The following-described processing procedure is repeatedly executed by the controller 3 of the in-vehicle device 1 every time a camera image is acquired.

[0064] As shown in FIG. 6, when the controller 3 acquires a camera image from the camera 50 (step S101), it extracts a rectangular area by camera image recognition (step S102). The controller 3 extracts, as the rectangular area, a bounding box in which the score of the traffic signal is equal to or higher than a certain value, obtained by a deep learning method such as YOLO.

[0065] Subsequently, the controller 3 detects a light-emitting body of the signal color corresponding to the traffic signal light from the extracted rectangular area (step S103). The controller 3 detects the light-emitting body of the signal color from the rectangular area using a learned model by a machine learning method such as deep learning.

[0066] Subsequently, the controller 3 determines whether or not the detected light-emitting body is within the detection range A (step S104). The controller 3 projects the light-emitting body onto a virtual plane and determines whether or not its coordinates are within the detection range A.

[0067] When the controller 3 determines that the light-emitting body is within the detection range A (step S104; Yes), it calculates the movement amount of the light-emitting body (step S105). The controller 3 calculates the movement amount of the light-emitting body from the coordinate change of the light-emitting body in the virtual plane. Note that the movement amount of the light-emitting body includes the movement distance and the movement speed of the light-emitting body.

[0068] Subsequently, the controller 3 determines whether or not the calculated movement amount of the light-emitting body satisfies the movement amount condition (step S106). The movement amount condition includes that the movement speed of the light-emitting body is equal to or higher than a threshold value and that the difference between the traveling speed of the vehicle V and the movement speed of the light-emitting body is equal to or lower than a threshold value.

[0069] That is, in step S106, the controller 3 excludes a light-emitting body such as a street lamp whose movement amount is small and which is actually located farther away than the detection range A. When the controller 3 determines in step S106 that the light-emitting body satisfies the movement amount condition (step S106; Yes), it determines whether or not the light-emitting time condition is satisfied (step S107).

[0070] The controller 3 determines whether the light emission time continuously issued by the light emitter is less than or equal to the threshold value. Then, the controller 3 determines that such a light emitter is likely to be a yellow signal and will be excluded at this stage.

[0071] When the controller 3 determines that the light emitter satisfies the light emission time condition (step S107; Yes), the controller 3 determines whether the light emitter satisfies the straight-ahead condition passing condition (step S108).

[0072] As shown in FIG. 3, the controller 3 determines whether the light emitter approaches the vehicle V straight ahead based on the locus of the light emitter in the virtual plane. When the controller 3 determines that the light emitter satisfies the straight-ahead passing condition (step S108; Yes), the controller 3 detects it as a traffic signal light (step S109) and ends the process.

[0073] Also, when the controller 3 determines that the condition is not satisfied in any of the determinations in step S104, step S106, step S107, and step S108 (step S104 / step S106 / step S107 / step S108: No), the controller 3 ends the process. That is, in this case, the controller 3 ends the process without detecting the light emitter as a traffic signal light.

[0074] As described above, the in-vehicle device 1 (an example of a detection device) according to the embodiment includes a controller 3. The controller 3 detects a light emitter corresponding to a traffic signal lamp from a camera image captured by the vehicle, calculates the movement amount of the light emitter from the change in the position of the light emitter, excludes a light emitter with a movement amount less than or equal to the threshold value from the detection target of the lamp, and detects a light emitter with a movement amount exceeding the threshold value as a lamp.

[0075] Therefore, according to the in-vehicle device 1 according to the embodiment, light emitters such as streetlights and building lights located far away can be excluded from the detection target of traffic signal lights, so that the detection accuracy of traffic signal lights can be improved.

[0076] Further effects and modifications can be easily derived by those skilled in the art. Therefore, a broader aspect of the present invention is not limited to the specific details and representative embodiments presented and described as above. Accordingly, various changes are possible without departing from the spirit or scope of the general inventive concept defined by the appended claims and their equivalents.

Explanation of Signs

[0077] 1 Vehicle-mounted device 3 Controller 4 Storage unit 50 Camera 51 Sensor 52 Notification unit 300 Traffic signal A Detection range V Vehicle

Claims

1. having a controller, the controller: detects a light-emitting body corresponding to a signal lamp from a camera image taken by a vehicle, calculates the movement amount of the light-emitting body from the change in the position of the light-emitting body, excludes the light-emitting body with a movement amount equal to or less than a threshold value from the detection target of the signal lamp, and detects the light-emitting body with a movement amount exceeding the threshold value as the signal lamp, a detection device.

2. the controller: excludes the light-emitting body with a difference between the movement amount of the vehicle and the movement amount of the light-emitting body equal to or less than a threshold value from the detection target, the detection device according to Claim 1.

3. the controller: excludes the light-emitting body with a movement speed of the light-emitting body obtained from the movement amount of the light-emitting body equal to or less than a threshold value from the detection target, the detection device according to Claim 1.

4. the controller: when the traveling speed of the vehicle exceeds a speed threshold, excludes the light-emitting body with a movement speed of the light-emitting body equal to or less than a threshold value from the detection target, the detection device according to Claim 3.

5. the controller: excludes the light-emitting body with a difference between the traveling speed of the vehicle and the movement speed of the light-emitting body equal to or more than a threshold value from the detection target, the detection device according to Claim 1.

6. the controller: excludes the light-emitting body with a light-emitting period of continuously emitting light in a specific color exceeding a threshold value from the detection target, the detection device according to Claim 1.

7. the controller: projects the position of the vehicle onto a virtual plane set based on the height of the signal lamp, and calculates the movement amount of the light-emitting body according to the positional relationship between the light-emitting body and the vehicle in the virtual plane, the detection device according to Claim 1.

8. a detection method executed by a controller, comprising: detecting a light-emitting body corresponding to a signal lamp from a camera image taken by a vehicle, calculating the movement amount of the light-emitting body from the change in the position of the light-emitting body, excluding the light-emitting body with a movement amount equal to or less than a threshold value from the detection target of the signal lamp, and detecting the light-emitting body with a movement amount exceeding the threshold value as the signal lamp, a detection method.

9. a detection procedure for detecting a light-emitting body corresponding to a signal lamp from a camera image taken by a vehicle, a calculation procedure for calculating the movement amount of the light-emitting body from the change in the position of the light-emitting body, a detection procedure for excluding the light-emitting body with a movement amount equal to or less than a threshold value from the detection target of the signal lamp, and detecting the light-emitting body with a movement amount exceeding the threshold value as the signal lamp A detection program that causes a computer to execute.

Images