Object detection device and object detection method
The object detection device enhances detection accuracy by dynamically adjusting thresholds based on consecutive object detection and environmental conditions, addressing the issue of reduced accuracy for distant objects.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- DENSO TEN LTD
- Filing Date
- 2022-06-17
- Publication Date
- 2026-05-11
AI Technical Summary
Conventional object detection methods, particularly using deep learning, struggle with reduced detection accuracy for distant objects due to smaller detection frames having fewer features and lower confidence levels, leading to missed detections.
An object detection device adjusts its confidence threshold dynamically by setting a second, lower threshold when detecting a series of consecutive objects, and resets to a first threshold based on predefined conditions such as distance or vehicle stability, enhancing detection accuracy for continuous objects like traffic signals.
The method improves the detection accuracy of continuous objects by preventing missed detections and maintaining reliable object recognition even at varying distances and conditions, particularly around intersections.
Smart Images
Figure 0007856503000001 
Figure 0007856503000002 
Figure 0007856503000003
Abstract
Description
Technical Field
[0004]
[0001] The disclosed embodiments relate to an object detection device and an object detection method.
Background Art
[0002] Conventionally, a technique for detecting a traffic signal from a camera image obtained by photographing the front of a vehicle with an in-vehicle camera is known. Some such techniques detect a traffic signal after setting the installation position of the traffic signal registered in advance in map information as a detection target area (see, for example, Patent Document 1).
[0003] In addition, some such techniques extract, as a detection frame, an area estimated to be a traffic signal in a camera image using a machine learning algorithm such as deep learning, and detect the traffic signal based on the confidence level of the calculated detection frame.
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0005] However, there is still room for further improvement in the above-described conventional techniques in terms of improving the detection accuracy of traffic signals.
[0006] For example, in the technique using deep learning described above, the detection frame of a distant traffic signal is likely to be extracted small. Since a small detection frame tends to have few features and a low confidence level, the traffic signal is likely to be determined as undetected.
[0007] Note that such problems are not limited to traffic signals, but are common to all detection targets that tend to appear continuously in a camera image like traffic signals.
[0008] One embodiment has been made in view of the above, and aims to provide an object detection device and an object detection method that can improve the detection accuracy of objects to be detected. [Means for solving the problem]
[0009] An object detection device according to one embodiment is an object detection device mounted on a moving object and includes a control unit. The control unit detects an object when the confidence level included in the image recognition result for a camera image of a learning model that has learned the features of the object to be detected exceeds a threshold. Furthermore, when the control unit detects a series of objects consisting of one or more consecutive objects of the same type in the camera image as the object to be detected, it sets a second threshold lower than a predetermined first threshold. If the recovery condition is met, which determines that the continuity of detection of the continuous detection object has been interrupted, the first threshold is set to the threshold; and if the recovery condition is met and the first continuous detection object is detected, the second threshold is set to the threshold. do. [Effects of the Invention]
[0010] According to one embodiment, the detection accuracy of the object to be detected can be improved. [Brief explanation of the drawing]
[0011] [Figure 1] Figure 1 shows an example of the mounting of an object detection device according to an embodiment. [Figure 2] Figure 2 is a schematic diagram (part 1) illustrating the object detection method according to the embodiment. [Figure 3] Figure 3 is a schematic diagram (part 2) illustrating the object detection method according to the embodiment. [Figure 4] Figure 4 is a block diagram showing an example configuration of an object detection device according to an embodiment. [Figure 5] Figure 5 is an explanatory diagram of the first modified example. [Figure 6] Figure 6 is an explanatory diagram of the second modified example. [Figure 7] Figure 7 is a flowchart showing the processing procedure performed by the object detection device according to the embodiment. [Modes for carrying out the invention]
[0012] The embodiments of the object detection device and object detection method disclosed herein will be described in detail below with reference to the attached drawings. However, the present invention is not limited to the embodiments described below.
[0013] Furthermore, in the following, we will assume that the objects to be detected are those that appear consecutively in the camera image, such as traffic lights, with one or more of the same type. Such objects will be referred to as "consecutive detection objects" as appropriate below. In this embodiment, we will give an example where such consecutive detection objects are traffic lights. Traffic lights rarely exist alone at a single intersection, so they can be considered consecutive detection objects.
[0014] First, an overview of the object detection device and object detection method according to the embodiment will be explained using Figures 1 to 3. Figure 1 is a diagram showing an example of the mounting of the object detection device 10 according to the embodiment. Figures 2 and 3 are schematic diagrams (part 1) and (part 2) illustrating the object detection method according to the embodiment.
[0015] As shown in Figure 1, the object detection device 10 according to this embodiment is mounted on a vehicle V, for example, and detects objects to be detected by image recognition processing based on camera images captured by the camera 11 (see Figure 4) of the object detection device 10.
[0016] In the example shown in Figure 1, the object detection device 10 is a drive recorder that is mounted to capture images in front of the vehicle V. Note that the object detection device 10 may be a separate device from the drive recorder, and may be configured to detect objects from camera images surrounding the vehicle V, including the rear and sides, in addition to the front of the vehicle V.
[0017] For example, the object detection device 10 uses a deep learning algorithm to perform image recognition processing for detecting a traffic signal, which is a detection target, from a camera image in which the front of the vehicle V is photographed. More specifically, the object detection device 10 uses a learning model obtained by learning the feature amounts of detection targets using a deep learning algorithm, and detects a detection frame whose confidence level included in the image recognition result of the learning model for the camera image exceeds a preset threshold as a detection target.
[0018] In such an object detection method using machine learning, the larger the detection target in the camera image, the more accurately it can be detected, and the smaller the detection target in the camera image, the more likely the detection accuracy is to decrease. That is, the farther away from the vehicle V, the smaller the detection target appears in the camera image, and thus the detection accuracy tends to decrease.
[0019] More specifically, as shown in FIG. 2, for example, the detection frames BB1 to BB5 of the traffic signal tend to be larger the closer they are to the vehicle V and smaller the farther away they are from the vehicle V. Further, if the size of the detection frame is small, the feature amount thereof also becomes small, and thus the confidence level also tends to become low.
[0020] In such a case, if the threshold value for the confidence level is a uniform value, although there is a tendency to more easily detect the traffic signal the closer it is to the vehicle V, on the other hand, there is a risk that it may not be detected the farther away it is from the vehicle V. In the example of FIG. 2, for example, when the uniform confidence level threshold value is "0.5", the detection frames BB1 and BB2 are detected as traffic signals, but the detection frames BB3 to BB5 are not detected as traffic signals. That is, a detection omission of a traffic signal that should appear in the camera image occurs.
[0021] Regarding this point, for example, it is also conceivable to make the threshold value variable according to the size of the detection frame for each detection frame. However, since complicated arithmetic processing taking into account the depth and the like in the three-dimensional space for each detection frame is required, it is generally not preferable to apply it to in-vehicle devices with limited processing capabilities.
[0022] Therefore, in the object detection method according to this embodiment, when the control unit 14 (see Figure 4) of the object detection device 10 detects a series of objects consisting of one or more consecutive objects of the same type in the camera image as the object to be detected, it sets a second confidence threshold that is lower than a predetermined first threshold.
[0023] As shown in Figure 3, there is a warning signal N in the direction of travel a1 of vehicle V, and further ahead there is an intersection containing signals A to C. Such warning signal N and signals A to C are "continuously detected objects" that appear one or more times in succession in the camera image and belong to the same type of traffic light.
[0024] First, as shown in Figure 3, the control unit 14 sets a predetermined first threshold value to the confidence threshold mentioned above when the continuity of detection of continuously detected objects is interrupted. In the example in Figure 3, the first threshold value is "0.5".
[0025] Then, when the control unit 14 detects a warning signal N at time T1, for example, while the vehicle V is in motion, through image recognition processing using the first threshold, it uses this detection as a trigger to set a second threshold that is smaller than the first threshold. In the example in Figure 3, the second threshold is "0.25".
[0026] Then, as the vehicle V continues to move, the control unit 14 detects signals A, B, C... through image recognition processing using the second threshold. Note that the spacing between locations where traffic lights are installed, such as intersections, varies depending on the layout of the city and road conditions. For example, there may be another traffic light immediately after passing an intersection containing signals A to C.
[0027] The control unit 14 detects signals A, B, C, etc. by image recognition processing using a second threshold until the continuity of detection, including that of the other traffic lights, is interrupted (see time points T1 to T2 in the figure). According to the image recognition processing using the second threshold, for example, detection frames BB1 to BB5 shown in Figure 2 will all be detected as traffic lights.
[0028] Then, when the control unit 14 detects that the detection of signals A, B, C, etc. is interrupted at time T2, and that a predetermined distance (30m in this case) has been moved from time T2 at time T3, it sets the threshold to a first threshold. In other words, the control unit 14 resets the confidence threshold to a predetermined first threshold, using the occurrence of each event at time T2 and time T3 as the threshold reset condition.
[0029] The control unit 14 detects events at time T3 by utilizing, for example, changes in GPS (Global Positioning System) positioning location or vehicle speed data based on vehicle speed pulses. The control unit 14 then repeats the series of object detection processes shown in Figure 3 while the vehicle V is in motion.
[0030] As a result, the object detection method according to this embodiment can improve the detection accuracy of continuous objects such as traffic lights. Of course, continuous objects are not limited to traffic lights. For example, guide signs are likely to appear continuously near road junctions and can therefore be considered continuous objects.
[0031] The configuration example of the object detection device 10 will be described in more detail below. Figure 4 is a block diagram showing a configuration example of the object detection device 10 according to this embodiment. Note that Figure 4 shows only the components necessary to explain the features of this embodiment, and descriptions of general components are omitted.
[0032] In other words, each component shown in Figure 4 is a functional concept and does not necessarily need to be physically configured as shown. For example, the specific forms of distribution and integration of each block are not limited to those shown, and it is possible to configure all or part of them by functionally or physically distributing and integrating them in any unit according to various loads and usage conditions.
[0033] Furthermore, in the explanation using Figure 4, explanations of components that have already been explained may be simplified or omitted.
[0034] As shown in Figure 4, the object detection device 10 according to this embodiment includes a camera 11, a sensor unit 12, a storage unit 13, and a control unit 14. The camera 11 is mounted at various locations on the vehicle V, such as the windshield and dashboard, and captures a predetermined shooting area in front of the vehicle V.
[0035] The sensor unit 12 is a group of various sensors. The sensor unit 12 includes GPS sensors, acceleration sensors, vehicle speed sensors, etc. The camera 11 and the sensor unit 12 may be located outside the object detection device 10 and connected to the object detection device 10 via an in-vehicle network such as CAN (Controller Area Network).
[0036] The memory unit 13 is implemented by a memory device such as RAM (Random Access Memory) or flash memory, and in the example shown in Figure 4, it stores continuous detection information 13a, image recognition AI (Artificial Intelligence) 13b, and threshold information 13c.
[0037] The continuous detection object information 13a is information defining the continuous detection object described above. The continuous detection object information 13a includes information on the labels that correspond to the continuous detection object among the labels indicating the object to be detected in each detection frame determined as an object to be detected by the determination unit 14c described later.
[0038] Image recognition AI13b corresponds to the aforementioned learning model. For example, image recognition AI13b is a DNN (Deep Neural Network) model trained using a deep learning algorithm.
[0039] After being loaded as a DNN model into the image recognition unit 14b (described later), the image recognition AI 13b outputs the detection frames of various objects to be detected in the camera image and the confidence level for each detection frame as image recognition results when a camera image captured by the camera 11 is input. In this embodiment, the image recognition AI 13b outputs at least the detection frames of traffic signals for vehicles and their confidence levels.
[0040] The threshold information 13c corresponds to the confidence thresholds mentioned above. The threshold information 13c is set to either a first threshold or a second threshold by the setting unit 14a, which will be described later.
[0041] The control unit 14 is a controller and is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing a program (not shown) stored in the memory unit 13 using RAM as the working area. The control unit 14 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).
[0042] The control unit 14 includes a setting unit 14a, an image recognition unit 14b, a determination unit 14c, and an output unit 14d, and realizes or executes the information processing functions and operations described below.
[0043] Based on the label of the detection frame and the continuous detection object information 13a determined by the determination unit 14c to have been detected, the setting unit 14a determines that the detection frame corresponds to a continuous detection object and sets the second threshold in the threshold information 13c if the threshold currently set is the first threshold.
[0044] Furthermore, the setting unit 14a sets the first threshold in the threshold information 13c when the determination unit 14c determines that the aforementioned threshold return condition has been met.
[0045] The image recognition unit 14b acquires the camera image captured by the camera 11 and inputs this camera image to the image recognition AI 13b. The image recognition unit 14b also acquires the detection frame and confidence level, which are the image recognition results output from the image recognition AI 13b, based on the input camera image. The image recognition unit 14b also outputs the image recognition results obtained from the image recognition AI 13b to the determination unit 14c.
[0046] The determination unit 14c determines whether the confidence level for each detection frame exceeds the threshold set in the threshold information 13c, based on the image recognition results input from the image recognition unit 14b. If it exceeds the threshold, the determination unit 14c detects the corresponding detection frame as a target object. If it is below the threshold, the determination unit 14c deems the corresponding detection frame undetected and deletes it. The determination unit 14c also outputs the determination result for each detection frame to the setting unit 14a and the output unit 14d.
[0047] Furthermore, the determination unit 14c determines whether the aforementioned threshold return condition is true (fulfilled) based on the detection frame determination result and sensor data from the sensor unit 12. In other words, the determination unit 14c determines the surrounding conditions of the vehicle V. If the return condition is true, the determination unit 14c instructs the setting unit 14a to set the first threshold in the threshold information 13c.
[0048] The output unit 14d outputs the detection frame determination result from the determination unit 14c to the external device 30. The external device 30 is a device that executes applications corresponding to various functions performed based on the detection frame determination result. The external device 30 is, for example, an ECU (Electronic Control Unit) or a car navigation system. The external device 30 may be configured integrally with the object detection device 10.
[0049] By the way, in the explanation using Figure 3, an example was given in which the control unit 14 sets a second threshold in the threshold information 13c triggered by the detection of the warning signal N. However, there are cases where there is no warning signal N before the intersection. A modified example of such a case will be explained using Figure 5.
[0050] Figure 5 is an explanatory diagram of the first modified example. Similar to Figure 3, the control unit 14 sets the confidence threshold to a predetermined first threshold of "0.5" when the continuity of detection of the continuously detected object is interrupted.
[0051] Then, while the vehicle V is in motion, if the control unit 14 detects any of the signals A to C shown in Figure 3 at time T1, for example, through image recognition processing using the first threshold, it sets the threshold to a second threshold of "0.25", which is smaller than the first threshold.
[0052] Then, as the vehicle V continues to move, the control unit 14 detects signals A, B, C… through image recognition processing using the second threshold. The control unit 14 continues to detect signals A, B, C… through image recognition processing using the second threshold until the continuity of detection, including other traffic signals besides signals A to C, is interrupted (see time points T1 to T2 in the figure).
[0053] Then, when the control unit 14 detects that the detection of signals A, B, C… has ceased at time T2, and that a predetermined distance has been moved from time T2 at time T3, it sets a first threshold value. The control unit 14 then repeats the series of object detection processes shown in Figure 5.
[0054] Furthermore, while the explanations using Figures 3 and 5 show an example where the threshold remains at the second threshold until the aforementioned threshold reset condition is met, the threshold may also be temporarily set to the first threshold depending on the speed of vehicle V. A modified example of this case will be explained using Figure 6.
[0055] Figure 6 is an explanatory diagram of the first modified example. Similar to Figures 3 and 5, the control unit 14 sets the confidence threshold to a predetermined first threshold of "0.5" when the continuity of detection of the continuously detected object is interrupted.
[0056] Then, while the vehicle V is in motion, if the control unit 14 detects any of the signals A to C shown in Figure 3 at time T1, for example, through image recognition processing using the first threshold, it sets the threshold to a second threshold of "0.25", which is smaller than the first threshold.
[0057] Then, as the vehicle V continues to move, the control unit 14 detects signals A, B, C… through image recognition processing using this second threshold. However, as shown in Figure 6, if the vehicle speed of vehicle V becomes 0, for example, when stopped at a traffic light at an intersection, the control unit 14 returns the threshold to the first threshold during the time the vehicle is stopped at the traffic light (see time points T11 to T12 in the figure).
[0058] When the vehicle speed is 0, for example, vibrations of the vehicle V are reduced, so the shooting conditions of camera 11 become more stable. In other words, the image quality of the camera image is more stable compared to when the vehicle is moving, so the confidence level is expected to be higher even when detecting distant traffic lights compared to when the vehicle is moving.
[0059] In this case, while the vehicle speed is 0, the likelihood of failure to detect anything even if the threshold is returned to the first threshold is low. Note that "vehicle speed of 0" here does not mean exactly 0, but may include the case where the vehicle speed is approximately 0 before and after the vehicle V comes to a complete stop. As shown in Figure 6, the confidence threshold can be appropriately changed in response to changes in the situation, such as vehicle speed.
[0060] Next, the processing procedure performed by the object detection device 10 will be explained using Figure 7. Figure 7 is a flowchart showing the processing procedure performed by the object detection device 10 according to this embodiment.
[0061] As shown in Figure 7, the control unit 14 of the object detection device 10 sets a first confidence threshold when the continuity of detection of continuously detected objects is interrupted (step S101). Then, the control unit 14 acquires the camera image from the camera 11 and performs image recognition processing (step S102).
[0062] The control unit 14 then determines the image recognition result using the set threshold (step S103). The control unit 14 also outputs the determination result to the external device 30 (step S104).
[0063] On the other hand, the control unit 14 determines whether or not it has detected a continuous object (step S105). If it has detected a continuous object (step S105, Yes), the control unit 14 determines whether or not the first threshold value is set (step S106).
[0064] If a first threshold is set (step S106, Yes), the control unit 14 sets a second threshold that is lower than the first threshold (step S107). If no continuous detection object is detected (step S105, No) or if a second threshold is set (step S106, No), the control unit 14 proceeds to step S108.
[0065] In step S108, the control unit 14 determines whether the threshold return condition described above is true (step S108). If the threshold return condition is true (step S108, Yes), the control unit 14 repeats the process from step S101. On the other hand, if the threshold return condition is false (step S108, No), the control unit 14 repeats the process from step S102.
[0066] As described above, the object detection device 10 according to the embodiment is an object detection device mounted on a vehicle V (corresponding to an example of a "moving object") and includes a control unit 14. The control unit 14 detects the object when the confidence level included in the image recognition result for the camera image of the image recognition AI 13b (corresponding to an example of a "learning model"), which has learned the features of the object to be detected, exceeds a threshold. Furthermore, when the control unit 14 detects a series of objects consisting of one or more consecutive objects of the same type in the camera image as the object to be detected, it sets a second threshold lower than a predetermined first threshold.
[0067] Therefore, according to the object detection device 10 of this embodiment, the detection accuracy of the object to be detected, especially continuous detection objects, can be improved.
[0068] Furthermore, the control unit 14 sets the first threshold to the threshold when it determines that the continuity in which the continuous detection of the continuous object has been interrupted, and sets the second threshold to the threshold when the return condition is met and the first continuous object is detected.
[0069] Therefore, according to the object detection device 10 of this embodiment, it is possible to prevent the failure to detect traffic signals, for example, which are numerous around intersections.
[0070] Furthermore, the control unit 14 sets the second threshold value to the threshold value from the time the continuous detection object is detected until the return condition is met.
[0071] Therefore, according to the object detection device 10 of this embodiment, it is possible to prevent, for example, failure to detect traffic signals while driving around an intersection.
[0072] Furthermore, the control unit 14 defines the return condition as the vehicle V moving a predetermined distance after the continuous detection of the object has ceased.
[0073] Therefore, according to the object detection device 10 of the embodiment, the detection accuracy of objects other than continuously detected objects can be improved after the situation in which the continuity is interrupted has stabilized. For example, the detection accuracy of objects other than traffic lights after passing around an intersection can be improved.
[0074] Furthermore, if the vehicle speed of the vehicle V (corresponding to an example of "movement speed") is 0, the control unit 14 sets the first threshold value to the threshold value.
[0075] Therefore, according to the object detection device 10 of this embodiment, when the vehicle V is stopped and the camera image capture conditions are stable, the detection accuracy of the object to be detected can be improved by using the first threshold for object detection.
[0076] Furthermore, the continuous detection objects mentioned above are traffic lights or road signs.
[0077] Therefore, according to the object detection device 10 of this embodiment, the detection accuracy of traffic lights or road signs can be improved in particular.
[0078] Furthermore, the object detection method according to the embodiment is an object detection method performed by an object detection device 10 mounted on a vehicle V, which includes detecting an object when the confidence level included in the image recognition result for a camera image of a learning model that has learned the features of the object to be detected exceeds a threshold, and setting a second threshold lower than a predetermined first threshold to the threshold when detecting a series of objects consisting of one or more consecutive objects of the same type in the camera image as the object to be detected.
[0079] Therefore, according to the object detection device 10 of this embodiment, the detection accuracy of the object to be detected, especially continuous detection objects, can be improved.
[0080] In the embodiment described above, the object detection device 10 performs object detection based on image recognition on its own. However, some of the functions of the object detection device 10 may be performed by other devices. Examples of other devices include a server device that is connected to the object detection device 10 via a mobile phone network or the like and has functions equivalent to the image recognition function performed by the image recognition unit 14b.
[0081] Furthermore, in the embodiments described above, specific numerical values such as "0.5," "0.25," and "30m" were given for the first threshold, second threshold, and threshold return conditions, but these are merely examples and do not limit the range of values.
[0082] Further effects and modifications can be readily derived by those skilled in the art. Therefore, broader aspects of the present invention are not limited to the specific details and representative embodiments expressed and described above. Accordingly, various modifications are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and their equivalents. [Explanation of Symbols]
[0083] 10. Object detection device 11 Cameras 12 Sensor section 13 Storage section 13a Continuous detection information 13b Image recognition AI 13c threshold information 14 Control Unit 14a Setting section 14b Image Recognition Unit 14c Judgment section 14d Output Section 30 External device
Claims
1. An object detection device mounted on a moving object, The system includes a control unit that detects the target object when the confidence level included in the image recognition result for a camera image of a learning model that has learned the features of the target object exceeds a threshold. The control unit, When detecting a series of objects consisting of one or more consecutive objects of the same type in the camera image as the object to be detected, a second threshold lower than a predetermined first threshold is set to the threshold. If the recovery condition is met, which determines that the continuity of detection of the continuous object has been interrupted, the first threshold is set to the threshold; if the recovery condition is met and the first continuous object is detected, the second threshold is set to the threshold. Object detection device.
2. The control unit, The second threshold is set to the threshold from the time the continuous detection object is detected until the return condition is met. The object detection device according to claim 1.
3. The control unit, The return condition is defined as the moving body moving a predetermined distance after the continuous detection of the object ceases to be detected. The object detection device according to claim 2.
4. The control unit, If the moving speed of the moving body is 0, the first threshold is set to the threshold. The object detection device according to claim 1.
5. The aforementioned continuous detection object is a traffic light or a road sign. An object detection device according to any one of claims 1 to 4.
6. An object detection method performed by an object detection device mounted on a moving object, The detection of the target object occurs when the confidence level included in the image recognition result for a camera image of a learning model that has learned the features of the target object exceeds a threshold. When detecting a series of objects consisting of one or more consecutive objects of the same type in the camera image as the object to be detected, a second threshold lower than a predetermined first threshold is set to the threshold. If the recovery condition is met, which determines that the continuity of detection of the continuous detection object has been interrupted, the first threshold is set to the threshold; if the recovery condition is met and the first continuous detection object is detected, the second threshold is set to the threshold; An object detection method, including the following.