Image processing device and road sign detection method

Abstract

To reduce the occurrence of false detections when detecting road signs from images. [Solution] The image processing device according to the embodiment includes a controller for detecting road signs from an image. The controller detects road signs using an image recognition model trained on training data in which annotation frames are set so that a relatively wide area of ​​the background is included relative to the road signs.

Description

Technical Field

[0001] The disclosed embodiments relate to an image processing apparatus and a road sign detection method.

Background Art

[0002] Conventionally, there has been known a technique for assisting a driver by detecting a road sign from an image captured by an in-vehicle camera and performing a driving operation according to the content indicated by the road sign.

[0003] For example, Patent Document 1 discloses a technique for warning a driver if the detected road sign is a regulatory sign and the driving situation of the vehicle conflicts with the regulation content indicated by the regulatory sign.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when the above-described conventional technique is used, there may be a case where a store sign or the like similar to a road sign is erroneously detected as a road sign. The occurrence of such an erroneous detection is likely to occur, for example, when a road sign or a signboard is present far away from the vehicle and is detected by a detection frame having a small size and a low resolution.

[0006] One aspect of the embodiment has been made in view of the above, and an object thereof is to provide an image processing apparatus and a road sign detection method capable of reducing the occurrence of erroneous detection when detecting a road sign from an image.

Means for Solving the Problems

[0007] One embodiment includes a controller for detecting road signs from an image. The controller detects the road signs using an image recognition model trained on training data in which annotation frames are set so that a relatively wide area of ​​the background is included relative to the road signs. [Effects of the Invention]

[0008] According to one embodiment, the controller detects road signs using an image recognition model trained on training data in which annotation frames are set so that a relatively wider background is included for road signs compared to other objects. As a result, the controller can detect road signs including the features of the area surrounding the road sign, and can distinguish road signs from, for example, signs that resemble road signs but have different surrounding features. In other words, according to one embodiment, the occurrence of false detections when detecting road signs from images can be reduced. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is an overview diagram (part 1) of the road sign detection method according to the first embodiment. [Figure 2] Figure 2 is an overview diagram (part 2) of the road sign detection method according to the first embodiment. [Figure 3] Figure 3 shows an example of the configuration of an image processing apparatus according to the first embodiment. [Figure 4] Figure 4 is a flowchart showing the processing procedure performed by the image processing device according to the first embodiment. [Figure 5] Figure 5 shows a part of the configuration example of the image processing apparatus according to the second embodiment. [Figure 6] Figure 6 is an schematic diagram (part 1) illustrating the road sign detection method according to the second embodiment. [Figure 7] Figure 7 is an overview diagram (part 2) of the road sign detection method according to the second embodiment. [Figure 8]Figure 8 is a flowchart showing the processing procedure performed by the image processing apparatus according to the second embodiment. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the image processing apparatus and road sign detection method disclosed in this application will be described in detail with reference to the attached drawings. However, the present invention is not limited to the embodiments described below.

[0011] Furthermore, in the following description, the image processing device according to the embodiment is assumed to be an in-vehicle device mounted on a vehicle. Also, the road sign detection method according to the embodiment is assumed to be executed by the controller 12 (see Figure 3) of the image processing device.

[0012] Furthermore, the road sign detection method according to the first embodiment will be described below using Figures 1 to 4. The road sign detection method according to the second embodiment will also be described using Figures 5 to 8.

[0013] <First Embodiment> Figure 1 is an overview diagram (part 1) of the road sign detection method according to the first embodiment. Figure 2 is an overview diagram (part 2) of the road sign detection method according to the second embodiment. The road sign detection method according to the first embodiment is executed by the controller 12 of the image processing device 10 (see Figure 3) according to the first embodiment.

[0014] In the road sign detection method according to the first embodiment, the controller 12 detects road signs from an image. At this time, the controller 12 detects road signs using an image recognition model trained on training data in which annotation frames are set so that a relatively wide area of ​​the background is included relative to the road sign.

[0015] Specifically, the controller 12 detects road signs from the images captured by the in-vehicle camera. At this time, the controller 12 detects each road sign by performing image recognition processing on the image using an image recognition model 11a (see FIG. 3), which is an AI (Artificial Intelligence) model for image recognition.

[0016] The image recognition model 11a is, for example, a DNN (Deep Neural Network) model learned using a machine learning algorithm. The image recognition model 11a is pre-learned so as to be able to detect at least the class (type), shape, color, etc. of each object including road signs, and functions as an AI for image recognition when loaded by the controller 12.

[0017] Prior to its learning, annotation is performed on each image of the image group prepared in advance as a learning dataset for the image recognition model 11a. Annotation is an operation of labeling each object to be detected among the objects appearing in each image. Such labeling is performed by setting an annotation frame for the object to be detected. The annotation frame is a frame indicating the image range in the labeling of learning.

[0018] Here, when a road sign is the object to be detected, usually, as shown in "Annotation (before improvement)" in FIG. 1, the annotation frame AF1 is set so as to surround the main sign portion of the road sign as narrowly as possible. The annotation frame AF1 corresponds to an example of a "reference annotation frame serving as a reference".

[0019] However, when performing image recognition processing using the image recognition model 11a learned with the teacher data with the annotation frame AF1 set in this way, as shown in "False detection" in FIG. 1, there may be a case of falsely detecting a store signboard or the like similar to a road sign as a road sign.

[0020] Therefore, in the road sign detection method according to the first embodiment, as shown in "Annotation (This Embodiment)" in Figure 1, an annotation frame AF2 is set that is enlarged to include a relatively wide background relative to the road sign. The controller 12 then detects the road sign using an image recognition model 11a that has been trained on training data with the annotation frame AF2 set in this manner.

[0021] As a result, the controller 12 can detect road signs, including, for example, the feature quantities of the mounting pole P1 of the road sign in Figure 1. Since the sign shown as "falsely detected" in Figure 1 does not have the features of such a mounting pole P1, the possibility of the controller 12 falsely detecting such a sign as a road sign is reduced according to the road sign detection method of the first embodiment.

[0022] Furthermore, as shown in "False Detection" in Figure 2, a sign resembling a road sign is assumed to have the same characteristics as a road sign, including the mounting pole P1. In this case, as shown in "Annotation (Before Improvement)" in Figure 2, if the road sign has an auxiliary sign AS1, it is preferable to set the annotation frame AF2 to include such auxiliary sign AS1, as shown in "Annotation (This Embodiment)" in Figure 2.

[0023] By using the image recognition model 11a trained on training data with annotation frame AF2 set in this manner, the controller 12 can detect road signs, including the feature quantities of the auxiliary sign AS1. The sign shown as "false detection" in Figure 2 does not have the features of such auxiliary sign AS1, so according to the road sign detection method of the first embodiment, the possibility of the controller 12 falsely detecting such a sign as a road sign is reduced.

[0024] Furthermore, annotation frame AF2 may be enlarged relative to annotation frame AF1 by the same aspect ratio in both width and height (for example, by about 1.5 times), as shown in Figure 1, or it may be enlarged by different aspect ratios in both width and height, as shown in Figure 2. This makes it possible to set annotation frame AF2 to include various features of the road sign, depending on, for example, the manner in which the road sign is installed.

[0025] Thus, in the road sign detection method according to the first embodiment, the controller 12 detects road signs from the image. The controller 12 also detects road signs using an image recognition model 11a trained on training data in which the annotation frame AF2 is set so that a relatively wide background is included relative to the road sign.

[0026] Thus, the controller 12 detects road signs using an image recognition model 11a trained on training data in which the annotation frame AF2 is set so that a relatively wider background is included for road signs compared to other objects. As a result, the controller 12 can detect road signs including the features of the area surrounding the road sign, and can distinguish road signs from, for example, signs that look similar to road signs but have different surrounding features. In other words, the road sign detection method according to the first embodiment can reduce the occurrence of false detections when detecting road signs from images.

[0027] Below, we will describe in more detail an example of the configuration of an image processing device 10 to which the road sign detection method according to the first embodiment described above is applied.

[0028] Figure 3 shows an example of the configuration of the image processing device 10 according to the first embodiment. The image processing device 10 is a computer mounted in a vehicle.

[0029] As shown in Figure 3, the image processing device 10 comprises a storage unit 11 and a controller 12. The image processing device 10 is also connected to a camera 3, various sensors 5, an output unit 7, and an external device 9.

[0030] Camera 3 is one or more on-board cameras mounted on various parts of the vehicle. Camera 3 is positioned to capture at least the area in front of the vehicle. Camera 3 may be positioned to capture not only the area in front of the vehicle, but also the area behind, to the sides of, or even inside the vehicle's interior. Camera 3 may also be a 360-degree camera capable of capturing the entire area around the vehicle.

[0031] The various sensors 5 are various sensors other than the camera 3 that are mounted in various parts of the vehicle. The various sensors 5 include, for example, GPS (Global Positioning System) sensors and acceleration sensors.

[0032] The output unit 7 is an output device that displays output information from the image processing device 10. The output unit 7 is implemented by a display, speaker, or the like.

[0033] External devices 9 are various devices that utilize the image recognition results (detection results) from the image processing device 10. Examples of external devices 9 include various devices that perform various functions of advanced driver-assistance systems (ADAS), car navigation systems, and various ECUs (Electronic Control Units). This allows these devices to perform various driver assistance functions based on the detection results from the image processing device 10.

[0034] The external device 9 is connected to the image processing device 10 via an in-vehicle network such as CAN (Controller Area Network). The external device 9 may also be connected to the image processing device 10 wirelessly via Bluetooth (registered trademark), Wi-Fi (registered trademark), UWB (Ultra Wide Band), etc.

[0035] The memory unit 11 is implemented by a memory device such as RAM (Random Access Memory) or flash memory. The memory unit 11 stores a program (not shown) according to an embodiment executed by the controller 12. The memory unit 11 also stores various types of information used in the information processing executed by the controller 12.

[0036] In the example shown in Figure 3, the memory unit 11 stores the image recognition model 11a as various types of information. The image recognition model 11a functions as an AI for image recognition when it is loaded as an AI model into the controller 12. When functioning as an AI for image recognition, the image recognition model 11a is pre-trained to output at least the road signs visible in each frame as detection frames associated with a prediction class, when each frame of video captured by the camera 3 is input. In addition, when functioning as an AI for image recognition, the image recognition model 11a is pre-trained to output an accuracy score indicating the accuracy of the prediction for each detected detection frame. The accuracy score takes a value between 0 and 1, with a value closer to 1 indicating higher accuracy.

[0037] The controller 12 corresponds to a so-called processor. The controller 12 can be implemented by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc. The controller 12 reads the program according to the embodiment stored in the memory unit 11 and executes it using RAM as the working area. The controller 12 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0038] The controller 12 performs information processing according to the processing procedure shown in Figure 4. An explanation using Figure 4 will be given later.

[0039] The image processing device 10 can be implemented, for example, as part of a drive recorder. In this case, the image processing device 10 is implemented by a microcontroller mounted on the drive recorder. In this case, part or all of the camera 3 is implemented by a camera unit mounted on the drive recorder, and part or all of the output unit 7 is implemented by a display, speaker, etc., mounted on the drive recorder.

[0040] Furthermore, the image processing device 10 can be implemented as an image processing ECU. In this case, the camera 3 is implemented by, for example, a camera unit mounted on a drive recorder or an in-vehicle camera unit other than a drive recorder. The output unit 7 is implemented by a display, speaker, etc. mounted on a drive recorder, or an in-vehicle output device other than a drive recorder, such as an in-vehicle display or in-vehicle speaker.

[0041] Next, the processing procedure for information processing performed by the controller 12 of the image processing device 10 will be explained using Figure 4. Figure 4 is a flowchart of the processing procedure performed by the image processing device 10 according to the first embodiment.

[0042] First, the controller 12 acquires the image captured by camera 3 from camera 3 (step S101). Then, the controller 12 detects the road sign using an image recognition model trained on training data in which the annotation frame AF2 is set so that a relatively wide background is included relative to the road sign (step S102).

[0043] The controller 12 then outputs the detection result from step S102 to the external device 9 (step S103). This enables the external device 9 to perform information processing, such as driving assistance functions, based on the detection result from the controller 12. When the controller 12 uses the detection result itself, it performs the necessary information processing based on the detection result as appropriate and outputs the processing result to the output unit 7.

[0044] The controller 12 then determines whether or not the system has terminated (step S104). The controller 12 determines that the system has terminated if, for example, the ignition switch is turned off.

[0045] If the system is not shutting down (step S104, No), the controller 12 repeats the process from step S101. If the system is shutting down (step S104, Yes), the controller 12 terminates the process.

[0046] As described above, the image processing device 10 according to the first embodiment includes a controller 12 that detects road signs from an image. The controller 12 detects road signs using an image recognition model 11a trained on training data in which the annotation frame AF2 is set so that the background is included relatively broadly relative to the road signs.

[0047] Thus, the controller 12 detects road signs using an image recognition model 11a trained on training data in which the annotation frame AF2 is set so that the background is relatively wider for road signs compared to other objects. As a result, the controller 12 can detect road signs including the features of the area surrounding the road sign, and can distinguish road signs from, for example, signs that look similar to road signs but have different surrounding features. In other words, the image processing device 10 according to the first embodiment can reduce the occurrence of false detections when detecting road signs from an image.

[0048] <Second Embodiment> Next, a second embodiment will be described. Note that the configuration example of the image processing apparatus 10A according to the second embodiment differs from that in Figure 3 only in some aspects. Therefore, only the differing aspects of the configuration example of the image processing apparatus 10A according to the second embodiment will be described.

[0049] Figure 5 shows a part of the configuration example of the image processing apparatus 10A according to the second embodiment. As shown in Figure 5, the image processing apparatus 10A differs from Figure 3 in that the storage unit 11 stores a first image recognition model 11aa and a second image recognition model 11ab. The first image recognition model 11aa corresponds to an example of the "first image recognition model". The second image recognition model 11ab corresponds to an example of the "second image recognition model".

[0050] The first image recognition model 11aa and the second image recognition model 11ab are AI models for image recognition, similar to image recognition model 11a. Both the first image recognition model 11aa and the second image recognition model 11ab, when loaded into the controller 12 and functioning as AI for image recognition, output at least the detection frame, prediction class, and accuracy of road signs in the image. However, the first image recognition model 11aa and the second image recognition model 11ab differ in the classification level of the predicted road sign class they output.

[0051] Figure 6 is an schematic diagram (part 1) illustrating the road sign detection method according to the second embodiment. Figure 7 is an schematic diagram (part 2) illustrating the road sign detection method according to the second embodiment.

[0052] As shown in Figure 6, when the first image recognition model 11aa functions as an AI for image recognition, it is trained to output predictive classes that classify road signs by their appearance, such as "red circular sign" or "red triangular sign," based on a combination of color and shape. The predictive classes output by the first image recognition model 11aa correspond to an example of the "first predictive class."

[0053] On the other hand, as shown in Figure 7, the second image recognition model 11ab, when functioning as an AI for image recognition, is trained to output predictive classes that classify different road signs individually. The predictive classes output by the second image recognition model 11ab are examples of the "second predictive classes." As shown in Figure 7, the individually classified predictive classes include, for example, "Maximum Speed ​​(30km)," "Maximum Speed ​​(50km)," "No Entry," "Road Closed," "Stop," "Slow Down," and so on.

[0054] In other words, the classification level of the prediction class of the second image recognition model 11ab is more refined than that of the first image recognition model 11aa. In the road sign detection method according to the second embodiment, the controller 12 of the image processing device 10A detects road signs in stages using the first image recognition model 11aa and the second image recognition model 11ab, which have different classification levels of the prediction class of the road signs they output. As the stages progress, the classification level of the prediction class becomes more refined, making it possible to gradually narrow down the detection results with high accuracy for each classification level, for example.

[0055] The specific processing procedure will be explained using Figure 8. Figure 8 is a flowchart showing the processing procedure executed by the image processing device 10A according to the second embodiment.

[0056] First, the controller 12 acquires the image captured by camera 3 (step S201). Then, the controller 12 detects the road sign using the first image recognition model 11aa (step S202).

[0057] Then, the controller 12 determines whether or not there are any road signs among those detected in step S202 whose accuracy is below a threshold (step S203). The threshold is, for example, 0.5.

[0058] If there are road signs below the threshold (step S203, Yes), the controller 12 removes the corresponding road signs from the detection results in step S202 (step S204). If there are no road signs below the threshold (step S203, No), the controller 12 proceeds to step S205 while retaining the detection results in step S202. In the following, the processes from steps S202 to S204 will be referred to as the "first detection process".

[0059] Next, the controller 12 extracts detection frames for road signs from the detection results of the first detection process (step S205). Then, the controller 12 uses the second image recognition model 11ab to detect road signs using each extracted detection frame as input (step S206).

[0060] Then, the controller 12 determines whether or not there are any road signs among those detected in step S206 whose accuracy is below a threshold (step S207). The threshold is, as in step S203, for example, 0.5.

[0061] If there are road signs below the threshold (step S207, Yes), the controller 12 removes the corresponding road signs from the detection results in step S206 (step S208). If there are no road signs below the threshold (step S207, No), the controller 12 proceeds to step S209 while retaining the detection results in step S206. In the following, the processes from steps S206 to S208 will be referred to as the "second detection process".

[0062] The controller 12 then outputs the detection result from the second detection process to the external device 9 (step S209). This enables the external device 9 to perform information processing such as driving assistance functions based on the detection result from the controller 12. When the controller 12 uses the detection result itself, it appropriately performs the necessary information processing based on the detection result and outputs the processing result to the output unit 7.

[0063] The controller 12 then determines whether or not the system has terminated (step S210). The controller 12 determines that the system has terminated if, for example, the ignition switch is turned off.

[0064] If the system is not shutting down (step S210, No), the controller 12 repeats the process from step S201. If the system is shutting down (step S210, Yes), the controller 12 terminates the process.

[0065] Thus, in the road sign detection method according to the second embodiment, the controller 12 of the image processing device 10A detects road signs in stages using a first image recognition model 11aa and a second image recognition model 11ab, which have different classification levels for the prediction class. First, in the first detection process that outputs a prediction class that classifies the road signs by their appearance, the controller 12 excludes detection results with low accuracy based on the external characteristics of the road signs.

[0066] Furthermore, the controller 12 extracts the detection frames of road signs with high accuracy from the first detection process and performs a second detection process that outputs prediction classes that individually classify different road signs using these frames as input. Then, in the second detection process, the controller 12 excludes detection results with low accuracy based on the individual characteristics of each road sign and adopts only the detection results with high accuracy. By adopting only the detection results with high accuracy overall through such stepwise detection processes, the occurrence of false detections when detecting road signs from images can be reduced.

[0067] As described above, the image processing device 10A according to the second embodiment includes a controller 12 that detects road signs from an image. The controller 12 detects road signs in stages using a first image recognition model 11aa and a second image recognition model 11ab (corresponding to an example of "multiple image recognition models") which have different classification levels for predicting the output road sign class.

[0068] Therefore, according to the image processing device 10A of the second embodiment, road signs can be detected by, for example, refining the classification level of the prediction class each time a plurality of image recognition models are used, and gradually narrowing down the detection results with higher accuracy. In other words, according to the image processing device 10A of the second embodiment, the occurrence of false detections when detecting road signs from images can be reduced.

[0069] <Other variations> In the embodiments described above, the image processing devices 10 and 10A were assumed to be in-vehicle devices, but they are not limited to in-vehicle devices. For example, they may be central devices that collect images captured by each camera 3 from each vehicle via a network. Furthermore, the image processing devices 10 and 10A may be terminal devices such as PCs (Personal Computers) or smartphones.

[0070] Furthermore, camera 3 may not be an in-vehicle camera fixedly mounted to the vehicle, but rather a camera that can be attached to and removed from the vehicle, such as a camera mounted on a PC or smartphone brought in by the vehicle's occupants. In this case, the output unit 7 may be a display or speaker provided by the PC or smartphone. Also, among the external devices 9, for example, a car navigation system may be application software with car navigation functionality that runs on a PC or smartphone.

[0071] Furthermore, although the second embodiment described above gave an example where there were two image recognition models, there may be three or more.

[0072] Furthermore, the first and second embodiments described above may be combined as appropriate. For example, the first image recognition model 11aa and the second image recognition model 11ab according to the second embodiment may each be trained using training data in which the annotation frame AF2 is set to include a relatively wide background relative to the road sign.

[0073] Further effects and modifications can be readily derived by those skilled in the art. Therefore, broader aspects of the present invention are not limited to the specific details and representative embodiments expressed and described above. Accordingly, various modifications are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and their equivalents. [Explanation of Symbols]

[0074] 3 cameras 5. Various Sensors 7 Output section 9 External device 10,10A Image Processing Device 11 Storage section 11a Image Recognition Model 11aa First Image Recognition Model 11ab Second Image Recognition Model 12 controllers

Claims

1. Equipped with a controller that detects road signs from images, The aforementioned controller, The road sign is detected using an image recognition model trained on training data in which the annotation frame is set so that a relatively wide background is included around the road sign. Image processing device.

2. The annotation frame is set by enlarging a standard annotation frame by an arbitrary ratio. The image processing apparatus according to claim 1.

3. The annotation frame is set to include the mounting portion of the road sign. The image processing apparatus according to claim 2.

4. The annotation frame is configured to include the supplementary sign of the road sign. The image processing apparatus according to claim 2.

5. Equipped with a controller that detects road signs from images, The aforementioned controller, The road signs are detected in stages using multiple image recognition models with different classification levels for the predicted class of the road signs to be output. Image processing device.

6. The aforementioned controller, The road sign is detected using the multiple image recognition models in which the classification level is subdivided as the stage progresses. The image processing apparatus according to claim 5.

7. The aforementioned multiple image recognition models are A first image recognition model trained to output a first prediction class that classifies the road signs by their appearance and the accuracy of the first prediction class, A second image recognition model trained to output a second prediction class that individually classifies different road signs and the accuracy of the second prediction class, The aforementioned controller, The first detection process is performed using the first image recognition model described above. A second detection process using the second image recognition model is executed, with the detection result of the road sign whose accuracy of the first prediction class output in the first detection process is greater than or equal to a threshold as input. The detection result for the road sign in which the accuracy of the second prediction class output in the second detection process is greater than or equal to a threshold is output. The image processing apparatus according to claim 6.

8. The aforementioned controller, The detection result of the aforementioned road sign is output to an external device that performs a driving assistance function based on the detection result. The image processing apparatus according to claim 1 or 5.

9. A road sign detection method performed by a controller, Detecting road signs from images, The road sign is detected using an image recognition model trained on training data in which the annotation frame is set so that a relatively wide background is included with respect to the road sign. A road sign detection method that includes [details omitted].

10. A road sign detection method performed by a controller, Detecting road signs from images, The process involves detecting the road sign in stages using multiple image recognition models with different classification levels for the predicted class of the road sign to be output, A road sign detection method that includes [details omitted].

Images