Image processing device and image processing program

Abstract

This invention provides an image processing device and an image processing program that improve the accuracy of road sign recognition. [Solution] The image processing device, which is a drive recorder mounted on a vehicle, has a controller that determines the type of road sign captured in an image taken by a camera mounted on the vehicle. The controller calculates a weight coefficient for each road sign based on past images, and uses a corrected value obtained by correcting the confidence level calculated for the target image (the image to be judged) with the weight coefficient to determine the type of road sign captured in the current image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and an image processing program.

Background Art

[0002] Conventionally, techniques for performing image processing on images captured by in-vehicle cameras have been disclosed. For example, Patent Document 1 discloses a technique for predicting the probability of an event occurring for each frame, calculating a weight coefficient for each frame according to the prediction result, and identifying road signs and the like based on the calculated weight coefficient.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, since the conventional technique employs the same weight coefficient for the entire frame, there is a possibility that the determination accuracy of road signs may vary for each frame. Therefore, there was room for improvement in the conventional technique in terms of improving the determination accuracy of road signs.

[0005] The present invention has been made in view of the above, and an object thereof is to provide an image processing apparatus and an image processing program capable of improving the determination accuracy of road signs.

Means for Solving the Problems

[0006] To solve the above-mentioned problems and achieve the objective, the image processing device according to the present invention has a controller that determines the type of road sign captured in an image taken by a camera mounted on a vehicle. The controller calculates a weight coefficient for each road sign based on past images, and determines the type of road sign captured in the current image using a corrected value obtained by correcting the confidence level calculated for the current image using the weight coefficient. [Effects of the Invention]

[0007] According to the present invention, road signs are determined using weight coefficients calculated for each road sign based on the detection results of past images, thereby improving the accuracy of road sign determination. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of an image processing device being installed. [Figure 2] Figure 2 is an overview diagram of the image processing. [Figure 3] Figure 3 shows specific examples of similar groups. [Figure 4] Figure 4 is a block diagram of the dashcam. [Figure 5] Figure 5 is an explanatory diagram of the judgment process. [Figure 6] Figure 6 is an explanatory diagram of the difference in confidence levels. [Figure 7] Figure 7 is a flowchart showing the processing steps performed by the dashcam. [Modes for carrying out the invention]

[0009] The following describes in detail, with reference to the drawings, the embodiments for implementing the image processing apparatus and image processing program according to the present application (hereinafter referred to as "embodiments"). However, these embodiments do not limit the image processing apparatus and image processing program according to the present application.

[0010] First, an example of mounting the image processing device according to the embodiment will be described using Figure 1. Figure 1 is a diagram showing an example of mounting the image processing device. As shown in Figure 1, the image processing device according to the embodiment is a drive recorder 1 mounted on a vehicle 100.

[0011] The drive recorder 1 is positioned to capture images of the area in front of the vehicle 100. In this disclosure, the drive recorder 1 analyzes images taken of the area in front of the vehicle 100 and determines the type of road sign captured in the images.

[0012] The drive recorder 1 uses a deep learning algorithm to perform image recognition processing to detect each road sign, which is the object to be detected, from an image taken in front of the vehicle 100. In this disclosure, the drive recorder 1 determines road signs that belong to a similar group, where the feature quantities of the road signs are similar. Specific examples of similar groups will be described later with reference to Figure 3.

[0013] Next, an overview of the image processing according to the embodiment will be explained using Figure 2. Figure 2 is an explanatory diagram of the image processing. As shown in Figure 2, in the image processing according to the embodiment, an object classifier is used to calculate the confidence level of each road sign for the target image that is to be identified as a road sign. The confidence level is an indicator of how reliable the identification of road signs in the image processing is.

[0014] For example, an object classifier is a learning model that has learned the features of each road sign using a deep learning algorithm. The object classifier recognizes road signs included in the target image and calculates the confidence level of each road sign included in the target image.

[0015] Next, in the image processing according to the embodiment, the road signs in the target image are determined based on the confidence level of each road sign calculated using an object classifier. In this case, the image processing according to the embodiment calculates a weight coefficient for each road sign using the confidence level of each road sign calculated from past images.

[0016] In the image processing according to the embodiment, the type of the road sign shown in the current image is determined using the corrected value of each road sign obtained by correcting the confidence level of each road sign calculated in the current image with the weight coefficient of each road sign.

[0017] Here, the corrected value is a value obtained by multiplying the current confidence level by the weight coefficient. That is, the higher the confidence level of the road sign in the past image, the higher the value of the weight coefficient in the current image, and the lower the confidence level of the road sign in the past image, the lower the value of the weight coefficient in the current image.

[0018] In the image processing according to the embodiment, the road sign with the highest corrected value obtained by multiplying the confidence level by the weight coefficient is determined as the road sign shown in the current image. Thus, in the image processing according to the embodiment, when determining the road sign shown in the current image, the confidence level of each road sign in the current image is corrected using the integrated value of the confidence level of each road sign in the past image.

[0019] Therefore, according to the image processing according to the embodiment, since the road sign is determined using the weight coefficient calculated for each road sign based on the detection result of the past image, the determination accuracy of the road sign can be improved.

[0020] Next, a specific example of the similar group of road signs with similar feature amounts will be described using FIG. 3. FIG. 3 is a diagram showing a specific example of the similar group. In the example of FIG. 3, similar groups C1 to C4 are illustrated as the similar groups.

[0021] Similar groups C1 to C4 are groups in which the feature amounts of the road signs are respectively similar. More specifically, similar group C1 is a group of general road signs with similar shapes and colors. Similar group C1 includes road signs in which the road sign has a round shape, the outer peripheral part is red, and the internal numbers, letters, and illustrations are blue respectively.

[0022] Furthermore, similar group C2 consists of road signs related to speed limits from similar group C1. More specifically, each road sign included in similar group C2 is a road sign related to speed limits, and each sign has a different number written on it.

[0023] Similar groups C3 and C4 are groups composed of road signs from similar group C1 that are similar down to the numbers written on them. Note that the similar groups shown in Figure 2 are just examples, and developers may create similar groups by combining any road signs they like.

[0024] Next, an example of the configuration of the drive recorder 1 according to the embodiment will be described using Figure 4. Figure 4 is a block diagram of the drive recorder 1. As shown in Figure 4, the drive recorder 1 comprises a camera 2, a sensor 3, a storage unit 4, and a controller 5. The drive recorder 1 may also include, for example, a communication unit for communicating with a server device, a display unit for displaying images captured by the camera 2, and an operation unit for receiving user input.

[0025] Camera 2 is an imaging device equipped with an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 2 photographs the area in front of vehicle 100.

[0026] Sensor 3 is, for example, a G-sensor that detects the acceleration of vehicle 100. Sensor 3 may also include a GPS sensor (antenna) that detects GPS signals transmitted from GPS (Global Positioning system) satellites.

[0027] The storage unit 4 is implemented using storage devices such as ROM (Read Only Memory), RAM (Random Access Memory), and flash memory. The storage unit 4 stores the image processing program according to the embodiment, images captured by the camera 2, and the like.

[0028] Controller 5 corresponds to a so-called processor. Controller 5 is implemented by a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphical Processing Unit), etc. Controller 5 executes a program according to an embodiment not shown in the diagram, stored in the memory unit 4, using RAM as the working area. Controller 5 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0029] Controller 5 determines the type of road sign captured in the image taken by the camera 2 mounted on the vehicle 100. In this disclosure, Controller 5 calculates a weight coefficient for each road sign based on the confidence level of each road sign calculated from past images. Controller 5 determines the type of road sign captured in the target image using a corrected value obtained by correcting the confidence level calculated from the current image using the weight coefficient.

[0030] First, controller 5 uses an object detection method to detect the detection frame of a road sign in the target image. The detection frame is a rectangular area that includes the road sign, for example, an area that is similar in shape to the road sign. If controller 5 does not detect a road sign detection frame in the target image, it omits the subsequent processing. In other words, controller 5 performs the following processing on target images in which a road sign detection frame has been detected.

[0031] Controller 5 uses an object classifier (learned model) that has learned the characteristics of each road sign to determine the type of road sign shown in the target image. For example, the object classifier is a learned model that has learned the characteristics of each road sign in one of the similar groups C1 to C4 shown in Figure 3. This disclosure describes a case where the object classifier is a learned model that has learned the characteristics of each road sign in similar group C2, which relates to speed limits, or similar group C3, which is similar even down to the numbers within the road signs.

[0032] The object classifier calculates the confidence level for each road sign within each detection frame of the target image. The confidence level takes a value between 0 and 1, with a value closer to 1 indicating a higher probability that it is the corresponding road sign. For example, controller 5 determines that the road sign with the highest corrected value (after correcting the confidence level with a weighting coefficient) is the road sign that appears in the target image.

[0033] Here, we will explain a specific example of the processing performed by the controller 5 using Figure 5. Figure 5 is an explanatory diagram of the determination process. In Figure 5, we will explain the case in which the type of road sign is determined to be a regulatory sign that restricts the maximum speed. In the example in Figure 5, we show the case where the regulatory signs restrict the maximum speed to 20 km / h, 30 km / h, 40 km / h, 50 km / h, and 60 km / h.

[0034] Furthermore, the horizontal axis in Figure 5 shows the time each image was captured, with the image being captured more recently from left to right. In other words, in Figure 5, the images are captured in the following order from most recent to most recent: past image 4, past image 3, past image 2, past image 1, and the current image.

[0035] As shown in Figure 5, the controller 5 uses an initial weight as a weighting coefficient for past images 4. The initial weight is "1". When the controller 5 inputs past images 4 to the object classifier, the object classifier calculates the confidence level of each regulatory sign.

[0036] In the example in Figure 5, the confidence level for the 20 km / h regulatory sign is "0.1", for the 30 km / h regulatory sign it is "0.2", for the 40 km / h regulatory sign it is "0.8", for the 50 km / h regulatory sign it is "0.5", and for the 60 km / h regulatory sign it is "0.6".

[0037] Controller 5 calculates the correction value in past image 4 by multiplying the confidence level of each regulatory sign by the initial weight of "1". In the example in Figure 5, the correction value for the 40 km regulatory sign is "0.8", indicating the highest value.

[0038] Next, we will explain the processing of past image 3. When controller 5 inputs past image 3 to the object classifier, the object classifier calculates the confidence level of each regulatory sign for past image 3.

[0039] In the example in Figure 5, the confidence level for the 20 km / h regulatory sign is "0.2", the confidence level for the 30 km / h regulatory sign is "0.2", the confidence level for the 40 km / h regulatory sign is "0.9", the confidence level for the 50 km / h regulatory sign is "0.8", and the confidence level for the 60 km / h regulatory sign is "0.3".

[0040] Controller 5 calculates the correction value in past image 3 by multiplying the confidence level of each regulatory sign by the weighting coefficient of each regulatory sign. The weighting coefficient here is the correction value in past image 4.

[0041] In other words, for the 20 km / h speed limit sign, the correction value for past image 4 is "0.1" and the confidence level for past image 3 is "0.2", so the correction value for past image 3 is "0.1 × 0.2 = 0.02".

[0042] Controller 5 performs these processes for each regulatory sign and calculates the correction value for each regulatory sign in past image 3. Controller 5 repeatedly performs these processes for past image 2, past image 1, and the current image.

[0043] This allows the weighting coefficient for each regulatory sign to be the integral of the confidence level of each past image. In other words, if the confidence level is low in any of the past images, the weighting coefficient for the current image will be low, so even if the confidence level is high in the current image, the corrected value will be low.

[0044] As a result, the controller 5 can keep the correction value low by using the weighting coefficient up to past image 3, even if the confidence level is high, such as "0.9" for the 60 km / h speed limit sign in past image 2.

[0045] In other words, for example, even if the confidence level of a road sign momentarily increases due to the influence of noise, the controller 5 corrects it using weight coefficients calculated from past images, thereby suppressing false detection of road signs.

[0046] In this way, the controller 5 calculates a weight coefficient for each regulatory sign for each frame and corrects the confidence level of the current image using the weight coefficient. Therefore, the controller 5 can suppress false detections of road signs and improve the accuracy of road sign recognition.

[0047] Furthermore, if the controller 5 finds frames in the past images in which no road sign detection frame is detected, it stops updating the weight coefficients for those frames. In other words, it calculates each weight coefficient for frames in which no road signs are visible, resulting in low weight coefficients for each road sign in those frames, and consequently, low correction values ​​for subsequent frames as well.

[0048] In response to this, the controller 5 stops updating the weight coefficients in frames where no road sign detection frame is detected, thus maintaining each weight coefficient for each road sign at an appropriate value.

[0049] Furthermore, the controller 5 resets the weight coefficients of each regulatory sign to their initial weights when predetermined conditions are met. For example, the controller 5 resets the correction values ​​of each road sign to their initial values ​​when the conditions are met that indicate that vehicle 100 has turned at an intersection.

[0050] For example, the controller 5 estimates whether the vehicle 100 has turned at an intersection based on the acceleration signal output from the sensor 3. For example, the controller 5 estimates that the conditions for the vehicle 100 to have turned at an intersection have been met when the horizontal component of the acceleration signal exceeds a threshold, that is, when horizontal acceleration occurs to the vehicle 100.

[0051] The controller 5 then resets the weight coefficients of each road sign when these conditions are met. This allows the controller 5 to appropriately determine the road signs on the road after the vehicle 100 has turned at an intersection and entered a road with different speed limits, etc. The controller 5 may also determine whether the conditions for the vehicle 100 to have turned at an intersection are met based on the changes in location information detected by the GPS sensor. In such a case, the controller 5 calculates the driving trajectory of the vehicle 100 from the changes in the vehicle 100's location information and determines whether the conditions for the vehicle 100 to have turned at an intersection are met based on the calculated driving trajectory.

[0052] In addition, the controller 5 may reset the weight coefficients of each road sign when the vehicle 100's mileage exceeds a threshold or when the vehicle 100's driving time exceeds a threshold. In such cases, the controller 5 can appropriately reset the weight coefficients through a simple process.

[0053] Furthermore, in addition to comparing the correction value with the threshold, the controller 5 may also determine the road signs visible in the current image based on the difference in the confidence level of each road sign in the current image. Here, a specific example of the difference condition will be explained using Figure 6. Figure 6 is an explanatory diagram of the confidence level difference condition.

[0054] Figure 6 illustrates this using two examples: a speed limit sign of 30 km / h and a speed limit sign of 20 km / h. Controller 5 calculates the difference in confidence levels for each speed limit sign in each frame, and identifies any speed limit sign whose calculated difference exceeds a threshold as a road sign currently visible in the image.

[0055] In the example in Figure 6, the differences from left to right are "0.4", "0", "0.1", "0.1", and "0.3". Also, as shown in Figure 6, the threshold for the difference in confidence is "0.3". In the frame where the difference in confidence is "0.4", Controller 5 makes the final identification decision result the regulatory sign with a maximum speed of 30 km / h, which has a high confidence level.

[0056] Furthermore, in frames where the confidence difference is "0.3", Controller 5 will determine the regulatory sign with a minimum speed of 30 km / h as the final identification result, as it has a higher confidence level. Also, in frames where the confidence difference is below the threshold, Controller 5 will leave the final identification result undetermined.

[0057] In other words, controller 5 can reduce the risk of misclassification by making a final identification decision based on the difference in confidence levels, for example, when both sides show high confidence levels.

[0058] Next, the processing procedure performed by the drive recorder 1 according to this embodiment will be explained using Figure 7. Figure 7 is a flowchart showing the processing procedure performed by the drive recorder 1. Note that the processing shown in Figure 7 is repeatedly performed by the controller 5 each time an image captured by the camera 2 is acquired.

[0059] As shown in Figure 7, when the controller 5 acquires an image (target image) from the camera 2 (step S101), it uses an object classifier to calculate the confidence level of each road sign in the target image (step S102).

[0060] Next, the controller 5 calculates a correction value using weight coefficients calculated based on the confidence level of past images (step S103). Specifically, the controller 5 calculates a weight coefficient for each road sign by multiplying it by the confidence level of each road sign in each past image, and then calculates a correction value for each road sign in the target image by multiplying the calculated weight coefficient by the confidence level of the road sign in the target image.

[0061] Next, the controller 5 determines the road sign with the highest correction value as a result of step S103 (step S104), and then determines whether the confidence difference exceeds a threshold for that road sign (step S105).

[0062] Controller 5 calculates the difference between the confidence level of the road sign with the highest correction value in the target image and the confidence level of the road sign with the second highest confidence level in the target image, and determines whether the calculated difference in confidence levels exceeds a threshold.

[0063] If controller 5 determines that the confidence difference exceeds a threshold (step S105; Yes), it determines that the road sign determined in step S104 is a road sign that appears in the target image (step S106).

[0064] Furthermore, if the controller 5 determines in step S105 that the confidence difference is within the threshold (step S105; No), it omits the processing in step S106. In other words, in this case, the controller 5 leaves the type of road sign in the target image undetermined.

[0065] Subsequently, the controller 5 updates the weight coefficients for the next target image (step S107) and terminates the process. That is, the controller 5 sets the correction value for each road sign in the current image to the weight coefficient for each road sign in the next target image, and then terminates the process.

[0066] As described above, the drive recorder 1 (an example of an image processing device) according to the embodiment has a controller 5 that determines the type of road sign captured in an image taken by a camera 2 mounted on a vehicle 100. The controller 5 calculates a weight coefficient for each road sign based on the confidence level of each road sign calculated from past images, and determines the type of road sign captured in the target image using a corrected value obtained by correcting the confidence level calculated from the target image, which is the image to be judged, with the weight coefficient.

[0067] Therefore, according to the drive recorder 1 of this embodiment, the confidence level of each road sign in the target image can be corrected using the weight coefficient calculated for each road sign, thereby improving the accuracy of road sign identification.

[0068] By the way, the above-described embodiment described a case where the image processing device is a drive recorder 1, but it is not limited to this. The image processing device may be, for example, a server device that acquires images from the drive recorder 1.

[0069] Further effects and modifications can be readily derived by those skilled in the art. Therefore, broader aspects of the present invention are not limited to the specific details and representative embodiments expressed and described above. Accordingly, various modifications are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and equivalents. [Explanation of Symbols]

[0070] 1. Dashcam 2 cameras 3 sensors 4 Storage section 5 Controllers 100 vehicles

Claims

1. The vehicle has a controller that determines the type of road sign shown in the image captured by the camera mounted on the vehicle. The aforementioned controller, Based on the aforementioned past images, a weighting coefficient for each road sign is calculated. The type of road sign shown in the current image is determined using a corrected value obtained by correcting the confidence level calculated from the current image using the weighting coefficient. Image processing device.

2. The aforementioned controller, In the current image, the road sign with the highest correction value is determined to be the road sign shown in the current image. The image processing apparatus according to claim 1.

3. The aforementioned controller, In the current image, if the difference in confidence levels between the road signs exceeds a threshold, the road sign with the higher confidence level is determined to be the road sign shown in the current image. The image processing apparatus according to claim 1.

4. The aforementioned controller, The correction value of the current image is used as the weighting coefficient for the next current image. The image processing apparatus according to claim 1.

5. The aforementioned controller, The system detects the detection frame of the road sign from the aforementioned image, and stops updating the weight coefficient in past images where the detection frame is not detected. The image processing apparatus according to claim 4.

6. The aforementioned controller, When the conditions for which the vehicle is presumed to have turned at an intersection are met, the correction value is reset to its initial value. The image processing apparatus according to claim 1.

7. The aforementioned controller, If the vehicle's mileage exceeds a threshold, or if the vehicle's operating time exceeds a threshold, the correction value is reset to its initial value. The image processing apparatus according to claim 1.

8. An image processing program executed by a controller that determines the type of road sign captured in an image taken by a camera mounted on a vehicle, A procedure for calculating the weight coefficient of each road sign based on past images, and A procedure for determining the type of road sign shown in the current image using a corrected value obtained by correcting the confidence level calculated from the current image using the weighting coefficient, and Includes image processing programs.

Images