Image processing device and road sign detection method

Abstract

To further improve the accuracy of road sign detection. [Solution] The image processing device according to the embodiment includes a controller for detecting road signs from an image. The controller detects road signs by image recognition processing on the image, with a first detection frame indicating a guide sign and a second detection frame indicating signs other than guide signs. The controller also determines that the sign is depicted within a guide sign when the first detection frame encompasses the second detection frame.

Description

Technical Field

[0001] The disclosed embodiments relate to an image processing apparatus and a road sign detection method.

Background Art

[0002] Conventionally, there is known a technique for assisting a driver by detecting a road sign from an image captured by an in-vehicle camera and performing a driving operation according to the content indicated by the road sign.

[0003] For example, Patent Document 1 discloses a technique for warning a driver if the driving situation of a vehicle conflicts with the regulation content indicated by a detected road sign when the detected road sign is a regulation sign.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, regulation signs may be installed integrated with guiding signs among road signs. A regulation sign integrated with a guiding sign often functions as a warning sign that预告 the regulation content at a forward position rather than indicating the regulation content at the current driving position.

[0006] The above-described conventional technology does not assume the case where the guiding sign and the regulation sign are integrated. Therefore, when the above-described conventional technology is used, it is impossible to appropriately determine whether or not the driving situation of the vehicle conflicts with the regulation content indicated by the regulation sign.

[0007] In addition to guide signs, there are also warning signs and informational signs, and these, like regulatory signs, may be integrated with guide signs. Therefore, the conventional technology described above has room for further improvement in terms of detecting road signs with greater accuracy, even when guide signs and other road signs are integrated in this way.

[0008] One embodiment of the invention has been made in view of the above, and aims to provide an image processing device and a road sign detection method that can further improve the detection accuracy of road signs. [Means for solving the problem]

[0009] One embodiment includes a controller for detecting road signs from an image. The controller detects the road signs in the image through image recognition processing, defining them as a first detection frame indicating a guide sign and a second detection frame indicating signs other than guide signs. The controller determines that the sign is depicted within the guide sign when the first detection frame encompasses the second detection frame. [Effects of the Invention]

[0010] According to one embodiment, the controller determines that a sign in a second detection frame is depicted within a guide sign when a first detection frame showing a guide sign among the detected road signs includes a second detection frame showing a sign other than a guide sign. This makes it possible to detect signs other than guide signs that are integrated with the guide sign and depicted within the guide sign. In other words, according to one embodiment, the detection accuracy of road signs can be further improved. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a schematic diagram illustrating the road sign detection method according to the first embodiment. [Figure 2] Figure 2 shows an example of the configuration of an image processing apparatus according to the first embodiment. [Figure 3]Figure 3 is a flowchart showing the processing procedure performed by the image processing apparatus according to the first embodiment. [Figure 4] Figure 4 shows an example where a warning sign is depicted within a guide sign. [Figure 5] Figure 5 shows an example where an directional sign is depicted within a guide sign. [Figure 6] Figure 6 is an schematic diagram (part 1) illustrating the road sign detection method according to the second embodiment. [Figure 7] Figure 7 is an overview diagram (part 2) of the road sign detection method according to the second embodiment. [Figure 8] Figure 8 is an overview diagram (part 3) of the road sign detection method according to the second embodiment. [Figure 9] Figure 9 is a flowchart showing the processing procedure performed by the image processing device according to the second embodiment. [Figure 10] Figure 10 is an explanatory diagram (part 1) of how to use the system when a road sign depicted within a guide sign is detected. [Figure 11] Figure 11 is an explanatory diagram (part 2) of how to use the system when a road sign depicted within a guide sign is detected. [Modes for carrying out the invention]

[0012] The embodiments of the image processing apparatus and road sign detection method disclosed herein will be described in detail below with reference to the attached drawings. However, the present invention is not limited to the embodiments described below.

[0013] Furthermore, in the following description, the image processing device according to the embodiment is assumed to be an in-vehicle device mounted on a vehicle. Also, the road sign detection method according to the embodiment is assumed to be executed by the controller 12 (see Figure 2) of the image processing device.

[0014] Next, the road sign detection method according to the first embodiment will be described below with reference to FIGS. 1 to 5. Further, the road sign detection method according to the second embodiment will be described with reference to FIGS. 6 to 9. Further, the usage method of the road signs detected using the first embodiment or the second embodiment will be described with reference to FIGS. 10 and 11.

[0015] <First Embodiment> FIG. 1 is a schematic explanatory diagram of the road sign detection method according to the first embodiment. The road sign detection method according to the first embodiment is executed by the controller 12 (see FIG. 2) of the image processing apparatus 10 (see FIG. 2) according to the first embodiment.

[0016] In the road sign detection method according to the first embodiment, the controller 12 detects road signs from an image. At this time, the controller 12 detects road signs as a first detection frame indicating a guiding sign and a second detection frame indicating signs other than guiding signs by performing image recognition processing on the image. Further, when the first detection frame includes the second detection frame, the controller 12 determines that a sign other than the guiding sign is drawn within the guiding sign.

[0017] Specifically, as shown in FIG. 1, the controller 12 detects road signs from the image P1 captured by the in-vehicle camera. At this time, the controller 12 detects each road sign by performing image recognition processing on the image P1 using an AI (Artificial Intelligence) model for image recognition.

[0018] The AI model is, for example, a DNN (Deep Neural Network) model learned using a machine learning algorithm. The AI model for image recognition according to the present embodiment is pre-learned to be able to detect guiding signs and signs other than guiding signs, and functions as an AI for image recognition when read by the controller 12.

[0019] As shown in Figure 1, the controller 12 detects guide signs using the AI ​​trained on guide signs as detection frame BB1 ​​(step S1). The controller 12 also detects signs other than guide signs, such as regulatory signs, using the AI ​​trained on regulatory signs as detection frame BB2 (step S2).

[0020] The controller 12 then determines whether the detection frame BB1 ​​of the guide sign encompasses the detection frame BB2 of the regulatory sign. As shown in Figure 1, if the detection frame BB1 ​​of the guide sign encompasses the detection frame BB2 of the regulatory sign, the controller 12 determines that the regulatory sign in the detection frame BB2 is the regulatory sign depicted within the guide sign (step S3).

[0021] As a result, in the example in Figure 1, it is determined that the regulatory sign in detection frame BB2 is a warning sign indicating a no-entry zone 300m ahead. For example, the driver assistance function that operates based on the image recognition result of the image processing device 10 can provide driver assistance at the position ahead according to the content of the warning sign.

[0022] On the other hand, although not shown in the diagram, if the detection frame BB1 ​​of the guide sign does not encompass the detection frame BB2 of the regulatory sign, the controller 12 determines that the regulatory sign in detection frame BB2 is a normal regulatory sign that is not integrated with the guide sign.

[0023] As a result, the driving assistance function, which operates based on the image recognition results of the image processing device 10, treats this ordinary regulatory sign as indicating the content of the regulations at the current driving location, enabling driving assistance at the current location in accordance with those regulations.

[0024] As described above, in the road sign detection method according to the first embodiment, the controller 12 detects road signs from an image. The controller 12 also detects road signs by image recognition processing on the image, and identifies them as a detection frame BB1 ​​indicating a guide sign (corresponding to an example of the "first detection frame") and a detection frame BB2 indicating signs other than guide signs (corresponding to an example of the "second detection frame"). The controller 12 also determines that a sign other than a guide sign is depicted within a guide sign when detection frame BB1 ​​encompasses detection frame BB2.

[0025] Therefore, according to the road sign detection method according to the first embodiment, the controller 12 determines that the sign in detection frame BB2 is depicted within the guide sign when the detection frame BB1 ​​showing a guide sign among the detected road signs includes the detection frame BB2 showing a sign other than a guide sign. This makes it possible to detect signs other than guide signs that are integrated with the guide sign and depicted within the guide sign. In other words, the road sign detection method according to the first embodiment can further improve the accuracy of road sign detection.

[0026] The following describes in more detail an example of the configuration of an image processing device 10 to which the road sign detection method according to the first embodiment described above is applied. In the following description of the first embodiment, the main example given is when the sign other than a guide sign is a regulatory sign ("No Entry"), as shown in Figure 1. An example where the sign other than a guide sign is a warning sign or an informational sign will be shown later in Figure 4 or Figure 5.

[0027] Figure 2 shows an example configuration of the image processing device 10 according to the first embodiment. The image processing device 10 is a computer mounted in a vehicle. The image processing device 10 performs at least the information processing steps S1 to S3 described using Figure 1.

[0028] As shown in Figure 2, the image processing device 10 comprises a storage unit 11 and a controller 12. The image processing device 10 is also connected to a camera 3, various sensors 5, an output unit 7, and an external device 9.

[0029] Camera 3 is one or more on-board cameras mounted on various parts of the vehicle. Camera 3 is positioned to capture at least the area in front of the vehicle. Camera 3 may be positioned to capture not only the area in front of the vehicle, but also the area behind, to the sides of, or even inside the vehicle's interior. Camera 3 may also be a 360-degree camera capable of capturing the entire area around the vehicle.

[0030] The various sensors 5 are various sensors other than the camera 3 that are mounted in various parts of the vehicle. The various sensors 5 include, for example, GPS (Global Positioning System) sensors and acceleration sensors.

[0031] The output unit 7 is an output device that displays output information from the image processing device 10. The output unit 7 is implemented by a display, speaker, or the like.

[0032] External devices 9 are various devices that utilize the image recognition results from the image processing device 10. Examples of external devices 9 include various devices that perform functions of advanced driver-assistance systems (ADAS), car navigation systems, and various ECUs (Electronic Control Units). This allows these devices to perform various driver assistance functions based on the image recognition results from the image processing device 10.

[0033] The external device 9 is connected to the image processing device 10 via an in-vehicle network such as CAN (Controller Area Network). The external device 9 may also be connected to the image processing device 10 wirelessly via Bluetooth (registered trademark), Wi-Fi (registered trademark), UWB (Ultra Wide Band), etc.

[0034] The memory unit 11 is implemented by a memory device such as RAM (Random Access Memory) or flash memory. The memory unit 11 stores a program (not shown) according to an embodiment executed by the controller 12. The memory unit 11 also stores various types of information used in the information processing executed by the controller 12.

[0035] In the example shown in Figure 2, the memory unit 11 stores the first image recognition model 11a and the second image recognition model 11b as various types of information.

[0036] The first image recognition model 11a functions as the guide sign trained AI described above when it is loaded as an AI model into the controller 12. When the first image recognition model 11a functions as the guide sign trained AI, it is pre-trained to output the guide sign shown in each frame as a first detection frame when each frame of video captured by the camera 3 is input. In addition, when the first image recognition model 11a functions as the guide sign trained AI, it is pre-trained to output the accuracy of each detected detection frame. The accuracy takes a value between 0 and 1, with the closer to 1, the higher the accuracy.

[0037] The second image recognition model 11b functions as the regulatory sign-trained AI described above when it is loaded into the controller 12 as an AI model. When the second image recognition model 11b functions as the regulatory sign-trained AI, it is pre-trained to output the regulatory sign shown in each frame as a second detection frame when each frame of the video captured by the camera 3 is input. In addition, when the second image recognition model 11b functions as the regulatory sign-trained AI, it is pre-trained to output the accuracy of each detected detection frame.

[0038] Furthermore, the second image recognition model 11b may be pre-trained to output warning signs and directional signs that appear in each frame along with regulatory signs as a second detection frame. In this case, the second image recognition model 11b functions as a trained AI for signs other than guide signs when loaded into the controller 12 as an AI model. This enables the detection of regulatory signs, warning signs, and directional signs as signs other than guide signs, and can handle any case in which these are depicted within guide signs.

[0039] Alternatively, the first image recognition model 11a and the second image recognition model 11b may be integrated and provided as a single image recognition model. In this case, the integrated image recognition model is loaded into the controller 12 as an AI model and functions as a road sign trained AI. When each frame of video captured by the camera 3 is input to this road sign trained AI, it outputs the guide signs shown in each frame as a first detection frame and signs other than guide signs as a second detection frame.

[0040] The controller 12 corresponds to a so-called processor. The controller 12 can be implemented by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc. The controller 12 reads the program according to the embodiment stored in the memory unit 11 and executes it using RAM as the working area. The controller 12 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0041] The controller 12 performs information processing according to the processing procedure shown in Figure 3. An explanation using Figure 3 will be given later.

[0042] The image processing device 10 can be implemented, for example, as part of a drive recorder. In this case, the image processing device 10 is implemented by a microcontroller mounted on the drive recorder. In this case, part or all of the camera 3 is implemented by a camera unit mounted on the drive recorder, and part or all of the output unit 7 is implemented by a display, speaker, etc., mounted on the drive recorder.

[0043] Furthermore, the image processing device 10 can be implemented as an image processing ECU. In this case, the camera 3 is implemented by, for example, a camera unit mounted on a drive recorder or an in-vehicle camera unit other than a drive recorder. The output unit 7 is implemented by a display, speaker, etc. mounted on a drive recorder, or an in-vehicle output device other than a drive recorder, such as an in-vehicle display or in-vehicle speaker.

[0044] Next, the processing procedure for information processing performed by the controller 12 of the image processing device 10 will be explained using Figure 3. Figure 3 is a flowchart of the processing procedure performed by the image processing device 10 according to the first embodiment.

[0045] First, the controller 12 acquires image P1 from camera 3 (step S101). Then, the controller 12 performs image recognition processing on the acquired image P1. In this image recognition processing, the controller 12 detects guide signs using the first image recognition model 11a (step S102). The controller 12 also detects regulatory signs using the second image recognition model 11b (step S103).

[0046] The controller 12 then determines whether both the guide sign and the regulatory sign have been detected, that is, whether both detection frame BB1 ​​and detection frame BB2 are present (step S104).

[0047] If both detection frame BB1 ​​and detection frame BB2 exist (step S104, Yes), the controller 12 determines whether or not there is a detection frame BB2 for a regulatory sign that is contained within the detection frame BB1 ​​of a guide sign (step S105).

[0048] If there is a detection frame BB2 for the relevant regulatory sign (Step S105, Yes), the controller 12 determines that the relevant regulatory sign among those detected in Step S103 is a regulatory sign depicted within the guide sign (Step S106). The controller 12 also determines that any regulatory signs among those detected in Step S103 that are not applicable (not included) are ordinary regulatory signs.

[0049] Furthermore, if in step S104 only one of the detection frames BB1 or BB2 exists (step S104, No), the controller 12 determines that the regulatory sign detected in step S103 is a normal regulatory sign (step S107). Also, if in step S105 there is no detection frame BB2 for the corresponding regulatory sign (step S105, No), the controller 12 determines that the regulatory sign detected in step S103 is a normal regulatory sign (step S107).

[0050] The controller 12 then outputs the image recognition results, including the determination results from steps S104 to S107, to the external device 9 (step S108). This enables the external device 9 to perform information processing, such as driving assistance functions, based on the image recognition results from the controller 12. When the controller 12 uses the image recognition results itself, it performs the necessary information processing based on the image recognition results as appropriate and outputs the processing results to the output unit 7.

[0051] The controller 12 then determines whether or not the system has terminated (step S109). The controller 12 determines that the system has terminated if, for example, the ignition switch is turned off.

[0052] If the system is not shutting down (step S109, No), the controller 12 repeats the process from step S101. If the system is shutting down (step S109, Yes), the controller 12 terminates the process.

[0053] Up until now, we have mainly given examples of regulatory signs that are not guide signs, but as mentioned above, warning signs and informational signs can also be integrated with guide signs and depicted within them. Specific examples of this are shown in Figure 4 or Figure 5.

[0054] Figure 4 shows an example where a warning sign is depicted within a guide sign. Figure 5 shows an example where an informational sign is depicted within a guide sign.

[0055] Figure 4 shows an example in which a warning sign indicating "narrowing of road width" is drawn within a guide sign detected as detection frame BB1. In this case, the controller 12 of the image processing device 10 detects the warning sign as detection frame BB3 (corresponding to an example of a "second detection frame"). The controller 12 then determines that the detected warning sign is drawn within the guide sign because detection frame BB1 ​​encompasses detection frame BB3.

[0056] Based on the image recognition results, including this determination, the external device 9 will, for example, provide driving assistance such as issuing a warning to the driver that caution is required because the road width will decrease when the vehicle enters the road corresponding to this warning sign 300m ahead.

[0057] Figure 5 also shows an example in which a "one-way" sign is drawn within the guide sign detected as detection frame BB1. In this case, the controller 12 of the image processing device 10 detects the sign as detection frame BB4 (corresponding to an example of a "second detection frame"). The controller 12 then determines that the detected sign is drawn within the guide sign because detection frame BB1 ​​encompasses detection frame BB4.

[0058] Based on the image recognition results, including this determination, the external device 9 will, for example, provide driving assistance such as issuing a warning to the driver that the road is one-way and only left turns are permitted when the vehicle reaches 300m ahead.

[0059] As described above, the image processing device 10 according to the first embodiment includes a controller 12 that detects road signs from an image P1 captured by a camera 3 (corresponding to an example of an "in-vehicle camera"). The controller 12 also detects road signs by image recognition processing of the image P1, identifying them as a detection frame BB1 ​​indicating a guide sign (corresponding to an example of a "first detection frame") and detection frames BB2, BB3, BB4 indicating signs other than guide signs (corresponding to an example of a "second detection frame"). The controller 12 also determines that the sign is depicted within a guide sign when detection frame BB1 ​​encompasses detection frames BB2, BB3, BB4.

[0060] Therefore, according to the image processing device 10 of the first embodiment, if the detection frame BB1 ​​indicating a guide sign among the detected road signs includes detection frames BB2 to BB4 indicating signs other than guide signs, it is determined that the signs in detection frames BB2 to BB4 are depicted within the guide sign. This makes it possible to detect signs other than guide signs that are integrated with the guide sign and depicted within the guide sign. In other words, the image processing device 10 of the first embodiment can further improve the detection accuracy of road signs.

[0061] <Second Embodiment> Next, a second embodiment will be described. Note that the configuration example of the image processing apparatus according to the second embodiment is the same as in Figure 2, so the description of this configuration example will be omitted. However, for convenience, the image processing apparatus according to the second embodiment will be denoted as "10A" to distinguish it from the image processing apparatus 10 according to the first embodiment. Furthermore, in the following description of the second embodiment, as with the first embodiment, the main example will be a case where signs other than guide signs are regulatory signs ("No Entry").

[0062] Figure 6 is an schematic diagram (part 1) illustrating the road sign detection method according to the second embodiment. Figure 7 is an schematic diagram (part 2) illustrating the road sign detection method according to the second embodiment. Figure 8 is an schematic diagram (part 3) illustrating the road sign detection method according to the second embodiment. The road sign detection method according to the second embodiment is executed by the controller 12 of the image processing device 10A according to the second embodiment.

[0063] In the road sign detection method according to the second embodiment, the controller 12 detects road signs from an image. At this time, the controller 12 detects road signs by image recognition processing on the image as a first detection frame indicating guide signs and a second detection frame indicating signs other than guide signs. The controller 12 also determines that a sign other than a guide sign is depicted within a guide sign if the background color around the sign in the second detection frame matches the base color of the guide sign in the first detection frame.

[0064] Specifically, the controller 12 first detects the guide sign as a detection frame BB1 ​​from the image P1 captured by the camera 3, as in the first embodiment, and also detects signs other than guide signs, such as regulatory signs, as a detection frame BB2 (steps S1 and S2 in Figure 1).

[0065] Then, after step S2, as shown in Figure 6, the controller 12 cuts out the detection frame BB2 of the regulatory sign (step S4) and determines whether the background color around the regulatory sign matches the base color of the guide sign.

[0066] Here, the "basic color of the guide sign" is, for example, the color that occupies the largest area (most frequent color) in the area R1 of the guide sign within the detection frame BB1. "Basic color" can also be read as "background color." The basic color of guide signs is often blue or green. On the other hand, the "background color around the regulatory sign" is, for example, the color that occupies the largest area (most frequent color) in the area R2 surrounding the regulatory sign within the detection frame BB2. This allows us to determine, for example, whether or not a regulatory sign is depicted within the guide sign based on each most frequent color.

[0067] Then, as shown in Figure 6, the controller 12 determines that the regulatory sign in detection frame BB2 is a regulatory sign depicted within the guide sign if the background color around the regulatory sign matches the base color of the guide sign (step S5).

[0068] As a result, in the example in Figure 6, it can be determined that the regulatory sign in detection frame BB2 is a warning sign indicating a no-entry zone 300m ahead. For example, the driver assistance function that operates based on the image recognition result of the image processing device 10 can provide driver assistance according to the content of this warning sign.

[0069] On the other hand, as shown in Figure 7, if the background color around the regulatory sign does not match the base color of the guide sign, the controller 12 determines that the regulatory sign in detection frame BB2 is a regular regulatory sign that is not integrated with the guide sign (step S6).

[0070] The term "ordinary regulatory signs" as used herein includes, for example, pole-type signs mounted on poles, as well as signboard-type signs as shown in Figure 8. Signboard-type signs are sometimes installed at highway entrances and service areas, for example. In the road sign detection method according to the second embodiment, the controller 12 determines that such signboard-type regulatory signs are ordinary regulatory signs if the background color around the regulatory sign does not match the base color of the guide sign.

[0071] If the sign is determined to be a normal regulatory sign, the driving assistance function, which operates based on the image recognition result of the image processing device 10, will treat this normal regulatory sign as indicating the content of the regulations at the current driving location, and will be able to provide driving assistance according to the content of those regulations.

[0072] Thus, in the road sign detection method according to the second embodiment, the controller 12 detects road signs from the image. The controller 12 also detects road signs by image recognition processing on the image, and identifies them as detection frame BB1 ​​(corresponding to an example of the "first detection frame") indicating guide signs and detection frame BB2 (corresponding to an example of the "second detection frame") indicating signs other than guide signs. The controller 12 also determines that a sign other than a guide sign is depicted within a guide sign if the background color around the sign in detection frame BB2 matches the base color of the guide sign in detection frame BB1.

[0073] Therefore, according to the road sign detection method according to the second embodiment, the controller 12 determines that the sign in detection frame BB2 is depicted within a guide sign when the color of the surrounding area R2 in detection frame BB2 matches the base color of the area R1 of the guide sign in detection frame BB1. This makes it possible to detect signs other than guide signs that are integrated with guide signs and depicted within guide signs. In other words, the road sign detection method according to the second embodiment can further improve the detection accuracy of road signs.

[0074] Next, the processing procedure for information processing performed by the controller 12 of the image processing device 10A will be explained using Figure 9. Figure 9 is a flowchart showing the processing procedure performed by the image processing device 10A according to the second embodiment.

[0075] First, the controller 12 acquires image P1 from camera 3 (step S201). Then, the controller 12 performs image recognition processing on the acquired image P1. In this image recognition processing, the controller 12 detects guide signs using the first image recognition model 11a (step S202). The controller 12 also detects regulatory signs using the second image recognition model 11b (step S203).

[0076] The controller 12 then determines whether both the guide sign and the regulatory sign have been detected, that is, whether both detection frame BB1 ​​and detection frame BB2 are present (step S204).

[0077] If both detection frame BB1 ​​and detection frame BB2 exist (step S204, Yes), the controller 12 cuts out detection frame BB2 of the regulatory sign (step S205). Then, the controller 12 determines whether the background color around the regulatory sign (the color of the surrounding area R2 mentioned above) matches the base color of the guide sign (the base color of the area R1 of the guide sign mentioned above) (step S206).

[0078] If there is a match (step S206, Yes), the controller 12 determines that the regulatory sign detected in step S203 is a regulatory sign depicted within the guide sign (step S207).

[0079] Furthermore, if there is no match (step S206, No), the controller 12 determines that the regulatory sign detected in step S203 is a normal regulatory sign (step S208). Also, if only one of the detection frames BB1 or BB2 exists in step S204 (step S204, No), the controller 12 determines that the regulatory sign detected in step S203 is a normal regulatory sign (step S208).

[0080] The controller 12 then outputs the image recognition results, including the determination results from steps S204 to S208, to the external device 9 (step S209). This enables the external device 9 to perform information processing, such as driving assistance functions, based on the image recognition results from the controller 12. When the controller 12 uses the image recognition results itself, it performs the necessary information processing based on the image recognition results as appropriate and outputs the processing results to the output unit 7.

[0081] The controller 12 then determines whether or not the system has terminated (step S210). The controller 12 determines that the system has terminated if, for example, the ignition switch is turned off.

[0082] If the system is not shutting down (step S210, No), the controller 12 repeats the process from step S201. If the system is shutting down (step S210, Yes), the controller 12 terminates the process.

[0083] As described above, the image processing device 10A according to the second embodiment includes a controller 12 that detects road signs from an image P1 captured by a camera 3 (corresponding to an example of an "in-vehicle camera"). The controller 12 also detects road signs by image recognition processing on the image P1, and identifies them as a detection frame BB1 ​​indicating a guide sign (corresponding to an example of a "first detection frame") and a detection frame BB2 indicating signs other than guide signs (corresponding to an example of a "second detection frame"). The controller 12 also determines that a sign other than a guide sign is depicted within a guide sign if the background color around the sign in detection frame BB2 matches the base color of the guide sign in detection frame BB1.

[0084] Therefore, according to the image processing device 10A of the second embodiment, if the color of the surrounding area R2 in the detection frame BB2 matches the base color of the area R1 of the guide sign in the detection frame BB1, it is determined that the sign in the detection frame BB2 is depicted within the guide sign. This makes it possible to detect signs other than the guide sign that are integrated with the guide sign and depicted within the guide sign. In other words, according to the image processing device 10A of the second embodiment, the detection accuracy of road signs can be further improved.

[0085] <How to use the feature when detecting road signs depicted within guide signs> Furthermore, if the image processing devices 10 and 10A detect road signs depicted within the guide signs, possible uses include those shown in Figures 10 and 11. Figure 10 is an explanatory diagram (part 1) of a method of use when road signs depicted within guide signs are detected. Figure 11 is an explanatory diagram (part 2) of a method of use when road signs depicted within guide signs are detected.

[0086] As shown in Figure 10, the image processing devices 10 and 10A are assumed to have detected a guide sign corresponding to detection frame BB1 ​​and a "No Entry" regulatory sign corresponding to detection frame BB5 drawn within the guide sign. Furthermore, the guide sign is assumed to have depicted a detour corresponding to this regulatory sign (see arrow in the figure).

[0087] In this case, the controller 12 of the image processing devices 10 and 10A outputs the image recognition results, including detours, to, for example, a car navigation system included in the external device 9. The car navigation system receives the output from the image processing devices 10 and 10A and, for example, reconfigures the route so that it can guide the driver along a detour corresponding to the image recognition results.

[0088] In this case, the car navigation system will detour due to a road closure, but it does not necessarily have to set the detour route exactly as indicated by the signs. The car navigation system may, for example, readjust the optimal route as needed, depending on the vehicle's destination. This point is also true in Figure 11 below.

[0089] Furthermore, as shown in Figure 11, it is assumed that the image processing devices 10 and 10A have detected a guide sign corresponding to detection frame BB1 ​​and a "height restriction" regulatory sign corresponding to detection frame BB6 drawn within the guide sign. It is also assumed that the guide sign included a detour corresponding to this regulatory sign (see arrow in the figure).

[0090] In this case, the controller 12 of the image processing devices 10 and 10A outputs the image recognition results, including the detour route, to, for example, the car navigation system included in the external device 9. The car navigation system receives the output from the image processing devices 10 and 10A and, for example, if the vehicle violates a height restriction, it resets the route so that it can guide the driver on a detour route according to the image recognition results. The car navigation system does not reset the route if the vehicle does not violate a height restriction.

[0091] In this case, instead of the car navigation system, the image processing devices 10 and 10A may determine whether the vehicle violates the height limit based on, for example, vehicle specifications pre-stored in the storage unit 11. As a result, the image processing devices 10 and 10A may output the image recognition result to the car navigation system only if the vehicle violates the height limit.

[0092] <Other variations> Furthermore, while the image processing devices 10 and 10A are described as in-vehicle devices in the embodiments described above, they are not limited to in-vehicle devices. For example, they may be central devices that collect images P1 captured by each camera 3 from each vehicle via a network. Also, the image processing devices 10 and 10A may be terminal devices such as PCs (Personal Computers) or smartphones.

[0093] Furthermore, camera 3 may not be an in-vehicle camera fixedly mounted to the vehicle, but rather a camera that can be attached to and removed from the vehicle, such as a camera mounted on a PC or smartphone brought in by the vehicle's occupants. In this case, the output unit 7 may be a display or speaker provided by the PC or smartphone. Also, among the external devices 9, for example, a car navigation system may be application software with car navigation functionality that runs on a PC or smartphone.

[0094] Furthermore, the embodiments described above can be combined as appropriate, provided that the processing content is not contradictory. For example, steps S4 to S6 shown in Figures 6 and 7 may be performed instead of step S3 shown in Figure 1. Alternatively, steps S4 to S6 shown in Figures 6 and 7 may be performed together with step S3 shown in Figure 1.

[0095] Further effects and modifications can be readily derived by those skilled in the art. Therefore, broader aspects of the present invention are not limited to the specific details and representative embodiments expressed and described above. Accordingly, various modifications are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and their equivalents. [Explanation of Symbols]

[0096] 3 cameras 5. Various Sensors 7 Output section 9 External device 10,10A Image Processing Device 11 Storage section 11a First Image Recognition Model 11b Second Image Recognition Model 12 controllers

Claims

1. Equipped with a controller that detects road signs from images, The aforementioned controller, Image recognition processing on the aforementioned image detects the road sign as a first detection frame indicating a guide sign and a second detection frame indicating signs other than the guide sign. When the first detection frame encompasses the second detection frame, it is determined that the sign is depicted within the guide sign. Image processing device.

2. Equipped with a controller that detects road signs from images, The aforementioned controller, Image recognition processing on the aforementioned image detects the road sign as a first detection frame indicating a guide sign and a second detection frame indicating signs other than the guide sign. When the background color around the sign in the second detection frame matches the base color of the guide sign in the first detection frame, it is determined that the sign is depicted within the guide sign. Image processing device.

3. The aforementioned sign is a regulatory sign, a warning sign, or an informational sign. The image processing apparatus according to claim 1 or 2.

4. The aforementioned controller, The image recognition result, including the determination result of whether or not the sign is depicted within the guide sign, is output to an external device that performs a driving assistance function based on the image recognition result. The image processing apparatus according to claim 3.

5. The aforementioned controller, If it is determined that the aforementioned sign is depicted within the guide sign, the system determines that the sign is a warning sign and causes the external device to execute a driving assistance function at the forward position corresponding to the content of the warning sign. The image processing apparatus according to claim 4.

6. The aforementioned controller, If it is determined that the aforementioned sign is not depicted within the guide sign, the system determines that the sign is a normal sign and causes the external device to execute a driving assistance function at the current location corresponding to the content of the sign. The image processing apparatus according to claim 4.

7. The external device is, It is at least one of the following: an ADAS (Advanced Driver-Assistance Systems) device, a car navigation system, or an ECU (Electronic Control Unit). The image processing apparatus according to claim 4.

8. The aforementioned controller, The most frequent color in the area surrounding the sign within the second detection frame is set as the background color around the sign, and the most frequent color in the area of ​​the guide sign within the first detection frame is set as the base color of the guide sign. The image processing apparatus according to claim 2.

9. A road sign detection method performed by a controller, Detecting road signs from images, Image recognition processing on the aforementioned image detects the road sign as a first detection frame indicating a guide sign and a second detection frame indicating signs other than the guide sign. When the first detection frame encompasses the second detection frame, it is determined that the sign is depicted within the guide sign. A road sign detection method that includes [details omitted].

10. A road sign detection method performed by a controller, Detecting road signs from images, Image recognition processing on the aforementioned image detects the road sign as a first detection frame indicating a guide sign and a second detection frame indicating signs other than the guide sign. When the background color around the sign in the second detection frame matches the base color of the guide sign in the first detection frame, it is determined that the sign is depicted within the guide sign. A road sign detection method that includes [details omitted].

Images