Image recognition device

Abstract

This invention provides an image recognition device that suppresses the misidentification of objects such as buildings as road markings, thereby achieving more accurate road marking recognition. [Solution] The image recognition device 10 includes a candidate region extraction unit 24 that extracts candidate regions for road markings by image recognition, a type identification unit 26 that identifies the type of object in a unit region of an image based on probability, a determination unit 28 that determines the type of object within the candidate region based on the result of the type identification unit 26, and a marking region setting unit 30 that sets the road marking region based on the determination result, and excludes candidate regions in which the probability of the presence of objects other than road markings is higher than a predetermined threshold.

Description

[Technical Field]

[0001] The present invention relates to an image recognition device capable of recognizing road markings. [Background Art]

[0002] Accurately recognizing road signs is an important issue in automatic driving and driving assistance of vehicles. Patent Document 1 below discloses an image recognition device that detects road sign candidates from an image captured by a camera, calculates the degree of overlap between the candidate and a moving object area as a matching degree, and determines whether the candidate is a road sign according to the matching degree. According to this conventional technique, it is possible to suppress misrecognition of stickers or the like attached to moving objects such as preceding vehicles as road signs. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Unexamined Patent Publication No. 2021-022026 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, although the above-mentioned conventional technique can suppress misrecognition of stickers or the like attached to moving objects as road signs, it has a problem of misrecognizing stationary objects such as building signs as road signs. Specifically, when building signs, decorations, or the like have colors and shapes similar to those of road signs, they may be misrecognized as road signs, which may lead to providing incorrect road information to the driver.

[0005] In view of the above, the present invention has been made in consideration of the above problems, and an object of the present invention is to provide an image recognition device capable of suppressing misrecognition of objects such as building signs as road markings, and realizing more accurate road marking recognition. [Means for Solving the Problems]

[0006] (1) The image recognition device of the present invention comprises: a candidate region extraction unit that extracts candidate regions containing road markings as candidate regions by image recognition; a type identification unit that identifies the types of objects contained in a unit region of an image and derives, based on probability, which types of objects exist in each of the unit regions; a determination unit that determines the types of objects contained in the candidate regions extracted by the candidate region extraction unit based on the results derived by the type identification unit; and a marking region setting unit that sets regions containing road markings as marking regions based on the determination results by the determination unit, wherein the device excludes from the candidate regions extracted by the candidate region extraction unit regions in which the probability of objects other than road markings existing is higher than a predetermined threshold, based on the determination by the determination unit, from the list of candidate marking regions.

[0007] As described in (1) above, the image recognition device of the present invention extracts candidate marking candidate regions using a candidate region extraction unit, and for these extracted candidate regions, a type identification unit identifies the type of object for each unit region. Then, the determination unit determines the probability that an object other than a road marking exists within the candidate region based on the object type information identified by the type identification unit. Based on this determination result, candidate regions in which the probability of an object other than a road marking exists is higher than a predetermined threshold are excluded, and the remaining candidate regions are set as road marking regions. In this way, by combining two stages of processing—candidate region extraction and object type identification—the image recognition device of the present invention can suppress the misrecognition of building signs and the like as road markings.

[0008] (2) In the image recognition device of the present invention described above, the determination unit calculates the total number of unit regions included in the candidate region extracted by the candidate region extraction unit as the total number of unit regions, and calculates the number of unit regions within the candidate region that are identified as objects of a type other than the road marking by the type identification unit as the number of foreign object unit regions, calculates the ratio of the total number of unit regions to the number of foreign object unit regions as the foreign object probability, and excludes the candidate region from the candidates for the marking region if the calculated foreign object probability exceeds a predetermined threshold.

[0009] As described in (2) above, the image recognition device of the present invention has a determination unit that evaluates all unit regions within a candidate region and calculates the proportion of unit regions identified as objects other than road markings. By using this proportion as the probability of a non-road marking, the distribution of object types within the candidate region can be quantitatively evaluated. Then, by excluding the candidate region when this probability of a non-road marking exceeds a predetermined threshold, the image recognition device of the present invention can distinguish between road markings and other objects more accurately.

[0010] (3) In the image recognition device of the present invention described above, the candidate region extracted by the candidate region extraction unit is set as the first candidate region, the region in which the probability of being a road marking is determined to be greater than or equal to a predetermined value is set as the second candidate region, and the marking region setting unit sets the region as the marking region on the condition that it is included in both the first candidate region and the second candidate region.

[0011] As described in (3) above, the image recognition device of the present invention uses two candidate regions obtained by different methods: a first candidate region by the candidate region extraction unit and a second candidate region by the type identification unit. By setting the overlapping portion of these two candidate regions as the road marking region, the advantages of each method can be utilized while compensating for their disadvantages. Therefore, the image recognition device of the present invention can further improve the detection accuracy of road markings.

[0012] (4) In the image recognition device of the present invention described above, the type identification unit may use semantic segmentation to identify the type of object contained in the unit region.

[0013] As described in (4) above, the image recognition device of the present invention can accurately classify each unit region of an image according to its object type by using semantic segmentation technology. Therefore, the image recognition device of the present invention can more reliably distinguish between road markings and other objects. The present invention can also be implemented using methods other than semantic segmentation.

[0014] (5) In the image recognition device of the present invention described above, the type identification unit may use a deep learning model to identify the type of object contained in the unit region.

[0015] As described in (5) above, the image recognition device of the present invention can automatically learn the characteristics of objects by using a deep learning model, enabling more flexible identification of object types. Therefore, the image recognition device of the present invention can appropriately handle objects with various shapes and appearances. It should be noted that the present invention can also be implemented using methods other than deep learning models.

[0016] (6) The image recognition device of the present invention described above may further include a marking type determination unit that determines the type of road marking based on the marking area which is set as an area that includes the road marking.

[0017] As described in (6) above, the image recognition device of the present invention can determine the type of road marking more accurately by suppressing misrecognition and determining the type of road marking based on the set marking area. Therefore, the image recognition device of the present invention can provide drivers with more accurate road information. The present invention can also be implemented in a configuration that does not include a marking type determination unit.

[0018] (7) The image recognition device of the present invention described above may further include a notification unit that visually or audibly notifies the type of road marking determined by the marking type determination unit.

[0019] As described in (7) above, the image recognition device of the present invention enables drivers to reliably recognize information on road markings by notifying them of the determination result visually or audibly. In particular, by using both visual and audible notification in combination, the optimal notification method can be selected according to the driver's situation. The present invention can also be implemented in a configuration without a notification unit.

[0020] (8) In the image recognition device of the present invention described above, the candidate region extraction unit may extract the candidate region using template matching.

[0021] As described in (8) above, the image recognition device of the present invention can efficiently detect known road marking patterns by using template matching. Therefore, the image recognition device of the present invention can extract candidate regions while suppressing the processing load. The present invention can also be implemented using methods other than template matching.

[0022] (9) In the image recognition device of the present invention described above, the road marking is preferably this marking.

[0023] As described in (9) above, the image recognition device of the present invention can prioritize the detection of important information necessary for driving by targeting road markings such as regulatory markings, directional markings, warning markings, and guide markings. Therefore, the image recognition device of the present invention can provide drivers with more useful information. The present invention can also be implemented by targeting road markings other than those described above.

[0024] The configurations described in (1) to (9) above can be combined in any way. In particular, by combining the determination based on the probability of foreign objects in (2) with the determination of overlap between two candidate regions in (3), it becomes possible to detect road markings with higher accuracy. [Effects of the Invention]

[0025] According to the present invention, it is possible to provide an image recognition device capable of suppressing misrecognition of objects such as signboards on buildings as road markings and realizing more accurate road marking recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] [Figure 1] It is a block diagram showing the overall configuration of an image recognition device according to an embodiment of the present invention. [Figure 2] It is a block diagram showing the hardware configuration of an image recognition device according to an embodiment of the present invention. [Figure 3] It is a block diagram showing the functional configuration of a sign recognition unit according to an embodiment of the present invention. [Figure 4] It is a flowchart showing the flow of road marking recognition processing in the image recognition device according to an embodiment of the present invention. [Figure 5] It is an explanatory diagram showing an example of misrecognition prevention processing for a building signboard in the image recognition device according to an embodiment of the present invention. MODE FOR CARRYING OUT THE INVENTION

[0027] Hereinafter, an image recognition device 10 according to an embodiment of the present invention will be described in detail with reference to the drawings. In the following description, an outline of the image recognition device 10 is first described, then the configuration of each part constituting the image recognition device 10 and the operation of the image recognition device 10 will be described in further detail.

[0028] <<Outline of Image Recognition Device 10>> FIG. 1 is a block diagram showing the overall configuration of the image recognition device 10 according to the present embodiment. The image recognition device 10 includes a camera 12 and a sign recognition unit 20. The sign recognition unit 20 includes an image acquisition unit 22, a candidate region extraction unit 24, a type identification unit 26, a determination unit 28, a marking region setting unit 30, a marking type determination unit 32, and a notification unit 34.

[0029] Camera 12 is mounted on the vehicle to photograph the area in front of the vehicle, and outputs the captured image data to the traffic sign recognition unit 20. By installing camera 12 on the upper inside of the vehicle's windshield, it can photograph a wide area in front of the vehicle without obstructing the driver's view.

[0030] The sign recognition unit 20 performs road marking recognition processing based on image data input from the camera 12 via the image acquisition unit 22. In the road marking recognition processing, the candidate area extraction unit 24 extracts areas that may contain road markings as candidate areas through image recognition. The type identification unit 26 identifies the type of object contained in each unit area of ​​the image. The determination unit 28 determines the type of object contained in the candidate area. The marking area setting unit 30 sets the area containing the actual road marking based on the determination result. The marking type determination unit 32 determines the type of road marking from the set area. The notification unit 34 notifies the driver of the determined type of road marking.

[0031] The image recognition device 10 configured in this way can detect road markings from images captured by the camera 12, determine their type, and notify the driver. Furthermore, the image recognition device 10 can suppress the misidentification of building signs and the like as road markings, thereby enabling it to provide the driver with accurate information about the road conditions.

[0032] Figure 2 is a block diagram showing the hardware configuration of the image recognition device 10. In addition to the camera 12, the image recognition device 10 includes a processor 40, memory 42, display device 44, and speaker 46.

[0033] The processor 40 implements each function of the sign recognition unit 20 by executing a program stored in the memory 42. By using a central processing unit (CPU), graphics processing unit (GPU), or application-specific integrated circuit (ASIC) as the processor 40, image processing can be performed efficiently.

[0034] Memory 42 stores the program, various parameters used for detecting road markings, and template images of road markings. Memory 42 includes read-only memory (ROM) and random access memory (RAM). RAM is used as a workspace for image processing by the processor 40.

[0035] The display device 44 is configured as a vehicle's meter panel or display and visually notifies the driver of the type of road marking detected. The display device 44 can be configured as a liquid crystal display, an organic EL display, or a head-up display, etc.

[0036] Speaker 46 notifies the driver of the type of road marking detected by voice. Speaker 46 can utilize the vehicle's existing audio system speakers. This allows the driver to receive information about road markings without missing them, while keeping their eyes focused on the road ahead.

[0037] ≪Detailed Configuration of Each Part Constituting the Image Recognition Device 10≫ Figure 3 is a block diagram showing the functional configuration of the sign recognition unit 20. In the sign recognition unit 20, the image acquisition unit 22 performs preprocessing such as image distortion correction and brightness correction on the image data input from the camera 12.

[0038] The candidate region extraction unit 24 extracts regions from the preprocessed image that may contain road markings. The candidate region extraction unit 24 may have a processing unit consisting of, for example, a template matching unit 24a and a deep learning processing unit 24b.

[0039] The template matching unit 24a extracts candidate regions using template images of road markings stored in memory 42. Specifically, the template matching unit 24a calculates the similarity between the template images of road markings stored in memory 42 and the input image, and extracts regions with high similarity as candidate regions.

[0040] The deep learning processing unit 24b extracts candidate regions using a deep learning model that has learned the features of road markings. The deep learning processing unit 24b may be composed of, for example, a convolutional neural network having multiple convolutional layers and pooling layers. The deep learning processing unit 24b extracts features from each region of the input image and calculates the probability that it is a road marking based on those features. For example, the deep learning processing unit 24b may be able to extract candidate regions with the highest accuracy when the calculated probability is above a predetermined threshold, extract candidate regions with practical accuracy when it is above a second threshold, and extract a wide range of candidate regions when it is above a third threshold.

[0041] The type identification unit 26 identifies the type of object contained in each unit region of the input image. The segmentation unit 26a classifies each pixel of the image into object types such as buildings, roads, signs, billboards, utility poles, and trees. The type identification unit 26 includes the segmentation unit 26a and a probability calculation unit 26b that calculates the probability for each object type based on the segmentation results by the segmentation unit 26a.

[0042] The segmentation unit 26a uses semantic segmentation technology to classify each pixel of an image into object types such as buildings, roads, signs, billboards, utility poles, and trees. In semantic segmentation processing, feature quantities are extracted from each pixel of the image, and the object type is determined based on these feature quantities. The deep learning model used in semantic segmentation processing is constructed by learning the features of each object type from training data.

[0043] The probability calculation unit 26b calculates the probability of existence of each object type in each unit region based on the segmentation results. The type identification unit 26 can process the image using regions divided into a grid as unit regions. The probability calculation unit 26b calculates the probability of existence of each object type in each unit region based on the distribution of pixels classified by the segmentation unit 26a.

[0044] Furthermore, the type identification unit 26 can use information such as color, shape, texture, and positional relationship as features of the object. For example, buildings have straight edges and regular textures, while road markings have specific colors and shape patterns. By using a combination of these features, the type identification unit 26 can more accurately identify the type of object.

[0045] The processing results from the type identification unit 26 are output as the type of object contained in each unit region and its probability of existence. These processing results are used as basic data in the subsequent determination unit 28 when determining whether or not an object contained in a candidate region is a road marking.

[0046] The determination unit 28 determines the type of object included in the candidate region. The determination unit 28 includes either or both of the foreign object probability calculation unit 28a and the feature quantity determination unit 28b. In this embodiment, the determination unit 28 includes both the foreign object probability calculation unit 28a and the feature quantity determination unit 28b.

[0047] The foreign object probability calculation unit 28a calculates the foreign object probability as the ratio of the total number of unit regions within the candidate region to the number of unit regions identified as objects other than road markings. The feature determination unit 28b determines the type of object included in the candidate region based on the features extracted by the convolutional neural network (CNN).

[0048] More specifically, the foreign object probability calculation unit 28a calculates the probability of the presence of objects other than road markings within the candidate region. The foreign object probability calculation unit 28a calculates the total number of unit regions included within the candidate region as the total number of unit regions, and also calculates the number of unit regions identified by the type identification unit 26 as objects other than road markings as the number of foreign object unit regions. The foreign object probability calculation unit 28a calculates the ratio of the total number of unit regions to the number of foreign object unit regions as the foreign object probability.

[0049] The foreign object probability calculation unit 28a performs an exclusion determination of the candidate area based on the calculated foreign object probability. The threshold (determination threshold) for the exclusion determination based on the foreign object probability can be set appropriately according to the required determination accuracy. Specifically, if the foreign object probability is set to the first threshold (e.g., 90%), it can be determined with high reliability that the object is not a road marking; if the determination threshold is set to the second threshold (e.g., 80%), which is lower than the first threshold, a practical determination can be made; and if the determination threshold is set to the third threshold (e.g., 70%), which is lower than the second threshold, a determination that is only useful as a reference can be made. The foreign object probability calculation unit 28a can set the determination threshold according to the required determination accuracy.

[0050] The feature determination unit 28b extracts features from the candidate region image using a convolutional neural network (CNN) and determines the type of object based on these features. The feature determination unit 28b may, for example, hierarchically extract features such as color, shape, edges, and texture through multiple convolutional layers and pooling layers. In this case, the extracted features are determined by a classifier that has been trained on the features of road markings.

[0051] The determination unit 28 determines whether a candidate region includes road markings by comprehensively evaluating the determination result from the foreign object probability calculation unit 28a and the determination result from the feature quantity determination unit 28b. When both determination results match, the determination can be made with high reliability, and the number of misdeterminants can be reduced compared to when only one determination result is used.

[0052] The marking area setting unit 30 sets the area that includes the actual road markings based on the determination result. The marking area setting unit 30 has a first candidate area setting unit 30a, a second candidate area setting unit 30b, and an overlap determination unit 30c. The first candidate area setting unit 30a sets the candidate area extracted by the candidate area extraction unit 24 as the first candidate area. The second candidate area setting unit 30b sets the area that the type identification unit 26 has derived to have a probability of being a road marking of a predetermined value or more as the second candidate area. The overlap determination unit 30c sets the overlapping area between the first candidate area and the second candidate area as the area that includes road markings.

[0053] More specifically, the first candidate region setting unit 30a sets the candidate regions extracted by the candidate region extraction unit 24 as the first candidate regions. The first candidate region setting unit 30a sets the regions remaining after excluding candidate regions where the probability of foreign objects exceeds a predetermined threshold as the first candidate regions. The threshold (judgment threshold) related to the exclusion judgment by the first candidate region setting unit 30a can be set appropriately according to the required judgment accuracy. Specifically, if a first threshold (e.g., 90%) is set as the judgment threshold, the first candidate regions can be set with high reliability; if a second threshold (e.g., 80%) lower than the first threshold is set as the judgment threshold, the first candidate regions can be set with practical reliability; and if a third threshold (e.g., 70%) lower than the second threshold is set as the judgment threshold, the first candidate regions can be set with a level of reliability that is useful for reference. The first candidate region setting unit 30a can set the judgment threshold according to the required judgment accuracy.

[0054] The second candidate area setting unit 30b sets as a second candidate area an area in which the probability of being a road marking is derived by the type identification unit 26 to be above a predetermined threshold (judgment threshold). The second candidate area setting unit 30b can also be set appropriately according to the required judgment accuracy. Specifically, if the first threshold (e.g., 0.9) is set as the judgment threshold, the second candidate area can be set with high reliability; if the second threshold (e.g., 0.8), which is lower than the first threshold, is set as the judgment threshold, the second candidate area can be set with practical reliability; and if the third threshold (e.g., 0.7), which is lower than the second threshold, is set as the judgment threshold, the third candidate area can be set with a level of reliability that is useful for reference. The second candidate area setting unit 30b can set the judgment threshold according to the required judgment accuracy.

[0055] The overlap determination unit 30c sets the overlapping area between the first candidate area and the second candidate area as the area containing road markings.

[0056] The marking area setting unit 30 is better configured to correct the shape and size of the set road marking area. For example, by shaping the outline of the set area into a regular shape such as a rectangle or a circle, the marking area setting unit 30 can be expected to facilitate the subsequent marking type determination process. Furthermore, the marking area setting unit 30 is better configured to adjust the size of the area based on the standard size and aspect ratio of road markings.

[0057] Furthermore, the marking area setting unit 30 should exclude inappropriate areas by considering the positional relationship of the multiple set road marking areas. For example, the marking area setting unit 30 can be expected to reduce false detections by excluding areas that are too close to each other or areas that have an unnatural arrangement relationship as road markings.

[0058] The marking type determination unit 32 determines the type of road marking from the area set by the marking area setting unit 30. The marking type determination unit 32 can determine the type of marking, such as regulatory markings, directional markings, warning markings, and guide markings, from the set road marking area. The marking type determination unit 32 has a feature extraction unit 32a and a type determination unit 32b.

[0059] The feature extraction unit 32a extracts feature quantities to identify the type of road marking from a set area. For example, the feature extraction unit 32a extracts information such as color information, shape information, and symbol information as feature quantities.

[0060] The type determination unit 32b determines the type of road marking based on the feature quantities extracted by the feature extraction unit 32a. The type determination unit 32b determines the type of road marking from among regulatory markings, directional markings, warning markings, and guide markings. The type determination unit 32b can determine the type of road marking with high accuracy by, for example, using a deep learning model.

[0061] The sign recognition unit 20 configured as described above can improve the detection accuracy of road markings by combining different methods: image recognition and object type identification. Furthermore, by effectively excluding objects other than road markings from the candidate area, misrecognition can be suppressed.

[0062] The notification unit 34 includes a display control unit 34a and an audio control unit 34b, and notifies the driver of the determined type of road marking via a display device 44 and a speaker 46. The display control unit 34a causes the display device 44 to display icons and text information indicating the type of road marking. The display control unit 34a can change the display mode, such as the size, color, and flashing, according to the importance of the road marking. The audio control unit 34b outputs an audio message indicating the type of road marking from the speaker 46.

[0063] Next, with reference to the flowchart in Figure 4, the process of recognizing road markings in the image recognition device 10 described above will be explained. The road marking recognition process according to this embodiment is performed through the following steps.

[0064] (Step 1) In step 1, the image acquisition unit 22 acquires image data captured by the camera 12. The image acquisition unit 22 performs preprocessing such as distortion correction and brightness correction on the acquired image data to generate image data suitable for subsequent processing. After that, the control flow proceeds to step 2.

[0065] (Step 2) In step 2, the candidate region extraction unit 24 extracts regions that may contain road markings from the image data obtained in step 1 as candidate regions. The candidate region extraction unit 24 performs pattern matching processing by the template matching unit 24a and processing using a trained model by the deep learning processing unit 24b to determine the candidate regions. After that, the control flow proceeds to step 3.

[0066] (Step 3) When the control flow proceeds to step 3, the type identification unit 26 performs semantic segmentation processing on the entire image. This allows the type identification unit 26 to calculate the type of object contained in each unit region and its probability of existence. The segmentation unit 26a classifies each pixel of the image according to its object type, and the probability calculation unit 26b derives the probability of existence of each object type in each unit region. The control flow then proceeds to step 4.

[0067] (Step 4) In step 4, the determination unit 28 calculates the probability of a foreign object within the candidate region using the foreign object probability calculation unit 28a, and performs feature analysis of the candidate region using the feature determination unit 28b. After that, the control flow proceeds to step 5.

[0068] (Step 5) When the control flow proceeds to step 5, the determination unit 28 determines, based on the result derived in step 4, whether or not the candidate area is an area containing road markings. If it is determined that the candidate area is an area containing road markings, the control flow proceeds to step 6. On the other hand, if it is determined that the candidate area is not an area containing road markings (i.e., the candidate area is a foreign object), the control flow proceeds to step 9.

[0069] (Step 6) If the control flow progresses from step 5 to step 6, the marking area setting unit 30 sets the road marking area based on the determination result. After that, the control flow proceeds to step 7.

[0070] (Step 7) In step 7, the marking type determination unit 32 determines the type of road marking from the set road marking area. Based on the feature quantities extracted by the feature extraction unit 32a, the type determination unit 32b determines the appropriate type from among the markings. The control flow then proceeds to step 8.

[0071] (Step 8) In step 8, the notification unit 34 notifies the driver of the type of road marking that has been determined. The display control unit 34a provides a visual notification to the display device 44, and the voice control unit 34b provides an audio notification from the speaker 46. After that, the control flow returns to step 1.

[0072] (Step 9) If, as a result of the determination in step 5 described above, it is determined that the candidate area is not an area containing road markings (i.e., the candidate area is a foreign object), then in step 9, the process of excluding the candidate area from the road marking area is performed. After that, the control flow returns to step 1.

[0073] The above processing flow is repeatedly executed in synchronization with the timing of image acquisition by camera 12. This process allows the image recognition device 10 to continuously perform tasks from detecting road markings to notifying the driver.

[0074] ≪Specific Examples of Road Marking Recognition Processing≫ Figure 5 shows a specific example of the road marking recognition process in the image recognition device 10 according to this embodiment. Below, we will describe an example of processing that suppresses the misrecognition of building signs as road markings.

[0075] The image captured by camera 12 includes actual road markings 100 on the road and a circular sign 102 installed on the wall of building 120, as shown in Figure 5(a). The candidate region extraction unit 24 performs template matching and deep learning processing on this image and extracts both the road markings 100 and the sign 102 as candidate regions 110 and 112, as shown in Figure 5(b).

[0076] The type identification unit 26 performs semantic segmentation processing on the entire image. As shown in Figure 5(c), the segmentation processing classifies the road markings 100 as road markings and the signboards 102 as part of the building 120.

[0077] The determination unit 28 calculates the probability of an object being an alien object for each of the extracted candidate regions 110 and 112. As shown in Figure 5(d), in candidate region 112, which includes the sign 102 of building 120, a high probability of an alien object is calculated because the majority of the region is classified as building 120. On the other hand, in candidate region 100, which includes the actual road marking 100, the probability of an alien object is low.

[0078] The marking area setting unit 30 excludes candidate areas with a high probability of being foreign objects and sets only the area of ​​actual road markings as the road marking area. This makes it possible to suppress the misidentification of building signs as road markings.

[0079] Thus, the image recognition device 10 according to this embodiment can extract areas that are candidates for road markings and determine the type of object within those areas, thereby suppressing misrecognition of building signs and the like as road markings. Furthermore, it can appropriately detect multiple road markings and effectively notify the driver.

[0080] <Effects> The image recognition device 10 described above has the following characteristic configurations (a) to (i). Therefore, the image recognition device 10 of this embodiment can achieve the following unique effects that cannot be achieved with conventional technology.

[0081] (a) The image recognition device 10 of this embodiment includes: a candidate region extraction unit 24 that extracts candidate regions containing road markings as candidate regions by image recognition; a type identification unit 26 that identifies the types of objects contained in the unit regions of the image and derives which type of object exists in each unit region based on probability; a determination unit 28 that determines the types of objects contained in the candidate regions extracted by the candidate region extraction unit 24 based on the results derived by the type identification unit 26; and a marking region setting unit 30 that sets regions containing road markings as marking regions based on the determination result by the determination unit 28. Among the candidate regions extracted by the candidate region extraction unit 24, regions in which the probability of objects other than road markings being present is higher than a predetermined threshold, as determined by the determination unit 28, are excluded from the candidate marking regions.

[0082] In this embodiment, the image recognition device 10 extracts candidate marking candidate regions using the candidate region extraction unit 24 as described in (a) above, and the type identification unit 26 identifies the type of object for each unit region of the extracted candidate regions. Then, the determination unit 28 determines the probability that an object other than a road marking exists within the candidate region based on the object type information identified by the type identification unit 26. Based on this determination result, candidate regions in which the probability of an object other than a road marking exists is higher than a predetermined threshold are excluded, and the remaining candidate regions are set as road marking regions. In this way, by combining two stages of processing—candidate region extraction and object type identification—the image recognition device 10 of this embodiment can suppress the misrecognition of building signs and the like as road markings.

[0083] (b) In the image recognition device 10 of this embodiment, the determination unit 28 calculates the total number of unit regions included in the candidate region extracted by the candidate region extraction unit 24 as the total number of unit regions, and calculates the number of unit regions within the candidate region that are identified as objects of a type other than road markings by the type identification unit 26 as the number of foreign object unit regions, calculates the ratio of the total number of unit regions to the number of foreign object unit regions as the foreign object probability, and excludes the candidate region from the list of marking regions if the calculated foreign object probability exceeds a predetermined threshold.

[0084] In this embodiment, the image recognition device 10, as described in (b) above, has a determination unit 28 that evaluates all unit regions within the candidate region and calculates the proportion of unit regions identified as objects other than road markings. By using this proportion as the probability of a foreign object, the distribution of object types within the candidate region can be quantitatively evaluated. Then, by excluding the candidate region when this probability of a foreign object exceeds a predetermined threshold, the image recognition device 10 of this embodiment can distinguish between road markings and other objects more accurately.

[0085] The image recognition device 10 of this embodiment has the configuration described in (b) above, but the present invention is not limited thereto. For example, it is possible to evaluate only a specific sub-region within the candidate region, or to use a machine learning model to determine the type of object. By adopting such a configuration, road markings can be detected while reducing the processing load.

[0086] (c) In the image recognition device 10 of this embodiment, the candidate region extracted by the candidate region extraction unit 24 is set as the first candidate region, the region in which the probability of being a road marking is determined to be greater than or equal to a predetermined value by the type identification unit 26 is set as the second candidate region, and the marking region setting unit 30 sets the region as a marking region on the condition that it is included in both the first candidate region and the second candidate region.

[0087] As described in (c) above, the image recognition device 10 of this embodiment uses two candidate regions obtained by different methods: a first candidate region by the candidate region extraction unit 24 and a second candidate region by the type identification unit 26. By setting the overlapping portion of these two candidate regions as the road marking region, the advantages of each method can be utilized while compensating for their disadvantages. Therefore, the image recognition device 10 of this embodiment can further improve the detection accuracy of road markings.

[0088] Although the image recognition device 10 of this embodiment has the configuration described in (c) above, the present invention is not limited thereto, and it is also possible to have an image recognition device that does not have the configuration described in (c) above.

[0089] (d) In the image recognition device 10 of this embodiment, the type identification unit 26 identifies the type of object contained in the unit region using the semantic segmentation unit 26a.

[0090] As described in (d) above, the image recognition device 10 of this embodiment can accurately classify each unit region of an image according to its object type by using the semantic segmentation unit 26a. Therefore, the image recognition device 10 of this embodiment can more reliably distinguish between road markings and other objects.

[0091] Although the image recognition device 10 of this embodiment has the configuration described in (d) above, the present invention is not limited thereto. For example, other image recognition technologies such as object detection and region segmentation can also be used. Furthermore, by using multiple image recognition technologies in combination, it is possible to identify object types with higher accuracy.

[0092] (e) In the image recognition device 10 of this embodiment, the type identification unit 26 uses the deep learning processing unit 24b to identify the type of object contained in the unit region.

[0093] As described in (e) above, the image recognition device 10 of this embodiment can automatically learn the characteristics of an object by using the deep learning processing unit 24b, enabling more flexible identification of the object type. Therefore, the image recognition device 10 of this embodiment can appropriately handle objects with various shapes and appearances.

[0094] The image recognition device 10 of this embodiment has the configuration described in (e) above, but the present invention is not limited thereto. For example, it is also possible to use conventional image processing algorithms or rule-based determination methods. With such a configuration, stable identification of object types can be performed even when the amount of training data is small.

[0095] (f) The image recognition device 10 of this embodiment is equipped with a marking type determination unit 32 that determines the type of road marking based on a marking area set as an area containing road markings.

[0096] As described in (f) above, the image recognition device 10 of this embodiment can perform more accurate determination of the type of road marking by having the marking type determination unit 32 determine the type of road marking based on the marking area set while suppressing misrecognition. Therefore, the image recognition device 10 of this embodiment can provide drivers with more accurate road information.

[0097] Although the image recognition device 10 of this embodiment has the configuration described in (f) above, the present invention is not limited thereto. For example, it is also possible to configure the device to detect only the location of road markings. By using such a configuration, the processing load can be reduced while informing the driver of the presence of road markings.

[0098] (g) The image recognition device 10 of this embodiment is equipped with a notification unit 34 that visually or audibly notifies the type of road marking determined by the marking type determination unit 32 via a display device 44 and / or a speaker 46.

[0099] In this embodiment, the image recognition device 10 allows the driver to reliably recognize road marking information by having the notification unit 34 notify the driver of the determination result visually or audibly, as described in (g) above. In particular, by using both visual and audible notification in combination, the optimal notification method can be selected according to the driver's situation.

[0100] Although the image recognition device 10 of this embodiment has the configuration described in (g) above, the present invention is not limited thereto. For example, it is also possible to configure it to provide road marking information to other vehicle control systems. With such a configuration, road marking information can be utilized in the control of an autonomous driving system or the like.

[0101] (h) In the image recognition device 10 of this embodiment, the candidate region extraction unit 24 extracts candidate regions using the template matching unit 24a.

[0102] As described in (h) above, the image recognition device 10 of this embodiment can efficiently detect known road marking patterns by using the template matching unit 24a. Therefore, the image recognition device 10 of this embodiment can extract candidate regions while suppressing the processing load.

[0103] The image recognition device 10 of this embodiment has the configuration described in (h) above, but the present invention is not limited thereto. For example, other image processing methods such as edge detection and pattern recognition can also be used. With such a configuration, road markings with shapes not registered in the template can also be detected.

[0104] (i) In the image recognition device 10 of this embodiment, road markings are main markings consisting of regulatory markings, directional markings, warning markings, guide markings, etc.

[0105] As described in (i) above, the image recognition device 10 of this embodiment can prioritize the detection of important information necessary for driving by targeting main signs such as regulatory signs, instruction signs, warning signs, and guidance signs. Therefore, the image recognition device 10 of this embodiment can provide drivers with more useful information.

[0106] While the image recognition device 10 of this embodiment has the configuration described in (i) above, the present invention is not limited thereto. For example, it is also possible to detect auxiliary signs and road markings. Such a configuration allows for the provision of a wider range of road information to the driver.

[0107] The configurations described in (a) to (i) above can be combined in any way. In particular, combining the determination by probability of foreign objects in (b) with the determination of overlap between two candidate regions in (c) enables more accurate detection of road markings. Furthermore, combining semantic segmentation in (d) with template matching in (h) can achieve highly accurate and efficient detection.

[0108] The present invention is not limited to the configurations described in the embodiments, etc. above, and can be modified as appropriate without departing from the scope of the technical idea of ​​the present invention. The components of each embodiment and modification described above can be arbitrarily selected and combined. Furthermore, any component of each embodiment and modification can be arbitrarily combined with any component described in the means for solving the problem, the form for carrying out the invention, etc., or a component that embodies any component described in the means for solving the problem, the form for carrying out the invention, etc. The present application also intends to obtain rights in this application or in divisional applications, amendment applications, etc. based on this application. [Industrial applicability]

[0109] The present invention can be suitably used in a wide range of image recognition devices capable of recognizing road markings. [Explanation of Symbols]

[0110] 10: Image recognition device 24: Candidate region extraction section 24a: Template Matching Section 24b: Deep Learning Processing Unit 26: Type identification part 26a: Semantic Segmentation Section 28: Judgment section 30: Display area setting section 32: Marking type determination section 34:Notification Department 44:Display device 46: Speaker

Claims

1. A candidate region extraction unit extracts candidate regions containing road markings using image recognition, A type identification unit that identifies the types of objects contained in a unit region of an image and derives, based on probability, which types of objects are present in each of the unit regions, A determination unit determines the type of object contained in the candidate region extracted by the candidate region extraction unit based on the result derived by the type identification unit, Based on the determination result by the determination unit, a marking area setting unit sets the area containing road markings as a marking area, It has, An image recognition device characterized by excluding from the candidate marking areas any area extracted by the candidate area extraction unit any area in which the probability of an object other than the road marking being present is higher than a predetermined threshold, as determined by the determination unit, from the candidate marking areas.

2. The determination unit, The total number of unit regions included in the candidate regions extracted by the candidate region extraction unit is calculated as the total number of unit regions, Within the candidate area, the number of unit areas identified by the type identification unit as being of a type other than the road marking is calculated as the number of non-uniform object unit areas. The ratio of the total number of unit regions to the number of foreign object unit regions is calculated as the foreign object probability. The image recognition device according to claim 1, characterized in that the candidate region is excluded from the candidate region for marking on the condition that the calculated probability of a foreign object exceeds a predetermined threshold.

3. The candidate region extracted by the candidate region extraction unit is designated as the first candidate region. The region in which the probability of being a road marking is derived to be greater than or equal to a predetermined value by the type identification unit is designated as the second candidate region. The image recognition device according to claim 1 or 2, characterized in that the marking area setting unit sets an area as the marking area on the condition that the area is included in both the first candidate area and the second candidate area.

4. The image recognition device according to claim 1 or 2, characterized in that the type identification unit identifies the type of object contained in the unit region using semantic segmentation.

5. The image recognition device according to claim 1 or 2, characterized in that the type identification unit identifies the type of object contained in the unit region using a deep learning model.

6. The image recognition device according to claim 1 or 2, further comprising a marking type determination unit that determines the type of road marking based on the marking area set as an area containing the road marking.

7. The image recognition device according to claim 1 or 2, characterized in that the candidate region extraction unit extracts the candidate region using template matching.

Images