A method for mining difficult examples of semantic segmentation in a crowd-sourced map and a readable storage medium

Abstract

The application protects a crowd-sourced map semantic segmentation difficult case mining method and a computer readable storage medium, the method comprises the following steps: step 1, the number of labels is counted: the number of each label in the training set and the validation set is counted for the labeled data; step 2, in the inference part of the semantic segmentation training model, the iou value of each label element on each picture is recorded; step 3, the recorded iou value is filtered, and the picture path and element with iou value below the threshold are recorded; step 4, the low iou element, the corresponding labeled picture and the inference effect picture are combined, and the element or scene with poor segmentation effect is obtained, that is, the segmented difficult case. The application can effectively mine the semantic segmentation difficult case for crowd-sourced map construction by comprehensively considering the label quantity and the inference effect, and extract the difficult case data.

Description

Technical Field

[0001] This invention is applicable to the field of deep learning, specifically relating to a method for mining difficult examples of semantic segmentation in crowdsourced maps. Background Technology

[0002] Image semantic segmentation is gradually becoming a research hotspot for computer vision and machine learning researchers. Semantic segmentation of 2D images, videos, and even 3D data is a key problem in the field of computer vision. Semantic segmentation has also been applied to crowdsourced maps and autonomous driving. During the operation of an autonomous vehicle, images detected by onboard cameras or LiDAR are input into a neural network. The trained algorithm model can then automatically segment and classify the images to avoid obstacles such as pedestrians and vehicles.

[0003] For autonomous driving, accurate maps are a crucial element for safe driving. Currently, crowdsourced maps are an effective supplement to high-precision maps. In crowdsourced map building, labeled road surface element data is typically processed through a semantic segmentation model to obtain segmented elements, which are then used for backend SLAM mapping. Generally, the larger the dataset, the better the semantic segmentation results; however, in reality, there is often insufficient data. The data used is real road surface video data collected by the front-facing camera of the experimental vehicle while it is driving normally on the road. This data is then processed to obtain image data, which is then labeled and used for the segmentation model. However, in practice, the segmented metrics of the various elements from normally collected data can vary significantly.

[0004] Furthermore, the purpose of hard example mining is to identify hard examples that are either missed or false positives. Currently, commonly used hard example mining methods can be broadly categorized into two types: supervised hard example mining methods, which require a large amount of labeled data. Data labeling consumes a significant amount of manpower, and with a large amount of data, the labor cost is extremely high (which is also the main reason why there is not enough data in actual training); and unsupervised hard example mining. In this type of method, how to efficiently extract hard examples is also a problem that needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a method for mining difficult examples of semantic segmentation in crowdsourced maps, which can realize the mining of difficult examples of semantic segmentation and extract difficult example data.

[0006] The technical solution of the present invention is as follows:

[0007] The method of this invention uses a semantic segmentation training model, which includes two stages: training and inference. The Intersection over Union (IoU) is an important evaluation metric; the higher the IoU value, the better the segmentation effect. This invention first counts the number of each label in the labeled data. Then, during the inference stage, by setting a threshold, images containing elements less than the threshold are filtered out. By comprehensively considering the number of labels and the inference effect, difficult examples of hard-to-segment elements are discovered.

[0008] The method of this invention specifically includes:

[0009] Step 1: Count the number of labels: For the labeled data, count the number of each label in the training set and the validation set.

[0010] Step 2: In the semantic segmentation training model, during the inference part, i.e., during the inference on the valset, record the IoU value of each label element on each image.

[0011] Step 3: Filter the recorded IOUs and record the image paths and elements with IOUs below the threshold.

[0012] Step 4: Combining low IOU elements with their corresponding labeled images and inference results, we can identify elements or scenes with poor segmentation performance, which are the difficult examples segmented.

[0013] Specifically, in step 1, the semantic segmentation categories are first determined and recorded in a txt file; then, based on the category txt file and the labeled json file, the number of each label is counted using a Python script.

[0014] Specifically, step 2 involves recording the inference result index (iou) for each image in the inference part of the semantic segmentation training model in txt format.

[0015] Specifically, step 3 involves using a Python script, combined with an annotated JSON file, to filter the IOU records in the txt file to obtain image paths below a threshold.

[0016] Specifically, in step 4, the inference module is first run separately using valset to obtain the inference result image on valset. Then, by combining the category quantity file, the annotation file, and the inference effect image of each element, the elements and scenes with poor segmentation effect are identified, which are the segmented difficult examples.

[0017] In a second aspect, the present invention also provides a computer-readable storage medium having stored computer-readable instructions thereon, which, when executed by a computer's processor, cause the computer to perform the steps of the crowdsourced map semantic segmentation hard example mining method.

[0018] By adopting the above technical solutions, this invention analyzes the elements of the semantic segmentation process in crowdsourced map construction by comprehensively considering the number of tags and the reasoning effect, and can effectively mine difficult semantic segmentation cases for crowdsourced map construction and extract difficult case data. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method. Detailed Implementation

[0020] The following will describe the implementation of this application through specific embodiments in conjunction with the accompanying drawings. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0021] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0022] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0023] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0024] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0025] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0026] Crowdsourced map update data can be divided into four main categories. First is trajectory data. Positioning must be from a high-precision source, typically derived from GNSS + inertial navigation location perception data and high-precision trajectory data obtained by fusing visual positioning, high-precision positioning, LiDAR positioning, and millimeter-wave radar positioning through SLAM or related technologies. However, single trajectory information has very low value; it must be scaled up. This mainly involves topology network updates and also includes dynamic traffic information. Second is image data. Image data contains a vast amount of information, mostly unstructured. Transforming this unstructured data into structured data requires calibration and AI algorithms to convert image data into vector data. Third is radar data, including LiDAR, millimeter-wave radar, and some ultrasonic radars. LiDAR offers very high precision. Millimeter-wave radar can cover all applications and is valuable for improving data accuracy in scenario reconstruction. Fourth is CAN data, containing millisecond-level output data from various in-vehicle components. Mining this data is highly beneficial for developing decision-making algorithms for autonomous driving.

[0027] Currently, most mainstream deep learning-based image segmentation methods are based on Fully Convolutional Network for Semantic Segmentation (FCN), a work presented at CVPR 2015. FCN's pixel-to-pixel approach is used to solve various pixel-level vision tasks, such as depth estimation, optical flow estimation, and image generation. Challenges faced by FCN algorithms include: low resolution: the resolution typically decreases as the network depth increases, leading to a loss of spatial information; lack of contextual information: the receptive field of pixel-level features is insufficient, and objects exhibit multi-scale issues (the absolute size of different object categories has multiple scales, and even objects of the same category have multiple scales due to varying distances from the camera); and boundary errors: because the feature representations of pixels in boundary regions typically have weak discriminative power, many semantic segmentation errors are distributed in these boundary regions.

[0028] Semantic segmentation is a fundamental task in computer vision. In semantic segmentation, we need to categorize visual input into different semantically interpretable categories—that is, categories that are meaningful in the real world. For example, we might need to distinguish all pixels in an image that belong to a car. Compared to image classification or object detection, semantic segmentation gives us a more detailed understanding of images and is crucial in many fields such as autonomous driving, robotics, and image search engines.

[0029] When an autonomous vehicle is in motion, its onboard cameras and lidar detect obstacles and generate images that are then input into a neural network. The trained semantic segmentation model automatically segments and categorizes the images to avoid obstacles such as pedestrians and vehicles.

[0030] See Figure 1 The following embodiment provides a detailed explanation of the steps involved in the hard example mining method for semantic segmentation in crowdsourced maps:

[0031] Step 1: Count the number of labels

[0032] When autonomous vehicles are driving on the road, they may encounter various complex situations, making the identification and judgment of road markings particularly important. However, solid lines, stop lines, and herringbone lines next to hard barriers are easily missed. Different elements appear at different frequencies on the road. This embodiment categorizes the labeled data elements into three types based on road markings and obstacle conditions. The first type of element is very common on the road, and there are many labels for this type of element in the labeled data, such as solid and dashed lane lines. The second type of element is less common on the road, and there are fewer labels for this type of element in the labeled data, such as speed bumps. Although the third type of element is rarely seen on the actual road surface, because it consists of multiple elements of the same shape, the presence of even one of these elements will result in many labels. Herringbone lines are another example of this type of element being rarely seen on the road in practice. This embodiment can use these three scenarios as challenging examples for semantic segmentation.

[0033] First, the labeled dataset is divided into training and validation sets in a 3:1 ratio using a random classification method, and the number of each label in the training set and validation set is counted.

[0034] In this embodiment, the training set (trainset) and validation set (valset) are counted separately because the amount of data is small. By counting them separately, we can see the general situation of the labels in the training set and the validation set, that is, whether all labels are included and the number of labels.

[0035] For example, the following are the statistics of the number of labels for the trainset and the valset, respectively:

[0036] background:0

[0037] void:816

[0038] dividing:2144

[0039] guiding:4471

[0040] stopping:129

[0041] chevron:45

[0042] parking: 160

[0043] zebra:2034

[0044] turn:770

[0045] reduction:14

[0046] fish_line:138

[0047] no parking:21

[0048] pole_light:2901

[0049] pole_sign:684

[0050] pole_other:1375

[0051] background:0

[0052] void:213

[0053] dividing:554

[0054] guiding: 1255

[0055] stopping:23

[0056] chevron:5

[0057] parking:19

[0058] zebra:295

[0059] turn:171

[0060] reduction:8

[0061] fish_line:62

[0062] no parking:2

[0063] pole_light:972

[0064] pole_sign:116

[0065] pole_other:334.

[0066] Step 2: Record IOU information

[0067] The semantic segmentation model is trained using the training set, and then the semantic segmentation model output from the training is used to infer the validation set. The path and IOU value of each semantic segmentation original image in the inference result are recorded.

[0068] The following is an example of a txt file containing IOU records:

[0069] {'filename': ' / home / ca_lyw / data / segdata_gan / images / val / data_video02_frame_175.jpg', 'Iou': tensor([0.9924, 0.9737, 0.6402, 0.6822, 0.0000,0.0000, 0.0000, 0.0000, 0.0000,0.0000, 0.0000, 0.0000, 0.0214, 0.0000,0.0000])}

[0070] {'filename':' / home / ca_lyw / data / segdata_gan / images / val / data_video02_frame_176.jpg', 'Iou': tensor([0.9941, 0.9785, 0.5652, 0.4160, 0.0000,0.0000, 0.0000, 0.0000, 0.0000,0.0000, 0.0000, 0.0000, 0.2438, 0.0000,0.0000])}

[0071] {'filename':' / home / ca_lyw / data / segdata_gan / images / val / data_video02_frame_177.jpg', 'Iou': tensor([0.9934, 0.9784, 0.3892, 0.6513, 0.0000,0.0000, 0.0000, 0.0000, 0.0000,0.0000, 0.0000, 0.0000, 0.4690, 0.0000,0.0000])}.

[0072] Step 3, IOU filtering

[0073] The filtering principle is as follows: it is generally considered that when the IoU value is greater than 0.5, the target segmentation result is acceptable. Therefore, in this embodiment, records with an IoU value greater than 0.5 will be filtered out, that is, only records with an IoU value of no more than 0.5 (unacceptable segmentation results) will be retained as the first basis for subsequent judgment of difficult cases. The IoU txt file and the valset json file are processed by a Python script to obtain the image paths and element types with IoU values ​​below the threshold and their IoU values, and saved in txt file format.

[0074] As shown below:

[0075] {'filename': 'data_video02_frame_176.jpg', 'Iou': {'guiding': 0.416}}

[0076] {'filename': 'data_video02_frame_177.jpg', 'Iou': {'dividing':0.3892}}

[0077] {'filename': 'data_video02_frame_178.jpg', 'Iou': {'dividing': 0.4026, 'guiding': 0.2024}}.

[0078] Step 4: Combine the filtered IOUs, labeled JSON files, and inference results to identify difficult examples: First, run the inference module separately using valset to obtain the inference result image on valset. Then, combine the category quantity file, label file, and inference effect diagram of each element to identify the elements and scenes with poor segmentation results, which are the difficult examples identified.

[0079] Specifically as follows:

[0080] 1. Use a semantic segmentation model to perform inference on the validation set to obtain an image with inference effect. That is, the segmentation result is directly superimposed on the original image, and the segmentation effect can be intuitively seen from the image.

[0081] 2. Intuitive observation and reasoning reveal significant missed detections of solid lines, stop lines, and herringbone lines next to hard barriers. By examining the corresponding annotation files and the total number of these labels, it's concluded that different elements appear at varying frequencies on the road. Therefore, the annotated data elements can be categorized into three types: First, elements that are common on the road surface have many labels, such as solid and dashed lane lines; second, elements that are less common on the road have fewer labels, such as speed bumps; and third, elements that are rare on the actual road surface, but are composed of multiple elements of the same shape, resulting in numerous labels even for a single element, such as herringbone lines. Therefore, these three scenarios (excluding categories because solid lines are a category, including solid lines next to hard barriers) are considered challenging examples for semantic segmentation. Subsequent model training should increase the amount of data for these three scenarios to improve segmentation quality.

[0082] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solution of the present invention, and these simple modifications all fall within the protection scope of the present invention.

[0083] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the present invention will not describe the various possible combinations separately. Furthermore, various different embodiments of the present invention can also be arbitrarily combined, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed by the present invention.

Claims

1. A method for mining hard examples of semantic segmentation in a crowd-sourced map, the method comprising: The method comprises: Step 1, counting the number of labels: counting the number of labels in the training set and the validation set, and identifying three types of difficult example scenarios according to the number of labels: the first type is an element that is common on the road and has a large number of labels, the second type is an element that is rare on the road and has a small number of labels, and the third type is an element that is rare on the actual road but has a large number of labels due to being composed of multiple identical shapes; Step 2, in the semantic segmentation training model, record the iou value of each label element on each picture in the inference part; Step 3, filter the recorded iou value, record the picture path and element with iou value below the threshold; Step 4, combine the low iou element, its corresponding labeled picture and inference effect picture to comprehensively obtain the element or scene with poor segmentation effect, that is, the difficult example obtained by segmentation, specifically, first run the inference module separately with the validation set valset to obtain the inference result picture on the validation set valset, then combine the category number file, the annotation file and the element inference effect picture to obtain the element and scene with poor segmentation effect, that is, the difficult example obtained by segmentation.

2. The method of claim 1, wherein, In the step 1, first determine the category of semantic segmentation, record the category in the txt file; then according to the category txt file and the annotation json file, count the number of labels through a python script.

3. The method of claim 1, wherein, The step 2 is to record the inference result index iou of each picture in the semantic segmentation training model in the inference part in the form of txt.

4. The method of claim 1, wherein, The step 3 is to filter the iou recorded in the txt by combining the annotation json file through a python script to obtain the picture path below the threshold.

5. A computer readable storage medium, characterized in that, It stores computer readable instructions, which, when executed by the processor of the computer, cause the computer to perform the steps of the difficult example mining method for semantic segmentation in the crowd-sourced map according to any one of claims 1 to 4.

Images