An example segmentation labeling method and device for vehicle exterior component damage images

Abstract

The application discloses a kind of vehicle appearance component damage image instance segmentation annotation method and device, comprising: preloading damage instance segmentation detection model inference high probability damage polygon target;First personnel labels the pre-labeled damage target object, and the second personnel inspects the labeling result of first personnel in the way of classification cleaning;After cleaning, the training sample GT that has been inspected is sent to model training, and iteration model A is obtained;Based on iteration model A, the training sample GT is inferred, and overlap is judged;The damage target inferred without overlapping with GT is screened out as the data to be cleaned, and is cleaned;Select the picture sample of high neglect type frequency for secondary correction;The sample after cleaning, classification and correction is sent to model training again, and iteration model B is obtained;The above steps are repeated N times, and the final iteration model is obtained.The method of the application can reduce the labeling cost, the data noise is smaller, and the model training effect is more stable.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, specifically to a method and apparatus for instance segmentation and annotation of vehicle exterior component damage images, which is used for data production required for training a vehicle exterior component damage instance detection model in damage assessment videos. Background Technology

[0002] Vehicle exterior components: These refer to the vehicle assembly components that are visible to the human eye, such as fenders, bumpers, and headlights.

[0003] Instance segmentation: The machine automatically uses object detection methods to outline different instances in the image, and then uses semantic segmentation methods to label each instance region pixel by pixel.

[0004] Since 2012, deep learning has made significant progress in the field of computer vision. Compared to traditional image recognition methods that use low-level visual features such as color and HOG, deep neural networks can learn more advanced and abstract features, which makes their performance far surpass that of traditional methods.

[0005] Since 2014, deep learning has made outstanding progress in computer vision fields such as object detection, semantic segmentation, and instance segmentation, giving rise to a series of methods such as YOLO, Faster R-CNN, Mask R-CNN, and Deeplab. In terms of recognition accuracy for specific tasks, it has approached or surpassed the level of human recognition, while in terms of recognition speed and the scale of data processing, it has far surpassed human capabilities.

[0006] There has been relatively little work on applying deep learning to damage detection and recognition of vehicle exterior components, with attempts beginning around 2016. However, due to limitations such as difficulty in acquiring data and the high training difficulty of early deep learning models, progress in this field has been slow, and there is currently no readily available system or method for practical application.

[0007] As is well known, damage detection and identification of vehicle exterior components is a crucial part of various automotive businesses. For example, in car insurance claims, car-sharing, and daily car rental processes, the current state of damage to vehicle exterior components needs to be assessed to determine the user's liability. Currently, some insurance companies are piloting self-service damage assessment and claims processing. Customers submit a damage assessment video or several damage assessment photos, upload them to the server, and an intelligent damage assessment system using a deep learning network model identifies the damage. Professional damage assessors then review and complete the assessment process.

[0008] There are two main approaches to academic research. One is based on small-sample deep learning techniques, exploring how to achieve better deep learning network model learning with a small amount of specialized sample data. However, this approach is not yet mature and there are no effective engineering projects to implement it. The other approach is still about how to produce more training sample data in a low-cost and efficient manner to conduct supervised deep learning network model learning. This approach is more mature.

[0009] Existing patent CN113160209A discloses a target annotation method and a target recognition method for detecting damage to building facades. This target annotation method allows for rapid fine-tuning of the target recognition model. This approach typically adjusts the target detection bounding box to improve its recognition performance, thereby enhancing stability and efficiency.

[0010] Existing patent CN 107358596 A discloses an image-based vehicle damage assessment method, device, electronic device, and system. This system uses user-uploaded images as input to an intelligent damage assessment system. However, regardless of whether images or videos are used as input, the detection and recognition of damage to exterior components in a single image will be involved. This invention mainly discusses the data training and annotation problem of damage instances in a single image of a vehicle's exterior components.

[0011] The existing system has the following problems:

[0012] 1. Over-reliance on professional annotators leads to high annotation costs. In the segmentation and detection annotation data of vehicle exterior component damage instances, a certain proportion of the images are of simple professional difficulty, which can be mastered by ordinary annotators with a little familiarity. At the same time, a certain proportion of the images are of moderate professional difficulty, and ordinary annotators can master most of the knowledge for these images after a certain period of professional learning, that is, the accuracy of the annotation target can reach 70% to 80%. The remaining proportion of images are of complex professional difficulty, which ordinary annotators cannot master through learning.

[0013] 2. The individual differences among annotators are significant, sometimes reaching 30%. Therefore, similar to medical image data annotation, when discrepancies are encountered, it is usually necessary to introduce senior experts to conduct sample quality checks, unify the differences, and ensure the reliability of the annotated data. Otherwise, the training data will be noisy, which is not conducive to model training. Therefore, the cost of data quality checks is also very high. Summary of the Invention

[0014] To address the aforementioned problems in the prior art, the present invention provides a method and apparatus for instance segmentation and annotation of damaged images of vehicle exterior components.

[0015] This invention discloses an instance segmentation and annotation method for vehicle exterior component damage images, comprising:

[0016] Step 1: Based on the initially trained damage instance segmentation and detection model, infer the vehicle exterior component images to be labeled, and use the damage polygon targets with a probability higher than the threshold as pre-labeled damage target objects.

[0017] Step 2: The first person marks the pre-marked damaged target object;

[0018] Step 3: The second person performs quality inspection on the annotation results of Step 2 using a classification and cleaning method. The quality inspection process includes: storing the correct annotation (GT) of the vehicle exterior component image to be annotated; the second person then classifies and cleans the GT; if an incorrect damage target type is encountered, it is adjusted to the correct damage target type; if a non-damaged target is encountered, it is adjusted to the background type; if a target's image quality cannot accurately determine the actual situation, it is adjusted to the ignore type; if the damage target's boundary is incorrect, it is adjusted to the ignore type.

[0019] Step 4: Send the cleaned and quality-checked training samples GT from Step 3 to the model for training to obtain iterative model A;

[0020] Step 5: Based on iterative model A, infer the training samples GT that have been cleaned and quality inspected in Step 3, and determine the overlap between the inferred damage targets and GT; select the inferred damage targets that do not overlap with GT as the data to be cleaned, and have a second person clean the data to be cleaned according to the method in Step 3;

[0021] Step 6: Perform frequency statistics on the ignored types of the image samples after classification and cleaning in Step 5, sort them from high to low, and select the top n% of image samples to send to the second person for secondary correction.

[0022] Step 7: Re-train the cleaned, classified, and corrected samples from Step 6 to obtain iterative model B; repeat the above steps N times to obtain the final iterative model.

[0023] As a further improvement of the present invention, in step 1, the training samples of the damage instance segmentation and detection model are vehicle appearance damage instance segmentation and detection samples with a small number of labeled samples, and damage polygon targets with a probability higher than 0.9 are used as pre-labeled damage target objects.

[0024] As a further improvement of the present invention, the first person is a general labeler and the second person is a professional.

[0025] As a further improvement of the present invention, in step 6, the first 20% of the image samples are selected and sent to a second person for secondary correction.

[0026] As a further improvement of the present invention, in step 7, the iteration is repeated 3 to 4 times to obtain the final iterative model.

[0027] The present invention also discloses an instance segmentation and annotation device for images of damage to vehicle exterior components, comprising:

[0028] The segmentation module is used to infer the vehicle exterior component images to be labeled based on the initially trained damage instance segmentation and detection model, and to use damage polygon targets with a probability higher than a threshold as pre-labeled damage target objects.

[0029] The annotation module is used by a first person to annotate the pre-annotated damage target object;

[0030] The cleaning module is used by a second person to perform quality inspection on the annotation results of the first person using a categorized cleaning method. The quality inspection process includes: storing the correct annotation (GT) of the vehicle exterior component image to be annotated; the second person then performs categorized cleaning on the GT; if an incorrect damage target type is encountered, it is adjusted to the correct damage target type; if a non-damaged target is encountered, it is adjusted to the background type; if a target's image quality cannot accurately determine its true condition, it is adjusted to the ignore type; if the damage target's boundary is incorrect, it is adjusted to the ignore type.

[0031] The iterative module is used to send the cleaned and quality-checked training samples GT to the model for training to obtain iterative model A; and to send the cleaned, classified and corrected samples back to the model for training to obtain iterative model B; repeat the above steps N times to obtain the final iterative model.

[0032] The inference and filtering module is used to infer the cleaned and quality-inspected training samples GT based on the iterative model A, and to judge the overlap between the inferred damage targets and GT; the inferred damage targets that do not overlap with GT are filtered out as data to be cleaned, and the second person cleans the data to be cleaned according to the cleaning method of the cleaning module.

[0033] The secondary correction module is used to perform frequency statistics on the ignored types of image samples after classification and cleaning by the inference and screening module, and select the top n% of image samples from high to low to send them to a second person for secondary correction.

[0034] As a further improvement of the present invention, in the segmentation module, the training samples of the damage instance segmentation and detection model are vehicle appearance damage instance segmentation and detection samples with a small number of labeled samples, and damage polygon targets with a probability higher than 0.9 are used as pre-labeled damage target objects.

[0035] As a further improvement of the present invention, the first person is a general labeler and the second person is a professional.

[0036] As a further improvement of the present invention, in the secondary correction module, the top 20% of the image samples are selected and sent to a second person for secondary correction.

[0037] As a further improvement of the present invention, the iteration module is repeated 3 to 4 times to obtain the final iterative model.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] 1. Reduce annotation costs, enabling ordinary annotators to undertake most of the annotation tasks for segmentation and detection of vehicle exterior component damage instances after a certain period of professional training, reducing the image processing scope and methods of professionals and improving the utilization rate of professionals; at the same time, ordinary annotators can also process damage images after a certain period of training with the help of iterative models, resulting in lower annotation costs.

[0040] 2. Compared to the conventional instance segmentation detection and single-image browsing review quality inspection process, the professionals of this invention use classification and cleaning to achieve data quality inspection, which not only improves the speed of data quality inspection, but also makes the noise of vehicle exterior component damage instance segmentation training data smaller and the model training effect more stable as the iteration process progresses. Attached Figure Description

[0041] Figure 1 This is a flowchart of an instance segmentation and annotation method for vehicle exterior component damage images disclosed in one embodiment of the present invention;

[0042] Figure 2 This is a frame diagram of an example segmentation and annotation device for vehicle exterior component damage images disclosed in one embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] The present invention will now be described in further detail with reference to the accompanying drawings:

[0045] like Figure 1 As shown, the present invention provides an instance segmentation and annotation method for images of damage to vehicle exterior components, comprising:

[0046] Step 1: Based on the initially trained damage instance segmentation and detection model, infer the vehicle exterior component images to be labeled, and use damage polygon targets with a probability higher than a threshold as pre-labeled damage target objects; whereby,

[0047] The damage instance segmentation and detection model was trained by professionals based on a small number of labeled vehicle exterior damage instance segmentation and detection samples. The model preferentially selects damage polygon targets with a probability higher than 0.9 as pre-labeled damage target objects.

[0048] Step 2: The first person marks the pre-marked damaged target object; among which,

[0049] The first person to label the target type may be a regular labeler. During this labeling process, there is a certain percentage of target type labeling errors, omissions, and mislabeling; these need to be improved through subsequent methods.

[0050] Step 3: The second person performs quality inspection on the labeling results from Step 2 using a categorized cleaning method; among which,

[0051] The second person can be a professional, that is, someone whose level of expertise is higher than that of a regular labeler;

[0052] The specific process of the above quality inspection and review includes: storing the correct ground truth (GT) images of the vehicle exterior parts to be labeled; classifying and cleaning the GT images by professionals; if an incorrect damage target type is encountered, it is adjusted to the correct damage target type; if a non-damaged target is encountered, it is adjusted to the background type; if a target with low image quality is encountered and the actual situation cannot be accurately determined, it is adjusted to the ignore type; if the boundary of the damaged target is incorrect, it is adjusted to the ignore type.

[0053] The above-mentioned classification and cleaning process is obviously more efficient than browsing a single image, and business personnel can process images faster.

[0054] Step 4: Send the cleaned and quality-checked training samples GT from Step 3 to the model for training, obtaining iterative model A; where,

[0055] The iterative model A obtained in this step may be inaccurate due to a certain proportion of missing labels in the non-GT training samples after the quality check in step 3; therefore, further improvement is needed through subsequent methods.

[0056] Step 5: Based on iterative model A, infer the training samples GT that have been cleaned and quality inspected in Step 3, and determine the overlap between the inferred damage targets and GT; select the inferred damage targets that do not overlap with GT as the data to be cleaned, and have professionals clean the data to be cleaned according to the classification and cleaning method in Step 3.

[0057] Step 6: Perform frequency statistics on the ignored types of the image samples after classification and cleaning in Step 5, sort them from high to low, and select the top n% of image samples to be sent to professionals for secondary correction; among them,

[0058] The top 20% of image samples are selected and sent to professionals for secondary correction.

[0059] Secondary correction of image samples usually involves samples with a high degree of difficulty, and ordinary annotators have a high error rate. These images undergo a detailed correction process by professionals to improve the quality of such samples.

[0060] Step 7: Re-train the cleaned, classified, and corrected samples from Step 6 to obtain iterative model B; repeat the above steps N times to obtain the final iterative model; where,

[0061] Typically, after 3-4 iterations of the model, the ground truth (GT) annotation of the training samples can be correctly achieved.

[0062] like Figure 2 As shown, the present invention provides an instance segmentation and annotation device for images of damage to vehicle exterior components, comprising:

[0063] The segmentation module is used to implement step 1 above;

[0064] The annotation module is used to implement step 2 above;

[0065] The cleaning module is used to implement step 3 above;

[0066] The iteration module is used to implement steps 4 and 7 above;

[0067] The reasoning and filtering module is used to implement step 5 above;

[0068] The secondary correction module is used to implement step 6 above.

[0069] The advantages of this invention are:

[0070] 1. Reduce annotation costs, enabling ordinary annotators to undertake most of the annotation tasks for segmentation and detection of vehicle exterior component damage instances after a certain period of professional training, reducing the image processing scope and methods of professionals and improving the utilization rate of professionals; at the same time, ordinary annotators can also process damage images after a certain period of training with the help of iterative models, resulting in lower annotation costs.

[0071] 2. Compared to the conventional instance segmentation detection and single-image browsing review quality inspection process, the professionals of this invention use classification and cleaning to achieve data quality inspection, which not only improves the speed of data quality inspection, but also makes the noise of vehicle exterior component damage instance segmentation training data smaller and the model training effect more stable as the iteration process progresses.

[0072] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of instance segmentation annotation of a vehicle exterior component damage image, characterized by, The method comprises the following steps: Step 1, inferring a vehicle appearance part picture to be labeled based on an initially trained damage instance segmentation detection model, and taking a damage polygon target with a probability higher than a threshold as a pre-labeled damage target object; Step 2, a first person labels the pre-labeled damage target object; Step 3, a second person performs quality inspection on the labeling result of step 2 in a classified cleaning manner; wherein the quality inspection process comprises: performing cutout storage on a correct label (GT) of the vehicle appearance part picture to be labeled, and the second person classifies and cleans the cutout GT; if an incorrect damage target type is encountered, it is adjusted to a correct damage target type; if a non-damage target is encountered, it is adjusted to a background type; if an imaging quality cannot accurately judge the real situation of the target, it is adjusted to an ignored type; if the damage target boundary is incorrect, it is adjusted to an ignored type; Step 4, sending the training sample GT after quality inspection in step 3 to model training to obtain an iterative model A; Step 5, inferring the training sample GT after quality inspection in step 3 based on the iterative model A, and performing overlap judgment on the inferred damage target and the GT; screening out the inferred damage target without overlap with the GT as cleaning data, and cleaning the cleaning data by the second person according to the method of step 3; Step 6, performing frequency statistics on the ignored type of the image sample after classification and cleaning in step 5, and selecting the top n% of picture samples in descending order to send to the second person for secondary correction; Step 7, sending the sample after classification and cleaning and correction in step 6 to model training again to obtain an iterative model B; repeating the above steps for N times to obtain a final iterative model.

2. The instance segmentation annotation method of claim 1, wherein, In the step 1, the damage instance segmentation detection model training sample is a small sample of vehicle appearance damage instance segmentation detection sample, and a damage polygon target with a probability higher than 0.9 is taken as a pre-labeled damage target object.

3. The instance segmentation annotation method of claim 1, wherein, The first person is an ordinary labeling person, and the second person is a professional.

4. The instance segmentation annotation method of claim 1, wherein, In the step 6, the top 20% of picture samples are sent to the second person for secondary correction.

5. The instance segmentation annotation method of claim 1, wherein, In the step 7, the final iterative model is obtained by repeating iteration for 3-4 times.

6. An apparatus for instance segmentation annotation of a vehicle exterior component damage image, characterized by The method comprises the following steps: a segmentation module, configured to infer a vehicle appearance part picture to be labeled based on an initially trained damage instance segmentation detection model, and take a damage polygon target with a probability higher than a threshold as a pre-labeled damage target object; a labeling module, configured to have a first person label the pre-labeled damage target object; The cleaning module is used for the first person to perform quality inspection on the labeling result of the first person in a classified cleaning manner; wherein, the quality inspection process comprises: performing cutout storage on correct labeling (GT) of a vehicle appearance component picture to be labeled, and performing classified cleaning on the cutout GT by the second person; if an incorrect damage target type is encountered, it is adjusted to a correct damage target type; if a non-damage target is encountered, it is adjusted to a background type; if an imaging quality cannot accurately judge a real situation of the target, it is adjusted to an ignored type; if a damage target boundary is incorrect, it is adjusted to an ignored type; The iteration module is used for sending the training sample GT after cleaning and quality inspection to model training to obtain an iteration model A; The reasoning and screening module is used for performing reasoning on the training sample GT after cleaning and quality inspection based on the iteration model A, and performing overlap judgment on the reasoned damage target and the GT; the reasoned damage target without overlap with the GT is screened as to-be-cleaned data, and the to-be-cleaned data is cleaned by the second person according to the cleaning method of the cleaning module; The secondary correction module is used for performing frequency statistics on the image sample after classified cleaning of the reasoning and screening module, and selecting the top n% of picture samples to send to the second person for secondary correction after sorting from high to low; The iteration module is used for sending the sample after classified cleaning and correction of the secondary correction module to model training again to obtain an iteration model B; the above steps are repeated for N times to obtain a final iteration model.

7. The instance segmentation annotation apparatus according to claim 6, wherein In the segmentation module, the damage instance segmentation detection model training sample is a vehicle appearance damage instance segmentation detection sample with a small amount of labeled samples, and a damage polygon target with a probability higher than 0.9 is taken as a pre-labeled damage target object.

8. The instance segmentation annotation apparatus according to claim 6, wherein The first person is an ordinary labeling person, and the second person is a professional.

9. The instance segmentation annotation apparatus according to claim 6, wherein In the secondary correction module, the top 20% of picture samples are selected to send to the second person for secondary correction.

10. The instance segmentation annotation apparatus according to claim 6, wherein In the iteration module, the iteration is repeated for 3-4 times to obtain a final iteration model. In the segmentation module, the damage instance segmentation detection model training sample is a vehicle appearance damage instance segmentation detection sample with a small amount of labeled samples, and a damage polygon target with a probability higher than 0.9 is taken as a pre-labeled damage target object. The first person is an ordinary labeling person, and the second person is a professional. In the secondary correction module, the top 20% of picture samples are selected to send to the second person for secondary correction. In the iteration module, the iteration is repeated for 3-4 times to obtain a final iteration model.

Images