Vehicle body exterior part image segmentation method based on deep learning and related equipment
Through the deep learning-based image segmentation method of body appearance parts, the random weight average training strategy and mixed task cascade model are used to solve the problem of poor robustness of body appearance parts, improve the segmentation accuracy and robustness, and reduce the fixed Loss cost.
Patent Information
- Application Number
- CN202411377587.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2025-01-07
AI Technical Summary
The existing car body appearance parts image segmentation method has the problem of poor segmentation robustness, especially when there are many categories and scarce data, it is difficult to accurately segment the car body appearance parts, resulting in frequent fraud incidents and high cost during the loss determination process.
Using the deep learning-based image segmentation method, the preset hybrid task cascade model is trained by obtaining multiple body images in the training dataset and their corresponding annotated images, using a random weight average training strategy, combined with CBSwin -Base backbone network, feature pyramid network and Mask IoU header, optimize the training strategy of the model to improve segmentation accuracy and robustness.
It improves the segmentation accuracy and robustness of the image segmentation model of the body exterior parts, and can more accurately segment the appearance parts with classification, reduces the cost of manual loss determination and reduces the occurrence of fraud incidents.
Smart Images
Figure CN119273921A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence and image processing technology, and in particular to a vehicle body appearance parts image segmentation method and related equipment based on deep learning. Background Art
[0002] Vehicle damage assessment is the main business of insurance companies. Professional assessors determine vehicle damage through on-site photos, which is the basis for claims of auto insurance companies and is the most important operation link in auto insurance claims. However, the damage assessment link has always been completed manually, and the scale of assessors is huge, the cost of manual damage assessment is high, and fraud incidents caused by this are frequent; in addition, it is difficult to form a unified standard for manual damage assessment, which easily leads to a gray area of damage assessment fraud. Therefore, the realization of intelligent damage assessment through artificial intelligence has important economic value and risk needs.
[0003] In the application scenario of vehicle damage assessment, a segmentation model is needed to segment different parts of the car, covering 140 vehicle exterior parts, which greatly enriches the business application scenarios of intelligent damage assessment. For the segmentation model, the more categories there are, the lower the model accuracy will be. As the number of categories increases, the problems of partial category occlusion and data scarcity become more prominent, which increases the difficulty of body exterior parts segmentation. It can be seen that the existing body exterior parts image segmentation method has the problem of poor segmentation robustness. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose a vehicle body appearance parts image segmentation method and related equipment based on deep learning to solve the problem of poor robustness of vehicle body appearance parts segmentation in the prior art.
[0005] In order to solve the above technical problems, the embodiment of the present application provides a vehicle body appearance parts image segmentation method based on deep learning, which adopts the following technical solutions:
[0006] Acquire a training data set, wherein the training data set includes a plurality of vehicle body images and annotated images corresponding to each vehicle body image, wherein the annotated images are images annotated with masks according to vehicle body appearance part category segmentation results of each vehicle body image;
[0007] According to the multiple vehicle body images in the training data set and the annotated images corresponding to each vehicle body image, a preset mixed task cascade model is trained using a random weighted average training strategy to obtain a vehicle body appearance part image segmentation model;
[0008] The vehicle body image to be segmented is input into the vehicle body appearance parts image segmentation model to perform vehicle body appearance parts segmentation, and obtain a vehicle body appearance parts segmentation image with classification.
[0009] Furthermore, the step of training a preset mixed task cascade model using a random weighted average training strategy based on the multiple vehicle body images in the training data set and the annotated image corresponding to each vehicle body image to obtain a vehicle body appearance part image segmentation model specifically includes:
[0010] Inputting the plurality of vehicle body images in the training data set and the annotated image corresponding to each vehicle body image into the preset mixed task cascade model for training, and after training to a predetermined training amount, continuing to train the preset mixed task cascade model to a predetermined additional training period, and saving a checkpoint for each training period;
[0011] After the predetermined additional training periods are completed, the saved predetermined checkpoints are averaged to obtain the final weights, and the trained preset mixed task cascade model is submitted to obtain a vehicle body appearance part image segmentation model.
[0012] Further, the preset hybrid task cascade model includes: a CBSwin-Base backbone network, a feature pyramid network and a Mask IoU head, and the step of inputting the training data set into the preset hybrid task cascade model for training includes:
[0013] Extracting the underlying features of each vehicle body image in the training data set through the CBSwin-Base backbone network to obtain an underlying feature image;
[0014] Performing bottom-level feature fusion on the bottom-level feature image through the feature pyramid network to obtain a high-dimensional feature image;
[0015] The high-dimensional feature image is classified and segmented based on the mask features of the annotated image of each vehicle body image through the Mask I oU header to obtain a classified vehicle body appearance part segmentation image corresponding to each vehicle body image.
[0016] Furthermore, after the step of acquiring the training data set, and before the step of training the preset mixed task cascade model using a random weighted average training strategy based on the multiple vehicle body images in the training data set and the annotated images corresponding to each vehicle body image to obtain the vehicle body appearance part image segmentation model, the method further includes:
[0017] Perform data enhancement on the training data set to obtain an enhanced training data set.
[0018] Furthermore, the step of performing data enhancement on the training data set to obtain an enhanced training data set includes:
[0019] Extracting body appearance part category images and their corresponding mask annotations that meet preset conditions from the training data set, and saving the body appearance part category images and their corresponding mask annotations that meet the preset conditions in a preset location;
[0020] Randomly select any vehicle body appearance part category image from the vehicle body appearance part category images that meet the preset conditions as a candidate object;
[0021] At the edge position of the image in the preset natural scene data set, a pasting area is defined for each candidate object, and the corresponding candidate object is copied and pasted into the corresponding pasting area;
[0022] The preset natural scene data set with the candidate objects copied and pasted is determined as the training data set enhanced by the preset mixed task cascade model.
[0023] Furthermore, the step of extracting the body appearance part category images and their corresponding mask annotations that meet the preset conditions from the training data set includes:
[0024] Determining whether the number of vehicle body appearance part categories marked in the training data set is less than a preset number;
[0025] If the number of body appearance part categories annotated in the training data set is less than a preset number, the body appearance part categories that meet the preset number are cropped out from the corresponding body image to obtain the corresponding body appearance part category image, and the mask annotations corresponding to the body appearance part category image are extracted at the same time.
[0026] In order to solve the above technical problems, the embodiment of the present application further provides a vehicle body appearance part image segmentation device based on deep learning, which adopts the following technical solution:
[0027] An acquisition module, used to acquire a training data set, wherein the training data set includes a plurality of vehicle body images and an annotated image corresponding to each vehicle body image, wherein the annotated image is an image annotated with a mask according to a vehicle body appearance part category segmentation result of each vehicle body image;
[0028] A training module, used to train a preset mixed task cascade model using a random weighted average training strategy according to a plurality of vehicle body images in the training data set and an annotated image corresponding to each vehicle body image, so as to obtain a vehicle body appearance part image segmentation model;
[0029] The segmentation module is used to input the vehicle body image to be segmented into the vehicle body appearance part image segmentation model to perform vehicle body appearance part segmentation and obtain a vehicle body appearance part segmentation image with classification.
[0030] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0031] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the vehicle body appearance parts image segmentation method based on deep learning described in the above embodiment are implemented.
[0032] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0033] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the vehicle body appearance parts image segmentation method based on deep learning described in the above embodiment are implemented.
[0034] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0035] By obtaining a training data set, the training data set includes multiple vehicle body images and annotated images corresponding to each vehicle body image, wherein the annotated images are images annotated with masks according to the vehicle body appearance part category segmentation results of each vehicle body image; according to the multiple vehicle body images in the training data set and the annotated images corresponding to each vehicle body image, a preset hybrid task cascade model is trained using a random weight average training strategy to obtain a vehicle body appearance part image segmentation model; the vehicle body image to be segmented is input into the vehicle body appearance part image segmentation model to perform vehicle body appearance part segmentation, and a vehicle body appearance part segmentation image with classification is obtained. In this way, a hybrid task cascade model with better performance is used as the main model architecture, and a random weight average training strategy is used to optimize the training strategy of the hybrid task cascade model, thereby obtaining a vehicle body appearance part image segmentation model that can directly segment a vehicle body appearance part segmentation image with classification, thereby improving the segmentation accuracy of the vehicle body appearance part image segmentation model and improving the robustness of the vehicle body appearance part image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0037] Figure 1 A flowchart of an embodiment of a vehicle body appearance part image segmentation method based on deep learning according to the present application;
[0038] Figure 2 yes Figure 1 A flowchart of a specific implementation of step S102;
[0039] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S1021;
[0040] Figure 4 is a flowchart of another embodiment of a vehicle body appearance part image segmentation method based on deep learning of the present application;
[0041] Figure 5 yes Figure 4 A flowchart of a specific implementation of step S104;
[0042] Figure 6 It is a structural schematic diagram of an embodiment of a vehicle body appearance part image segmentation device based on deep learning according to the present application;
[0043] Figure 7 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0045] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0046] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0047] refer to Figure 1 , shows a flow chart of an embodiment of a vehicle body appearance part image segmentation method based on deep learning according to the present application. The vehicle body appearance part image segmentation method based on deep learning comprises the following steps:
[0048] Step S101, obtaining a training data set.
[0049] In this embodiment, the training data set includes a plurality of vehicle body images and annotated images corresponding to each vehicle body image, and the annotated images are images annotated with masks according to the vehicle body appearance part category segmentation results of each vehicle body image. The vehicle body images in the training data set can be acquired by a road camera or manually photographed. The training data set can also be called a test set, and the training samples in the training data set are a plurality of vehicle body images, each of which includes one or more vehicle body appearance parts. The vehicle body image can be an overall vehicle body image of the entire vehicle or a partial vehicle body image. The number of annotated images is the same as the number of vehicle body images, and they correspond one to one. There can be multiple categories of body exterior parts. For example, the body exterior parts categories may include doors (left rear door, right rear door, driver's door, co-pilot door, trunk door, etc.), mirrors (left mirror, right mirror), front of the car, bumper, wheels (left rear wheel, left front wheel, right rear wheel, right front wheel, etc.), roof, bumper, windshield, windows (left rear window, driver's window, right rear window, co-pilot window, etc.), lights (may include left rear light, left front light, right rear light, right front light, etc.), and other vehicle parts categories.
[0050] Specifically, multiple vehicle body images are collected, and the labeled images corresponding to each vehicle body image are obtained by manually performing mask annotation according to the vehicle body appearance part category segmentation results of each vehicle body image. The collected multiple vehicle body images and the corresponding labeled images form a training data set.
[0051] Step S102 , according to the multiple vehicle body images in the training data set and the annotated image corresponding to each vehicle body image, a random weighted average training strategy is adopted to train the preset mixed task cascade model to obtain a vehicle body appearance part image segmentation model.
[0052] Among them, the body appearance parts image segmentation model (Hyb rid Task Cascade, HTC) is the main model architecture.
[0053] In this embodiment, if Figure 2 As shown, step S102 uses a random weighted average training strategy to train the preset mixed task cascade model according to multiple vehicle body images in the training data set and the annotated image corresponding to each vehicle body image, and obtains a vehicle body appearance image segmentation model including:
[0054] Step S1021, input multiple vehicle body images in the training data set and the annotated image corresponding to each vehicle body image into the preset hybrid task cascade model for training. After training to a predetermined training amount, continue training the preset hybrid task cascade model to a predetermined additional training period, and save a checkpoint for each training period.
[0055] Among them, the predetermined training amount can be a pre-set main training amount; the predetermined additional training periods can be set according to the predetermined cycle; a checkpoint needs to be taken after each training period; for example, the predetermined training amount is set to 300, the predetermined additional training periods can be set to 12, and the predetermined cycle is 10. When the training amount exceeds the predetermined training amount (300), a training period is obtained after each training of 10 data, and a checkpoint is saved accordingly. For example, the first checkpoint is saved after training to 310 data, and the training is repeated for 10 minutes. The second checkpoint is saved when the data reaches 320, the third checkpoint is saved when the data reaches 330, the fourth checkpoint is saved when the data reaches 340, the fifth checkpoint is saved when the data reaches 350, the sixth checkpoint is saved when the data reaches 360, the seventh checkpoint is saved when the data reaches 370, the eighth checkpoint is saved when the data reaches 380, the ninth checkpoint is saved when the data reaches 390, the tenth checkpoint is saved when the data reaches 400, the eleventh checkpoint is saved when the data reaches 410, and the twelfth checkpoint is saved when the data reaches 420. In this way, 12 checkpoints corresponding to 12 training periods are obtained.
[0056] Step S1022, after the predetermined additional training period is completed, the saved predetermined checkpoints are averaged to obtain the final weight, and the trained preset mixed task cascade model is submitted to obtain the body appearance part image segmentation model.
[0057] In this embodiment, after obtaining the checkpoints of 12 training periods, the average value of these 12 checkpoints is taken, and this average value is used as the weight of the entire preset hybrid task cascade model, indicating that the preset hybrid task cascade model has been trained. At the same time, the trained preset hybrid task cascade model is submitted to obtain the final body appearance parts image segmentation model.
[0058] In this embodiment, adopting a random weight averaging training strategy to train the preset mixed task cascade model can improve the generalization ability of the preset mixed task cascade model, thereby improving the generalization ability of the vehicle body appearance part image segmentation model.
[0059] Furthermore, the preset hybrid task cascade model includes: CBSwin-Base backbone network, feature pyramid network and Mask IoU head. Among them, the feature pyramid network can be called CB-FPN feature pyramid network. Figure 3 As shown, step S1021 inputs a plurality of vehicle body images in the training data set and the annotated image corresponding to each vehicle body image into a preset hybrid task cascade model for training, including:
[0060] Step S10211, extracting the underlying features of each vehicle body image in the training data set through the CBSwin-Base backbone network to obtain an underlying feature image.
[0061] Among them, the underlying features can be representative features of various parts of the car body, such as door handle features, window features, etc., and of course they can also be more subtle features of the car body parts.
[0062] Step S10212, performing bottom-level feature fusion on the bottom-level feature image through a feature pyramid network to obtain a high-dimensional feature image.
[0063] In this embodiment, after obtaining the underlying features, the feature pyramid network can fuse these underlying features to obtain higher-dimensional features, and then obtain the corresponding high-dimensional feature image. For example, the underlying features include handlebar features and window features. By fusing the handlebar features and window features, it can be inferred that the corresponding body part is the door. Of course, when the underlying features include headlight features, license plate features, and bumper features, it can be inferred that it is the front of the car.
[0064] Step S10213, classifying and segmenting the high-dimensional feature image based on the mask features of the annotated image of each vehicle body image through the MaskIoU header, and obtaining a classified vehicle body appearance part segmentation image corresponding to each vehicle body image.
[0065] In this embodiment, the Mask I o U header includes a RO IPoo ling layer, wherein the size of the mask feature extracted by the RO IPoo ling layer is 20×20. Specifically, in the case of occluded scenes, the quality of the segmentation mask is critical, so enhanced segmentation mask prediction is selected. To achieve this goal, the size of the mask feature extracted by the RO IPoo ling layer is expanded from 14×14 to 20×20. This modification not only improves the mask quality and enhances the segmentation mask prediction, but also helps the model to reconstruct the details of the object in the case of severe occlusion.
[0066] Specifically, the body appearance part category to which each high-dimensional feature image belongs is determined through the mask features of the annotated image of each body image in the MaskIoU header, and the segmentation mask of each high-dimensional feature image is determined at the same time, so as to classify and segment each high-dimensional feature image, and then obtain the classified body appearance part segmentation image corresponding to each body image. Each body image can have one or more classified body appearance part segmentation images. When the body image has multiple body appearance parts, there can be multiple classified body appearance part segmentation images. When the body image has only one body appearance part, there is only one classified body appearance part segmentation image. It should be noted that when a large body appearance part contains another small body appearance part, the body appearance part segmentation images of the two overlap in the pixel dimension, but can be divided into different categories.
[0067] In this embodiment, by designing the model architecture, a mixed task cascade is selected as the main model architecture. Unlike the traditional use of the ResNet backbone network, it is decided to use the CBSwin-Base backbone network with stronger feature extraction capabilities and lighter weight, and combined with a feature pyramid network called CB-FPN, which can balance the model size and available training resources, so as to improve the model performance under limited training resources. At the same time, the MaskIoU head is used to help the model learn to generate high-quality segmentation masks, which is different from traditional methods, which usually share confidence scores with classification heads. In the case of occluded scenes, the quality of the segmentation mask is crucial, so choosing to enhance the segmentation mask prediction can obtain a more accurate segmentation image of the body appearance parts with classification.
[0068] Step S103 , inputting the vehicle body image to be segmented into a vehicle body appearance parts image segmentation model to perform vehicle body appearance parts segmentation, and obtaining a vehicle body appearance parts segmentation image with classification.
[0069] In this embodiment, the vehicle body image to be segmented may be an image for which the user needs to segment the vehicle body exterior parts. The classified vehicle body exterior parts segmented image may be a vehicle component image with a category name obtained after the vehicle body image is segmented by the vehicle body exterior parts image segmentation model. For example, after segmentation by the vehicle body exterior parts image segmentation model, a left rear door image may be obtained. Then, left rear door is the category name of the door, and the image is a left rear door image.
[0070] Specifically, after the body appearance parts image segmentation model is trained, the image that needs to be segmented can be input into the body appearance parts image segmentation model, and the body appearance parts image segmentation model can output the body appearance parts segmentation image with classification. Then, the categories of all body appearance parts in the body image and the corresponding segmented images can be obtained. For example, the current body image contains two parts, the front passenger door and the right reflector. After segmentation by the body appearance parts image segmentation model, the front passenger door image and its corresponding category name (front passenger door), the right reflector image and its corresponding category name (right reflector) can be obtained.
[0071] The present application obtains a training data set, which includes multiple vehicle body images and annotated images corresponding to each vehicle body image, wherein the annotated images are images annotated with masks according to the vehicle body appearance part category segmentation results of each vehicle body image; according to the multiple vehicle body images in the training data set and the annotated images corresponding to each vehicle body image, a preset hybrid task cascade model is trained using a random weight average training strategy to obtain a vehicle body appearance part image segmentation model; the vehicle body image to be segmented is input into the vehicle body appearance part image segmentation model to perform vehicle body appearance part segmentation, thereby obtaining a vehicle body appearance part segmentation image with classification. In this way, a hybrid task cascade model with better performance is used as the main model architecture, and a random weight average training strategy is used to optimize the training strategy of the hybrid task cascade model, thereby obtaining a vehicle body appearance part image segmentation model that can directly segment a vehicle body appearance part segmentation image with classification, thereby improving the segmentation accuracy of the vehicle body appearance part image segmentation model and improving the robustness of the vehicle body appearance part image segmentation.
[0072] In some optional implementations of this embodiment, the vehicle appearance parts can be segmented based on the deep learning-based body appearance parts image segmentation method provided in this embodiment, which can reduce the cost of manual damage assessment. Intelligent damage assessment through artificial intelligence has important economic value and risk needs.
[0073] In some optional implementations of this embodiment, such as Figure 4 As shown, after obtaining the training data set in step S101, step S102 trains the preset mixed task cascade model using a random weight average training strategy according to the multiple vehicle body images in the training data set and the annotated images corresponding to each vehicle body image, and before obtaining the vehicle body appearance part image segmentation model, the electronic device may further perform the following steps:
[0074] Step S104, performing data enhancement on the training data set to obtain an enhanced training data set.
[0075] In this embodiment, data augmentation is critical to achieving consistent model performance. Therefore, in addition to traditional geometric and photometric transformations (such as translation, rotation, or color perturbations), data augmentation is considered for very few categories of appearance parts. This helps train the model to better handle the situation where the image appearance parts are occluded.
[0076] Specifically, Figure 5 As shown, step S104 performs data enhancement on the training data set, and the enhanced training data set includes:
[0077] Step S1041 , extracting the body appearance part category images and their corresponding mask annotations that meet the preset conditions from the training data set, and saving the body appearance part category images and their corresponding mask annotations that meet the preset conditions in a preset location.
[0078] In this embodiment, the preset condition is a preset condition for determining whether there are body appearance part category images that meet the requirements in the training data set. The preset condition may be that the number of annotated body appearance part categories meets a certain number. The body appearance part category image is obtained by cropping the body appearance part category images that meet the preset conditions in the body image. The mask annotation is obtained by manually annotating each body image. The preset location may be a preset storage space for storing body appearance part category images that meet the preset conditions and their corresponding mask annotations, such as being saved locally or in the cloud.
[0079] Specifically, step S1041 extracts the body appearance part category images and their corresponding mask annotations that meet the preset conditions from the training data set, including the following steps:
[0080] Determine whether the number of vehicle body appearance part categories annotated in the training data set is less than a preset number.
[0081] If the number of body appearance part categories annotated in the training data set is less than a preset number, the body appearance part categories that meet the preset number are cropped out from the corresponding body image to obtain the corresponding body appearance part category image, and the mask annotations corresponding to the body appearance part category image are extracted at the same time.
[0082] In this embodiment, the number of annotated body appearance part categories may be the actual number of annotated body appearance part categories in the training data set, and the preset number may be a pre-set standard for determining whether the annotated body appearance part categories in the training data set meet the preset conditions. For example, when there are 10,000 annotated data in the training data set, of which 2,000 are for the roof, 3,000 are for the headlights, 2,000 are for the bumpers, 800 are for the doors, 500 are for the windshields, and 1,200 are for the mirrors, if the preset number is set to 1,000, then the 500 for the doors and the 400 for the windshields are both less than the preset number of 1,000, and it can be determined that the number of doors and the number of windshields meet the preset conditions. There may be multiple body appearance part categories that meet the preset number at the same time.
[0083] Specifically, when it is determined that there are less than a preset number of body appearance parts categories, the corresponding position of the body appearance parts category in the body image can be cropped to obtain the image of the body appearance parts category, and the mask annotation corresponding to the body appearance parts category can be extracted at the same time, so as to obtain the body appearance parts category image and its corresponding mask annotation that meet the preset conditions. In this way, the body appearance parts category images with less training data can be quickly obtained in the training data set through the preset number, so as to perform data enhancement on the body appearance parts category images under limited training resources to improve the model segmentation performance.
[0084] Step S1042 : randomly selecting any vehicle body appearance part category image from the vehicle body appearance part category images that meet preset conditions as a candidate object.
[0085] After determining the body appearance part category image that meets the preset conditions, a body appearance part category image may be randomly selected from the body appearance part categories in the preset position as a candidate object to be copied and pasted.
[0086] Step S1043 , defining a pasting area for each candidate object at an edge position of the image in the preset natural scene data set, and copying and pasting the corresponding candidate object to the corresponding pasting area.
[0087] In this embodiment, the above-mentioned preset natural scene data set can be a natural scene data set such as roads and communities prepared in advance. Specifically, to accurately locate and paste these candidate objects, it is necessary to first determine a pasting area at the edge position of the image in the preset natural scene data set, and then randomly copy the candidate objects to the pasting area at the edge position of the image of the natural scene data set such as roads and communities prepared in advance. After copying and pasting, an occlusion module is also applied. That is, once a candidate object is pasted, this module will have a 70% probability of superimposing another candidate object on the upper left quadrant of the initial entity, thereby simulating the realistic occlusion scenario observed in the training set. This method helps the training model to better handle occlusion situations.
[0088] In this embodiment, after copying and pasting, other basic transformation preprocessing can be continued, including random resizing, photometric distortion, and geometric transformation. Then, the enhanced image is cropped and padded to a fixed size to obtain a better training data set.
[0089] Step S1044, determining the preset natural scene data set with the candidate objects copied and pasted as the preset training data set enhanced by the hybrid task cascade model.
[0090] After the copy and paste is completed, the images in the preset natural scene data set all carry the pasted images of each body appearance part category, so that the preset natural scene data set with the candidate objects copied and pasted is used as the training data set after the preset mixed task cascade model is enhanced, so as to facilitate the training of the preset mixed task cascade model to be able to segment the body images in the occluded scenes in reality. At the same time, when the training data of the body appearance part category is very small, the body appearance part category can also be enhanced to improve the model performance under limited training resources, thereby improving the robustness of the body appearance part image segmentation.
[0091] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0092] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0093] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0094] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0095] Further references Figure 6 , as a response to the above Figure 1 The present application provides an embodiment of a vehicle body appearance part image segmentation device based on deep learning, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0096] like Figure 6 As shown, the vehicle body appearance part image segmentation device 600 based on deep learning in this embodiment includes: an acquisition module 601, a training module 602 and a segmentation module 603. Among them:
[0097] An acquisition module 601 is used to acquire a training data set, where the training data set includes a plurality of vehicle body images and an annotated image corresponding to each vehicle body image, where the annotated image is an image annotated with a mask according to a vehicle body appearance part category segmentation result of each vehicle body image;
[0098] The training module 602 is used to train the preset mixed task cascade model using a random weight average training strategy according to the multiple vehicle body images in the training data set and the annotated images corresponding to each vehicle body image, so as to obtain a vehicle body appearance part image segmentation model;
[0099] The segmentation module 603 is used to input the vehicle body image to be segmented into the vehicle body appearance part image segmentation model to perform vehicle body appearance part segmentation and obtain a vehicle body appearance part segmentation image with classification.
[0100] Furthermore, the training module 602 includes:
[0101] A training submodule, for inputting a plurality of vehicle body images in a training data set and an annotated image corresponding to each vehicle body image into a preset mixed task cascade model for training, and after training to a predetermined training amount, continuing to train the preset mixed task cascade model to a predetermined additional training period, and saving a checkpoint for each training period;
[0102] The submission submodule is used to average the saved predetermined checkpoints to obtain the final weight after the training of the predetermined additional training periods is completed, and submit the trained preset mixed task cascade model to obtain the body appearance image segmentation model.
[0103] Furthermore, the preset hybrid task cascade model includes: CBSwin-Base backbone network, feature pyramid network and Mask IoU head, and the training submodules include:
[0104] An extraction unit, used to extract the underlying features of each vehicle body image in the training data set through the CBSwin-Base backbone network to obtain an underlying feature image;
[0105] A fusion unit is used to fuse the underlying features of the underlying feature image through a feature pyramid network to obtain a high-dimensional feature image;
[0106] The segmentation unit is used to classify and segment the high-dimensional feature image based on the mask features of the annotated image of each vehicle body image through the Mask I oU head, so as to obtain a classified vehicle body appearance part segmentation image corresponding to each vehicle body image.
[0107] Furthermore, before the training module 602, the apparatus further includes:
[0108] The data enhancement module is used to enhance the training data set to obtain an enhanced training data set.
[0109] Furthermore, the data enhancement module includes:
[0110] An extraction submodule, used to extract body appearance part category images and their corresponding mask annotations that meet preset conditions from the training data set, and save the body appearance part category images and their corresponding mask annotations that meet the preset conditions in a preset location;
[0111] A random selection submodule, used for randomly selecting any body appearance part category image from body appearance part category images that meet preset conditions as a candidate object;
[0112] A copy and paste submodule is used to define a paste area for each candidate object at an edge position of an image in a preset natural scene dataset, and copy and paste the corresponding candidate object to the corresponding paste area;
[0113] The determination submodule is used to determine the preset natural scene data set with the candidate objects copied and pasted as the training data set enhanced by the preset mixed task cascade model.
[0114] Furthermore, the extraction submodule includes:
[0115] A judgment unit, used to judge whether the number of vehicle body appearance parts categories marked in the training data set is less than a preset number;
[0116] A cropping unit is used to crop the body appearance part categories that meet the preset number from the corresponding body image if the number of body appearance part categories annotated in the training data set is less than a preset number, to obtain the corresponding body appearance part category image, and to extract the mask annotation corresponding to the body appearance part category image.
[0117] In this embodiment, the operations performed by the above modules, sub-modules, units or sub-units correspond one-to-one to the steps of the vehicle body appearance parts image segmentation method based on deep learning in the aforementioned embodiment, and are not repeated here.
[0118] In this embodiment, a training data set is obtained, the training data set includes multiple vehicle body images and annotated images corresponding to each vehicle body image, and the annotated images are images annotated with masks according to the vehicle body appearance part category segmentation results of each vehicle body image; a preset hybrid task cascade model is trained using a random weight average training strategy based on multiple vehicle body images in the training data set and the annotated images corresponding to each vehicle body image, so as to obtain a vehicle body appearance part image segmentation model; the vehicle body image to be segmented is input into the vehicle body appearance part image segmentation model to perform vehicle body appearance part segmentation, so as to obtain a vehicle body appearance part segmentation image with classification. In this way, a hybrid task cascade model with better performance is used as the main model architecture, and a random weight average training strategy is used to optimize the training strategy of the hybrid task cascade model, so as to obtain a vehicle body appearance part image segmentation model that can directly segment a vehicle body appearance part segmentation image with classification, thereby improving the segmentation accuracy of the vehicle body appearance part image segmentation model and improving the robustness of the vehicle body appearance part image segmentation.
[0119] In some optional implementations of this embodiment, the training module 602 includes:
[0120] A training submodule, for inputting a plurality of vehicle body images in a training data set and an annotated image corresponding to each vehicle body image into a preset mixed task cascade model for training, and after training to a predetermined training amount, continuing to train the preset mixed task cascade model to a predetermined additional training period, and saving a checkpoint for each training period;
[0121] The submission submodule is used to average the saved predetermined checkpoints to obtain the final weight after the training of the predetermined additional training periods is completed, and submit the trained preset mixed task cascade model to obtain the body appearance image segmentation model.
[0122] In this embodiment, the operations performed by the above modules, sub-modules, units or sub-units correspond one-to-one to the steps of the vehicle body appearance parts image segmentation method based on deep learning in the aforementioned embodiment, and are not repeated here.
[0123] In some optional implementations of this embodiment, the preset hybrid task cascade model includes: a CBSwin-Base backbone network, a feature pyramid network, and a Mask IoU head, and the training submodule includes:
[0124] An extraction unit, used to extract the underlying features of each vehicle body image in the training data set through the CBSwin-Base backbone network to obtain an underlying feature image;
[0125] A fusion unit is used to fuse the underlying features of the underlying feature image through a feature pyramid network to obtain a high-dimensional feature image;
[0126] The segmentation unit is used to classify and segment the high-dimensional feature image based on the mask features of the annotated image of each vehicle body image through the Mask I oU head, so as to obtain a classified vehicle body appearance part segmentation image corresponding to each vehicle body image.
[0127] In this embodiment, the operations performed by the above modules, sub-modules, units or sub-units correspond one-to-one to the steps of the vehicle body appearance parts image segmentation method based on deep learning in the aforementioned embodiment, and are not repeated here.
[0128] In some optional implementations of this embodiment, the size of the mask feature extracted by the Mask I o U header is 20×20.
[0129] In this embodiment, the operations performed by the above modules, sub-modules, units or sub-units correspond one-to-one to the steps of the vehicle body appearance parts image segmentation method based on deep learning in the aforementioned embodiment, and are not repeated here.
[0130] In some optional implementations of this embodiment, after the acquisition module 601 and before the training module 602, the vehicle body appearance part image segmentation device based on deep learning further includes:
[0131] The data enhancement module is used to enhance the training data set to obtain an enhanced training data set.
[0132] In this embodiment, the operations performed by the above modules, sub-modules, units or sub-units correspond one-to-one to the steps of the vehicle body appearance parts image segmentation method based on deep learning in the aforementioned embodiment, and are not repeated here.
[0133] In some optional implementations of this embodiment, the data enhancement module includes:
[0134] An extraction submodule, used to extract body appearance part category images and their corresponding mask annotations that meet preset conditions from the training data set, and save the body appearance part category images and their corresponding mask annotations that meet the preset conditions in a preset location;
[0135] A random selection submodule, used for randomly selecting any body appearance part category image from body appearance part category images that meet preset conditions as a candidate object;
[0136] A copy and paste submodule is used to define a paste area for each candidate object at an edge position of an image in a preset natural scene dataset, and copy and paste the corresponding candidate object to the corresponding paste area;
[0137] The determination submodule is used to determine the preset natural scene data set with the candidate objects copied and pasted as the training data set enhanced by the preset mixed task cascade model.
[0138] In this embodiment, the operations performed by the above modules, sub-modules, units or sub-units correspond one-to-one to the steps of the vehicle body appearance parts image segmentation method based on deep learning in the aforementioned embodiment, and are not repeated here.
[0139] In some optional implementations of this embodiment, the extraction submodule includes:
[0140] A judgment unit, used to judge whether the number of vehicle body appearance parts categories marked in the training data set is less than a preset number;
[0141] A cropping unit is used to crop the body appearance part categories that meet the preset number from the corresponding body image if the number of body appearance part categories annotated in the training data set is less than a preset number, to obtain the corresponding body appearance part category image, and to extract the mask annotation corresponding to the body appearance part category image.
[0142] In this embodiment, the operations performed by the above modules, sub-modules, units or sub-units correspond one-to-one to the steps of the vehicle body appearance parts image segmentation method based on deep learning in the aforementioned embodiment, and are not repeated here.
[0143] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.
[0144] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 having components 61-63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.
[0145] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.
[0146] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the body appearance part image segmentation method based on deep learning, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.
[0147] In some embodiments, the processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for running a vehicle body appearance part image segmentation method based on deep learning.
[0148] The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.
[0149] In this embodiment, a training data set is obtained, the training data set includes multiple vehicle body images and annotated images corresponding to each vehicle body image, and the annotated images are images annotated with masks according to the vehicle body appearance part category segmentation results of each vehicle body image; a preset hybrid task cascade model is trained using a random weight average training strategy based on multiple vehicle body images in the training data set and the annotated images corresponding to each vehicle body image, so as to obtain a vehicle body appearance part image segmentation model; the vehicle body image to be segmented is input into the vehicle body appearance part image segmentation model to perform vehicle body appearance part segmentation, so as to obtain a vehicle body appearance part segmentation image with classification. In this way, a hybrid task cascade model with better performance is used as the main model architecture, and a random weight average training strategy is used to optimize the training strategy of the hybrid task cascade model, so as to obtain a vehicle body appearance part image segmentation model that can directly segment a vehicle body appearance part segmentation image with classification, thereby improving the segmentation accuracy of the vehicle body appearance part image segmentation model and improving the robustness of the vehicle body appearance part image segmentation.
[0150] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the vehicle body appearance parts image segmentation method based on deep learning as described above.
[0151] In this embodiment, a training data set is obtained, the training data set includes multiple vehicle body images and annotated images corresponding to each vehicle body image, and the annotated images are images annotated with masks according to the vehicle body appearance part category segmentation results of each vehicle body image; a preset hybrid task cascade model is trained using a random weight average training strategy based on multiple vehicle body images in the training data set and the annotated images corresponding to each vehicle body image, so as to obtain a vehicle body appearance part image segmentation model; the vehicle body image to be segmented is input into the vehicle body appearance part image segmentation model to perform vehicle body appearance part segmentation, so as to obtain a vehicle body appearance part segmentation image with classification. In this way, a hybrid task cascade model with better performance is used as the main model architecture, and a random weight average training strategy is used to optimize the training strategy of the hybrid task cascade model, so as to obtain a vehicle body appearance part image segmentation model that can directly segment a vehicle body appearance part segmentation image with classification, thereby improving the segmentation accuracy of the vehicle body appearance part image segmentation model and improving the robustness of the vehicle body appearance part image segmentation.
[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0153] Obviously, the embodiments described above are only a part of the embodiments of the present application, rather than all the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific embodiments, or to replace some of the technical features therein with equivalents. All equivalent structures made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, are similarly within the scope of patent protection of this application. The non-company software tools or components that appear in the embodiments of this application are merely examples and do not represent actual use.
Claims
1. A vehicle body appearance parts image segmentation method based on deep learning, characterized in that: The steps include: Acquire a training data set, wherein the training data set includes a plurality of vehicle body images and annotated images corresponding to each vehicle body image, wherein the annotated images are images annotated with masks according to vehicle body appearance part category segmentation results of each vehicle body image; According to the multiple vehicle body images in the training data set and the annotated images corresponding to each vehicle body image, a preset mixed task cascade model is trained using a random weighted average training strategy to obtain a vehicle body appearance part image segmentation model; The vehicle body image to be segmented is input into the vehicle body appearance parts image segmentation model to perform vehicle body appearance parts segmentation, and obtain a vehicle body appearance parts segmentation image with classification.
2. The vehicle body appearance parts image segmentation method based on deep learning according to claim 1, characterized in that: The step of training a preset mixed task cascade model using a random weighted average training strategy based on the multiple vehicle body images in the training data set and the annotated image corresponding to each vehicle body image to obtain a vehicle body appearance part image segmentation model specifically includes: Inputting the plurality of vehicle body images in the training data set and the annotated image corresponding to each vehicle body image into the preset mixed task cascade model for training, and after training to a predetermined training amount, continuing to train the preset mixed task cascade model to a predetermined additional training period, and saving a checkpoint for each training period; After the predetermined additional training periods are completed, the saved predetermined checkpoints are averaged to obtain the final weights, and the trained preset mixed task cascade model is submitted to obtain a vehicle body appearance part image segmentation model.
3. The vehicle body appearance parts image segmentation method based on deep learning according to claim 2, characterized in that: The preset hybrid task cascade model includes: a CBSwin-Base backbone network, a feature pyramid network and a MaskIoU head, and the step of inputting the training data set into the preset hybrid task cascade model for training includes: Extracting the underlying features of each vehicle body image in the training data set through the CBSwin-Base backbone network to obtain an underlying feature image; Performing bottom-level feature fusion on the bottom-level feature image through the feature pyramid network to obtain a high-dimensional feature image; The high-dimensional feature image is classified and segmented based on the mask features of the annotated image of each vehicle body image through the MaskIoU header to obtain a classified vehicle body appearance part segmentation image corresponding to each vehicle body image.
4. The vehicle body appearance parts image segmentation method based on deep learning according to any one of claims 1 to 3, characterized in that: After the step of acquiring the training data set, and before the step of training the preset mixed task cascade model using a random weighted average training strategy based on the multiple vehicle body images in the training data set and the annotated images corresponding to each vehicle body image to obtain the vehicle body appearance part image segmentation model, the method further includes: Perform data enhancement on the training data set to obtain an enhanced training data set.
5. The vehicle body appearance parts image segmentation method based on deep learning according to claim 4, characterized in that: The step of performing data enhancement on the training data set to obtain an enhanced training data set includes: Extracting the body appearance part category images and their corresponding mask annotations that meet the preset conditions from the training data set, and saving the body appearance part category images and their corresponding mask annotations that meet the preset conditions in a preset location; Randomly select any vehicle body appearance part category image from the vehicle body appearance part category images that meet the preset conditions as a candidate object; At the edge position of the image in the preset natural scene data set, a pasting area is defined for each candidate object, and the corresponding candidate object is copied and pasted into the corresponding pasting area; The preset natural scene data set with the candidate objects copied and pasted is determined as the training data set enhanced by the preset mixed task cascade model.
6. The vehicle body appearance parts image segmentation method based on deep learning according to claim 5, characterized in that: The step of extracting the body appearance part category images and their corresponding mask annotations that meet the preset conditions from the training data set includes: Determining whether the number of vehicle body appearance part categories marked in the training data set is less than a preset number; If the number of body appearance part categories annotated in the training data set is less than a preset number, the body appearance part categories that meet the preset number are cropped out from the corresponding body image to obtain the corresponding body appearance part category image, and the mask annotations corresponding to the body appearance part category image are extracted at the same time.
7. A vehicle body appearance parts image segmentation device based on deep learning, characterized in that: include: An acquisition module, used to acquire a training data set, wherein the training data set includes a plurality of vehicle body images and an annotated image corresponding to each vehicle body image, wherein the annotated image is an image annotated with a mask according to a vehicle body appearance part category segmentation result of each vehicle body image; A training module, used to train a preset mixed task cascade model using a random weighted average training strategy according to a plurality of vehicle body images in the training data set and an annotated image corresponding to each vehicle body image, so as to obtain a vehicle body appearance part image segmentation model; The segmentation module is used to input the vehicle body image to be segmented into the vehicle body appearance part image segmentation model to perform vehicle body appearance part segmentation and obtain a vehicle body appearance part segmentation image with classification.
8. The vehicle body appearance parts image segmentation device based on deep learning according to claim 7, characterized in that: The training module includes: A training submodule, for inputting the plurality of vehicle body images in the training data set and the annotated image corresponding to each vehicle body image into the preset mixed task cascade model for training, and after training to a predetermined training amount, continuing to train the preset mixed task cascade model to a predetermined additional training period, and saving a checkpoint for each training period; The submission submodule is used to average the saved predetermined checkpoints to obtain the final weight after the predetermined additional training period is completed, and submit the trained preset mixed task cascade model to obtain the body appearance image segmentation model.
9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the vehicle body appearance part image segmentation method based on deep learning as described in any one of claims 1 to 6 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the vehicle body appearance parts image segmentation method based on deep learning as described in any one of claims 1 to 6.