Image segmentation method, system, electronic device and storage medium

Abstract

This invention provides an image segmentation method, system, electronic device, and storage medium. The method includes: acquiring autonomous driving image data under different scenarios, performing data augmentation processing on the autonomous driving image data, obtaining a training dataset, and training a DLinkNet network model; extracting feature values ​​of each pixel in the image data to be processed based on the trained DLinkNet network model; performing clustering post-processing on the feature values ​​of each pixel in the image data to be processed, and obtaining the image segmentation result of the image data to be processed. This invention performs clustering post-processing on the segmentation features extracted by the model to obtain more accurate image segmentation results, solving the problem of inaccurate segmentation results caused by insufficient data and limited model learning ability in existing methods. This improves the accuracy of image segmentation in autonomous driving.

Description

Technical Field

[0001] This invention relates to the field of image segmentation, and more specifically, to an image segmentation method, system, electronic device, and storage medium. Background Technology

[0002] Image segmentation is one of the most important technologies in autonomous driving perception. The results of image semantic segmentation can provide information such as the vehicle's drivable area and obstacles ahead. However, traditional or deep learning methods for image semantic segmentation often result in inaccurate results due to limited data and the model's limited learning capabilities. Therefore, improving the accuracy of image segmentation is one of the challenges in autonomous driving. Summary of the Invention

[0003] This invention addresses the technical problems existing in the prior art by providing an image segmentation method, system, electronic device, and storage medium.

[0004] According to a first aspect of the present invention, an image segmentation method is provided, comprising:

[0005] Collect autonomous driving image data in different scenarios, and perform data augmentation processing on the autonomous driving image data to obtain a training dataset;

[0006] The DLinkNet network model was trained based on the aforementioned training dataset;

[0007] The feature value of each pixel in the image data to be processed is extracted based on the trained DLinkNet network model.

[0008] Clustering is performed on the feature values ​​of each pixel in the image data to be processed to obtain the image segmentation result of the image data to be processed. The image segmentation result refers to the segmentation of the target object from the image data to be processed.

[0009] According to a second aspect of the present invention, an image segmentation system is provided, comprising:

[0010] The acquisition module is used to collect autonomous driving image data under different scenarios, and to perform data augmentation processing on the autonomous driving image data to obtain a training dataset;

[0011] The training module is used to train the DLinkNet network model based on the training dataset;

[0012] The extraction module is used to extract the feature value of each pixel in the image data to be processed based on the trained DLinkNet network model.

[0013] The clustering module is used to perform clustering and post-processing on the feature values ​​of each pixel in the image data to be processed, and to obtain the image segmentation result of the image data to be processed. The image segmentation result refers to the segmentation of the target object from the image data to be processed.

[0014] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, the processor being configured to implement the steps of an image segmentation method when executing a computer management program stored in the memory.

[0015] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer management class program stored thereon, which, when executed by a processor, implements the steps of an image segmentation method.

[0016] This invention provides an image segmentation method, system, electronic device, and storage medium. It performs clustering processing on segmentation features extracted by a model to obtain more accurate image segmentation results, solving the problem of inaccurate segmentation results caused by insufficient data and limited model learning capabilities in existing methods. This improves the accuracy of image segmentation in autonomous driving. Attached Figure Description

[0017] Figure 1 A flowchart illustrating an image segmentation method provided by the present invention;

[0018] Figure 2 A schematic diagram of the overall process of the image segmentation method;

[0019] Figure 3 A schematic diagram of the structure of an image segmentation system provided by the present invention;

[0020] Figure 4 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0021] Figure 5 This is a schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined with each other to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0023] Figure 1 The image segmentation method provided by this invention mainly includes the following steps:

[0024] S1. Collect autonomous driving image data under different scenarios, and perform data augmentation processing on the autonomous driving image data to obtain a training dataset.

[0025] As an example, the process of collecting autonomous driving image data under different scenarios and performing data augmentation processing on the autonomous driving image data to obtain a training dataset includes: collecting multiple autonomous driving image data under different scenarios; rotating or flipping all or part of the autonomous driving image data; adding the rotated or flipped autonomous driving image data and the original autonomous driving image data to the training dataset; and annotating the target objects in each autonomous driving image data in the training dataset with polygonal contours to generate a mask image containing target type and location information as a label.

[0026] Understandably, before training the model, it is necessary to obtain a training dataset, collect autonomous driving image data in different scenarios, perform data augmentation (rotation and flipping) on ​​some of the autonomous driving image data, expand the original dataset, and form a training dataset. The training dataset includes the rotated and flipped autonomous driving image data as well as the original autonomous driving image data.

[0027] Each autonomous driving image in the training dataset is labeled. In this invention, closed polygons are used to label the contours of target objects in the autonomous driving image data. After labeling, an XML file containing target category and location information is obtained. The XML file is then converted to PNG format to generate a mask image file as a label, which can be used for training the DLinkNet network model.

[0028] S2, Train the DLinkNet network model based on the training dataset.

[0029] Understandably, the present invention uses the DLinkNet network model as the training model. This network model uses an encoder-decoder structure, dilated convolution, and a pre-trained encoder to complete the image segmentation task. The backbone network is based on the Residual Network (ResNet), the Neck part is based on dilated convolution, and the Head part is based on the deconvolution structure.

[0030] During training, a loss function is constructed for the DLinkNet network model, and the DLinkNet network model is trained based on the loss function and the training dataset.

[0031] Wherein, the loss function is the cross-entropy loss function. When the DLinkNet network model is a binary classification model, the expression for the cross-entropy loss function is:

[0032]

[0033] Among them, y i This represents the label of sample i, with 1 for positive class and 0 for negative class, p i This represents the probability that sample i is predicted to be of the positive class.

[0034] When the DLinkNet network model is a multi-class classification model, the expression for the cross-entropy loss function is:

[0035]

[0036] Where M represents the number of categories output by the DLinkNet network model;

[0037] y ic The sign function is p, which takes the value 1 if the true class of the i-th sample is equal to c, and 0 otherwise. ic Let be the predicted probability that observed sample i belongs to category c.

[0038] Understandably, the function of the DLinkNet network model is to segment target objects in autonomous driving image data, that is, to classify each pixel in the autonomous driving image data, whether each pixel belongs to the target object, the background, or something else.

[0039] Depending on the requirements of the image segmentation task, the DLinkNet network model can be a binary classification model or a multi-class classification model, and the loss function for the binary classification model and the loss function for the multi-class classification model can be defined respectively.

[0040] The DLinkNet network model is trained based on the training dataset and loss function to obtain the trained network model.

[0041] S3 extracts the feature value of each pixel in the image data to be processed based on the trained DLinkNet network model.

[0042] S4, perform clustering processing on the feature values ​​of each pixel in the image data to be processed to obtain the image segmentation result of the image data to be processed, wherein the image segmentation result refers to the segmentation of the target object from the image data to be processed.

[0043] Understandably, the feature values ​​of each pixel in the image data to be processed are extracted based on the trained DLinkNet network model. In order to obtain more accurate image segmentation results, the feature values ​​of each pixel in the image data to be processed extracted by the model are post-processed. The post-processing method adopts a clustering-based post-processing method.

[0044] Among them, the FCM algorithm is a partition-based clustering algorithm. Its idea is to maximize the similarity between objects belonging to the same cluster and minimize the similarity between different clusters. The fuzzy C-means algorithm is an improvement on the ordinary C-means algorithm. The ordinary C-means algorithm is rigid in its data partitioning, while FCM is a flexible fuzzy partitioning.

[0045] The steps for clustering the feature values ​​of each pixel in the image data to be processed include:

[0046] a. Obtain the samples to be clustered, D = [d0, d1, d2, ..., d]. m ], D represents the feature value of each pixel, m represents the number of pixels, and the feature values ​​of each pixel are floating-point numbers;

[0047] b. Initialize the elements in D by uniform distribution to satisfy the constraints, and set the initialized set of elements as O, O = [θ0, θ1, ..., θ]. m ], N is the number of elements after initialization, and the value of each element after initialization is a floating-point number between 0 and 1;

[0048] c. Calculate the cluster center set Z based on O and the iterative formula, Z = (z0, z1, z2, ..., zn). j ),z j represents the cluster center of each category, and j represents the number of categories;

[0049] d. Based on the cluster center set Z, cluster the element set O, and calculate the objective function J based on the clustering results;

[0050] e. Compare whether the objective function meets the preset conditions. If it does, output the clustering result, which is the image segmentation result. If it does not meet the conditions, repeat steps c, d, and e until the objective function value meets the preset conditions.

[0051] The preset condition is that the objective function value is less than a first preset threshold or the change between the current objective function value and the objective function value of the previous iteration is less than a second preset threshold.

[0052] Clustering is performed continuously through steps a to e until the objective function value meets the condition, at which point the image segmentation result is output.

[0053] Among them, clustering of the element set O is performed based on the cluster center set Z, and the objective function J is calculated based on the clustering results, including:

[0054] Define the size of the dataset to be clustered as m, the number of cluster centers as C, and u ij Let be the membership degree of the i-th data point to the j-th centroid and 0. ij <1,

[0055] The objective function can be defined similarly to the distance concept in the K-means algorithm, as follows:

[0056]

[0057] Where α is a hyperparameter, ||x i -c j || represents the distance from the i-th data point to the j-th center. Obviously, in order to make the classification more reasonable, J should be as small as possible. The algorithm ends when J no longer decreases or decreases by a small amount.

[0058] This invention combines functions and constraints using the Lagrange multiplier method. Calculate the objective function J based on the concept of distance:

[0059]

[0060] Where ω1 is the weighting coefficient.

[0061] This invention addresses the input parameter c. j and u ij By taking the derivative, we can obtain the condition under which the objective function reaches its minimum value.

[0062] First, regarding c j Differentiate:

[0063]

[0064]

[0065] Then for u ij Differentiate:

[0066] ​

[0067]

[0068] This allows us to obtain the minimum value of c when J is minimized. j and u ij The corresponding value, before iteration, is c. j and u ij One of them sets an initial value, which is random.

[0069] It should be noted that after segmenting the image data using the DLinkNet network model, clustering the segmentation results of the network model can correct outliers of each pixel and make the differences between feature classes extracted by the model larger and the differences within classes smaller, ultimately resulting in better segmentation results.

[0070] After obtaining the clustering results, the evaluation process includes assessing the clustering results based on the Intersection over Union (mIOU) and Mean Accuracy (mACC) metrics. The mIOU is the intersection-over-union ratio between the clustered image segmentation result and the actual image segmentation result, and its calculation formula is as follows:

[0071]

[0072] Where i represents the actual value, j represents the predicted value, and p ij This represents the number of pixels that predict i as j.

[0073] The mean accuracy (mACC) is the average accuracy across all classes, calculated using the following formula:

[0074]

[0075] Among them, P i N represents the number of correctly predicted categories. i This represents the number of class prediction errors, where N is the total number of classes.

[0076] Understandably, after obtaining the clustering results through the clustering method, the clustering results are evaluated based on the Intersection over Union (mIOU) and Mean Accuracy (mACC) metrics. If the evaluation of the clustering results is good, the clustering result, i.e., the image segmentation result, is output; if the evaluation of the clustering results is poor, ...

[0077] Then, perform clustering again until a better segmentation result is obtained.

[0078] The output of the post-processing method described above is returned as the final result. Mapping this output onto the original image data yields better image segmentation results. Table 1 shows the evaluation metrics for image segmentation with and without clustering post-processing.

[0079] Table 1 Image segmentation results

[0080]

[0081]

[0082] As can be seen from Table 1, the image segmentation accuracy is improved by adding clustering post-processing, with mIOU increasing by 3.7% and mACC increasing by 2.2%.

[0083] See Figure 2 The diagram illustrates the entire process of the image segmentation method of the present invention. First, an image segmentation training dataset is obtained. Then, data augmentation is performed on a portion of the image data in the training dataset to expand the training dataset. Each image in the expanded training dataset is labeled to obtain image segmentation labels.

[0084] Based on the expanded training dataset and corresponding labels, the DLinkNet network model is trained to obtain the trained DLinkNet network model. The feature values ​​of each pixel in the image data to be processed are extracted based on the trained DLinkNet network model. Clustering is then performed on the feature values ​​of each pixel to obtain the final clustering result, i.e., the image segmentation result. Compared to not performing post-clustering processing, the image segmentation accuracy is better.

[0085] See Figure 3 The present invention provides an image segmentation system, which includes an acquisition module 301, a training module 302, an extraction module 303, and a clustering module 304, wherein:

[0086] The acquisition module 301 is used to collect autonomous driving image data under different scenarios, and to perform data augmentation processing on the autonomous driving image data to obtain a training dataset;

[0087] Training module 302 is used to train the DLinkNet network model based on the training dataset;

[0088] The extraction module 303 is used to extract the feature value of each pixel in the image data to be processed based on the trained DLinkNet network model.

[0089] Clustering module 304 is used to perform clustering and post-processing on the feature values ​​of each pixel in the image data to be processed, and to obtain the image segmentation result of the image data to be processed. The image segmentation result refers to the segmentation of the target object from the image data to be processed.

[0090] It is understood that the image segmentation system provided by the present invention corresponds to the image segmentation methods provided in the foregoing embodiments. The relevant technical features of the image segmentation system can be referred to the relevant technical features of the image segmentation methods, and will not be repeated here.

[0091] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 4 As shown, an embodiment of the present invention provides an electronic device 400, including a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 411, it implements the steps of an image segmentation method.

[0092] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 5 As shown, this embodiment provides a computer-readable storage medium 500 on which a computer program 511 is stored. When the computer program 511 is executed by a processor, it implements the steps of an image segmentation method.

[0093] This invention provides an image segmentation method, system, electronic device, and storage medium. First, image data is acquired, augmented, and labeled. Data preprocessing, including flipping and rotation, is then performed. Next, the preprocessed data is fed into a DLinkNet network model for training. After training, the trained DLinkNet network model can be used to segment the image data, and then clustering post-processing yields a more accurate and complete image segmentation result. These steps address the problem of inaccurate segmentation results caused by insufficient data and limited model learning capabilities.

[0094] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0095] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0097] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0099] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0100] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. An image segmentation method, characterized in that, include: Collect autonomous driving image data in different scenarios, and perform data augmentation processing on the autonomous driving image data to obtain a training dataset; The DLinkNet network model was trained based on the aforementioned training dataset; The feature value of each pixel in the image data to be processed is extracted based on the trained DLinkNet network model. Clustering is performed on the feature values ​​of each pixel in the image data to be processed to obtain the image segmentation result of the image data to be processed. The image segmentation result refers to the segmentation of the target object from the image data to be processed. The step of clustering the features of each pixel in the image data to be processed to obtain the image segmentation result of the image data to be processed includes: Step a: Obtain samples to be clustered D represents the feature value of each pixel, m represents the number of pixels, and the feature values ​​of each pixel are floating-point numbers; Step b: Initialize the elements in D by uniformly distributing them to satisfy the constraints, and set the initialized set of elements as... , Each initialized element has a value that is a floating-point number between 0 and 1; Step c, according to The cluster center set is calculated using the iterative formula. , represents the cluster center of each category, and j represents the number of categories; Step d: Based on the cluster center set For the set of elements Perform clustering and calculate the objective function J based on the clustering results: Define the size of the dataset to be clustered as n, and the number of cluster centers as C. Let be the membership degree of the i-th data point to the j-th centroid, and 0 < 0. <1, ; Merging functions and constraints using the Lagrange multiplier method The objective function J is calculated based on the concept of distance. ； in, For hyperparameters, This represents the distance from the i-th data point to the j-th cluster center. These are weighting coefficients; Step e: Compare whether the objective function meets the preset conditions. If it does, output the clustering result, which is the image segmentation result. If it does not meet the conditions, repeat steps c, d and e until the objective function value meets the preset conditions.

2. The image segmentation method according to claim 1, characterized in that, The process involves collecting autonomous driving image data under different scenarios and performing data augmentation processing on the autonomous driving image data to obtain a training dataset, including: Collect multiple autonomous driving image data under different scenarios, rotate or flip all or part of the autonomous driving image data, and add the rotated or flipped autonomous driving image data and the original autonomous driving image data to the training dataset. For each autonomous driving image in the training dataset, polygonal contour annotations are performed on the target objects to generate mask images containing target type and location information as labels.

3. The image segmentation method according to claim 1 or 2, characterized in that, Training the DLinkNet network model based on the aforementioned training dataset includes: Construct the loss function of the DLinkNet network model, and train the DLinkNet network model based on the loss function and the training dataset; The loss function is the cross-entropy loss function. When the DLinkNet network model is a binary classification model, the expression for the cross-entropy loss function is: ； in, The label represents sample i, with 1 for positive and 0 for negative. This represents the probability that sample i is predicted to be of the positive class; When the DLinkNet network model is a multi-class classification model, the expression for the cross-entropy loss function is: ； Where M represents the number of categories output by the DLinkNet network model; The sign function is set to 1 if the true class of the i-th sample is equal to c, and 0 otherwise. Let be the predicted probability that observed sample i belongs to category c.

4. The image segmentation method according to claim 1, characterized in that, The preset condition is that the objective function value is less than a first set threshold or the change between the current objective function value and the objective function value of the previous iteration is less than a second set threshold.

5. The image segmentation method according to claim 1, characterized in that, In step d, based on the cluster center set Clustering of the set of elements O, followed by: Clustering results were evaluated based on the intersection-union ratio (mIOU) and mean accuracy (mACC) metrics. The intersection-over-union ratio (mIOU) is the ratio between the clustered image segmentation result and the actual image segmentation result, and its calculation formula is as follows: ； Where i represents the actual value and j represents the predicted value. This represents the number of pixels that predict i as j; The average accuracy mACC is the average accuracy across all categories, and the calculation formula is as follows: ； in, This indicates the number of correctly predicted categories. This represents the number of class prediction errors, where N is the total number of classes.

6. An image segmentation system, characterized in that, include: The acquisition module is used to collect autonomous driving image data under different scenarios, and to perform data augmentation processing on the autonomous driving image data to obtain a training dataset; The training module is used to train the DLinkNet network model based on the training dataset; The extraction module is used to extract the feature value of each pixel in the image data to be processed based on the trained DLinkNet network model. The clustering module is used to perform clustering and post-processing on the feature values ​​of each pixel in the image data to be processed, and to obtain the image segmentation result of the image data to be processed. The image segmentation result refers to the segmentation of the target object from the image data to be processed. The clustering module is used to perform clustering and post-processing on the features of each pixel in the image data to be processed, and to obtain the image segmentation result of the image data to be processed, including: Obtain samples to be clustered ; D represents the feature value of each pixel, m represents the number of pixels, and the feature values ​​of each pixel are floating-point numbers; Initialize the elements in D by uniformly distributing them to satisfy the constraints, and set the initialized set of elements as... , Each initialized element has a value that is a floating-point number between 0 and 1; according to The cluster center set is calculated using the iterative formula. , represents the cluster center of each category, and j represents the number of categories; Based on the cluster center set For the set of elements Perform clustering and calculate the objective function J based on the clustering results: Define the size of the dataset to be clustered as n, and the number of cluster centers as C. Let be the membership degree of the i-th data point to the j-th centroid, and 0 < 0. <1, ; Merging functions and constraints using the Lagrange multiplier method The objective function J is calculated based on the concept of distance. ； in, For hyperparameters, This represents the distance from the i-th data point to the j-th cluster center. These are weighting coefficients; Compare whether the objective function meets the preset conditions. If it does, output the clustering result, which is the image segmentation result. If it does not meet the conditions, repeat the iteration until the objective function value meets the preset conditions.

7. An electronic device, characterized in that, It includes a memory and a processor, wherein the processor is used to implement the steps of the image segmentation method as described in any one of claims 1-5 when executing a computer management program stored in the memory.

8. A computer-readable storage medium, characterized in that, It stores a computer management program, which, when executed by a processor, implements the steps of the image segmentation method as described in any one of claims 1-5.

Images