A training method, apparatus, and device for image segmentation networks and image segmentation.

Abstract

This disclosure provides a training method, apparatus, and device for an image segmentation network and image segmentation. The training method involves obtaining a first training image sample set and a second training image sample set. Each first training image sample in the first training image sample set is labeled with a category label, while each second training image sample in the second training image sample set is not labeled with a category label. Based on the first training image sample set and a first image segmentation network trained on the first training image sample set, a first category feature of the category is determined. Based on the first image segmentation network, a first predicted category label of each second training image sample is obtained. Based on the first predicted category label of each second training image sample and the first category feature, the first image segmentation network is iteratively updated to obtain an iteratively updated second image segmentation network, thereby reducing labeling costs and improving accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a training method, apparatus, and device for image segmentation networks and image segmentation. Background Technology

[0002] Image semantic segmentation is one of the main tasks of computer vision based on deep learning. It can classify each pixel in an image for segmentation. In related technologies, image segmentation methods mainly use image segmentation networks trained by supervised learning. Supervised learning-based image segmentation networks need to pre-label a large number of image category labels, while related technologies mainly use manual labeling, which is costly and inefficient. Summary of the Invention

[0003] This disclosure provides at least one method, apparatus, and device for training an image segmentation network and for image segmentation.

[0004] In a first aspect, embodiments of this disclosure provide a method for training an image segmentation network, comprising:

[0005] Obtain a first training image sample set and a second training image sample set, wherein each first training image sample in the first training image sample set is labeled with a category label, and each second training image sample in the second training image sample set is not labeled with a category label.

[0006] Based on the first training image sample set and the first image segmentation network trained on the first training image sample set, the first category feature of the category is determined. Based on the first image segmentation network, the first predicted category label of each second training image sample is obtained. Based on the first predicted category label of each second training image sample and the first category feature, the first image segmentation network is iteratively updated to obtain the iteratively updated second image segmentation network.

[0007] In this embodiment of the disclosure, a first image segmentation network is trained based on the first training image sample set. A first category feature of the category is determined based on the first training image sample set and the first image segmentation network. Then, the first image segmentation network is iteratively updated based on the first category feature and the second training image sample set to obtain an iteratively updated second image segmentation network. This allows training to be performed on a small number of labeled first training image sample sets and an unlabeled second training image sample set, reducing labeling costs and improving efficiency. Furthermore, training can automatically label unlabeled data, enabling the segmentation effect of supervised learning training to be achieved or exceeded with less labeled data, thus improving efficiency and accuracy.

[0008] In one possible embodiment, determining the first category feature of the category based on the first training image sample set and a first image segmentation network trained based on the first training image sample set includes:

[0009] A first image segmentation network is trained based on the first training image sample set;

[0010] Based on the first image segmentation network, feature extraction is performed on each of the first training image samples to obtain the first feature vector of each pixel in each of the first training image samples.

[0011] The first category feature of the category is determined based on the first feature vector of each pixel in each of the first training image samples and the category label corresponding to each pixel.

[0012] In this embodiment of the disclosure, a first category feature of a category can be determined based on a first training image sample set, providing a possible method for determining the first category feature. Furthermore, since the first training image sample set is labeled with category tags, the first category feature determined based on the first training image sample set is more accurate.

[0013] In one possible embodiment, iteratively updating the target loss function of the first image segmentation network includes summing the first loss functions of each of the first training image samples and summing the second loss functions of each of the second training image samples;

[0014] The first loss function is determined based on the first similarity between the first feature vector of each pixel in the first training image sample and the first category feature, and the corresponding category label. The first feature vector is obtained by feature extraction of the first training image sample by the first image segmentation network.

[0015] The second loss function is determined based on the second similarity between the second feature vector of each pixel in the second training image sample and the first category feature, as well as the corresponding first predicted category label, wherein the second feature vector is obtained by feature extraction of the second training image sample based on the first image segmentation network.

[0016] In this embodiment of the disclosure, based on the first category feature, the target loss function is determined according to the similarity between the second training image sample and the first training image sample and the first category feature, respectively, to guide the update learning of the first image segmentation network. This can enable the second image segmentation network to be updated and iterated even when the second training image sample set has no category label annotation, and improve training efficiency.

[0017] In one possible embodiment, the step of determining a first category feature of the category based on the first training image sample set and a first image segmentation network trained based on the first training image sample set, obtaining a first predicted category label for each second training image sample based on the first image segmentation network, and iteratively updating the first image segmentation network based on the first predicted category label and the first category feature of each second training image sample includes:

[0018] Based on the first training image sample set and the first image segmentation network trained on the first training image sample set, a first category feature of the category is determined;

[0019] The first image segmentation network is iteratively updated based on the second training image sample set;

[0020] Specifically, for each iteration update of the first image segmentation network, in the current iteration update, based on the first image segmentation network after the previous iteration update, the first predicted category label of each second training image sample is obtained. Based on the first predicted category label of each second training image sample and the target category feature of the category obtained after the previous iteration update, the first image segmentation network after the current iteration update is obtained. In the case of the first iteration update, the target category feature is the first category feature.

[0021] Furthermore, after this round of iteration and update, based on the first image segmentation network and the first training image sample set after this round of iteration and update, the second category feature of the category is determined, and based on the first image segmentation network and the second training image sample set after this round of iteration and update, the third category feature of the category is determined, and based on the second category feature and the third category feature, the fourth category feature of the category is obtained, and the fourth category feature is used as the target category feature to enter the next round of iteration and update.

[0022] In this embodiment of the disclosure, during the iterative update of the first image segmentation network, the category features can be updated in each round of iteration, which improves the efficiency and accuracy of the iterative update and improves the accuracy and performance of the final obtained second image segmentation network.

[0023] In one possible embodiment, obtaining the fourth category feature of the category based on the second category feature and the third category feature includes:

[0024] Determine the first product of the second category feature and the preset first weight value, and determine the second product of the third category feature and the preset second weight value;

[0025] The fourth category feature of the category is obtained by summing the first product and the second product.

[0026] In this embodiment of the disclosure, the category features are updated by weighted calculation of the category features obtained from the first training image sample set and the second training image sample set, thereby improving accuracy.

[0027] In one possible embodiment, the method further includes:

[0028] Based on the second image segmentation network, the second predicted category label of each second training image sample is obtained;

[0029] Pixels in each of the second training image samples whose probability value of the second predicted category label is greater than the confidence threshold of the corresponding category are selected.

[0030] Based on the selected pixels, the image regions corresponding to each selected pixel are obtained. The obtained image regions and the corresponding second predicted category labels are updated in the first training image sample set. The image regions are then deleted from the second training image sample set to update the second training image sample set. Based on the updated first training image sample set and the updated second training image sample set, the step of determining the first category feature of the category based on the first training image sample set and the first image segmentation network trained based on the first training image sample set is returned to continue execution until a preset number of iterations is reached or the second training image sample set is empty, at which point the final second image segmentation network is obtained.

[0031] In this embodiment of the disclosure, during the training and iterative update process, the second training image samples can be continuously labeled with category labels. Then, based on the confidence level, the first training image sample set and the second training image sample set can be continuously updated. Based on the updated first training image sample set and the second training image sample set, the first image segmentation network is trained and iteratively updated to obtain the second image segmentation network, thereby improving the accuracy of the final second image segmentation network.

[0032] In one possible embodiment, it further includes:

[0033] Based on the second predicted category label of each second training image sample, determine the first number of pixels belonging to the category in each second training image sample;

[0034] The confidence threshold of the category is determined based on the total number of pixels included in each of the second training image samples and the first number.

[0035] In this embodiment of the disclosure, the confidence threshold can be dynamically determined based on the second predicted category label of the second training image sample set by the second image segmentation network, thereby improving accuracy.

[0036] In one possible embodiment, the initial data of the first training image sample set is each of the first training image samples labeled with category tags, generated based on simulation software.

[0037] The initial data for the second training image sample set consists of each of the second training image samples collected for real business scenarios.

[0038] In this embodiment of the disclosure, the initial data of the first training image sample set and the second training image sample set can come from different scenarios, such as simulation and real business scenarios, which can improve the diversity of samples, improve the training effect, and make the image segmentation performance of the second image segmentation network after final training better.

[0039] Secondly, embodiments of this disclosure also provide an image segmentation method, including:

[0040] Obtain the image to be processed;

[0041] The trained target image segmentation network, obtained using an image segmentation network-based training method, is used to perform semantic segmentation on the image to be processed, thereby obtaining the segmentation result of the image to be processed.

[0042] Thirdly, embodiments of this disclosure also provide a training apparatus for an image segmentation network, comprising:

[0043] The acquisition module is used to acquire a first training image sample set and a second training image sample set, wherein each first training image sample in the first training image sample set is labeled with a category label, and each second training image sample in the second training image sample set is not labeled with a category label.

[0044] The iterative update module is used to determine the first category feature of the category based on the first training image sample set and the first image segmentation network trained based on the first training image sample set, obtain the first predicted category label of each second training image sample based on the first image segmentation network, and iteratively update the first image segmentation network based on the first predicted category label of each second training image sample and the first category feature to obtain the iteratively updated second image segmentation network.

[0045] Fourthly, embodiments of this disclosure also provide an image segmentation apparatus, comprising:

[0046] The acquisition module is used to acquire the image to be processed;

[0047] The segmentation module is used to perform semantic segmentation on the image to be processed using a trained target image segmentation network obtained by a training method based on image segmentation networks, and to obtain the segmentation result of the image to be processed.

[0048] Fifthly, an optional implementation of this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the steps of any of the possible implementations of the first aspect or the second aspect described above.

[0049] In a sixth aspect, an optional implementation of this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the possible implementations of the first aspect or the second aspect described above.

[0050] For a description of the effects of the training device, image segmentation device, electronic device, and computer-readable storage medium of the above-mentioned image segmentation network, please refer to the description of the training method of the above-mentioned image segmentation network and the image segmentation method, which will not be repeated here.

[0051] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure.

[0052] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating a training method for an image segmentation network provided in an embodiment of this disclosure is shown;

[0055] Figure 2 A schematic diagram of the structure of the image segmentation network provided in the embodiments of this disclosure is shown;

[0056] Figure 3 A flowchart illustrating another method for training an image segmentation network provided in an embodiment of this disclosure is shown;

[0057] Figure 4 A flowchart of an image segmentation method provided by an embodiment of this disclosure is shown;

[0058] Figure 5 A schematic diagram of a training apparatus for an image segmentation network provided in an embodiment of this disclosure is shown;

[0059] Figure 6 A schematic diagram of an image segmentation apparatus provided in an embodiment of this disclosure is shown;

[0060] Figure 7 A schematic diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0061] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0063] To facilitate understanding of the technical solutions disclosed herein, the technical terms used in the embodiments of this disclosure will first be explained:

[0064] Image semantic segmentation is a crucial step in image understanding within image processing and computer vision technologies. It operates at the pixel level, requiring the classification of each pixel in an image to determine its category, such as whether it belongs to the background, people, or other categories, thereby dividing the image into regions.

[0065] Research has found that in related technologies, image segmentation methods mainly use image segmentation networks trained through supervised learning. Supervised learning-based image segmentation networks require pre-labeling a large number of image category labels, while related technologies mainly use manual labeling, which is costly and inefficient.

[0066] Based on the above research, this disclosure provides an image segmentation network training method. The method involves obtaining a first training image sample set labeled with category labels and a second training image sample set without labels. Based on the first training image sample set and the first image segmentation network trained on the first training image sample set, a first category feature of each category is determined. Then, based on the first image segmentation network, a first predicted category label is obtained for each second training image sample. Based on the first predicted category label and the first category feature of each second training image sample, the first image segmentation network is iteratively updated to obtain an iteratively updated second image segmentation network. This allows training to be performed on a small set of labeled first training image samples and a set of unlabeled second training image samples, reducing labeling costs and improving efficiency. Furthermore, through training, the labeled data information can be utilized to automatically label unlabeled data, enabling the segmentation effect to reach or exceed that of supervised learning training with a smaller amount of labeled data.

[0067] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.

[0068] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0069] To facilitate understanding of this embodiment, a detailed description of the image segmentation network training method disclosed in this disclosure is provided first. The execution entity of the image segmentation network training method provided in this disclosure is generally an electronic device with a certain computing capability. This electronic device may include, for example, a terminal device, a server, or other processing devices. The terminal device can be a user equipment (UE), mobile device, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. A personal digital assistant is a handheld electronic device that possesses some functions of a computer. It can be used to manage personal information, browse the internet, send and receive emails, etc., and generally does not have a keyboard; it can also be called a handheld computer. In some possible implementations, the image segmentation network training method can be implemented by a processor calling computer-readable instructions stored in memory.

[0070] The training method of the image segmentation network provided in this embodiment will be described below, taking the server as the execution subject as an example.

[0071] See Figure 1 The diagram shows a flowchart of a training method for an image segmentation network provided in this embodiment of the present disclosure. The method includes:

[0072] S101: Obtain a first training image sample set and a second training image sample set, wherein each first training image sample in the first training image sample set is labeled with a category label, and each second training image sample in the second training image sample set is not labeled with a category label.

[0073] This disclosure can be applied to any scenario with semantic segmentation tasks and is not limited thereto. For example, in a road lane segmentation scenario, the segmentation categories may include lanes, ramps, lane lines, etc., requiring the collection of diverse training data and data annotations. However, data collection from real road lane scenarios is often insufficient to meet the diverse data requirements, and semantic segmentation is pixel-level, requiring classification for each pixel, which increases the time and cost of annotation. While readily available simulated road lane data can be used to address the diverse requirements and high annotation costs, simulated data differs from real-world scene data in terms of image texture and pixel distribution. Pixel distributions belong to different domains, and directly mixing data from different domains for supervised learning training can easily lead to confusion, negatively impacting the training performance of the image segmentation network.

[0074] Therefore, in this embodiment of the disclosure, it is not necessary to label all training data in advance. Iterative training can be performed based on a small amount of labeled training data and a large amount of unlabeled training data.

[0075] In this embodiment, the initial data of the first training image sample set consists of first training image samples labeled with category tags generated by simulation software. The simulation software can automatically generate image samples simulating real-world scenarios and corresponding category tags, eliminating the need for manual annotation and improving efficiency. Furthermore, to improve the accuracy of the final network semantic segmentation, the initial data of the second training image sample set consists of second training image samples collected for real-world business scenarios. However, in this embodiment, the second training image sample set does not require initial manual annotation. Instead, through continuous iterative training, the second training image sample set can be cross-domain feature aligned with the first training image samples, and automatic annotation after domain adaptation is achieved, improving the accuracy and effectiveness of training.

[0076] S102: Determine the first category feature of the category based on the first training image sample set and the first image segmentation network trained on the first training image sample set.

[0077] When performing step S102, this disclosure provides possible implementation methods, specifically including:

[0078] 1) A first image segmentation network is trained based on the first training image sample set.

[0079] In this embodiment of the disclosure, a first image segmentation network can be obtained by supervised learning training based on a first training image sample set with labeled category labels. The specific supervised learning method is not limited.

[0080] For example, the cross-entropy loss function used to optimize the training of the first image segmentation network via gradient descent is denoted as... The first training image sample set is denoted as Data. S ,but It can be defined as:

[0081]

[0082] Where, N s Data for the first training image sample set s The first training image sample has a number of samples, H is the height of the first training image sample, W is the width of the first training image sample, and C is the number of segmentation category labels. The one-hot encoding representation of class c at pixel coordinates (i,j) for the class label of the first training image sample. The predicted class probability of the first training image sample for class c at pixel coordinates (i,j) is obtained by the first image segmentation network.

[0083] 2) Based on the first image segmentation network, feature extraction is performed on each first training image sample to obtain the first feature vector of each pixel in each first training image sample.

[0084] In this embodiment of the disclosure, after the first image segmentation network is trained, each first training image sample can be input into the first image segmentation network, and the first image segmentation network can extract features from each first training image sample to obtain the first feature vector of each pixel in each first training image sample.

[0085] 3) Determine the first category feature of the category based on the first feature vector of each pixel in each first training image sample and the category label corresponding to each pixel.

[0086] In this embodiment of the disclosure, a first category feature corresponding to each category c can be calculated for the category labels annotated in the first training image sample set. The first category feature can be understood as feature information representing the category.

[0087] For example, for any category c, the first category feature is denoted as Specifically It can be determined in the following ways:

[0088]

[0089] Where, N s Data for the first training image sample set s The data volume, H is the height of the first training image sample, W is the width of the first training image sample, and c is the value for each category. The first feature vector is obtained by encoding the first training image sample through the first image segmentation network. For category labels, It is an indicator function for the category label, indicating that the value is 1 when the category label of pixel coordinate (i,j) in the nth first training image sample is c, and 0 when it is not c.

[0090] S103: Based on the first image segmentation network, obtain the first predicted category label of each second training image sample. According to the first predicted category label and the first category feature of each second training image sample, iteratively update the first image segmentation network to obtain the iteratively updated second image segmentation network.

[0091] In this embodiment of the disclosure, the first image segmentation network is iteratively updated according to the second training image sample set. The second training image sample set and the first training image sample set may have different sources, and the distribution domains of image pixels from different sources may be different. In order to improve training accuracy, cross-domain data feature alignment is achieved based on the first category feature. Specifically, the target loss function is determined based on the first category feature and the first predicted category label to constrain the training and learning of the second image segmentation network.

[0092] Specifically, in this embodiment of the disclosure, when iteratively updating the first image segmentation network to obtain the second image segmentation network, a possible implementation is provided for the target loss function of iteratively updating the first image segmentation network: the target loss function includes the sum of the first loss functions of each first training image sample and the sum of the second loss functions of each second training image sample;

[0093] The first loss function is determined based on the first similarity between the first feature vector of each pixel in the first training image sample and the first category feature, as well as the corresponding category label. The first feature vector is obtained by feature extraction from the first training image sample using the first image segmentation network. The second loss function is determined based on the second similarity between the second feature vector of each pixel in the second training image sample and the first category feature, as well as the corresponding first predicted category label. The second feature vector is obtained by feature extraction from the second training image sample using the first image segmentation network.

[0094] For example, the second similarity between the second training image sample and the first category feature of category c can be denoted as: Specifically:

[0095]

[0096] in, Data for the second training image sample set T The similarity between the nth second training image sample at pixel coordinates (i,j) and the first category feature of category c. The first category feature corresponding to category c, The second training image sample is input into the second feature vector obtained by the first image segmentation network after feature extraction. τ is a constant parameter, for example, it takes the value of 50 and is not restricted. C is the number of segmentation categories.

[0097] Then, for the nth second training image sample in the second training image sample set, its corresponding second loss function is denoted as... but:

[0098]

[0099] in, Let c be the one-hot representation of the first predicted class label of the nth second training image sample for class c at pixel position (i,j).

[0100] Similarly, for the first training image sample set, the first similarity between the first training image sample and the first category feature of category c can be calculated, which can be denoted as: And for the nth first training image sample in the first training image sample set, its corresponding first loss function is denoted as... but:

[0101]

[0102] in, Let (i,j) be the one-hot representation of the class label of the first training image sample for class c at pixel position (i,j).

[0103] Then, by summing the first loss functions of each first training image sample and summing the second loss functions of each second training image sample, the target loss function is obtained. For example, the target loss function is denoted as... The objective loss function can be calculated as follows:

[0104]

[0105] Where, N s N represents the number of first training image samples included in the first training image sample set. t The number of second training image samples included in the second training image sample set.

[0106] In this way, the feature alignment loss between the second training image sample set and the first category features can be calculated using the first category features. Since the first category feature is determined based on the first training image sample set, therefore... This can also be understood as the cross-domain alignment loss from the source domain corresponding to the first training image sample set to the target domain corresponding to the second training image sample set. Furthermore, to further improve accuracy, the same-domain feature alignment loss between the first training image sample set and the first category features can also be calculated. Then, based on the first loss function and the second loss function, the target loss function is determined, and the first image segmentation network is iteratively updated according to the target loss function to obtain the iteratively updated second image segmentation network.

[0107] Furthermore, in order to further improve training accuracy during iterative updates of the first image segmentation network, the first category feature of the category can be updated synchronously once more during each iteration. Specifically, this disclosure provides a possible implementation method: determining the first category feature of the category based on the first training image sample set and the first image segmentation network trained on the first training image sample set; obtaining the first predicted category label of each second training image sample based on the first image segmentation network; and iteratively updating the first image segmentation network based on the first predicted category label and the first category feature of each second training image sample, including: 1) determining the first category feature of the category based on the first training image sample set and the first image segmentation network trained on the first training image sample set; 2) iteratively updating the first image segmentation network based on the second training image sample set.

[0108] Specifically, for each iteration update of the first image segmentation network, in the current iteration update, based on the first image segmentation network after the previous iteration update, the first predicted category label of each second training image sample is obtained. Based on the first predicted category label of each second training image sample and the target category feature of the category obtained after the previous iteration update, the first image segmentation network after the current iteration update is obtained. In the case of the first iteration update, the target category feature is the first category feature.

[0109] Furthermore, after this round of iteration and update, based on the first image segmentation network and the first training image sample set after this round of iteration and update, the second category feature of the category is determined, and based on the first image segmentation network and the second training image sample set after this round of iteration and update, the third category feature of the category is determined, and based on the second category feature and the third category feature, the fourth category feature of the category is obtained, and the fourth category feature is used as the target category feature to enter the next round of iteration and update.

[0110] In particular, regarding obtaining the fourth category feature of a category based on the second category feature and the third category feature, this disclosure also provides a possible implementation method: determining a first product of the second category feature and a preset first weight value, and determining a second product of the third category feature and a preset second weight value; obtaining the fourth category feature of the category based on the sum of the first product and the second product.

[0111] For example, after this round of iteration, the second category feature of the first image segmentation network and the first training image sample set determined by this round of iteration is: Based on the updated first image segmentation network and the second training image sample set, the third category feature is determined as follows: If the first weight is m and the second weight is (1-m), then the fourth category feature can be represented as:

[0112]

[0113] The value of m can be set according to actual conditions and experience. For example, the value of m can be 0.75. This embodiment of the present disclosure does not impose any restrictions.

[0114] In this embodiment, a first set of labeled training image samples and a second set of unlabeled training image samples are obtained. Based on the first set of training image samples and a first image segmentation network trained on the first set of training image samples, a first category feature of the category is determined. Then, based on the first image segmentation network, a first predicted category label for each second training image sample is obtained. Based on the first predicted category label and the first category feature of each second training image sample, the first image segmentation network is iteratively updated to obtain an iteratively updated second image segmentation network. In this way, during the training process, the loss is optimized based on the first category feature, realizing the alignment of feature information belonging to the same category between the first and second training image samples. This not only effectively reduces the labeling cost and improves efficiency, but also allows training data from different domains with different distributions to be used for training, improving the accuracy and effect of training, thereby improving the semantic segmentation accuracy of the final trained second image segmentation network.

[0115] Furthermore, based on the above embodiments, in this embodiment, the first training image sample set needs to be labeled with category labels. Initially, the first training image sample set may only include a small number of first training image samples. As training iterates, the second training image samples in the unlabeled second training image sample set can obtain predicted label categories. Since the first category features determined based on the first training image sample set are needed during the training process, in order to further improve the training accuracy, the first training image sample set can be continuously updated based on the second training image samples that have obtained predicted label categories, thereby increasing the number of first training image samples included in the first training image sample set, improving the accuracy of the first category features representing the category, and further improving the semantic segmentation accuracy of the final second image segmentation network.

[0116] Specifically, this disclosure also provides possible implementation methods, including:

[0117] 1) Based on the second image segmentation network, obtain the second predicted category label for each second training image sample.

[0118] 2) Filter out the pixels in each second training image sample whose probability value of the second predicted category label is greater than the confidence threshold of the corresponding category.

[0119] For example, for N t The second prediction category label The probability values ​​of all pixels that contain category c and are predicted to be category c. in, The second predicted category label represents the probability value. Pixels predicted to be class c can be labeled as high-confidence class labels.

[0120] The confidence threshold can be predetermined or dynamically determined. This embodiment of the present disclosure also provides a method for determining the confidence threshold, which is to determine the confidence threshold of the category based on the second predicted category label.

[0121] In this embodiment of the disclosure, after iteratively updating and training based on the first training image sample set and the second training image sample set, a second image segmentation network is obtained. The second predicted category label of each second training image sample in the second training image sample set can be obtained based on the second image segmentation network. There may be errors between the second predicted category label and the true category label. In order to improve the accuracy, the confidence threshold of each category can be determined first, and the second predicted category label with higher confidence can be selected.

[0122] Regarding the confidence threshold for determining a category, this disclosure provides a possible implementation: determining a first number of pixels belonging to a category in each second training image sample based on the second predicted category label of each second training image sample; and determining the confidence threshold for the category based on the total number of pixels included in each second training image sample and the first number.

[0123] For example, the second training image sample set includes N t By inputting each of the second training image samples into the second image segmentation network, the second predicted category label corresponding to each second training image sample can be obtained, i.e., N can be obtained. t The second predicted category label for semantic segmentation Each second prediction category label Both are semantic segmentation graphs, assuming N t indivual The total number is N sum-pixel N pixels, of which the first number of pixels divided into category c is N c-pixel If there are 1, then the confidence threshold for category c can be expressed as Thh c = c-pixel / N sum-pixel Additionally, if there are multiple categories, the confidence threshold for each category can be obtained separately using this method.

[0124] 3) Based on the selected pixels, obtain the image regions corresponding to each selected pixel, update the obtained image regions and the corresponding second predicted category labels to the first training image sample set, and delete the image regions in the second training image sample set to update the second training image sample set. Based on the updated first training image sample set and the updated second training image sample set, return to step S102 above, that is, the step of determining the first category features of the category based on the first training image sample set and the first image segmentation network trained based on the first training image sample set, and continue to execute until the preset number of iterations is reached or the second training image sample set is empty, and obtain the final second image segmentation network.

[0125] In this embodiment of the disclosure, semantic segmentation is pixel-level, with each second predicted category label... Both are semantic segmentation maps. A semantic segmentation map may divide pixels into multiple different categories. In a second training image sample, the probability value of the second predicted category label of some pixels is greater than the confidence threshold of the corresponding category, while the probability value of the second predicted category label of some pixels may be less than the confidence threshold of the corresponding category. Therefore, in the embodiments disclosed in this disclosure, only the image regions corresponding to each pixel that is selected as greater than the confidence threshold can be obtained and updated to the first training image sample set, instead of updating the entire second training image sample to the first training image sample set.

[0126] Furthermore, to ensure image size consistency, all pixels in the second training image sample except for the high-confidence image region are processed as background or as pure white or pure black, etc. Then, the image containing only the real pixels of the high-confidence image region is updated to the first training image sample set. Similarly, the pixels in the high-confidence image region of the second training image sample are processed as background or as pure white or pure black, etc., while the regions other than the high-confidence image region are retained in the second training image sample set.

[0127] Then, the updated first training image sample set and the updated second training image sample set can be obtained. Then, the above step S102 is returned to perform the loop iteration until the preset number of iterations is reached. For example, the number of iterations is set to 10 times. It can be set according to the actual situation and experience. This embodiment of the disclosure does not impose any restrictions. Alternatively, since the number of samples in the second training image sample set will decrease as the loop iteration continues, the loop iteration end condition can also be set to the second training image sample set being empty. After the loop iteration ends, the second image segmentation network obtained in the last loop iteration can be obtained.

[0128] Thus, in this embodiment of the present disclosure, by continuously iterating, each second training image sample included in the second training image sample set can be labeled with a second predicted category label with high confidence, and the first training image sample set and the second training image sample set can be continuously updated, thereby continuously updating the training image segmentation network, improving the training accuracy of the finally obtained second image segmentation network, and thus improving the accuracy of semantic segmentation based on the second image segmentation network.

[0129] The training method of the image segmentation network in this embodiment will be described below using a specific application scenario.

[0130] First, a brief explanation of the structure of the image segmentation network will be given. For ease of understanding, the first image segmentation network and the second image segmentation network in the embodiments of this disclosure will be collectively referred to as the image segmentation network. See reference [link to documentation]. Figure 2 The diagram shown is a schematic of the structure of an image segmentation network in this embodiment of the present disclosure. The image segmentation network includes a feature encoder and a segmentation decoder. The feature encoder is mainly used to extract features to obtain feature vectors, for example, using a ResNet-101 network. The segmentation decoder is mainly used to perform semantic segmentation, for example, using a semantic segmentation framework Deeplab-V2. Of course, in this embodiment of the present disclosure, there are no restrictions on the network structure of the image segmentation network.

[0131] Using the first training image sample set as Data S The second training image sample set is Data T For example, the initial data for the first training image sample set can be automatically generated using simulation software, producing each first training image sample with category labels. The initial data for the second training image sample set can be collected from real-world business scenarios and does not require labeling. Furthermore, there may be multiple category labels for different scenarios. Taking any single category c as an example, please refer to [reference needed]. Figure 3 The diagram shown is a flowchart of another image segmentation network training method in this embodiment of the present disclosure, which specifically includes:

[0132] S301: Determine the first category feature of the category based on the first training image sample set and the first image segmentation network trained on the first training image sample set.

[0133] S302: Based on the second training image sample set and the first category features, iteratively update the first image segmentation network to obtain the iteratively updated second image segmentation network.

[0134] In the process of iteratively updating the first image segmentation network, based on the target loss function, the first category feature is updated synchronously once in each round of iteration, and then the next round of iteration is performed based on the updated first category feature, until the target loss function converges, and the second image segmentation network can be obtained.

[0135] S303: Based on the second image segmentation network, update the first training image sample set and the second training image sample set. Then return to continue executing the above step S301 until the preset number of iterations is reached or the second training image sample set is empty.

[0136] Thus, in this embodiment of the present disclosure, a first training image sample set with different domains and labeled and a second training image sample set without labeling can be used to train the final image segmentation network. It is not necessary to label all training image samples, which reduces the labeling cost. Furthermore, it can be applied to cases where the first training image sample set and the second training image sample set belong to different domains, and can adaptively perform cross-domain category feature alignment, thereby improving the accuracy of the image segmentation network.

[0137] Based on the training method of the image segmentation network in the embodiments of this disclosure, a second image segmentation network can be finally obtained. Image semantic segmentation can be performed based on the second image segmentation network. The embodiments of this disclosure also provide an image segmentation method based on the image segmentation network. The execution subject of the training method of the image segmentation network provided in the embodiments of this disclosure is generally an electronic device with certain computing power. The electronic device includes, for example, a terminal device, a server, or other processing devices. Taking a server as the execution subject as an example.

[0138] See Figure 4 The diagram shows a flowchart of an image segmentation method provided in an embodiment of this disclosure. The method includes:

[0139] S401: Obtain the image to be processed.

[0140] S402: Using the trained target image segmentation network obtained by the image segmentation network training method, perform semantic segmentation on the image to be processed to obtain the segmentation result of the image to be processed.

[0141] The target image segmentation network is the second image segmentation network finally obtained in the above embodiments. The specific training method of the target image segmentation network is the same as that in the above embodiments, and will not be described again here.

[0142] Furthermore, in this embodiment, the application scenarios of the image segmentation method and the image segmentation network are not limited, such as autonomous driving, roads, smart security, etc. For semantic segmentation tasks in different application scenarios, the target image segmentation network can be trained by obtaining the first training image sample set and the second training image sample set required for the semantic segmentation task. Then, based on the target image segmentation network, semantic segmentation can be performed on the image to be processed in the application scenario, that is, the category of each pixel point identified in the image to be processed can be obtained.

[0143] Thus, in this embodiment of the disclosure, lower annotation costs and higher accuracy can be achieved to obtain the final target image segmentation network. Then, based on the target image segmentation network, semantic segmentation is performed on the image to be processed to obtain the segmentation result of the image to be processed, which can improve the accuracy of semantic segmentation.

[0144] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0145] Based on the same inventive concept, this disclosure also provides an image segmentation network training device corresponding to the image segmentation network training method. Since the principle of the device in this disclosure for solving the problem is similar to the image segmentation network training method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0146] Reference Figure 5 The diagram shown is a schematic of a training apparatus for an image segmentation network provided in an embodiment of this disclosure. The apparatus includes:

[0147] The acquisition module 51 is used to acquire a first training image sample set and a second training image sample set, wherein each first training image sample in the first training image sample set is labeled with a category label, and each second training image sample in the second training image sample set is not labeled with a category label.

[0148] The iterative update module 52 is used to determine the first category feature of the category based on the first training image sample set and the first image segmentation network trained based on the first training image sample set, obtain the first predicted category label of each second training image sample based on the first image segmentation network, and iteratively update the first image segmentation network based on the first predicted category label of each second training image sample and the first category feature to obtain the iteratively updated second image segmentation network.

[0149] This disclosure also provides an image segmentation apparatus corresponding to the image segmentation method, see reference. Figure 6 The diagram shown is a schematic representation of an image segmentation apparatus provided in an embodiment of this disclosure. The apparatus includes:

[0150] Acquisition module 61 is used to acquire the image to be processed;

[0151] The segmentation module 62 is used to perform semantic segmentation on the image to be processed using a trained target image segmentation network obtained by a training method based on image segmentation networks, so as to obtain the segmentation result of the image to be processed.

[0152] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0153] This disclosure also provides an electronic device, such as... Figure 7 The diagram shown is a schematic representation of the structure of an electronic device provided in an embodiment of this disclosure, including:

[0154] A processor 71 and a memory 72; the memory 72 stores machine-readable instructions executable by the processor 71, and the processor 71 executes the machine-readable instructions stored in the memory 72. When the machine-readable instructions are executed by the processor 71, the processor 71 performs the following steps:

[0155] Obtain a first training image sample set and a second training image sample set, wherein each first training image sample in the first training image sample set is labeled with a category label, and each second training image sample in the second training image sample set is not labeled with a category label.

[0156] Based on the first training image sample set and the first image segmentation network trained on the first training image sample set, the first category feature of the category is determined. Based on the first image segmentation network, the first predicted category label of each second training image sample is obtained. Based on the first predicted category label of each second training image sample and the first category feature, the first image segmentation network is iteratively updated to obtain the iteratively updated second image segmentation network.

[0157] Alternatively, the processor 71 is configured to perform the following steps:

[0158] Obtain the image to be processed;

[0159] The trained target image segmentation network, obtained using an image segmentation network-based training method, is used to perform semantic segmentation on the image to be processed, thereby obtaining the segmentation result of the image to be processed.

[0160] The aforementioned memory 72 includes a main memory 721 and an external memory 722; the main memory 721, also known as internal memory, is used to temporarily store the computational data in the processor 71, as well as the data exchanged with external memory 722 such as a hard disk. The processor 71 exchanges data with the external memory 722 through the main memory 721.

[0161] The specific execution process of the above instructions can be referred to the training method of the image segmentation network or the steps of the image segmentation method described in the embodiments of this disclosure, and will not be repeated here.

[0162] This disclosure also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program performs the steps of the image segmentation network training method or image segmentation method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0163] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the image segmentation network training method or image segmentation method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0164] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0165] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0166] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0167] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0168] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0169] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A method for training an image segmentation network, characterized in that, The method comprises: obtaining a first training image sample set and a second training image sample set, wherein each first training image sample in the first training image sample set is labeled with a category label, and each second training image sample in the second training image sample set is not labeled with a category label; determining a first category feature of the category according to the first training image sample set and a first image segmentation network trained based on the first training image sample set, obtaining a first predicted category label of each second training image sample based on the first image segmentation network, and iteratively updating the first image segmentation network according to the first predicted category label of each second training image sample and the first category feature to obtain an iteratively updated second image segmentation network; wherein the determination of the first category feature of the category according to the first training image sample set and the first image segmentation network trained based on the first training image sample set comprises: training a first image segmentation network based on the first training image sample set; performing feature extraction on each first training image sample according to the first image segmentation network to obtain a first feature vector of each pixel point in each first training image sample; determining the first category feature of the category according to the first feature vector of each pixel point in each first training image sample and the category label corresponding to each pixel point; wherein the target loss function for iteratively updating the first image segmentation network comprises a sum of first loss functions of each first training image sample and a sum of second loss functions of each second training image sample; wherein the first loss function is determined according to a first similarity between the first feature vector of each pixel point in the first training image sample and the first category feature, and the corresponding category label, wherein the first feature vector is obtained by performing feature extraction on the first training image sample according to the first image segmentation network; the second loss function is determined according to a second similarity between a second feature vector of each pixel point in the second training image sample and the first category feature, and the corresponding first predicted category label, wherein the second feature vector is obtained by performing feature extraction on the second training image sample according to the first image segmentation network.

2. The method of claim 1, wherein, The determination of the first category feature of the category according to the first training image sample set and the first image segmentation network trained based on the first training image sample set, the obtaining of the first predicted category label of each second training image sample based on the first image segmentation network, and the iteratively updating of the first image segmentation network according to the first predicted category label of each second training image sample and the first category feature comprises: determining a first category feature of the category according to the first training image sample set and a first image segmentation network trained based on the first training image sample set; iteratively updating the first image segmentation network according to the second training image sample set; In the method, for each iteration update of the first image segmentation network, in the current iteration update, first prediction class labels of the second training image samples are obtained based on the first image segmentation network after the last iteration update, and then the first prediction class labels and the target class feature of the class obtained after the last iteration update are used to obtain the first image segmentation network after the current iteration update, wherein in the case of the first iteration update, the target class feature is the first class feature; After the current iteration update, a second class feature of the class is determined based on the first image segmentation network after the current iteration update and the first training image sample set, a third class feature of the class is determined based on the first image segmentation network after the current iteration update and the second training image sample set, a fourth class feature of the class is obtained based on the second class feature and the third class feature, and the fourth class feature is taken as the target class feature for the next iteration update.

3. The method of claim 2, wherein, The fourth class feature of the class is obtained based on the second class feature and the third class feature, including: a first product of the second class feature and a preset first weight value is determined, and a second product of the third class feature and a preset second weight value is determined; the fourth class feature of the class is obtained based on a sum of the first product and the second product.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: second prediction class labels of the second training image samples are obtained based on the second image segmentation network; pixel points in the second training image samples whose probability values of the second prediction class labels are greater than a confidence threshold of the corresponding class are screened out; image regions corresponding to the screened pixel points are obtained based on the screened pixel points, the obtained image regions and the corresponding second prediction class labels are updated to the first training image sample set, and the image regions are deleted from the second training image sample set, so as to update the second training image sample set, and the step of determining the first class feature of the class based on the first training image sample set and the first image segmentation network trained based on the first training image sample set is continued to be executed until a preset iteration number is reached or the second training image sample set is empty, and finally the second image segmentation network is obtained.

5. The method of claim 4, wherein, The method further includes: a first number of pixel points belonging to the class in the second training image samples is determined based on the second prediction class labels of the second training image samples; a confidence threshold of the class is determined based on a total number of pixel points included in the second training image samples and the first number.

6. The method of claim 1, wherein, The initial data of the first training image sample set is each first training image sample labeled with a class label generated based on simulation software. The initial data of the second training image sample set is each second training image sample collected for a real business scenario.

7. An image segmentation method characterized by, Comprise: Obtaining an image to be processed; Using the trained target image segmentation network obtained by the image segmentation network training method of any one of claims 1 to 6, performing semantic segmentation on the image to be processed to obtain a segmentation result of the image to be processed.

8. A training device for an image segmentation network, characterized in that, Comprise: An acquisition module is configured to acquire a first training image sample set and a second training image sample set, wherein each first training image sample in the first training image sample set is labeled with a category label, and each second training image sample in the second training image sample set is not labeled with a category label; An iterative updating module is configured to determine a first category feature of the category based on the first training image sample set and a first image segmentation network trained based on the first training image sample set, obtain a first predicted category label of each second training image sample based on the first image segmentation network, iteratively update the first image segmentation network based on the first predicted category label of each second training image sample and the first category feature, and obtain an iteratively updated second image segmentation network. The iterative updating module is configured to, when determining the first category feature of the category based on the first training image sample set and the first image segmentation network trained based on the first training image sample set: Train a first image segmentation network based on the first training image sample set; Extract features of each first training image sample based on the first image segmentation network to obtain a first feature vector of each pixel point in each first training image sample; Determine the first category feature of the category based on the first feature vector of each pixel point in each first training image sample and the category label corresponding to each pixel point; The target loss function for iteratively updating the first image segmentation network comprises a sum of first loss functions of each first training image sample and a sum of second loss functions of each second training image sample. The first loss function is determined based on a first similarity between the first feature vector of each pixel point in the first training image sample and the first category feature and the corresponding category label, wherein the first feature vector is obtained by extracting features of the first training image sample based on the first image segmentation network; The second loss function is determined based on a second similarity between a second feature vector of each pixel point in the second training image sample and the first category feature and the corresponding first predicted category label, wherein the second feature vector is obtained by extracting features of the second training image sample based on the first image segmentation network.

9. An image segmentation apparatus characterized by comprising: Comprise: An acquisition module is configured to acquire an image to be processed; A segmentation module is configured to use a trained target image segmentation network obtained by the image segmentation network training method of any one of claims 1 to 6 to perform semantic segmentation on the image to be processed to obtain a segmentation result of the image to be processed.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor, when executing the program, implements the steps of the method of any one of claims 1-6 or 7.

11. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program, when executed by the processor, implements the steps of the method of any one of claims 1-6 or 7.

Images