Training method of semantic segmentation model, semantic segmentation method and device of image

Abstract

The embodiments of the present disclosure disclose a training method of a semantic segmentation model, a semantic segmentation method and device of an image, wherein the method comprises: updating a first semi-supervised semantic segmentation model based on a first processing result of the first semi-supervised semantic segmentation model on first labeled training image data and a second processing result of the first semi-supervised semantic segmentation model on first unlabeled training image data, to obtain a second semi-supervised semantic segmentation model; determining at least one image to be labeled by using a preset active learning rule based on the second processing result; updating the first labeled training image data and the first unlabeled training image data based on the obtained each image to be labeled and the corresponding label; and training the second semi-supervised semantic segmentation model based on the updated second labeled training image data and the second unlabeled training image data to obtain a target semantic segmentation model. The embodiments of the present disclosure can effectively improve the distribution of labeled data, thereby effectively improving the performance of the semantic segmentation model.

Description

Technical Field

[0001] This disclosure relates to computer vision technology, and in particular to a method for training a semantic segmentation model, a method for semantic segmentation of images, and an apparatus. Background Technology

[0002] Semi-supervised semantic segmentation is a hot topic in computer vision recently. It utilizes a small amount of labeled image data and a large amount of unlabeled image data to train a model, obtaining a semantic segmentation model, which is then used to classify each pixel in the image. Existing semi-supervised semantic segmentation methods typically train an initial model on labeled image data, then use this initial model to predict unlabeled image data, generating pseudo-labels for the unlabeled data, and continuing to train the model. However, the performance of existing semi-supervised semantic segmentation methods is limited by confirmation errors caused by prediction mistakes on unlabeled data, resulting in poor model performance. Summary of the Invention

[0003] To address the aforementioned technical problems such as the poor performance of semi-supervised semantic segmentation models, this disclosure is proposed. Embodiments of this disclosure provide a method for training a semantic segmentation model, a method for semantic segmentation of images, and an apparatus for such segmentation.

[0004] According to one aspect of the present disclosure, a method for training a semantic segmentation model is provided, comprising: acquiring a first processing result of a first semi-supervised semantic segmentation model on first labeled training image data and a second processing result of the first semi-supervised semantic segmentation model on first unlabeled training image data; updating the first semi-supervised semantic segmentation model based on the first processing result and the second processing result to obtain a second semi-supervised semantic segmentation model; determining at least one unlabeled image in the first unlabeled training image data based on the second processing result and using a preset active learning rule; acquiring labels corresponding to each unlabeled image; updating the first labeled training image data and the first unlabeled training image data based on each unlabeled image and the labels corresponding to each unlabeled image to obtain updated second labeled training image data and second unlabeled training image data; training the second semi-supervised semantic segmentation model based on the second labeled training image data and the second unlabeled training image data; and obtaining a target semantic segmentation model in response to satisfying a training termination condition.

[0005] According to another aspect of the present disclosure, a training apparatus for a semantic segmentation model is provided, comprising: a first acquisition module, configured to acquire a first processing result of a first semi-supervised semantic segmentation model on first labeled training image data and a second processing result of the first semi-supervised semantic segmentation model on first unlabeled training image data; a first processing module, configured to update the first semi-supervised semantic segmentation model based on the first processing result and the second processing result to obtain a second semi-supervised semantic segmentation model; a second processing module, configured to determine at least one unlabeled image in the first unlabeled training image data based on the second processing result and using a preset active learning rule; a second acquisition module, configured to acquire labels corresponding to each unlabeled image; a third processing module, configured to update the first labeled training image data and the first unlabeled training image data based on each unlabeled image and the labels corresponding to each unlabeled image to obtain updated second labeled training image data and second unlabeled training image data; a fourth processing module, configured to train the second semi-supervised semantic segmentation model based on the second labeled training image data and the second unlabeled training image data; and a fifth processing module, configured to obtain a target semantic segmentation model in response to satisfying a training termination condition.

[0006] According to another aspect of the present disclosure, a semantic segmentation method for an image is provided, comprising: acquiring image data to be processed; performing semantic segmentation on the image data to be processed based on a pre-obtained target semantic segmentation model to obtain a semantic segmentation result corresponding to the image data to be processed; wherein the target semantic segmentation model is obtained by training the semantic segmentation model as described in any of the above embodiments.

[0007] According to another aspect of the present disclosure, an image semantic segmentation apparatus is provided, comprising: a third acquisition module for acquiring image data to be processed; and a sixth processing module for performing semantic segmentation on the image data to be processed based on a pre-obtained target semantic segmentation model to obtain a semantic segmentation result corresponding to the image data to be processed; wherein the target semantic segmentation model is obtained by training the semantic segmentation model as described in any of the above embodiments.

[0008] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for executing a training method for a semantic segmentation model as described in any of the above embodiments of the present disclosure; or for executing a semantic segmentation method for an image as described in any of the above embodiments of the present disclosure.

[0009] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the training method of the semantic segmentation model described in any of the above embodiments of the present disclosure, or to implement the semantic segmentation method of the image described in any of the above embodiments of the present disclosure.

[0010] Based on the semantic segmentation model training method, image semantic segmentation method, and apparatus provided in the above embodiments of this disclosure, during the training process of the semi-supervised semantic segmentation model, high-value images to be labeled are determined from the unlabeled training data through active learning rules, and the labels of the images to be labeled are obtained to improve the distribution of labeled data. The training of the semi-supervised semantic segmentation model continues. By iteratively training based on semi-supervised learning and active learning, the distribution of labeled data can be continuously improved, thereby effectively improving the performance of the semantic segmentation model.

[0011] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0012] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 This is an exemplary application scenario of the training method for the semantic segmentation model provided in this disclosure;

[0014] Figure 2 This is a flowchart illustrating a training method for a semantic segmentation model provided in an exemplary embodiment of this disclosure;

[0015] Figure 3 This is a flowchart illustrating a training method for a semantic segmentation model provided in another exemplary embodiment of this disclosure;

[0016] Figure 4 This is a flowchart illustrating step 2031 provided in an exemplary embodiment of this disclosure;

[0017] Figure 5 This is a schematic diagram illustrating the overall principle of cross-supervision and active learning provided in an exemplary embodiment of this disclosure;

[0018] Figure 6This is a schematic flowchart of an image semantic segmentation method provided in an exemplary embodiment of this disclosure;

[0019] Figure 7 This is a schematic diagram of the structure of a training device for a semantic segmentation model provided in an exemplary embodiment of this disclosure;

[0020] Figure 8 This is a schematic diagram of the structure of a training device for a semantic segmentation model provided in another exemplary embodiment of this disclosure;

[0021] Figure 9 This is a schematic diagram of the structure of the fifth processing unit 5031 provided in an exemplary embodiment of the present disclosure;

[0022] Figure 10 This is a schematic diagram of the structure of an image semantic segmentation apparatus provided in an exemplary embodiment of the present disclosure;

[0023] Figure 11 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0024] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0025] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0026] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0027] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0028] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0029] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0030] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0031] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0032] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0033] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0034] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0035] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0036] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0037] This disclosure outlines

[0038] In developing this disclosure, the inventors discovered that semi-supervised semantic segmentation is a recent hot topic in the field of computer vision. It utilizes a small amount of labeled image data and a large amount of unlabeled image data for model training to obtain a semantic segmentation model, which is then used to classify each pixel in the image. Existing semi-supervised semantic segmentation methods typically train an initial model using labeled image data, then use this initial model to predict unlabeled image data, generating pseudo-labels for the unlabeled data, and continuing to train the model. However, the performance of the semantic segmentation model in these existing semi-supervised methods is limited by confirmation errors caused by prediction mistakes on unlabeled data, resulting in poor model performance.

[0039] Exemplary Overview

[0040] Figure 1 This is an exemplary application scenario of the training method for the semantic segmentation model provided in this disclosure.

[0041] For training the semantic segmentation model, the training method of the semantic segmentation model disclosed herein is executed using a semantic segmentation model training device. Based on a pre-obtained first semi-supervised semantic segmentation model, first labeled training image data and first unlabeled training image data are processed respectively to obtain a first processing result corresponding to the first labeled training image and a second processing result corresponding to the first unlabeled training image data. The first semi-supervised semantic segmentation model is updated based on the first and second processing results to obtain an updated second semi-supervised semantic segmentation model. Simultaneously, based on the second processing result and combined with active learning rules, at least one image to be labeled in the first unlabeled training image data can be determined. Each image to be labeled is then labeled to obtain the corresponding label. The labeling method can include automatic labeling, semi-automatic labeling, manual labeling, etc., and is not specifically limited. The first labeled training image data and the first unlabeled training image data are updated based on each image to be labeled and its corresponding label. Each image to be labeled is removed from the first unlabeled training image data, and each image to be labeled and its corresponding label are added to the first labeled training image data to improve the distribution of the labeled data. The semi-supervised semantic segmentation model is then trained based on the updated second labeled training image data and the second unlabeled training image data. This process of training based on semi-supervised learning and active learning is repeated until the model meets the training termination condition, thus obtaining the target semantic segmentation model. By continuously improving the distribution of labeled data, the performance of the semantic segmentation model is effectively improved.

[0042] The training method of the semantic segmentation model disclosed herein can be applied to the training of semantic segmentation models required in any scenario or domain, such as autonomous driving scenarios, geographic information systems, medical image analysis, robotics, etc., without any specific limitations.

[0043] Exemplary methods

[0044] Figure 2 This is a flowchart illustrating a training method for a semantic segmentation model provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically servers, terminals, and other electronic devices. Figure 2 As shown, it includes the following steps:

[0045] Step 201: Obtain the first processing result of the first semi-supervised semantic segmentation model on the first labeled training image data and the second processing result of the first semi-supervised semantic segmentation model on the first unlabeled training image data.

[0046] The first semi-supervised semantic segmentation model can be an initialized model or a model that has undergone some training; there are no specific limitations. The first semi-supervised semantic segmentation model can include at least one branch network, each of which is a semantic segmentation network. For example, including two branch networks can be used to achieve cross-supervision of unlabeled training image data. The specific structure of the branch networks can be set according to actual needs; for example, it can use a semantic segmentation network based on deeplabv3+, a semantic segmentation network based on UNet, a semantic segmentation network based on FCN (Fully Convolutional Networks), etc. The network structures of each branch network can be the same, but different initializations can be performed.

[0047] The first set of labeled training image data can include multiple labeled training images and their corresponding labels. The labels for the labeled training images can be obtained using any feasible annotation method, such as manual annotation, automatic annotation, etc., without any specific limitation. The first set of unlabeled training image data can include multiple unlabeled training images. Both labeled and unlabeled training images can be collected in advance using any feasible method. For example, in the field of autonomous driving, a data collection vehicle equipped with image sensors, such as cameras, can be used to drive in the area or road to be collected. During the journey, the vehicle collects images of the surrounding environment through the cameras, and labeled training images are obtained by annotating these images. Unlabeled images can be used as unlabeled training images.

[0048] The first processing result may include probability data corresponding to the first labeled training image data. This probability data may include probability maps corresponding to each labeled training image in the first labeled training image data. Each probability map includes the probability value of each pixel in the labeled training image belonging to each type. The probability map corresponds to the training image, and the pixel value at each pixel position on the probability map represents the probability that the pixel at that position belongs to a certain type. When there are multiple types, the probability map may include multiple channels, each channel corresponding to one type. The pixel value at each pixel position in that channel represents the probability that the image at that pixel position belongs to the type corresponding to that channel. The specific representation of the probability map is not limited. Taking a two-branch network as an example, the first processing result includes the first probability data predicted by the first branch network and the second probability data predicted by the second branch network. Similarly, the second processing result may include probability data corresponding to the first unlabeled training image data. Further details will not be elaborated further.

[0049] Step 202: Update the first semi-supervised semantic segmentation model based on the first processing result and the second processing result to obtain the second semi-supervised semantic segmentation model.

[0050] Updating the first semi-supervised semantic segmentation model refers to updating its network parameters. Specifically, the loss for labeled training can be determined based on the first processing result and the first label data (including the labels corresponding to each labeled training image) in the first labeled training image data. Then, pseudo-label data is generated based on the second processing result for cross-supervision to determine the loss for unlabeled training. The combined loss of the two losses is then used to update the network parameters, resulting in the updated second semi-supervised semantic segmentation model. Specific update rules can be set according to actual needs; this embodiment does not impose limitations.

[0051] Step 203: Based on the second processing result, a preset active learning rule is used to determine at least one image to be labeled in the first unlabeled training image data.

[0052] The preset active learning rules can be set according to actual needs. For example, they can be combined with the uncertainty of the prediction results of the second processing and the pseudo-label inconsistency of the cross-supervision used for the first unlabeled training image data when updating the first semi-supervised semantic segmentation model to determine the images with higher value to be labeled in the first unlabeled training image data. The specific rules are not limited.

[0053] Step 204: Obtain the labels corresponding to each image to be labeled.

[0054] The labels corresponding to the images to be labeled can be obtained through any feasible labeling method, such as manual labeling, semi-automatic labeling, automatic labeling, etc., and the specific method is not limited.

[0055] Step 205: Update the first labeled training image data and the first unlabeled training image data based on each image to be labeled and the corresponding label for each image to be labeled, to obtain the updated second labeled training image data and the second unlabeled training image data.

[0056] The update of the first labeled training image data and the first unlabeled training image data can be achieved by filtering out each image to be labeled from the first unlabeled training image data and adding each image to be labeled as a labeled training image along with its label to the first labeled training image data, thereby improving the distribution of the labeled training image data.

[0057] Step 206: Train the second semi-supervised semantic segmentation model based on the second labeled training image data and the second unlabeled training image data.

[0058] The training of the second semi-supervised semantic segmentation model based on the second labeled training image data and the second unlabeled training image data is similar to that based on the first labeled training image data and the first unlabeled training image data.

[0059] In an optional embodiment, the second labeled training image data can be used as the first labeled training image data, the second unlabeled training image data can be used as the first unlabeled training image data, and the second semi-supervised semantic segmentation model can be used as the first semi-supervised semantic segmentation model. The above process is repeated, and each repetition can determine whether the training termination condition is met. For example, after updating and obtaining the second semi-supervised semantic segmentation model, it can be determined whether the second semi-supervised semantic segmentation model meets the training termination condition through test data, or after each execution of step 201, the network loss is determined based on the first processing result and the second processing result to determine whether the training termination condition is met. The specific judgment node can be set according to actual needs, and this disclosure does not limit it.

[0060] Step 207: In response to the completion of training, the target semantic segmentation model is obtained.

[0061] The training termination condition can be set according to actual needs. For example, the training termination condition may include network loss convergence, the number of iterations reaching a preset threshold, etc., and there are no specific limitations. If the training termination condition is met, training can be terminated, and any branch network in the finally updated second semi-supervised semantic segmentation model can be used as the target semantic segmentation model.

[0062] The semantic segmentation model training method provided in this embodiment can identify high-value images to be labeled from unlabeled training data through active learning rules during the training process of a semi-supervised semantic segmentation model. The labels of the images to be labeled are obtained to improve the distribution of labeled data. The training of the semi-supervised semantic segmentation model can then continue. By iteratively training based on semi-supervised learning and active learning, the distribution of labeled data can be continuously improved, thereby effectively improving the performance of the semantic segmentation model.

[0063] Figure 3 This is a flowchart illustrating a training method for a semantic segmentation model provided in another exemplary embodiment of this disclosure.

[0064] In an optional embodiment, the second processing result includes the third probability data corresponding to the first unlabeled training image data in the first branch network of the first semi-supervised semantic segmentation model and the fourth probability data corresponding to the second branch network of the first semi-supervised semantic segmentation model; the method of this disclosure further includes:

[0065] Step 310: Determine the first pseudo-label data based on the third probability data included in the second processing result.

[0066] Step 320: Determine the second pseudo-label data based on the fourth probability data included in the second processing result.

[0067] The third probability data may include third probability maps corresponding to each unlabeled training image in the first unlabeled training image data. Each third probability map for an unlabeled training image includes the probability values ​​predicted by the first branch network for each pixel in the unlabeled training image to belong to each type. The fourth probability data includes fourth probability maps corresponding to each unlabeled training image. Each fourth probability map for an unlabeled training image includes the probability values ​​predicted by the second branch network for each pixel in the unlabeled training image to belong to each type. The first pseudo-label data may include type pseudo-labels (which can be called first type pseudo-labels) corresponding to each pixel in each unlabeled training image determined based on the third probability data. The second pseudo-label data may include type pseudo-labels (which can be called second type pseudo-labels) corresponding to each pixel in each unlabeled training image determined based on the fourth probability data. The type pseudo-label for each pixel may include pseudo-labels for each segmentation type to which the pixel belongs. The segmentation type can be set according to actual needs; for example, the segmentation type may include vehicles, pedestrians, lane lines, zebra crossings, signs, etc., without specific limitations. The first and second pseudo-label data can be obtained by encoding the third and fourth probability data respectively. The encoding method can be set according to actual needs, such as one-hot encoding.

[0068] This embodiment obtains pseudo-label data by encoding the probability data corresponding to the unlabeled training image data, which is used for cross-supervision and can effectively promote the convergence of the model.

[0069] In an optional embodiment, the first processing result includes first probability data corresponding to the first branch network of the first labeled training image data in the first semi-supervised semantic segmentation model and second probability data corresponding to the second branch network of the first semi-supervised semantic segmentation model; step 202, updating the first semi-supervised semantic segmentation model based on the first and second processing results to obtain a second semi-supervised semantic segmentation model, includes:

[0070] Step 2021: Determine the first cross-entropy loss based on the first probability data, the second probability data, and the first label data in the first labeled training image data.

[0071] The first label data includes the labels corresponding to each labeled training image in the first labeled training image data. The first cross-entropy loss of the labeled training is determined by comparing the first probability data and the second probability data with the first label data.

[0072] For example, the first cross-entropy loss L S It can be obtained through the following loss function:

[0073]

[0074] Among them, D l For the labeled dataset (i.e., the first labeled training image data), |D l | represents the number of samples in the labeled dataset, and x represents D. l The text contains labeled training images, where W and H represent the width and height of the labeled training image x, respectively. ce This refers to cross-entropy calculation. Y represents the prediction results of the first branch network and the second branch network, respectively, that is, the probability value corresponding to the i-th pixel of the labeled training image x in the first probability data and the probability value corresponding to the i-th pixel of the labeled training image x in the second probability data. i 1 Y i 2 Y is the label value corresponding to the i-th pixel of the labeled training image x in the first labeled data. For the same pixel, Y i 1 =Y i 2 .

[0075] Step 2022: Determine the second cross-entropy loss based on the third probability data, the second pseudo-label data, the fourth probability data, and the first pseudo-label data.

[0076] Specifically, the second pseudo-label data determined by the fourth probability data is used as the label data corresponding to the third probability data, and the first pseudo-label data determined by the third probability data is used as the label data corresponding to the fourth probability data. Cross-supervision is performed on the first branch network and the second branch network to determine the second cross-entropy loss.

[0077] For example, the second cross-entropy loss L U It can be obtained through the following loss function:

[0078]

[0079] Among them, D u Let x represent the unlabeled dataset (i.e., the first unlabeled training image data), and let D represent the dataset. u In the unlabeled training image, W and H are the width and height of the unlabeled training image x, respectively. Let l be the pseudo-label of the type corresponding to the i-th pixel of the unlabeled training image x in the first pseudo-label data and the pseudo-label of the type corresponding to the i-th pixel of the unlabeled training image x in the second pseudo-label data. ce This refers to cross-entropy calculation. This represents the first cross-entropy corresponding to the i-th pixel. P represents the second cross-entropy corresponding to the i-th pixel. i 1 P i 2 These are the prediction results of the first branch network and the second branch network, respectively, namely the probability value corresponding to the i-th pixel of the unlabeled training image x in the third probability data and the probability value corresponding to the i-th pixel of the unlabeled training image x in the fourth probability data.

[0080] Step 2023: Determine the network loss based on the first cross-entropy loss and the second cross-entropy loss.

[0081] The first cross-entropy loss and the second cross-entropy loss can be weighted according to preset weights to obtain the network loss. The specific weights can be set according to actual needs.

[0082] For example, the network loss L can be represented as follows:

[0083] L=a1L S +α2L U

[0084] Where α1 and α2 represent the first cross-entropy loss L, respectively. S Second cross-entropy loss L UThe weights can be set according to actual needs. For example, they can be set to α1 = 1 and α2 = 1. Of course, they can also be set to other values ​​according to actual needs. There are no specific limitations.

[0085] Step 2024: Update the network parameters of the first semi-supervised semantic segmentation model based on the network loss to obtain the second semi-supervised semantic segmentation model.

[0086] Specifically, any feasible gradient descent method can be used to update the network parameters of the first semi-supervised semantic segmentation model to obtain the second semi-supervised semantic segmentation model.

[0087] This embodiment updates the semi-supervised semantic segmentation model by combining cross-supervised learning with labeled training image data using unlabeled training image data, thereby promoting model convergence and continuously improving model performance.

[0088] In an optional embodiment, step 203, based on the second processing result, uses a preset active learning rule to determine at least one image to be labeled in the first unlabeled training image data, including:

[0089] Step 2031: Based on the third probability data, the fourth probability data, the first pseudo-label data, the second pseudo-label data, and the preset active learning rules, determine the image to be labeled in the first unlabeled training image data.

[0090] The third probability data and the fourth probability data are the prediction results of the first branch network and the second branch network for the first unlabeled training image data, respectively. The first pseudo-label data and the second pseudo-label data are obtained by encoding based on the third probability data and the fourth probability data, respectively. Active learning rules can be set to address the confirmation error caused by the prediction error of the unlabeled training image data. Then, based on the third probability data, the fourth probability data, the first pseudo-label data, the second pseudo-label data and the active learning rules, images with higher value to be labeled can be determined.

[0091] This embodiment combines the third and fourth probability data of the prediction results of the two branch networks on the unlabeled training image data, as well as the first pseudo-label data determined by the third probability data and the second pseudo-label data determined by the fourth probability data, to identify images with higher value to be labeled. This is used to improve the distribution of labeled training image data and helps to improve model performance.

[0092] In an optional embodiment, step 310, which determines the first pseudo-label data based on the third probability data included in the second processing result, includes: performing one-hot encoding on the third probability data to obtain the first pseudo-label data.

[0093] In an optional embodiment, step 320, which determines the second pseudo-label data based on the fourth probability data included in the second processing result, includes: performing one-hot encoding on the fourth probability data to obtain the second pseudo-label data.

[0094] In this embodiment, the third and fourth probability data are encoded using one-hot encoding to obtain the first and second pseudo-label data, which are used for cross-supervised learning of unlabeled training image data. This can promote model convergence and optimize model performance.

[0095] Figure 4 This is a flowchart illustrating step 2031 provided in an exemplary embodiment of this disclosure.

[0096] In an optional embodiment, step 2031, which determines at least one image to be labeled from the first unlabeled training image data based on the third probability data, the fourth probability data, the first pseudo-label data, the second pseudo-label data, and the preset active learning rules, includes:

[0097] Step 20311: Based on the third probability data, the fourth probability data, and the preset uncertainty rules, determine the prediction result uncertainty data. The prediction result uncertainty data represents the uncertainty of the prediction probability of the first branch network and the second branch network. The prediction result uncertainty data includes the prediction result uncertainty map corresponding to each unlabeled training image in the first unlabeled training image data.

[0098] The preset uncertainty rules can be set according to actual needs. For example, they can include at least one of entropy-based uncertainty rules and KL (Kullback-Leibler) divergence-based uncertainty rules. Specifically, for entropy-based uncertainty rules, the entropy of each pixel in each unlabeled training image can be determined based on third and fourth probability data, respectively, obtaining the entropy data corresponding to the third and fourth probability data, and then combining the entropy data of both to determine the prediction result uncertainty data. For KL divergence-based uncertainty rules, the KL divergence of each pixel in each unlabeled training image can be determined based on third and fourth probability data, respectively, obtaining the divergence data corresponding to the third and fourth probability data, and then combining the divergence data of both to determine the prediction result uncertainty data. The prediction result uncertainty map is a tensor corresponding to the unlabeled training image, and includes the uncertainty value corresponding to each pixel of the corresponding unlabeled training image. This uncertainty value characterizes the degree of uncertainty of the prediction result for that pixel.

[0099] Step 20312: Based on the first pseudo-label data and the second pseudo-label data, determine the pseudo-label inconsistency data. The pseudo-label inconsistency data characterizes the inconsistency between the pseudo-labels generated by the prediction probabilities of the first branch network and the second branch network, respectively. The pseudo-label inconsistency data includes the pseudo-label inconsistency map corresponding to each unlabeled training image in the first unlabeled training image data.

[0100] The rules for determining pseudo-label inconsistency data can be set according to actual needs. Similar to the uncertainty map of prediction results, the pseudo-label inconsistency map is a tensor corresponding to the unlabeled training image, including the inconsistency value corresponding to each pixel of the unlabeled training image. This inconsistency value represents whether the pseudo-labels obtained by the pixel in the two branch networks are consistent. For example, 1 indicates consistency and 0 indicates inconsistency, or it can be the other way around. The specific settings can be set according to actual needs.

[0101] Steps 20311 and 20312 are not in any particular order.

[0102] Step 20313: Based on the uncertainty data of the prediction results, the inconsistency data of pseudo-labels, and the preset query rules, determine the query scores corresponding to each unlabeled training image in the first unlabeled training image data.

[0103] The preset query rules can be set according to actual needs. For example, based on the inconsistency data of pseudo-labels, the uncertainty value of inconsistent pixels can be queried from the uncertainty data of prediction results to determine the query score corresponding to each unlabeled training image. The query score corresponding to each unlabeled training image is a score value.

[0104] Step 20314: Based on the query scores corresponding to each unlabeled training image, determine the first number of unlabeled training images with larger query scores as images to be labeled.

[0105] The initial number can be set according to actual needs. Based on the query scores corresponding to each unlabeled training image, the largest initial number of unlabeled training images are selected as the images to be labeled. For example, ranked by score from highest to lowest, the top 10 unlabeled training images are selected as the images to be labeled. Specific limitations are not specified.

[0106] It should be noted that, for the case where 1 represents inconsistency (or mismatch) and 0 represents consistency (or a match) in pseudo-label inconsistent data, the first number of unlabeled training images with higher query scores are selected as the images to be labeled. Conversely, for the case where 0 represents inconsistency (or mismatch) and 1 represents consistency (or a match) in pseudo-label inconsistent data, the first number of unlabeled training images with lower query scores are selected as the images to be labeled.

[0107] In an optional embodiment, after obtaining the query scores corresponding to each unlabeled training image in step 20313, the process may include: sorting the unlabeled training images according to the sorting method corresponding to the inconsistency representation of the pseudo-label inconsistency map based on the query scores, and selecting the first number of unlabeled training images with the highest sorting order as images to be labeled. The inconsistency representation includes a first method and a second method. The sorting method corresponding to the first method is ascending order, and the sorting method corresponding to the second method is descending order. In the first method, 0 indicates inconsistency (or mismatch), and 1 indicates consistency (or matching). In the second method, 1 indicates inconsistency (or mismatch), and 0 indicates consistency (or matching).

[0108] This embodiment determines the uncertainty map of the prediction results and the pseudo-label inconsistency map of each unlabeled training image, and then combines the two to determine the query score of the unlabeled training image. Based on the score, the images with higher value to be labeled are selected for labeling, which further improves the effectiveness of labeling and can better improve the distribution of labeled training image data.

[0109] In an optional embodiment, the third probability data includes a third probability map corresponding to each unlabeled training image, and the third probability map corresponding to each unlabeled training image includes the probability values ​​of each pixel in the unlabeled training image belonging to each type predicted by the first branch network; the fourth probability data includes a fourth probability map corresponding to each unlabeled training image, and the fourth probability map corresponding to each unlabeled training image includes the probability values ​​of each pixel in the unlabeled training image belonging to each type predicted by the second branch network; step 20311, based on the third probability data, the fourth probability data, and the preset uncertainty rules, determines the uncertainty data of the prediction result, including:

[0110] For any unlabeled training image, based on the third probability map corresponding to the unlabeled training image and the preset entropy determination rule, the first entropy corresponding to each pixel of the third probability map is determined; based on the fourth probability map corresponding to the unlabeled training image and the preset entropy determination rule, the second entropy corresponding to each pixel of the fourth probability map is determined; based on the first entropy corresponding to each pixel of the third probability map and the second entropy corresponding to each pixel of the fourth probability map, the prediction result uncertainty map corresponding to the unlabeled training image is determined; based on the prediction result uncertainty maps corresponding to each unlabeled training image, the prediction result uncertainty data is determined.

[0111] The preset entropy determination rule can be set according to actual needs.

[0112] For example, for each unlabeled training image x, any pixel of it can be represented as x. (i，j) (i, j) represents the coordinates or index of a pixel in the image, x (i，j)entropy It can be represented as follows:

[0113]

[0114] Wherein, P(x (i，j，c) ) represents pixel x in the probability map (third probability map or fourth probability map) corresponding to the unlabeled training image x. (i，j) The probability of belonging to the c-th type, where C represents the total number of types.

[0115] pixel x (i，j) The corresponding first and second entropies can be obtained from the third and fourth probability diagrams respectively, according to the above formula, which will not be elaborated further. (i，j) The first entropy can be expressed as x (i，j) The second entropy can be expressed as Then, based on the first entropy corresponding to each pixel, we can obtain the first entropy map corresponding to the unlabeled training image x, which can be represented as follows: Similarly, based on the second entropy corresponding to each pixel, we can obtain the second entropy map corresponding to the unlabeled training image x, which can be represented as follows: The first entropy diagram Second Entropy Diagram The fusion process yields the uncertainty map of the prediction result corresponding to the unlabeled training image x. The uncertainty map of the prediction result can be represented as H(x). The fusion of the first entropy map and the second entropy map can be carried out in any feasible way, such as averaging or summing the two. The specific method can be set according to actual needs.

[0116] In an optional embodiment, the first and second entropies of each pixel can be fused first (by averaging or summing, etc.), and then an uncertainty map of the prediction result can be obtained based on the fused entropy of each pixel. The specific processing flow can be set according to actual needs.

[0117] This embodiment calculates the entropy map of the third probability map and the fourth probability map for each unlabeled training image, and then fuses the entropy maps obtained from the two to obtain the uncertainty map of the prediction result of the unlabeled training image. This map is used to determine the image to be labeled, which can help obtain the image to be labeled with higher value, thereby effectively improving the performance of the model with a smaller labeling cost.

[0118] In an optional embodiment, the first pseudo-label data includes first-type pseudo-labels corresponding to each pixel of each unlabeled training image determined based on the third probability data; the second pseudo-label data includes second-type pseudo-labels corresponding to each pixel of each unlabeled training image determined based on the fourth probability data; step 20312, determining pseudo-label inconsistency data based on the first and second pseudo-label data, includes:

[0119] For any unlabeled training image, based on the matching of the first type of pseudo-label and the second type of pseudo-label corresponding to each pixel of the unlabeled training image, determine the inconsistency value corresponding to each pixel; based on the inconsistency value corresponding to each pixel, determine the pseudo-label inconsistency map corresponding to the unlabeled training image; based on the pseudo-label inconsistency maps corresponding to each unlabeled training image, determine the pseudo-label inconsistency data.

[0120] The matching condition can include both matching and non-matching. Matching means that the first type of pseudo-label and the second type of pseudo-label of the pixel are the same, that is, they belong to the same type. The specific representation of the inconsistency value corresponding to each pixel can be set according to actual needs. For example, it can be represented by 1 and 0, where 1 indicates that the first type of pseudo-label and the second type of pseudo-label of the pixel match, and 0 indicates non-matching. Alternatively, it can be represented by 1 for non-matching and 0 for matching, or other methods can be used. There is no specific limitation. Each pixel in the unlabeled training image corresponds to an inconsistency value, and the inconsistency values ​​of all pixels constitute the pseudo-label inconsistency map corresponding to the unlabeled training image. The pseudo-label inconsistency map of all unlabeled training images serves as the pseudo-label inconsistency data corresponding to the first unlabeled training image data.

[0121] This embodiment obtains a pseudo-label inconsistency map corresponding to the unlabeled training image by matching the first type of pseudo-label and the second type of pseudo-label corresponding to each pixel of the unlabeled training image. This map is then used to address the uncertainty in the prediction results. Figure 1 This method determines the query score corresponding to the unlabeled training images, thereby identifying higher-value images to be labeled from among the unlabeled training images, further improving the effectiveness of labeled data for model training, and thus better improving model performance.

[0122] In an optional embodiment, based on the matching of the first type of pseudo-label and the second type of pseudo-label corresponding to each pixel of the unlabeled training image, the inconsistency value corresponding to each pixel is determined, including:

[0123] In response to a match between the first type pseudo-label and the second type pseudo-label corresponding to any pixel in the unlabeled training image, the inconsistency value corresponding to that pixel is determined to be 0; in response to a mismatch between the first type pseudo-label and the second type pseudo-label corresponding to any pixel in the unlabeled training image, the inconsistency value corresponding to that pixel is determined to be 1.

[0124] In this process, the inconsistency value of matching pixels is set to 0, and the inconsistency value of non-matching pixels is set to 1. The resulting pseudo-label inconsistency map is equivalent to a mask map of inconsistent pixels, which facilitates the determination of subsequent query scores and improves processing efficiency.

[0125] In an optional embodiment, step 20313, which determines the query score corresponding to each unlabeled training image in the first unlabeled training image data based on the prediction result uncertainty data, pseudo-label inconsistency data, and preset query rules, includes:

[0126] For any unlabeled training image, the pseudo-label inconsistency map and the prediction result uncertainty map corresponding to the unlabeled training image are multiplied element by element to obtain the multiplied result map; the elements in the multiplied result map are averaged to obtain the average value; the average value is used as the query score corresponding to the unlabeled training image.

[0127] Element-wise multiplication refers to multiplying the pixel values ​​at corresponding positions in the pseudo-label inconsistency map and the prediction result uncertainty map. Since the pseudo-label inconsistency map is essentially a mask of inconsistent pixels, element-wise multiplication allows us to retrieve the uncertainty values ​​of the inconsistent pixel portion from the prediction result uncertainty map, resulting in a multiplied image. In this multiplied image, the pixel values ​​of the inconsistent pixel portion retain their values ​​from the prediction result uncertainty map, while the pixel values ​​of the consistent pixel portion are set to 0. The elements in the multiplied image are then averaged, and this average is used as the query score for the unlabeled training image. This query score comprehensively considers both the uncertainty of the prediction probability and the inconsistency of the pseudo-label. Therefore, it can obtain images with high uncertainty and inconsistency, indicating that the current model has difficulty predicting these images. These images can then be used as unlabeled images, labeled using a specific method to obtain accurate labels, and then used as labeled training images for supervised model training. This allows the model to learn the predictions of these difficult samples, thereby improving model performance.

[0128] In one optional example, Figure 5This is a schematic diagram illustrating the overall principle of cross-supervision and active learning provided in an exemplary embodiment of this disclosure. This example only considers the processing of an unlabeled training image. Pseudo-label 1 includes first-type pseudo-labels corresponding to each pixel of the unlabeled training image obtained by encoding based on a third probability map. Pseudo-label 2 includes second-type pseudo-labels corresponding to each pixel of the unlabeled training image determined based on fourth probability data. The pseudo-label inconsistency rule can refer to the rule for determining the inconsistency value corresponding to each pixel based on the matching of the aforementioned first-type and second-type pseudo-labels corresponding to each pixel of the unlabeled training image. For any unlabeled training image in the first unlabeled training image data, a third probability map corresponding to the unlabeled training image is obtained based on a first branch network, and a fourth probability map corresponding to the unlabeled training image is obtained based on a second branch network. Pseudo-label 1 is obtained by encoding the third probability map, serving as the supervision label corresponding to the fourth probability map. Pseudo-label 2 is obtained by encoding the fourth probability map, serving as the supervision label corresponding to the third probability map. The third and fourth probability maps are processed according to preset uncertainty rules to obtain a prediction result uncertainty map. Pseudo-label 1 and pseudo-label 2 are processed according to pseudo-label inconsistency rules to obtain a pseudo-label inconsistency map. The pseudo-label inconsistency map and the prediction result uncertainty map are processed according to preset query rules to obtain the query score corresponding to the unlabeled training image. Then, all unlabeled training images can be sorted in ascending order (0 indicates no match, 1 indicates a match) or descending order (1 indicates no match, 0 indicates a match) according to the query score. The first number of unlabeled training images with the highest ranking are selected as the images to be labeled.

[0129] In an optional embodiment, step 205, which updates the first labeled training image data and the first unlabeled training image data based on each image to be labeled and the labels corresponding to each image to be labeled, to obtain updated second labeled training image data and second unlabeled training image data, includes:

[0130] Step 2051: Take each image to be labeled and the corresponding label of each image to be labeled, and the first labeled training image data as the second labeled training image data.

[0131] Step 2052: Filter out each image to be labeled from the first unlabeled training image data to obtain the second unlabeled training image data.

[0132] Steps 2051 and 2052 are not in any particular order.

[0133] This embodiment obtains the labels of the hard-to-label images and transfers them from the unlabeled training image data to the labeled training image data, which can effectively improve the distribution of the labeled training image data and thus help improve model performance.

[0134] In an optional embodiment, step 206, training the second semi-supervised semantic segmentation model based on the second labeled training image data and the second unlabeled training image data, includes:

[0135] Step 2061: Use the second labeled training image data as the first labeled training image data, use the second unlabeled training image data as the first unlabeled training image data, use the second semi-supervised semantic segmentation model as the first semi-supervised semantic segmentation model, and repeatedly execute the steps of obtaining the first processing result of the first semi-supervised semantic segmentation model on the first labeled training image data and the second processing result of the first semi-supervised semantic segmentation model on the first unlabeled training image data.

[0136] Specifically, the second labeled training image data is used as the first labeled training image data, the second unlabeled training image data is used as the first unlabeled training image data, and the second semi-supervised semantic segmentation model is used as the first semi-supervised semantic segmentation model. Starting from step 201, the next iteration begins. The specific iteration process is the same as the above process, and will not be repeated here.

[0137] Step 207, in response to the satisfaction of the training termination condition, obtains the target semantic segmentation model, including:

[0138] Step 2071: In response to the network loss meeting the preset conditions or the number of iterations reaching the preset threshold, determine that the training termination condition is met and obtain the target semi-supervised semantic segmentation model.

[0139] Step 2072: Use the first or second branch network of the target semi-supervised semantic segmentation model as the target semantic segmentation model.

[0140] The determination of whether the second semi-supervised semantic segmentation model obtained by any iteration process meets the training termination condition can be implemented during the execution of step 202 after step 201 is completed in the next iteration process of the iteration process. The determination of whether the network loss meets the preset condition or whether the number of iterations reaches the preset number threshold can be based on the determined network loss situation. The specific settings can be set according to actual needs.

[0141] The embodiments described above can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit them.

[0142] The training method for any semantic segmentation model provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, the training method for any semantic segmentation model provided in this disclosure can be executed by a processor, such as by a processor executing the training method for any semantic segmentation model mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0143] Figure 6 This is a schematic flowchart of an image semantic segmentation method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically, for example, in-vehicle computing platforms. Figure 6 As shown, it includes the following steps:

[0144] Step 401: Obtain the image data to be processed.

[0145] Step 402: Based on the pre-obtained target semantic segmentation model, perform semantic segmentation on the image data to be processed to obtain the semantic segmentation result corresponding to the image data to be processed.

[0146] The target semantic segmentation model is obtained through the training method of the semantic segmentation model provided in any of the above embodiments, as detailed in the foregoing embodiments, and will not be repeated here.

[0147] The semantic segmentation method for any image provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, the semantic segmentation method for any image provided in this disclosure can be executed by a processor, such as by a processor executing the semantic segmentation method for any image mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0148] Exemplary device

[0149] Figure 7 This is a schematic diagram of the structure of a training apparatus for a semantic segmentation model provided in an exemplary embodiment of this disclosure. The apparatus of this embodiment can be used to implement corresponding training method embodiments of the semantic segmentation model of this disclosure, such as... Figure 7 The device shown includes: a first acquisition module 501, a first processing module 502, a second processing module 503, a second acquisition module 504, a third processing module 505, a fourth processing module 506, and a fifth processing module 507.

[0150] The first acquisition module 501 is used to acquire the first processing result of the first semi-supervised semantic segmentation model on the first labeled training image data and the second processing result of the first semi-supervised semantic segmentation model on the first unlabeled training image data.

[0151] The first processing module 502 is used to update the first semi-supervised semantic segmentation model based on the first processing result and the second processing result to obtain the second semi-supervised semantic segmentation model.

[0152] The second processing module 503 is used to determine at least one image to be labeled in the first unlabeled training image data based on the second processing result and by adopting a preset active learning rule.

[0153] The second acquisition module 504 is used to acquire the labels corresponding to each image to be labeled.

[0154] The third processing module 505 is used to update the first labeled training image data and the first unlabeled training image data based on each image to be labeled and the corresponding label of each image to be labeled, so as to obtain the updated second labeled training image data and the second unlabeled training image data.

[0155] The fourth processing module 506 is used to train the second semi-supervised semantic segmentation model based on the second labeled training image data and the second unlabeled training image data.

[0156] The fifth processing module 507 is used to obtain the target semantic segmentation model in response to the satisfaction of the training termination condition.

[0157] Figure 8 This is a schematic diagram of the structure of a training apparatus for a semantic segmentation model provided in another exemplary embodiment of this disclosure.

[0158] In an optional embodiment, the second processing result includes the third probability data corresponding to the first unlabeled training image data in the first branch network of the first semi-supervised semantic segmentation model and the fourth probability data corresponding to the second branch network of the first semi-supervised semantic segmentation model; the apparatus of this disclosure further includes:

[0159] The first determining module 610 is used to determine the first pseudo-label data based on the third probability data included in the second processing result.

[0160] The second determining module 620 is used to determine the second pseudo-label data based on the fourth probability data included in the second processing result.

[0161] In an optional embodiment, the first processing result includes first probability data corresponding to the first branch network of the first labeled training image data in the first semi-supervised semantic segmentation model and second probability data corresponding to the second branch network of the first semi-supervised semantic segmentation model; the first processing module 502 includes: a first processing unit 5021, a second processing unit 5022, a third processing unit 5023 and a fourth processing unit 5024.

[0162] The first processing unit 5021 is used to determine the first cross-entropy loss based on the first probability data, the second probability data, and the first label data in the first labeled training image data.

[0163] The second processing unit 5022 is used to determine the second cross-entropy loss based on the third probability data, the second pseudo-label data, the fourth probability data, and the first pseudo-label data.

[0164] The third processing unit 5023 is used to determine the network loss based on the first cross-entropy loss and the second cross-entropy loss.

[0165] The fourth processing unit 5024 is used to update the network parameters of the first semi-supervised semantic segmentation model based on the network loss to obtain the second semi-supervised semantic segmentation model.

[0166] In an optional embodiment, the second processing module 503 includes:

[0167] The fifth processing unit 5031 is used to determine the image to be labeled in the first unlabeled training image data based on the third probability data, the fourth probability data, the first pseudo-label data, the second pseudo-label data, and the preset active learning rules.

[0168] In an optional embodiment, the first determining module 610 is specifically used to: perform one-hot encoding on the third probability data to obtain the first pseudo-label data.

[0169] In an optional embodiment, the second determining module 620 is specifically used to: perform one-hot encoding on the fourth probability data to obtain the second pseudo-label data.

[0170] Figure 9 This is a schematic diagram of the structure of the fifth processing unit 5031 provided in an exemplary embodiment of the present disclosure.

[0171] In an optional embodiment, the fifth processing unit 5031 includes: a first processing subunit 50311, a second processing subunit 50312, a third processing subunit 50313, and a fourth processing subunit 50314.

[0172] The first processing subunit 50311 is used to determine the prediction result uncertainty data based on the third probability data, the fourth probability data and the preset uncertainty rules. The prediction result uncertainty data represents the uncertainty of the prediction probability of the first branch network and the second branch network. The prediction result uncertainty data includes the prediction result uncertainty map corresponding to each unlabeled training image in the first unlabeled training image data.

[0173] The second processing subunit 50312 is used to determine pseudo-label inconsistency data based on the first pseudo-label data and the second pseudo-label data. The pseudo-label inconsistency data characterizes the inconsistency of pseudo-labels generated by the prediction probabilities of the first branch network and the second branch network, respectively. The pseudo-label inconsistency data includes pseudo-label inconsistency maps corresponding to each unlabeled training image in the first unlabeled training image data.

[0174] The third processing subunit 50313 is used to determine the query score corresponding to each unlabeled training image in the first unlabeled training image data based on the uncertainty data of the prediction results, the inconsistency data of pseudo-labels and the preset query rules.

[0175] The fourth processing subunit 50314 is used to determine the first number of unlabeled training images with larger query scores as images to be labeled based on the query scores corresponding to each unlabeled training image.

[0176] In an optional embodiment, the third probability data includes a third probability map corresponding to each unlabeled training image, and the third probability map corresponding to each unlabeled training image includes the probability values ​​of each pixel in the unlabeled training image belonging to each type predicted by the first branch network; the fourth probability data includes a fourth probability map corresponding to each unlabeled training image, and the fourth probability map corresponding to each unlabeled training image includes the probability values ​​of each pixel in the unlabeled training image belonging to each type predicted by the second branch network; the first processing subunit 50311 is specifically used for:

[0177] For any unlabeled training image, based on the third probability map corresponding to the unlabeled training image and the preset entropy determination rule, the first entropy corresponding to each pixel of the third probability map is determined; based on the fourth probability map corresponding to the unlabeled training image and the preset entropy determination rule, the second entropy corresponding to each pixel of the fourth probability map is determined; based on the first entropy corresponding to each pixel of the third probability map and the second entropy corresponding to each pixel of the fourth probability map, the prediction result uncertainty map corresponding to the unlabeled training image is determined; based on the prediction result uncertainty maps corresponding to each unlabeled training image, the prediction result uncertainty data is determined.

[0178] In an optional embodiment, the first pseudo-label data includes a first type of pseudo-label corresponding to each pixel of each unlabeled training image determined based on the third probability data; the second pseudo-label data includes a second type of pseudo-label corresponding to each pixel of each unlabeled training image determined based on the fourth probability data; the second processing subunit 50312 is specifically used for:

[0179] For any unlabeled training image, based on the matching of the first type of pseudo-label and the second type of pseudo-label corresponding to each pixel of the unlabeled training image, determine the inconsistency value corresponding to each pixel; based on the inconsistency value corresponding to each pixel, determine the pseudo-label inconsistency map corresponding to the unlabeled training image; based on the pseudo-label inconsistency maps corresponding to each unlabeled training image, determine the pseudo-label inconsistency data.

[0180] In an optional embodiment, the second processing subunit 50312 is specifically used for:

[0181] In response to a match between the first type pseudo-label and the second type pseudo-label corresponding to any pixel in the unlabeled training image, the inconsistency value corresponding to that pixel is determined to be 0; in response to a mismatch between the first type pseudo-label and the second type pseudo-label corresponding to any pixel in the unlabeled training image, the inconsistency value corresponding to that pixel is determined to be 1.

[0182] In an optional embodiment, the third processing subunit 50313 is specifically used for:

[0183] For any unlabeled training image, the pseudo-label inconsistency map and the prediction result uncertainty map corresponding to the unlabeled training image are multiplied element by element to obtain the multiplied result map; the elements in the multiplied result map are averaged to obtain the average value; the average value is used as the query score corresponding to the unlabeled training image.

[0184] In an optional embodiment, the third processing module 505 includes:

[0185] The first determining unit 5051 is used to take each image to be labeled and the corresponding label of each image to be labeled, and the first labeled training image data as the second labeled training image data.

[0186] The second determining unit 5052 is used to filter out each image to be labeled from the first unlabeled training image data to obtain the second unlabeled training image data.

[0187] In an optional embodiment, the fourth processing module 506 includes:

[0188] The sixth processing unit 5061 is used to take the second labeled training image data as the first labeled training image data, take the second unlabeled training image data as the first unlabeled training image data, take the second semi-supervised semantic segmentation model as the first semi-supervised semantic segmentation model, and repeatedly execute the steps of obtaining the first processing result of the first semi-supervised semantic segmentation model on the first labeled training image data and the second processing result of the first semi-supervised semantic segmentation model on the first unlabeled training image data.

[0189] The fifth processing module 507 includes:

[0190] The seventh processing unit 5071 is used to determine that the training termination condition is met and obtain the target semi-supervised semantic segmentation model in response to the network loss meeting the preset condition or the number of iterations reaching the preset threshold.

[0191] The eighth processing unit 5072 is used to take the first branch network or the second branch network of the target semi-supervised semantic segmentation model as the target semantic segmentation model.

[0192] Figure 10 This is a schematic diagram of the structure of an image semantic segmentation apparatus provided in an exemplary embodiment of this disclosure. The apparatus of this embodiment can be used to implement the corresponding image semantic segmentation method embodiments of this disclosure, such as... Figure 10 The apparatus shown includes a third acquisition module 701 and a sixth processing module 702.

[0193] The third acquisition module 701 is used to acquire the image data to be processed.

[0194] The sixth processing module 702 is used to perform semantic segmentation on the image data to be processed based on the pre-obtained target semantic segmentation model, and obtain the semantic segmentation result corresponding to the image data to be processed.

[0195] The target semantic segmentation model is obtained through the training method of the semantic segmentation model provided in any of the above embodiments, as detailed in the foregoing embodiments, and will not be repeated here.

[0196] Exemplary electronic devices

[0197] This disclosure also provides an electronic device, including: a memory for storing computer programs;

[0198] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the training method of the semantic segmentation model described in any of the above embodiments of the present disclosure, or implements the semantic segmentation method of the image described in any of the above embodiments of the present disclosure.

[0199] Figure 11 This is a schematic diagram of an application embodiment of the electronic device disclosed herein. In this embodiment, the electronic device 10 includes one or more processors 11 and a memory 12.

[0200] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0201] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0202] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0203] For example, the input device 13 may be the microphone or microphone array described above, used to capture the input signal of the sound source.

[0204] In addition, the input device 13 may also include, for example, a keyboard, a mouse, etc.

[0205] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0206] Of course, for the sake of simplicity, Figure 11 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0207] Exemplary computer program products and computer-readable storage media

[0208] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0209] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0210] Furthermore, embodiments of this disclosure may also be computer-readable storage media having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0211] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0212] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0213] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0214] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0215] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0216] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0217] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0218] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for training a semantic segmentation model, comprising: Obtain the first processing result of the first semi-supervised semantic segmentation model on the first labeled training image data and the second processing result of the first semi-supervised semantic segmentation model on the first unlabeled training image data; The first semi-supervised semantic segmentation model is updated based on the first processing result and the second processing result to obtain the second semi-supervised semantic segmentation model; Based on the second processing result, a preset active learning rule is used to determine at least one image to be labeled in the first unlabeled training image data; Obtain the labels corresponding to each of the images to be labeled; Based on each of the images to be labeled and the labels corresponding to each of the images to be labeled, the first labeled training image data and the first unlabeled training image data are updated to obtain the updated second labeled training image data and the second unlabeled training image data. The second semi-supervised semantic segmentation model is trained based on the second labeled training image data and the second unlabeled training image data. The target semantic segmentation model is obtained in response to the satisfaction of the training termination condition; The second processing result includes the third probability data corresponding to the first branch network of the first unlabeled training image data in the first branch network of the first semi-supervised semantic segmentation model and the fourth probability data corresponding to the second branch network of the first semi-supervised semantic segmentation model. The method further includes: Based on the third probability data, the first pseudo-label data is determined; Based on the fourth probability data, the second pseudo-label data is determined; Based on the second processing result, the step of determining at least one unlabeled image in the first unlabeled training image data using a preset active learning rule includes: Based on the third probability data, the fourth probability data, and the preset uncertainty rule, the prediction result uncertainty data is determined. The prediction result uncertainty data characterizes the uncertainty of the prediction probability of the first branch network and the second branch network. The prediction result uncertainty data includes the prediction result uncertainty map corresponding to each unlabeled training image in the first unlabeled training image data. Based on the first pseudo-label data and the second pseudo-label data, pseudo-label inconsistency data is determined. The pseudo-label inconsistency data characterizes the inconsistency of pseudo-labels generated by the prediction probabilities of the first branch network and the second branch network, respectively. The pseudo-label inconsistency data includes pseudo-label inconsistency maps corresponding to each unlabeled training image in the first unlabeled training image data. Based on the uncertainty data of the prediction results, the inconsistency data of the pseudo-labels, and the preset query rules, the query scores corresponding to each of the unlabeled training images in the first unlabeled training image data are determined. Based on the query scores corresponding to each of the unlabeled training images, a first number of unlabeled training images with larger query scores are determined as the images to be labeled.

2. The method according to claim 1, wherein, The third probability data includes a third probability map corresponding to each of the unlabeled training images, and the third probability map corresponding to each of the unlabeled training images includes the probability values ​​of each pixel in the unlabeled training image belonging to each type predicted by the first branch network; the fourth probability data includes a fourth probability map corresponding to each of the unlabeled training images, and the fourth probability map corresponding to each of the unlabeled training images includes the probability values ​​of each pixel in the unlabeled training image belonging to each type predicted by the second branch network. The determination of prediction result uncertainty data based on the third probability data, the fourth probability data, and preset uncertainty rules includes: For any of the unlabeled training images, based on the third probability map corresponding to the unlabeled training image and the preset entropy determination rule, the first entropy corresponding to each pixel of the third probability map is determined. Based on the fourth probability map corresponding to the unlabeled training image and the preset entropy determination rule, the second entropy corresponding to each pixel of the fourth probability map is determined. Based on the first entropy corresponding to each pixel of the third probability map and the second entropy corresponding to each pixel of the fourth probability map, the uncertainty map of the prediction result corresponding to the unlabeled training image is determined. Based on the uncertainty map of the prediction result corresponding to each of the unlabeled training images, the uncertainty data of the prediction result is determined.

3. The method according to claim 1, wherein, The first pseudo-label data includes a first type of pseudo-label corresponding to each pixel of each of the unlabeled training images determined based on the third probability data; the second pseudo-label data includes a second type of pseudo-label corresponding to each pixel of each of the unlabeled training images determined based on the fourth probability data. The step of determining the inconsistency data of pseudo-labels based on the first pseudo-label data and the second pseudo-label data includes: For any of the unlabeled training images, based on the matching situation of the first type of pseudo-label and the second type of pseudo-label corresponding to each pixel of the unlabeled training image, the inconsistency value corresponding to each pixel is determined. Based on the inconsistency values ​​corresponding to each pixel, the pseudo-label inconsistency map corresponding to the unlabeled training image is determined. Based on the pseudo-label inconsistency maps corresponding to each of the unlabeled training images, the pseudo-label inconsistency data is determined.

4. The method according to claim 3, wherein, The determination of the inconsistency value for each pixel based on the matching of the first type of pseudo-label and the second type of pseudo-label corresponding to each pixel of the unlabeled training image includes: In response to the first type of pseudo-label and the second type of pseudo-label matching for any pixel in the unlabeled training image, the inconsistency value corresponding to that pixel is determined to be 0; In response to a mismatch between the first type of pseudo-label and the second type of pseudo-label corresponding to any pixel in the unlabeled training image, the inconsistency value corresponding to that pixel is determined to be 1.

5. The method according to claim 1, wherein, The step of determining the query score corresponding to each unlabeled training image in the first unlabeled training image data based on the uncertainty data of the prediction results, the inconsistency data of the pseudo-labels, and the preset query rules includes: For any of the unlabeled training images, the pseudo-label inconsistency map and the prediction result uncertainty map corresponding to the unlabeled training image are multiplied element-wise to obtain the multiplication result map. The average value is obtained by averaging the elements in the multiplication result graph. The average value is used as the query score corresponding to the unlabeled training image.

6. The method according to claim 1, wherein, The determination of the first pseudo-label data based on the third probability data includes: The third probability data is one-hot encoded to obtain the first pseudo-label data; Based on the fourth probability data, the second pseudo-label data is determined, including: The fourth probability data is one-hot encoded to obtain the second pseudo-label data.

7. The method according to claim 1, wherein, The first processing result includes the first probability data corresponding to the first labeled training image data in the first branch network and the second probability data corresponding to the first branch network; The step of updating the first semi-supervised semantic segmentation model based on the first processing result and the second processing result to obtain a second semi-supervised semantic segmentation model includes: Based on the first probability data, the second probability data, and the first label data in the first labeled training image data, the first cross-entropy loss is determined. Based on the third probability data, the second pseudo-label data, the fourth probability data, and the first pseudo-label data, the second cross-entropy loss is determined. The network loss is determined based on the first cross-entropy loss and the second cross-entropy loss; The network parameters of the first semi-supervised semantic segmentation model are updated based on the network loss to obtain the second semi-supervised semantic segmentation model.

8. The method according to claim 1, wherein, The training of the second semi-supervised semantic segmentation model based on the second labeled training image data and the second unlabeled training image data includes: The second labeled training image data is used as the first labeled training image data, the second unlabeled training image data is used as the first unlabeled training image data, and the second semi-supervised semantic segmentation model is used as the first semi-supervised semantic segmentation model. The steps of obtaining the first processing result of the first semi-supervised semantic segmentation model on the first labeled training image data and the second processing result of the first semi-supervised semantic segmentation model on the first unlabeled training image data are repeated. The step of obtaining the target semantic segmentation model in response to the satisfaction of the training termination condition includes: When the network loss meets a preset condition or the number of iterations reaches a preset threshold, the training termination condition is determined to be met, and the target semi-supervised semantic segmentation model is obtained. The first branch network or the second branch network of the target semi-supervised semantic segmentation model is used as the target semantic segmentation model.

9. The method according to claim 1, wherein, The step of updating the first labeled training image data and the first unlabeled training image data based on each of the images to be labeled and the labels corresponding to each of the images to be labeled, to obtain the updated second labeled training image data and second unlabeled training image data, includes: Each of the images to be labeled and the labels corresponding to each of the images to be labeled, along with the first labeled training image data, are used as the second labeled training image data. Each of the images to be labeled is filtered out from the first unlabeled training image data to obtain the second unlabeled training image data.

10. A semantic segmentation method for images, comprising: Acquire the image data to be processed; Based on the pre-obtained target semantic segmentation model, semantic segmentation is performed on the image data to be processed to obtain the semantic segmentation result corresponding to the image data to be processed. The target semantic segmentation model is obtained by the training method of the semantic segmentation model as described in any one of claims 1-9.

11. A training device for a semantic segmentation model, comprising: The first acquisition module is used to acquire the first processing result of the first semi-supervised semantic segmentation model on the first labeled training image data and the second processing result of the first semi-supervised semantic segmentation model on the first unlabeled training image data. The first processing module is used to update the first semi-supervised semantic segmentation model based on the first processing result and the second processing result to obtain a second semi-supervised semantic segmentation model. The second processing module is used to determine at least one image to be labeled in the first unlabeled training image data based on the second processing result and by using a preset active learning rule. The second acquisition module is used to acquire the labels corresponding to each of the images to be labeled. The third processing module is used to update the first labeled training image data and the first unlabeled training image data based on each of the images to be labeled and the labels corresponding to each of the images to be labeled, so as to obtain the updated second labeled training image data and the second unlabeled training image data. The fourth processing module is used to train the second semi-supervised semantic segmentation model based on the second labeled training image data and the second unlabeled training image data; The fifth processing module is used to obtain the target semantic segmentation model in response to the satisfaction of the training termination condition; The second processing result includes the third probability data corresponding to the first branch network of the first unlabeled training image data in the first branch network of the first semi-supervised semantic segmentation model and the fourth probability data corresponding to the second branch network of the first semi-supervised semantic segmentation model. The device further includes: The first determining module is used to determine the first pseudo-label data based on the third probability data; The second determining module is used to determine the second pseudo-label data based on the fourth probability data; The second processing module includes: a fifth processing unit; The fifth processing unit includes: The first processing subunit is used to determine prediction result uncertainty data based on the third probability data, the fourth probability data and the preset uncertainty rule. The prediction result uncertainty data characterizes the uncertainty of the prediction probability of the first branch network and the second branch network. The prediction result uncertainty data includes prediction result uncertainty maps corresponding to each unlabeled training image in the first unlabeled training image data. The second processing subunit is used to determine pseudo-label inconsistency data based on the first pseudo-label data and the second pseudo-label data. The pseudo-label inconsistency data characterizes the inconsistency of pseudo-labels generated by the prediction probabilities of the first branch network and the second branch network, respectively. The pseudo-label inconsistency data includes pseudo-label inconsistency maps corresponding to each unlabeled training image in the first unlabeled training image data. The third processing subunit is used to determine the query score corresponding to each of the unlabeled training images in the first unlabeled training image data based on the prediction result uncertainty data, the pseudo-label inconsistency data and the preset query rules. The fourth processing subunit is used to determine, based on the query scores corresponding to each of the unlabeled training images, a first number of unlabeled training images with larger query scores as the images to be labeled.

12. A semantic segmentation apparatus for an image, comprising: The third acquisition module is used to acquire the image data to be processed; The sixth processing module is used to perform semantic segmentation on the image data to be processed based on the pre-obtained target semantic segmentation model, and obtain the semantic segmentation result corresponding to the image data to be processed; The target semantic segmentation model is obtained by the training method of the semantic segmentation model as described in any one of claims 1-9.

13. A computer-readable storage medium storing a computer program for executing a training method for a semantic segmentation model according to any one of claims 1-9, or executing a semantic segmentation method for an image according to claim 10.

14. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the training method of the semantic segmentation model according to any one of claims 1-9, or to implement the semantic segmentation method of the image according to claim 10.

Images