Semantic segmentation model training method and device, equipment and storage medium

Abstract

The application discloses a training method and device of a semantic segmentation model, equipment and a storage medium, and belongs to the field of semantic segmentation. The method comprises the following steps: acquiring a labeled sample and an unlabeled sample; inputting the labeled sample into a semantic segmentation model to obtain a labeled prediction map corresponding to the labeled sample, and obtaining a labeled loss based on the labeled prediction map and a true value label; performing an enhancement operation on the unlabeled sample to generate at least two enhanced unlabeled samples; inputting the at least two enhanced unlabeled samples into the semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two enhanced unlabeled samples respectively, and obtaining an unlabeled loss based on the at least two unlabeled prediction maps; and training the semantic segmentation model based on the labeled loss and the unlabeled loss to obtain a pre-trained semantic segmentation model. By enabling the semantic segmentation model to learn features of the unlabeled sample from different perspectives, potential information of the unlabeled sample is fully utilized, and the performance of the semantic segmentation model is improved.

Description

Technical Field

[0001] This application relates to the field of semantic segmentation, and in particular to a method, apparatus, device, and storage medium for training a semantic segmentation model. Background Technology

[0002] Semantic segmentation, a fundamental task in computer vision, is used to assign semantic categories to each pixel in an image and is widely applied in scenarios such as scene recognition, autonomous driving, and medical image analysis.

[0003] In related technologies, due to the high cost of labeling labeled image data (pixel-level annotation), semi-supervised learning is typically used to train semantic segmentation models. During semi-supervised learning training, the semantic segmentation model is usually trained simultaneously on a small number of labeled images (labeled samples) and a large number of unlabeled images (unlabeled samples).

[0004] However, the semi-supervised learning training method in related technologies simply combines labeled and unlabeled data, which limits the performance of semantic segmentation models. How to improve the performance of semantic segmentation models is a problem that needs to be solved. Summary of the Invention

[0005] This application provides a training method, apparatus, device, and storage medium for a semantic segmentation model, and the technical solution includes the following aspects.

[0006] According to one aspect of this application, a method for training a semantic segmentation model is provided, the method comprising:

[0007] Acquire labeled samples and unlabeled samples, wherein the labeled samples are image samples labeled with ground truth labels, and the unlabeled samples are image samples without ground truth labels;

[0008] The labeled sample is input into the semantic segmentation model to obtain the labeled prediction map corresponding to the labeled sample, and the labeled loss is obtained based on the labeled prediction map and the ground truth label;

[0009] An enhancement operation is performed on the unlabeled sample to generate at least two enhanced unlabeled samples, wherein the enhancement operation is used to perturb the unlabeled sample from at least two perspectives;

[0010] The at least two enhanced unlabeled samples are input into the semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two enhanced unlabeled samples, and unlabeled loss is obtained based on the at least two unlabeled prediction maps;

[0011] The semantic segmentation model is trained based on the labeled loss and the unlabeled loss to obtain a pre-trained semantic segmentation model.

[0012] According to one aspect of this application, a training apparatus for a semantic segmentation model is provided, the apparatus comprising:

[0013] The sample acquisition module is used to acquire labeled samples and unlabeled samples, wherein the labeled samples are image samples labeled with ground truth labels, and the unlabeled samples are image samples without the ground truth labels.

[0014] The sample prediction module is used to input the labeled sample into the semantic segmentation model to obtain the labeled prediction map corresponding to the labeled sample, and to obtain the labeled loss based on the labeled prediction map and the ground truth label;

[0015] An enhancement generation module is used to perform enhancement operations on the unlabeled samples to generate at least two enhanced unlabeled samples, wherein the enhancement operations are used to perturb the unlabeled samples from at least two perspectives.

[0016] The sample prediction module is used to input the at least two enhanced unlabeled samples into the semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two enhanced unlabeled samples respectively, and to obtain unlabeled loss based on the at least two unlabeled prediction maps;

[0017] The model training module is used to train the semantic segmentation model based on the labeled loss and the unlabeled loss to obtain a pre-trained semantic segmentation model.

[0018] According to another aspect of this application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores at least one computer program, the at least one computer program being loaded and executed by the processor to implement the training method of the semantic segmentation model as described above.

[0019] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to implement the training method of the semantic segmentation model as described above.

[0020] According to another aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium; a processor reads from the computer-readable storage medium and executes the computer program to implement the training method of the semantic segmentation model as described above.

[0021] The beneficial effects of the technical solutions provided in this application include at least the following:

[0022] By combining labeled and unlabeled samples for semi-supervised training of the semantic segmentation model, the unlabeled samples are first randomly augmented from at least two different perspectives to obtain at least two augmented unlabeled samples. An unlabeled loss is then derived based on these two augmented samples. This unlabeled loss is combined with the labeled loss corresponding to the labeled samples to train the semantic segmentation model. The unlabeled loss obtained from at least two augmented samples allows the semantic segmentation model to learn the features of unlabeled samples from different perspectives, enabling it to more comprehensively understand the semantic information within the unlabeled samples. This allows for more efficient utilization of the latent information in the unlabeled samples, improving the learning effect and performance of the semantic segmentation model. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of a training method for a semantic segmentation model provided in one embodiment of this application;

[0025] Figure 2 This is a schematic diagram of a computer system provided in one embodiment of this application;

[0026] Figure 3 This is a flowchart of a training method for a semantic segmentation model provided in one embodiment of this application;

[0027] Figure 4 This is a flowchart of a training method for a semantic segmentation model provided in one embodiment of this application;

[0028] Figure 5 This is a flowchart of a training method for a semantic segmentation model provided in one embodiment of this application;

[0029] Figure 6 This is a flowchart of a training method for a semantic segmentation model provided in one embodiment of this application;

[0030] Figure 7 This is a flowchart of a training method for a semantic segmentation model provided in one embodiment of this application;

[0031] Figure 8 This is a flowchart of a training method for a semantic segmentation model provided in one embodiment of this application;

[0032] Figure 9This is a flowchart of a training method for a semantic segmentation model provided in one embodiment of this application;

[0033] Figure 10 This is a structural block diagram of a training device for a semantic segmentation model provided in one embodiment of this application;

[0034] Figure 11 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0037] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0038] It should be understood that although the terms first, second, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0039] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user data. These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without user confirmation), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0040] First, let me introduce the relevant terms used in this application:

[0041] Semi-supervised learning (SSL) is a method in machine learning that falls between supervised learning and unsupervised learning. In supervised learning, the model learns from labeled samples, while in unsupervised learning, the model learns from unlabeled samples. Semi-supervised learning combines these two approaches, using a small number of labeled samples and a large number of unlabeled samples for training.

[0042] Semantic segmentation: Semantic segmentation refers to pixel-level image segmentation. For a given image (taking a two-dimensional image as an example), semantic segmentation assigns a category to each pixel in the image, ultimately outputting a semantic segmentation map. In the semantic segmentation map, the image is divided into several non-overlapping regions according to the category assigned to each pixel. Each region consists of pixels belonging to the same category, and pixels within the same region will exhibit consistency or similarity, while different regions will show obvious differences.

[0043] The purpose of semantic segmentation is to understand the content of an image at the pixel level and assign an object class to each pixel in the image. For example, an image containing elements such as people, vehicles, and buildings can be segmented into multiple regions, each representing a specific object.

[0044] True label: A true label refers to the correct and accurate label (category or value) for each sample in the dataset.

[0045] Pseudo-labels: Pseudo-labels are labels used when no real labels exist. In this embodiment, pseudo-labels are predicted labels generated by a pre-trained semantic segmentation model and used as real labels in subsequent training.

[0046] In related technologies, supervised learning is used to train semantic segmentation models. During supervised learning training, the model is trained using a large amount of labeled images. However, due to the high cost of labeling labeled image data (pixel-level annotation), semi-supervised learning has emerged as a training method for semantic segmentation models. Semi-supervised learning typically trains the model simultaneously using a small number of labeled images and a large number of unlabeled images. However, semi-supervised learning simply combines labeled and unlabeled data, limiting the performance of the semantic segmentation model. Improving the performance of semantic segmentation models is a problem that needs to be solved.

[0047] To address the aforementioned issues, this application proposes a training method for a semantic segmentation model, dividing the training into two stages. In the first stage, the semantic segmentation model is trained simultaneously using labeled and unlabeled samples. Random augmentation is performed on the unlabeled samples to generate at least two augmented unlabeled samples. An unlabeled loss is obtained based on the unlabeled prediction maps corresponding to these two augmented samples. The semantic segmentation model is then trained in the first stage using both labeled and unlabeled losses to obtain a pre-trained semantic segmentation model. In the second stage, the unlabeled samples are input into the pre-trained semantic segmentation model obtained in the first stage to obtain pseudo-labels for the unlabeled samples. These pseudo-labels are then combined with the unlabeled samples to obtain pseudo-labeled samples. These pseudo-labeled samples are then combined with labeled samples to obtain expanded samples. An expanded label loss is obtained based on the ground truth label and the expanded label prediction map of the expanded samples. The pre-trained semantic segmentation model is then trained using this expanded label loss, resulting in a fully trained semantic segmentation model.

[0048] This application employs a two-stage training approach. In the first stage, it combines the labeled loss (supervised loss) for labeled samples with the unlabeled loss (consistency loss) for unlabeled samples, enabling supervised and unsupervised learning to collaborate and complement each other during the semantic segmentation model training process. In the second stage, through pseudo-label generation and multiple training optimizations, the connection between the two (pseudo-labels and multi-round training) is further strengthened. After each training iteration, the parameters of the semantic segmentation model are updated, generating more accurate pseudo-labels, thus forming an organic whole. This tight integration allows the semantic segmentation model to better balance the learning of labeled and unlabeled samples, fully leveraging the advantages of semi-supervised learning and further improving the performance and generalization ability of the semantic segmentation model.

[0049] Reference Figure 1 The training method of the semantic segmentation model in the embodiments of this application is briefly described below:

[0050] For the first phase

[0051] In some embodiments, labeled sample 10 and unlabeled sample 20 are obtained. Optionally, labeled sample 10 is an image sample labeled with a truth value label 40, and unlabeled sample 20 is an image sample without a truth value label 40.

[0052] The truth labels are used to indicate the semantic category of each pixel in an image sample. For example, image sample A contains four categories: roads, pedestrians, buildings, and sky. The truth labels indicate the semantic category of each pixel in image sample A. For instance, the truth labels might indicate that the first set of pixels in image sample A belongs to "roads," the second set to "pedestrians," the third set to "buildings," and the fourth set to "sky."

[0053] • 10 labeled samples from the first phase

[0054] In some embodiments, labeled sample 10 is input into semantic segmentation model 30, and semantic segmentation model 30 predicts the labeled sample 10 to obtain the labeled prediction corresponding to labeled sample 10. Figure 11 Among them is label prediction. Figure 11 It contains the semantic prediction category for each pixel in the 10 labeled samples.

[0055] Optionally, based on labeled prediction Figure 11 The labeled loss is obtained by combining the ground truth label 40 with the labeled predictions of the semantic segmentation model 30. The labeled loss is used to indicate the labeled predictions made by the semantic segmentation model 30. Figure 11 The error between the semantic segmentation model 30 and the ground truth label 40. It can also be understood as the labeled loss used to indicate the error between the semantic predicted category of each pixel in the labeled sample 10 predicted by the semantic segmentation model 30 and the semantic category (semantic true category) of each pixel in the ground truth label 40.

[0056] In some embodiments, the labeled loss includes similarity coefficient loss and cross-entropy loss. Optionally, for labeled prediction... Figure 11 The first operation is performed on the ground truth label 40 to obtain the similarity coefficient loss. The similarity coefficient loss is used to measure the accuracy of labeled predictions. Figure 11 The degree of overlap with the ground truth label 40. Optionally, for labeled predictions... Figure 11 The second operation is performed with the ground truth label 40 to obtain the cross-entropy loss, which is used to measure the labeled prediction. Figure 11 The difference between the true label 40 and the ground truth label 40. The labeled loss is calculated based on the similarity coefficient loss and cross-entropy loss.

[0057] • Regarding the 20 unlabeled samples in the first phase

[0058] In some embodiments, an enhancement operation is performed on the unlabeled sample 20 to generate at least two enhanced unlabeled samples, wherein the enhancement operation is used to perturb the unlabeled sample 20 from at least two perspectives.

[0059] The following explanation uses at least two enhanced unlabeled samples, including a first enhanced unlabeled sample 21 and a second enhanced unlabeled sample 22, as an example. Optionally, a first enhancement operation is performed on the unlabeled sample 20 to generate the first enhanced unlabeled sample 21; a second enhancement operation is performed on the unlabeled sample 20 to generate the second enhanced unlabeled sample 22.

[0060] In some embodiments, the first enhancement operation and the second enhancement operation are enhancement operations from different perspectives. The first enhancement operation and the second enhancement operation include, but are not limited to, at least one of the following operations: a geometric transformation operation based on a first probability; a color transformation operation based on a second probability; a noise injection operation based on a third probability; and a mixing operation based on a fourth probability. Optionally, the geometric transformation operation includes, but is not limited to, at least one of the following: a random horizontal flip operation; a random vertical flip operation; a random rotation operation; a random cropping operation; and a random scaling operation. This application does not limit the scope of the geometric transformation operation.

[0061] For example, the first enhancement operation is a random horizontal flip operation, which yields a first enhanced unlabeled sample 21 after performing a random horizontal flip operation on the unlabeled sample 20. The first enhanced unlabeled sample 21 is flipped along the x-axis compared to the unlabeled sample 20. The second enhancement operation is a random rotation flip operation, which yields a second enhanced unlabeled sample 22 after performing a random rotation flip operation on the unlabeled sample 20. The second enhanced unlabeled sample 22 is rotated 90 degrees compared to the unlabeled sample 20.

[0062] In some embodiments, at least two enhanced unlabeled samples are input into the semantic segmentation model 30 to obtain unlabeled prediction maps corresponding to at least two enhanced unlabeled samples respectively. Optionally, a first enhanced unlabeled sample 21 is input into the semantic segmentation model 30, and the semantic segmentation model 30 predicts the first enhanced unlabeled sample 21 to obtain a first unlabeled prediction map 21a corresponding to the first enhanced unlabeled sample 21. The first unlabeled prediction map 21a contains the semantic prediction category for each pixel in the first enhanced unlabeled sample 21. Optionally, a second enhanced unlabeled sample 22 is input into the semantic segmentation model 30, and the semantic segmentation model 30 predicts the second enhanced unlabeled sample 22 to obtain a second unlabeled prediction map 22a corresponding to the second enhanced unlabeled sample 22. The second unlabeled prediction map 22a contains the semantic prediction category for each pixel in the second enhanced unlabeled sample 22.

[0063] Optionally, an unlabeled loss is obtained based on the first unlabeled prediction map 21a and the second unlabeled prediction map 22a. The unlabeled loss is used to indicate the error between the first unlabeled prediction map 21a and the second unlabeled prediction map 22a predicted by the semantic segmentation model 30. Alternatively, the unlabeled loss can be understood as indicating the error between the semantic prediction category of each pixel in the first enhanced unlabeled sample 21 and the semantic category (true semantic category) of each pixel in the second enhanced unlabeled sample 22.

[0064] In some embodiments, when calculating the unlabeled loss based on at least two unlabeled prediction maps, an inverse transformation operation is performed on the at least two unlabeled prediction maps to obtain at least two inversely transformed unlabeled prediction maps, and the unlabeled loss is obtained based on the at least two inversely transformed unlabeled prediction maps. Optionally, a first inverse transformation operation is performed on the first unlabeled prediction map 21a to obtain an inversely transformed first unlabeled prediction map, the first inverse transformation operation corresponding to the first enhancement operation. Optionally, a second inverse transformation operation is performed on the second unlabeled prediction map 22a to obtain an inversely transformed second unlabeled prediction map, the second inverse transformation operation corresponding to the second enhancement operation. The unlabeled loss is obtained based on the inversely transformed first unlabeled prediction map and the inversely transformed second unlabeled prediction map.

[0065] For example, the first enhancement operation is a random horizontal flip operation, where the first enhanced unlabeled sample 21 is flipped along the x-axis compared to the unlabeled sample 20. The second enhancement operation is a random rotation flip operation, where the second enhanced unlabeled sample 22 is rotated 90 degrees compared to the unlabeled sample 20. After predicting the first enhanced unlabeled sample 21 and the second enhanced unlabeled sample 22 based on the semantic segmentation model 30 to obtain the first unlabeled prediction image 21a and the second unlabeled prediction image 22a, a first inverse transformation operation is performed on the first unlabeled prediction image 21a, which is an inverse flip operation along the x-axis. A second inverse transformation operation is performed on the second unlabeled prediction image 22a, which is a 90-degree rotation inverse flip operation.

[0066] In some embodiments, the semantic segmentation model 30 is trained based on the labeled loss and the unlabeled loss described above, resulting in a pre-trained semantic segmentation model 30. Optionally, the semantic segmentation model 30 is trained based on a weighted sum of the labeled loss and the unlabeled loss.

[0067] For the second phase

[0068] In some embodiments, a pre-trained semantic segmentation model 30 is obtained through a first stage. Optionally, unlabeled samples 20 are input into the pre-trained semantic segmentation model 30, which predicts the unlabeled samples 20 to obtain pseudo-labels 50 corresponding to them. The pseudo-labels 50 and unlabeled samples 20 are combined to obtain pseudo-labeled samples 60. Optionally, expanded samples 70 are obtained based on the pseudo-labeled samples 60 and the labeled samples 10.

[0069] In some embodiments, the extended sample 70 is input into the pre-trained semantic segmentation model 30, and the semantic segmentation model 30 predicts the extended sample 70 to obtain the extended label prediction map 71 corresponding to the extended sample 70. The extended label prediction map 71 contains the semantic prediction category for each pixel in the extended sample 70.

[0070] Optionally, an expanded label loss is obtained based on the expanded label prediction map 71 and the ground truth label 40. The expanded label loss indicates the error between the expanded label prediction map 71 and the ground truth label 40 predicted by the pre-trained semantic segmentation model 30. Optionally, the pre-trained semantic segmentation model 30 is trained based on the expanded label loss to obtain the trained semantic segmentation model 30.

[0071] In some embodiments, the training of the semantic segmentation model 30 in the second stage of training includes multiple rounds of iterative training. For example, in the second stage of training, the unlabeled sample 20 is predicted based on the pre-trained semantic segmentation model 30 to obtain initial pseudo-labels. Then, initial pseudo-labeled samples are obtained based on the initial pseudo-labels and the unlabeled samples. The initial pseudo-labeled samples are combined with the labeled sample 10 to obtain initial expanded samples. The initial expanded samples are predicted using the pre-trained semantic segmentation model 30, and the initial expanded label loss is obtained by combining it with the ground truth label 40. The network parameters of the pre-trained semantic segmentation model 30 in the first round are updated based on the initial expanded label loss.

[0072] After obtaining the semantic segmentation model trained in the first round, the unlabeled sample 20 is predicted again using the same model to obtain the updated pseudo-label. Based on the updated pseudo-label and the unlabeled sample 20, updated extended samples are obtained. The updated extended samples are predicted using the semantic segmentation model trained in the first round, and the updated extended label loss is obtained by combining it with the ground truth label 40. The network parameters of the semantic segmentation model trained in the first round are updated in the second round based on the updated extended label loss. The above steps are repeated until the maximum number of training iterations or the convergence condition is met, at which point the iterative training process is stopped, and the trained semantic segmentation model is obtained.

[0073] Figure 2This is a schematic diagram of a computer system provided in an exemplary embodiment of this application. The computer system may include: a terminal 110 and a server 120.

[0074] Terminal 110 can be an electronic device such as a mobile phone, tablet computer, in-vehicle terminal (vehicle system), wearable device, personal computer (PC), or vehicle terminal. A client application for the target application can be installed and run on terminal 110. This target application can be an application used for training a semantic segmentation model, an application providing training functions for a semantic segmentation model, or an application providing usage functions for a generative language model; this application does not limit the specific form of the target application. Furthermore, this application does not limit the form of the target application, including but not limited to applications (Apps), mini-programs, clients, etc., installed on terminal 110, and can also be in web page form.

[0075] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence handheld image recognition platforms. Server 120 can be the backend server of the aforementioned target application, used to provide backend services to the client of the target application. In some optional embodiments, server 120 can also be implemented as a node in a blockchain system.

[0076] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Based on the cloud computing business model, cloud technology encompasses network technology, information technology, integration technology, management platform technology, and application technology. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0077] Terminal 110 and server 120 can communicate via a network, such as a wired or wireless network.

[0078] The semantic segmentation model training method provided in this application embodiment can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. Figure 1 Taking the computer system shown as an example, the semantic segmentation model training method can be executed by terminal 110, or by server 120, or by terminal 110 and server 120 interacting and cooperating to execute the semantic segmentation model training method. This application does not limit this.

[0079] Those skilled in the art will understand that the number of terminals 110 described above can be more or less. For example, there may be only one terminal 110, or there may be dozens or hundreds, or more. This application does not limit the number or type of terminals 110 in its embodiments.

[0080] The training methods for semantic segmentation models will be described in detail below.

[0081] Figure 3 This is a flowchart illustrating a training method for a semantic segmentation model provided in an exemplary embodiment of this application. The semantic segmentation model can be implemented using any neural network architecture with image analysis capabilities. This method is executed by a computer device used by the developer, and the computer device is... Figure 2 Terminal 110 in the middle, or by Figure 2 The method is executed on server 120. A generative language model runs on the computer device (terminal 110) or server 120. The method includes at least one of the following steps.

[0082] Step 210: Obtain labeled and unlabeled samples;

[0083] Labeled samples are image samples with ground truth labels, while unlabeled samples are image samples without ground truth labels. Ground truth labels can also be called ground truth map. In the field of image semantic segmentation, image samples are usually two-dimensional image samples, or they can be three-dimensional image samples. This application does not limit this.

[0084] In some embodiments, when the image sample is a two-dimensional image sample, the truth label is used to indicate the semantic category of each pixel in the image sample. Here, a pixel is the smallest unit that makes up a two-dimensional image sample, and each pixel represents the color or grayscale value of a point in the two-dimensional image sample.

[0085] In some embodiments, when the image sample is a three-dimensional image sample, the truth label is used to indicate the semantic category of each voxel in the three-dimensional image. Here, a voxel is the smallest unit that makes up a three-dimensional image sample. A voxel is similar to a pixel, but it contains depth (or third-dimensional) information.

[0086] Optionally, the image sample is a two-dimensional image sample. For a two-dimensional image, the ground truth label is represented using a mask image of the two-dimensional image. For example, a pixel value of 1 in the mask image indicates that the pixel at the corresponding position in the two-dimensional image belongs to the first category; a pixel value of 0 in the mask image indicates that the pixel at the corresponding position in the two-dimensional image belongs to the second category. As another example, a pixel value of 0 in the mask image indicates that the pixel at the corresponding position in the two-dimensional image belongs to the first category; a pixel value of 1 in the mask image indicates that the pixel at the corresponding position in the two-dimensional image belongs to the second category.

[0087] Optionally, the image sample is a three-dimensional image sample. For three-dimensional images, the ground truth label can be represented using a mask of voxels in the three-dimensional image. For example, a voxel mask value of 1 in the three-dimensional image represents belonging to the first category; a voxel mask value of 0 in the three-dimensional image represents belonging to the second category. As another example, a voxel mask value of 0 in the three-dimensional image represents belonging to the first category; a voxel mask value of 1 in the three-dimensional image represents belonging to the second category.

[0088] The embodiments of this application are mainly illustrated using two-dimensional image samples (hereinafter referred to as image samples). In some scenarios, the "pixel" in the two-dimensional image sample can be replaced with the "voxel" in the three-dimensional image sample.

[0089] For example, the pixels of image sample A include four categories. Taking image sample A as an example where image sample A contains roads, pedestrians, buildings, and sky, the truth labels are used to indicate the semantic category of each pixel in image sample A. For example, the truth labels are used to indicate that the first part of the pixels (or the first part of the pixel area) in image sample A belongs to "road", the second part of the pixels (or the second part of the pixel area) in image sample A belongs to "pedestrian", the third part of the pixels (or the third part of the pixel area) in image sample A belongs to "building", and the fourth part of the pixels (or the fourth part of the pixel area) in image sample A belongs to "sky".

[0090] Step 220: Input the labeled samples into the semantic segmentation model to obtain the labeled prediction map corresponding to the labeled samples, and obtain the labeled loss based on the labeled prediction map and the ground truth label;

[0091] In some embodiments, labeled samples are input into a semantic segmentation model, and predictions are made based on the semantic segmentation model to obtain a labeled prediction map corresponding to the labeled samples. The labeled prediction map contains the semantic prediction category for each pixel in the labeled samples.

[0092] In some embodiments, the semantic segmentation model may also be referred to as the image semantic segmentation model. The semantic segmentation model can be implemented using any neural network architecture with image segmentation capabilities.

[0093] For example, this application uses RWKV (Receptance Weighted KeyValue)-UNet as an example to illustrate the semantic segmentation model. RWKV is a deep learning architecture that combines a recurrent neural network (RNN) and a Transformer model. RWKV-UNet can achieve accurate semantic segmentation (i.e., image semantic segmentation).

[0094] In some embodiments, labeled samples are input into a semantic segmentation model to obtain image feature representations corresponding to the labeled samples. Based on these image feature representations, the semantic segmentation model is used to predict the labeled prediction map of the labeled samples. Optionally, a transformer in the semantic segmentation model converts the labeled samples into corresponding image feature representations. These image feature representations indicate the content features of the labeled samples; that is, they indicate the content features extracted from the labeled samples.

[0095] In some embodiments, the image feature representation corresponding to a labeled sample includes features directly related to the pixel values ​​of the labeled sample, such as color, texture, and edge features; or, the image feature representation corresponding to a labeled sample includes semantic features describing the shape, local structure, or pattern in the labeled sample; or, the image feature representation corresponding to a labeled sample includes semantic features describing the object category, scene category, or event in the labeled sample. This application does not limit this.

[0096] In some embodiments, the semantic segmentation model outputs a labeled prediction map corresponding to the input labeled samples. The labeled prediction map contains the semantic prediction category for each pixel in the labeled samples. Optionally, a labeled loss is obtained based on the labeled prediction map and ground truth labels. The labeled loss is used to indicate the error between the labeled prediction map predicted by the semantic segmentation model (an untrained image semantic segmentation model, or a general semantic segmentation model) and the ground truth labels. Alternatively, the labeled loss can be understood as indicating the error between the semantic prediction category of each pixel in the labeled samples predicted by the semantic segmentation model and the semantic category (semantic true category) of each pixel in the ground truth labels.

[0097] It should be noted that the aforementioned labeled loss can also be called supervised loss or supervised loss.

[0098] Step 230: Perform augmentation operations on the unlabeled samples to generate at least two augmented unlabeled samples. The augmentation operations are used to perturb the unlabeled samples from at least two perspectives.

[0099] The augmentation operation is used to perturb unlabeled samples from at least two perspectives. Optionally, the augmentation operation is also called a random augmentation operation. At least two perspectives means processing the unlabeled samples from at least two angles or in at least two ways.

[0100] In some embodiments, an augmentation operation is performed on the unlabeled sample to perturb the unlabeled sample from at least two perspectives, generating at least two augmented unlabeled samples, that is, generating augmented unlabeled samples from at least two perspectives. Here, an augmented unlabeled sample refers to an unlabeled sample obtained after performing a random augmentation operation on the unlabeled sample, or it can be understood as an augmented unlabeled sample obtained after perturbing the unlabeled sample.

[0101] For example, the illustration uses at least two enhanced unlabeled samples, including a first enhanced unlabeled sample and a second enhanced unlabeled sample. Optionally, a first enhancement operation is performed on the unlabeled sample to generate a first enhanced unlabeled sample, the first enhancement operation being used to perturb the unlabeled sample from a first viewpoint. Optionally, a second enhancement operation is performed on the unlabeled sample to generate a second enhanced unlabeled sample, the second enhancement operation being used to perturb the unlabeled sample from the first viewpoint. Here, the first enhanced unlabeled sample refers to the unlabeled sample obtained after performing the first enhancement operation on the unlabeled sample. The second enhanced unlabeled sample refers to the unlabeled sample obtained after performing the second enhancement operation on the unlabeled sample.

[0102] In some embodiments, the first enhancement operation and the second enhancement operation are enhancement operations from different perspectives.

[0103] Optionally, the first enhancement operation and the second enhancement operation include, but are not limited to, at least one of the following operations:

[0104] • Geometric transformation operations based on the first probability;

[0105] • Color transformation operation based on the second probability;

[0106] • Noise injection operation based on third probability;

[0107] • Hybrid operations based on the fourth probability.

[0108] In some embodiments, the first enhancement operation or the second enhancement operation is a geometric transformation operation based on a first probability. The first probability indicates the probability of performing a geometric transformation operation on the first enhancement operation or the second enhancement operation. A geometric transformation operation refers to an operation that performs geometric changes on unlabeled samples (i.e., unlabeled image samples) to alter their geometric features. For example, changing the shape or spatial position of the unlabeled sample. Optionally, geometric transformation operations include, but are not limited to, at least one of: horizontal flipping, vertical flipping, rotation, cropping, scaling, or other geometric transformation operations. For example, performing a cropping operation on an unlabeled sample crops the image size of the unlabeled sample from a first size to a second size. By performing geometric transformation operations on unlabeled samples, the geometric transformation operation enables the semantic segmentation model to have stronger robustness to changes in position, scale, and rotation, improving its adaptability to different image conditions.

[0109] In some embodiments, the first enhancement operation or the second enhancement operation is a color transformation operation based on a second probability. The second probability indicates the probability of performing the color transformation operation on the first enhancement operation or the second enhancement operation. A color transformation operation refers to an operation that transforms the color of an unlabeled sample to change its color characteristics. Optionally, color transformation operations include, but are not limited to, at least one of: brightness change operations, contrast change operations, saturation change operations, hue change operations, or other color transformation operations. For example, performing a hue change operation on an unlabeled sample transforms the image color of the unlabeled sample from a first hue to a second hue. By performing color transformation operations on unlabeled samples, color transformation operations can help the semantic segmentation model adapt to different lighting conditions and color changes, improving the generalization ability of the semantic segmentation model in diverse environments.

[0110] In some embodiments, the first enhancement operation or the second enhancement operation is a noise injection operation based on a third probability. The third probability indicates the probability of performing the noise injection operation on the first or second enhancement operation. A noise injection operation refers to the operation of adding random noise to unlabeled samples. For example, performing a noise injection operation on unlabeled samples adds Gaussian noise to them. By performing noise injection on unlabeled samples, the noise injection operation helps the semantic segmentation model cope with uncertainties or interference in the unlabeled samples, improving the noise robustness of the semantic segmentation model.

[0111] In some embodiments, the first or second enhancement operation is a blending operation based on a fourth probability. The fourth probability indicates the probability of performing the blending operation on the first or second enhancement operation. A blending operation refers to the operation of mixing unlabeled samples with blended samples (other labeled samples or other unlabeled samples). For example, performing a blending operation on unlabeled samples swaps a local region of an unlabeled sample with a local region of another image. By performing blending operations on unlabeled samples, the blending operation helps the semantic segmentation model learn how to process synthetic data, enhancing its ability to understand complex scenes or combined information.

[0112] It should be noted that the above is only an exemplary description of the enhancement operation on unlabeled samples, and this application does not limit the type of enhancement operation.

[0113] In some embodiments, a geometric transformation operation based on a first probability is performed on the unlabeled sample, generating a geometrically enhanced unlabeled sample according to the first probability. The geometrically enhanced unlabeled sample is the unlabeled sample after the geometric transformation operation. In some embodiments, a color transformation operation based on a second probability is performed on the unlabeled sample, generating a color-transformed enhanced unlabeled sample according to the second probability. The color-transformed enhanced unlabeled sample is the unlabeled sample after the color transformation operation. In some embodiments, a noise injection operation based on a third probability is performed on the unlabeled sample, generating a noise-enhanced unlabeled sample according to the third probability. The noise-enhanced unlabeled sample is the unlabeled sample after the noise injection operation. In some embodiments, a mixing operation based on a fourth probability is performed on the unlabeled sample, generating a mixed-enhanced unlabeled sample according to the fourth probability. The mixed-enhanced unlabeled sample is the unlabeled sample after the mixing operation.

[0114] In this embodiment, perturbing unlabeled sample images from multiple angles enhances the semantic segmentation model's adaptability to different conditions. For example, geometric transformation operations make the semantic segmentation model more robust to changes in position, scale, and rotation, improving its adaptability to different image conditions. Color transformation operations help the semantic segmentation model adapt to different lighting conditions and color changes. Noise injection operations help the semantic segmentation model cope with uncertainties or interference in unlabeled samples, improving its noise resistance. Hybrid operations enhance the semantic segmentation model's ability to understand complex scenes or combined information.

[0115] Step 240: Input at least two augmented unlabeled samples into the semantic segmentation model to obtain unlabeled prediction maps corresponding to at least two augmented unlabeled samples, and obtain unlabeled loss based on at least two unlabeled prediction maps;

[0116] Among them, augmented unlabeled samples refer to unlabeled samples obtained after performing random augmentation operations on unlabeled samples.

[0117] In some embodiments, at least two enhanced unlabeled samples are input into a semantic segmentation model, and predictions are made on the at least two enhanced unlabeled samples based on the semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two enhanced unlabeled samples. The unlabeled prediction map contains the semantic prediction category for each pixel in the enhanced unlabeled sample.

[0118] In some embodiments, at least two enhanced unlabeled samples are respectively input into a semantic segmentation model to obtain image feature representations corresponding to the at least two enhanced unlabeled samples. Based on the semantic segmentation model, predictions are made on the image feature representations corresponding to the at least two enhanced unlabeled samples to obtain unlabeled prediction maps corresponding to the at least two enhanced unlabeled samples. Optionally, a transformer in the semantic segmentation model converts each of the at least two enhanced unlabeled samples into a corresponding image feature representation. The image feature representation is used to indicate the content features of the enhanced unlabeled samples; that is, the image feature representation is used to indicate the content features extracted from the enhanced unlabeled samples.

[0119] In some embodiments, the image feature representation corresponding to the enhanced unlabeled sample includes features directly related to the pixel values ​​of the enhanced unlabeled sample, such as color, texture, and edge features; or, the image feature representation corresponding to the enhanced unlabeled sample includes semantic features describing the shape, local structure, or pattern in the enhanced unlabeled sample; or, the image feature representation corresponding to the enhanced unlabeled sample includes semantic features describing the object category, scene category, or event in the enhanced unlabeled sample. This application does not limit this.

[0120] In some embodiments, the semantic segmentation model outputs at least two unlabeled augmented samples as input, corresponding to at least two distributions of these augmented unlabeled samples. Each unlabeled prediction map contains a semantic prediction category for each pixel in the corresponding augmented unlabeled sample. Optionally, an unlabeled loss is obtained based on the at least two augmented unlabeled samples. This unlabeled loss indicates the error between the at least two unlabeled prediction maps predicted by the semantic segmentation model (an untrained image semantic segmentation model, or a general semantic segmentation model). Alternatively, the unlabeled loss can be understood as indicating the error between the semantic segmentation model's predictions of each pixel in the at least two augmented unlabeled samples.

[0121] In some embodiments, an unlabeled loss is obtained based on at least two enhanced unlabeled samples, and the unlabeled loss is used to indicate the error between the unlabeled prediction maps corresponding to the at least two enhanced unlabeled samples. In some embodiments, an unlabeled loss is obtained based on unlabeled samples and enhanced unlabeled samples, and the unlabeled loss is used to indicate the error between the unlabeled prediction map corresponding to the unlabeled sample and the unlabeled prediction map corresponding to the enhanced unlabeled sample. In some embodiments, an unlabeled loss is obtained based on unlabeled samples and average enhanced unlabeled samples, and the unlabeled loss is used to indicate the error between the unlabeled prediction map corresponding to the unlabeled sample and the unlabeled prediction map corresponding to the average enhanced unlabeled sample. The average enhanced unlabeled sample is obtained based on at least two enhanced unlabeled samples, and the image feature representation of the average enhanced unlabeled sample (i.e., the average image feature representation) is the average representation of the image feature representations corresponding to the at least two enhanced unlabeled samples. This application mainly uses the example of an unlabeled loss obtained based on at least two enhanced unlabeled samples for illustration, but this does not constitute a limitation on the unlabeled loss.

[0122] It should be noted that the aforementioned unlabeled loss can also be called consistency loss.

[0123] Step 250: Train the semantic segmentation model based on labeled loss and unlabeled loss to obtain the pre-trained semantic segmentation model.

[0124] In some embodiments, the labeled loss is obtained based on the labeled prediction map corresponding to the labeled sample and the ground truth label. The labeled loss is used to indicate the error between the labeled prediction map predicted by the semantic segmentation model and the ground truth label. Alternatively, it can be understood as the labeled loss indicating the error between the semantic prediction category of each pixel in the labeled sample and the semantic category of each pixel in the ground truth label.

[0125] In some embodiments, the unlabeled loss is obtained based on at least two augmented unlabeled samples. The unlabeled loss is used to indicate the error between at least two unlabeled prediction maps predicted by the semantic segmentation model. Alternatively, it can be understood as indicating the error between each pixel in the semantic segmentation model's predictions of at least two augmented unlabeled samples.

[0126] Optionally, the semantic segmentation model can be trained by combining labeled and unlabeled losses to obtain a pre-trained semantic segmentation model. The pre-trained semantic segmentation model learns the features of unlabeled samples from different perspectives, enabling a more comprehensive understanding of the distribution and semantic information of unlabeled samples. Therefore, the pre-trained semantic segmentation model exhibits better image semantic segmentation capabilities.

[0127] In summary, the method provided in this application combines labeled and unlabeled samples for semi-supervised training of the semantic segmentation model. When processing unlabeled samples, it first performs random augmentation operations, perturbing the unlabeled samples from at least two different perspectives to obtain at least two augmented unlabeled samples. An unlabeled loss is then obtained based on these at least two augmented unlabeled samples. This unlabeled loss is combined with the labeled loss corresponding to the labeled samples to train the semantic segmentation model. The unlabeled loss obtained from at least two augmented unlabeled samples allows the semantic segmentation model to learn the features of unlabeled samples from different perspectives, enabling the model to more comprehensively understand the semantic information in the unlabeled samples. This allows for more efficient utilization of the latent information in the unlabeled samples, improving the learning effect and performance of the semantic segmentation model.

[0128] In this embodiment of the application, the training of the semantic segmentation model is divided into two stages.

[0129] In the first stage (initial stage), the semantic segmentation model is first semi-supervised by combining labeled and unlabeled samples. The labeled loss is obtained based on the labeled prediction map and the ground truth label, and the unlabeled loss is obtained based on at least two unlabeled prediction maps. The semantic segmentation model is trained by the labeled loss and the unlabeled loss, and a pre-trained semantic segmentation model can be obtained. The pre-trained semantic segmentation model has high prediction accuracy for unlabeled samples.

[0130] In the second stage (subsequent stage), the pre-trained semantic segmentation model obtained in the first stage is used to predict unlabeled samples, which can obtain pseudo-labels for unlabeled samples. Through multiple rounds of iterative training, the pseudo-labels are incorporated into the supervised learning in the second stage. After each round of training, the updated semantic segmentation model is used to regenerate more accurate pseudo-labels, forming a positive cycle of "pseudo-label optimization - model enhancement".

[0131] Optionally, the semantic segmentation models for the first and second stages can be the same; or, the semantic segmentation models for the first and second stages can be different, and this application does not limit this. When the semantic segmentation models for the first and second stages are the same, the training in the second stage can be considered as reinforcement training (or enhancement training) of the semantic segmentation model. When the semantic segmentation models for the first and second stages are different, the initial pseudo-labels used in the second-stage semantic segmentation model are obtained by predicting unlabeled samples based on the semantic segmentation model of the first stage. Because the semantic segmentation model trained in the first stage already has strong predictive performance (able to accurately predict unlabeled samples), the initial pseudo-labels predicted using the semantic segmentation model of the first stage have high prediction accuracy, laying a good foundation for the subsequent training of the second-stage semantic segmentation model. This application's embodiments mainly use the example of the first and second stages using the same semantic segmentation model, but this does not constitute a limitation on the model.

[0132] The training for the two phases will be introduced below.

[0133] For the first phase

[0134] In some embodiments, a pre-trained semantic segmentation model can be obtained by training a semantic segmentation model using both labeled and unlabeled loss. The labeled loss is derived from labeled prediction maps and ground truth labels, and includes similarity coefficient loss and cross-entropy loss. The unlabeled loss is derived from at least two unlabeled prediction maps. Calculating the unlabeled loss requires performing an inverse transformation on at least two unlabeled prediction maps, and the unlabeled loss is obtained from the transformed unlabeled prediction maps.

[0135] • Calculate the unlabeled loss

[0136] like Figure 4 As shown, step 240 above can be replaced by steps 241 and 242.

[0137] Step 241: Input at least two augmented unlabeled samples into the semantic segmentation model to obtain unlabeled prediction maps corresponding to at least two augmented unlabeled samples respectively;

[0138] Among them, at least two augmented unlabeled samples are obtained by performing augmentation operations on unlabeled samples (i.e., performing at least two augmentation operations); or it can be understood as at least two augmented unlabeled samples are obtained by perturbing unlabeled samples from at least two perspectives.

[0139] Optionally, the enhancement operation includes at least one of the following operations: a geometric transformation operation based on a first probability; a color transformation operation based on a second probability; a noise injection operation based on a third probability; and a mixing operation based on a fourth probability.

[0140] In some embodiments, at least two enhanced unlabeled samples are input into a semantic segmentation model, and predictions are made on the at least two enhanced unlabeled samples based on the semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two enhanced unlabeled samples. The unlabeled prediction map contains the semantic prediction category for each pixel in the enhanced unlabeled sample.

[0141] Step 242: Perform an inverse transformation operation on at least two unlabeled prediction maps to obtain at least two inversely transformed unlabeled prediction maps, and obtain the unlabeled loss based on the at least two inversely transformed unlabeled prediction maps.

[0142] The inverse transformation operation refers to the reverse processing (inverse processing) of the unlabeled prediction map corresponding to the unlabeled samples after augmentation. It can also be understood as the inverse processing (inverse processing) of the augmentation operation, that is, the operation of reversing (inverse processing) the unlabeled prediction map obtained after augmentation.

[0143] It should be noted that, in practice, the inverse transformation operation mainly refers to the inverse transformation operation for geometric transformations. This application does not limit this.

[0144] For example, the inverse transformation operation of geometric transformation includes, but is not limited to, at least one of the following: inverse transformation operation of horizontal flip, inverse transformation operation of vertical flip, inverse transformation operation of rotation, inverse transformation operation of clipping, and inverse transformation operation of scaling.

[0145] In some embodiments, the enhancement operation is a geometric transformation operation, and the inverse transformation operation is the inverse of the geometric transformation. Optionally, a geometric transformation operation is performed on the unlabeled samples to generate geometrically enhanced unlabeled samples. The geometrically enhanced unlabeled samples are input into the semantic segmentation model to obtain the unlabeled prediction map corresponding to the geometrically enhanced unlabeled samples. Optionally, the inverse of the geometric transformation operation is performed on the unlabeled prediction map corresponding to the geometrically enhanced unlabeled samples to obtain the inverse-transformed unlabeled prediction map.

[0146] For example, the augmentation operation is a horizontal flip operation (a type of geometric transformation operation). After performing a horizontal flip operation on the unlabeled sample, an augmented unlabeled sample A is obtained. Augmented unlabeled sample A has undergone an x-axis flip operation compared to the original unlabeled sample. The semantic segmentation model predicts the augmented unlabeled sample A, obtaining the corresponding unlabeled prediction map A. An inverse transformation operation is then performed on the unlabeled prediction map A. This inverse transformation operation is the inverse of the horizontal flip operation; that is, the inverse transformation operation reverses the x-axis flip operation on the unlabeled prediction map A, resulting in the inversely transformed unlabeled prediction map A*.

[0147] In some embodiments, an inverse transformation operation is performed on at least two unlabeled prediction maps, and a different inverse transformation operation is performed on each of the at least two unlabeled prediction maps. The inverse transformation operation performed on each unlabeled prediction map corresponds to the augmentation operation corresponding to that unlabeled prediction map, resulting in at least two inversely transformed unlabeled prediction maps. Optionally, an unlabeled loss is obtained based on the at least two inversely transformed unlabeled prediction maps.

[0148] In this context, the inversely transformed unlabeled prediction map is the same as or similar to the original unlabeled prediction map corresponding to the unlabeled sample. Alternatively, it can be understood that the inversely transformed unlabeled prediction map is the same as or similar to the original unlabeled prediction map predicted by the semantic segmentation model for the unlabeled sample before the augmentation operation.

[0149] In this embodiment, the inverse transformation operation is the reverse operation performed after the data has undergone certain transformations. Its purpose is to recover or approximate the features of the original data (the original unlabeled prediction map corresponding to the unlabeled samples before the augmentation operation). By performing the inverse transformation operation on the unlabeled prediction map, the augmented unlabeled prediction map can be mapped back to the coordinate system of the original unlabeled samples. Calculating the unlabeled loss on at least two inversely transformed unlabeled prediction maps helps improve the accuracy and effectiveness of model training.

[0150] In some embodiments, the at least two enhanced unlabeled samples include a first enhanced unlabeled sample and a second enhanced unlabeled sample. The first enhanced unlabeled sample is the sample obtained by performing a first enhancement operation on the unlabeled sample. The second enhanced unlabeled sample is the sample obtained by performing a second enhancement operation on the unlabeled sample.

[0151] In some embodiments, a first augmentation operation is performed on the unlabeled sample to generate a first augmented unlabeled sample. The first augmentation operation is used to perturb the unlabeled sample from a first viewpoint. Optionally, the first augmented unlabeled sample is input into a semantic segmentation model, which predicts the first augmented unlabeled sample to obtain a first unlabeled prediction map corresponding to the first augmented unlabeled sample. The first unlabeled prediction map contains the semantic prediction category for each pixel in the first augmented unlabeled sample.

[0152] In some embodiments, a second augmentation operation is performed on the unlabeled sample to generate a second augmented unlabeled sample. The second augmentation operation is used to perturb the unlabeled sample from a first viewpoint. Optionally, the second augmented unlabeled sample is input into a semantic segmentation model, which predicts the second augmented unlabeled sample to obtain a second unlabeled prediction map corresponding to the second augmented unlabeled sample. The second unlabeled prediction map contains the semantic prediction category for each pixel in the second augmented unlabeled sample.

[0153] like Figure 5 As shown, step 242 above can be replaced by steps 242a and 242b.

[0154] Step 242a: Perform a first inverse transformation operation on the first unlabeled prediction map to obtain the inverse-transformed first unlabeled prediction map; and perform a second inverse transformation operation on the second unlabeled prediction map to obtain the inverse-transformed second unlabeled prediction map;

[0155] The first unlabeled prediction map is the prediction map obtained by the semantic segmentation model after predicting the first augmented unlabeled sample. The first unlabeled prediction map contains the semantic prediction category for each pixel in the first augmented unlabeled sample. The first inverse transformation operation refers to the operation of reversing the processing (inverse processing) of the first unlabeled prediction map corresponding to the unlabeled sample after the first augmentation operation (i.e., the first augmented unlabeled sample).

[0156] The second unlabeled prediction map is the prediction map obtained by the semantic segmentation model after predicting the second enhanced unlabeled sample. The second unlabeled prediction map contains the semantic prediction category for each pixel in the second enhanced unlabeled sample. The second inverse transformation operation refers to the reverse processing (inverse processing) of the second unlabeled prediction map corresponding to the unlabeled sample after the second enhancement operation (i.e., the second enhanced unlabeled sample).

[0157] In some embodiments, the first inverse transform operation corresponds to the first enhancement operation; that is, the first inverse transform operation is the reverse of the first enhancement operation. The second inverse transform operation corresponds to the second enhancement operation; that is, the second inverse transform operation is the reverse of the second enhancement operation.

[0158] For example, the first enhancement operation is a horizontal flip operation, in which the first enhanced unlabeled sample is flipped along the x-axis compared to the unlabeled sample. The second enhancement operation is a rotation flip operation, in which the second enhanced unlabeled sample is rotated 90 degrees compared to the unlabeled sample.

[0159] After predicting the first and second enhanced unlabeled samples based on the semantic segmentation model to obtain the first and second unlabeled prediction maps, a first inverse transformation operation is performed on the first unlabeled prediction map, which is to flip the first unlabeled prediction map in reverse along the x-axis. A second inverse transformation operation is performed on the second unlabeled prediction map, which is to rotate the second unlabeled prediction map by 90 degrees and then flip it in reverse.

[0160] Step 242b: Based on the first unlabeled prediction map after inverse transformation and the second unlabeled prediction map after inverse transformation, obtain the unlabeled loss.

[0161] In some embodiments, a first inverse transform operation is performed on the first unlabeled prediction map to obtain the inversely transformed first unlabeled prediction map. Optionally, the inversely transformed first unlabeled prediction map is the same as or similar to the original unlabeled prediction map corresponding to the unlabeled sample. It can also be understood that the inversely transformed first unlabeled prediction map refers to the result of restoring the first unlabeled prediction map to the original image space (the space where the original unlabeled prediction map is located) through the inverse transform operation.

[0162] In some embodiments, a second inverse transform operation is performed on the second unlabeled prediction map to obtain the inversely transformed second unlabeled prediction map. Optionally, the inversely transformed second unlabeled prediction map is the same as or similar to the original unlabeled prediction map corresponding to the unlabeled sample. Alternatively, the inversely transformed second unlabeled prediction map can be understood as the result of restoring the second unlabeled prediction map to the original image space (the space where the original unlabeled prediction map is located) through the inverse transform operation.

[0163] In some embodiments, an unlabeled loss is obtained based on the inversely transformed first unlabeled prediction map and the inversely transformed second unlabeled prediction map. The unlabeled loss is used to indicate the error between the first and second unlabeled prediction maps predicted by the semantic segmentation model. Further, the unlabeled loss is used to indicate the error between the inversely transformed first and second unlabeled prediction maps.

[0164] In this embodiment, the first and second unlabeled prediction maps after inverse transformation are prediction results recovered from enhanced unlabeled samples after different enhancement operations and then after inverse transformation. The differences between them reflect the adaptability and accuracy of the semantic segmentation model to sample transformations. Calculating the unlabeled loss based on the inversely transformed unlabeled prediction maps means recovering the original features of the unlabeled samples through inverse transformation operations, and then calculating the differences between different versions of unlabeled samples to optimize the semantic segmentation model and improve its robustness to data changes.

[0165] • Calculate labeled loss

[0166] like Figure 6 As shown, step 220 can be replaced by steps 221, 222, 223 and 224.

[0167] Step 221: Input the labeled samples into the semantic segmentation model to obtain the labeled prediction map corresponding to the labeled samples;

[0168] The labeled prediction map contains the semantic prediction category for each pixel in the labeled sample.

[0169] Step 222: Perform the first operation on the labeled prediction graph and the ground truth labels to obtain the similarity coefficient loss;

[0170] The truth label is used to indicate the semantic category of each pixel in the labeled sample.

[0171] Optionally, after obtaining the labeled prediction map corresponding to the labeled sample based on the semantic segmentation model, a first operation is performed on the labeled prediction map and the ground truth label, and a similarity coefficient loss is obtained based on the first operation. Here, the first operation refers to the operation of performing a similarity measurement on the labeled prediction map and the ground truth label.

[0172] In some embodiments, the similarity coefficient loss is used to measure the degree of overlap between the labeled prediction graph and the ground truth graph. The similarity coefficient measures the ratio of the intersection to the union between the labeled prediction graph and the ground truth graph (i.e., the ground truth graph). For example, the similarity coefficient can be the Dice coefficient or the Jaccard coefficient.

[0173] Step 223: Perform the second operation on the labeled prediction graph and the ground truth labels to obtain the cross-entropy loss;

[0174] The truth label is used to indicate the semantic category of each pixel in the labeled sample.

[0175] Optionally, after obtaining the labeled prediction map corresponding to the labeled sample based on the semantic segmentation model, a second operation is performed on the labeled prediction map and the ground truth label, and the cross-entropy loss is obtained based on the second operation. Here, the second operation refers to performing the operation on the probability distribution difference between the labeled prediction map and the ground truth label.

[0176] In some embodiments, cross-entropy loss is used to measure the difference between the labeled prediction graph and the ground truth label. It can also be understood as measuring the difference in probability distribution between the labeled prediction graph and the ground truth label (i.e., the ground truth label graph).

[0177] Step 224: Calculate the labeled loss based on the similarity coefficient loss and cross-entropy loss.

[0178] The similarity coefficient loss measures the overlap between the labeled prediction graph and the ground truth label. The cross-entropy loss measures the difference between the labeled prediction graph and the ground truth label.

[0179] Optionally, the similarity coefficient loss and cross-entropy loss can be summed, or the similarity coefficient loss and cross-entropy loss can be weighted and summed to calculate the labeled loss of the semantic segmentation model.

[0180] In this embodiment, by simultaneously calculating the similarity coefficient loss and the cross-entropy loss, the performance of the semantic segmentation model on labeled samples can be comprehensively measured. The similarity coefficient loss measures the degree of overlap between the labeled prediction map and the ground truth label, while the cross-entropy loss focuses on the difference between the probability distributions of the labeled prediction map and the ground truth label. Combining these two losses ensures that the semantic segmentation model is not only accurate in category prediction but also optimized in terms of structural or region overlap. Since the similarity coefficient loss and the cross-entropy loss focus on different aspects, their combined use can help the semantic segmentation model obtain more diverse training feedback, thereby improving the generalization ability of the semantic segmentation model.

[0181] like Figure 7 As shown, step 224 above can be replaced by steps 224a, 224b and 224c.

[0182] Step 224a: Obtain the first weight corresponding to the similarity coefficient loss and the second weight corresponding to the cross-entropy loss;

[0183] The first weight is the proportion of similarity coefficient loss in the labeled loss, and the second weight is the proportion of cross-entropy loss in the labeled loss.

[0184] Optionally, the first weight and the second weight are pre-set weights (or fixed weights), or the first weight and the second weight are weights determined based on the number of training iterations (or dynamic weights). This application does not limit this.

[0185] In some embodiments, the first weight and the second weight are preset weights. For example, the first weight corresponding to the similarity coefficient loss is set to 0.5, and the second weight corresponding to the cross-entropy loss is set to 0.5.

[0186] In some embodiments, the first weight and the second weight are weights determined based on the number of training iterations (the number of training iterations in the first phase). Optionally, the first weight corresponding to the labeled loss and the second weight corresponding to the cross-entropy loss are determined based on the current number of training iterations and the maximum number of training iterations in the training process (the training process in the first phase).

[0187] Step 224b: Calculate the first product of the similarity coefficient loss and the first weight, and calculate the second product of the cross-entropy loss and the second weight;

[0188] Optionally, after obtaining the first weight corresponding to the similarity coefficient loss and the second weight corresponding to the cross-entropy loss, the first product of the similarity coefficient loss and the first weight is calculated, and the second product of the cross-entropy loss and the second weight is calculated.

[0189] In some embodiments, based on the current training iterations and the maximum training iterations, a first weight corresponding to the similarity coefficient loss and a second weight corresponding to the cross-entropy loss are determined. A first product of the similarity coefficient loss and the first weight, and a second product of the cross-entropy loss and the second weight are calculated; the sum of the first and second products is determined as the labeled loss.

[0190] For example, suppose that in the first stage of training, the ratio between the current number of training attempts and the maximum number of training attempts is the percentage of training attempts. The first weight is negatively correlated with the percentage of training attempts, that is, the larger the percentage of training attempts, the smaller the first weight; the second weight is positively correlated with the percentage of training attempts, that is, the larger the percentage of training attempts, the larger the second weight.

[0191] Step 224c: The sum of the first product and the second product is determined as the labeled loss.

[0192] In this embodiment, by assigning different weights to each loss (similarity coefficient loss and cross-entropy loss), the contribution of each task to model optimization can be more precisely controlled. Different tasks may have different impacts on the semantic segmentation model. In one implementation, by dynamically adjusting the weights of the losses (first weight and second weight), the semantic segmentation model can achieve a better balance among different tasks. This weighting strategy ensures that the semantic segmentation model does not overfit to a specific task, thereby improving the generalization ability of the semantic segmentation model in practical applications.

[0193] • The loss value of the semantic segmentation model is obtained by combining labeled and unlabeled loss.

[0194] Reference Figure 8 Step 250 above can be replaced by steps 251, 252 and 253.

[0195] Step 251: Obtain the third weight corresponding to the labeled loss and the fourth weight corresponding to the unlabeled loss;

[0196] Labeled loss can also be called supervised loss or supervised loss. Unlabeled loss can also be called consistency loss. The third weight is the proportion of labeled loss in the semantic segmentation model's loss value (first stage), and the fourth weight is the proportion of unlabeled loss in the semantic segmentation model's loss value (first stage).

[0197] Optionally, the third and fourth weights are pre-set weights (or fixed weights), or the third and fourth weights are weights determined based on the number of training iterations (or dynamic weights). This application does not limit this.

[0198] In some embodiments, the third and fourth weights are pre-defined weights. For example, the third weight corresponding to labeled loss is set to 0.5, and the fourth weight corresponding to unlabeled loss is set to 0.5.

[0199] In some embodiments, the third and fourth weights are weights determined based on the number of training iterations (the number of training iterations in the first phase). Optionally, the third weight corresponding to labeled loss and the fourth weight corresponding to unlabeled loss are determined based on the current number of training iterations and the maximum number of training iterations in the training process (the training process in the first phase).

[0200] Step 252: Calculate the third product of the labeled loss and the third weight, and calculate the fourth product of the unlabeled loss and the fourth weight;

[0201] Optionally, after obtaining the third weight corresponding to the labeled loss and the fourth weight corresponding to the unlabeled loss, the third product of the labeled loss and the third weight is calculated, and the fourth product of the unlabeled loss and the fourth weight is calculated.

[0202] In some embodiments, based on the current training iterations and the maximum training iterations, a third weight corresponding to the labeled loss and a fourth weight corresponding to the unlabeled loss are determined. A third product of the labeled loss and the third weight, and a fourth product of the unlabeled loss and the fourth weight are calculated; the sum of the third and fourth products is determined as the loss value of the semantic segmentation model during the first stage of training.

[0203] For example, suppose that in the first stage of training, the ratio between the current number of training attempts and the maximum number of training attempts is the percentage of training attempts. The third weight is negatively correlated with the percentage of training attempts, that is, the larger the percentage of training attempts, the smaller the third weight. The fourth weight is positively correlated with the percentage of training attempts, that is, the larger the percentage of training attempts, the larger the fourth weight.

[0204] Step 253: Determine the sum of the third and fourth products as the loss value of the semantic segmentation model, train the semantic segmentation model based on the loss value, and obtain the pre-trained semantic segmentation model.

[0205] The sum of the third product obtained by the labeled loss and the third weight, and the fourth product obtained by the unlabeled loss and the fourth weight, is determined as the loss value of the semantic segmentation model in the first stage of training. The semantic segmentation model is trained based on this loss value to obtain the pre-trained semantic segmentation model.

[0206] By combining labeled and unlabeled samples to train the semantic segmentation model, the pre-trained semantic segmentation model can learn the features of unlabeled samples from different perspectives, and thus gain a more comprehensive understanding of the distribution and semantic information of unlabeled samples. The pre-trained semantic segmentation model has good image semantic segmentation capabilities.

[0207] In this embodiment, by assigning different weights to labeled and unlabeled losses, the contributions of supervised and unsupervised tasks to the model can be more precisely controlled. Different tasks may have different impacts on the semantic segmentation model. In one implementation, by dynamically adjusting the weights of the loss (the third and fourth weights), the semantic segmentation model can achieve a better balance between supervised and unsupervised tasks. This weighting strategy ensures that the semantic segmentation model does not overfit to a specific task, thereby improving the generalization ability of the semantic segmentation model in practical applications.

[0208] For the second phase

[0209] In some embodiments, a pre-trained semantic segmentation model can be obtained by performing semi-supervised training (combining labeled and unlabeled samples) on the semantic segmentation model in the first stage. The pre-trained semantic segmentation model has good image semantic segmentation capabilities for unlabeled samples. In the second stage, the performance of the semantic segmentation model is further improved through pseudo-label generation and multiple training optimizations.

[0210] Figure 9 This is a flowchart of a training method for a semantic segmentation model provided in an exemplary embodiment of this application, the method including at least one of the following steps.

[0211] Step 310: Input the unlabeled samples into the pre-trained semantic segmentation model to obtain the pseudo-labels corresponding to the unlabeled samples;

[0212] In this context, pseudo-labels are labels obtained by the pre-trained semantic segmentation model after predicting unlabeled samples. These labels are then used as real labels in subsequent training (the second stage).

[0213] Since the pre-trained semantic segmentation model is obtained after the first stage of training, it has good image semantic segmentation capabilities. Compared with the pseudo-labels predicted by the general semantic segmentation model for unlabeled samples, the accuracy of the pseudo-labels predicted by the pre-trained semantic segmentation model in this embodiment is greatly improved.

[0214] Step 320: Obtain pseudo-labeled samples based on unlabeled samples and pseudo-labels, and obtain expanded samples based on pseudo-labeled samples and labeled samples;

[0215] In some embodiments, unlabeled samples are input into a pre-trained semantic segmentation model, which then predicts pseudo-labels from the unlabeled samples. The unlabeled samples and pseudo-labels are combined to obtain pseudo-labeled samples. The pseudo-labeled samples are image samples annotated with pseudo-labels.

[0216] Optionally, the semantic segmentation models for the first stage and the second stage are the same semantic segmentation model; or, the semantic segmentation models for the first stage and the second stage are different semantic segmentation models, which is not limited in this application.

[0217] When the semantic segmentation models in the first and second stages are the same, the training in the second stage can be viewed as reinforcement training (or enhancement training) of the semantic segmentation model. When the semantic segmentation models in the first and second stages are different, the initial pseudo-labels used in the second stage are obtained by predicting unlabeled samples based on the semantic segmentation model in the first stage. Because the semantic segmentation model trained in the first stage already possesses strong predictive performance (able to accurately predict unlabeled samples), the initial pseudo-labels predicted using the first stage semantic segmentation model have high prediction accuracy, laying a good foundation for the subsequent training of the second stage semantic segmentation model.

[0218] This application mainly uses a semantic segmentation model where the first and second stages are the same as an example for illustration, but this does not constitute a limitation on the semantic segmentation model.

[0219] In some embodiments, during the second-stage training process, pseudo-labeled samples can be used to expand the training set in the second-stage training process, utilizing the pseudo-labels generated by the pre-trained semantic segmentation model obtained in the first stage to assist the training in the second stage. Optionally, after obtaining pseudo-labeled samples based on unlabeled samples and pseudo-labels, expanded samples are obtained based on pseudo-labeled samples and labeled samples.

[0220] In some embodiments, pseudo-labeled samples and labeled samples are aligned based on an alignment method to obtain extended samples. The extended samples are the combination of aligned pseudo-labeled samples and labeled samples. Optionally, the alignment method includes either a splicing method or an association method. The splicing method refers to directly merging pseudo-labeled samples and labeled samples in a data structure. The association method refers to establishing a relationship or mapping between pseudo-labeled samples and labeled samples in a data structure. This application does not limit the scope of the method.

[0221] For example, alignment methods include association methods, which logically bind pseudo-labeled samples to labeled samples using identifiers (such as indexes or key values), rather than directly merging them. This can also be understood as associating pseudo-labeled samples with labeled samples using a common identifier.

[0222] Step 330: Input the expanded sample into the pre-trained semantic segmentation model to obtain the expanded label prediction map corresponding to the expanded sample, and obtain the expanded label loss based on the expanded label prediction map and the ground truth label;

[0223] This application mainly uses a semantic segmentation model where the first and second stages are the same as an example for illustration.

[0224] In some embodiments, the extended samples are input into a pre-trained semantic segmentation model, and predictions are made on the extended samples based on the pre-trained semantic segmentation model to obtain an extended label prediction map corresponding to the extended samples. The extended label prediction map contains the semantic prediction category for each pixel in the extended samples.

[0225] In some embodiments, the extended samples are input into a pre-trained semantic segmentation model to obtain the image feature representations corresponding to the extended samples. Based on the pre-trained semantic segmentation model, predictions are made on the image feature representations corresponding to the extended samples to obtain the extended label prediction map corresponding to the extended samples. Optionally, a transformer in the pre-trained semantic segmentation model converts the extended samples into corresponding image feature representations. The image feature representations are used to indicate the content features of the extended samples; that is, the image feature representations are used to indicate the content features extracted from the extended samples.

[0226] In some embodiments, the pre-trained semantic segmentation model outputs an expanded label prediction map corresponding to the expanded samples based on the input expanded samples. The expanded label prediction map contains the semantic prediction category for each pixel in the expanded samples. Optionally, an expanded label loss is obtained based on the expanded label prediction map and the ground truth labels. The expanded label loss is used to indicate the error between the expanded label prediction map predicted by the pre-trained semantic segmentation model and the ground truth labels. Alternatively, the expanded label loss can be understood as indicating the error between the semantic prediction category of each pixel in the expanded samples predicted by the pre-trained semantic segmentation model and the semantic category (semantic true category) of each pixel in the ground truth labels.

[0227] Step 340: Train the pre-trained semantic segmentation model based on the extended label loss to obtain the trained semantic segmentation model.

[0228] The expanded label loss is used to indicate the error between the expanded label prediction map and the ground truth label predicted by the pre-trained semantic segmentation model. Optionally, the pre-trained semantic segmentation model is trained using the expanded label loss to optimize the network parameters (or model parameters) of the pre-trained semantic segmentation model, thus obtaining the trained semantic segmentation model.

[0229] In some embodiments, the training process of the pre-trained semantic segmentation model in the second stage includes multiple rounds of iterative training. Optionally, unlabeled samples are input into the semantic segmentation model pre-trained in the i-th round in the second stage to obtain the i-th round pseudo-labels corresponding to the unlabeled samples; i-th round pseudo-labeled samples are obtained based on the unlabeled samples and the i-th round pseudo-labels, and i-th round expanded samples are obtained based on the i-th round pseudo-labeled samples and labeled samples; the i-th round expanded samples are input into the semantic segmentation model pre-trained in the i-th round to obtain the i-th round expanded label prediction map corresponding to the i-th round expanded samples, and the i-th round expanded label loss is obtained based on the i-th round expanded label prediction map and ground truth labels; the semantic segmentation model pre-trained in the i-th round is trained based on the i-th round expanded label loss, and the network parameters trained in the i-th round are updated to obtain the semantic segmentation model for training in the (i+1)-th round, where i is a positive integer.

[0230] Optionally, based on the extended label loss of round i, the network parameters of the pre-trained semantic segmentation model trained in round i are updated to obtain the semantic segmentation model for training in round i+1. The semantic segmentation model trained in round i+1 generates pseudo-labels for unlabeled samples in round i+1. The pseudo-labels in round i+1 are pseudo-labels optimized from the pseudo-labels in round i, where i is a positive integer.

[0231] Specifically, unlabeled samples are input into the semantic segmentation model trained in the (i+1)th round in the second stage to obtain pseudo-labels for the unlabeled samples in the (i+1)th round, and expanded samples for the (i+1)th round are obtained based on the pseudo-labeled samples and labeled samples in the (i+1)th round. The expanded samples for the (i+1)th round are input into the semantic segmentation model pre-trained in the (i+1)th round to obtain the expanded label prediction map for the (i+1)th round, and expanded label loss for the (i+1)th round is obtained based on the expanded label prediction map and ground truth labels. The semantic segmentation model pre-trained in the (i+1)th round is trained based on the expanded label loss for the (i+1)th round, and the network parameters for the (i+1)th round are updated to obtain the semantic segmentation model for training in the (i+2)th round, and so on. The above steps are repeated until the maximum number of training iterations or the convergence condition is reached, at which point the iterative training process stops, and the trained semantic segmentation model is obtained.

[0232] For example, in the second stage of training, the unlabeled samples are predicted based on the pre-trained semantic segmentation model to obtain initial pseudo-labels. Initial pseudo-labeled samples are obtained based on the initial pseudo-labels and the unlabeled samples. The initial pseudo-labeled samples are combined with the labeled samples to obtain initial expanded samples. The initial expanded samples are predicted by the pre-trained semantic segmentation model. The initial expanded label loss is obtained by combining the ground truth labels. The pre-trained semantic segmentation model is trained based on the initial expanded label loss, and the network parameters of the pre-trained semantic segmentation model are updated in the first round.

[0233] After obtaining the semantic segmentation model trained in the first round, the model is used to predict unlabeled samples again to obtain updated pseudo-labels for the unlabeled samples. Updated expanded samples are obtained based on the updated pseudo-labels and unlabeled samples. The updated expanded samples are predicted using the semantic segmentation model trained in the first round, and the updated expanded label loss is obtained by combining the ground truth labels. The network parameters of the semantic segmentation model trained in the first round are updated based on the updated expanded label loss in the second round. The above steps are repeated until the maximum number of training iterations or the convergence condition is met, at which point the iterative training process is stopped, and the trained semantic segmentation model is obtained.

[0234] In summary, the method provided in this application, through pseudo-label generation, can fully exploit the value of unlabeled samples, transforming a large number of unlabeled samples into pseudo-labeled samples with certain label information, thus expanding the training dataset in the second stage of training. By incorporating pseudo-labels into supervised learning through multiple rounds of iterative training, and regenerating more accurate pseudo-labels using the updated model after each round of training, a positive loop of "pseudo-label optimization - model enhancement" is formed, further enhancing the image semantic segmentation capability of the semantic segmentation model and achieving efficient utilization of unlabeled samples.

[0235] Image semantic segmentation (SMS) is an important technology in computer vision and a crucial part of image understanding. The training method of the aforementioned semantic segmentation model is applicable to both two-dimensional and three-dimensional image segmentation scenarios. In practical applications, image semantic segmentation can be used in a wide range of image domains. For example, in medical image processing, it can be used to identify lesion areas from images, helping doctors make more accurate diagnoses. In satellite remote sensing image processing, it can more accurately identify various information about the Earth's surface, such as forest cover areas, from remote sensing images. In autonomous driving, it can be used to identify road elements such as obstacles in road images, thereby improving the accuracy of target recognition and ensuring driving safety. In agriculture, it can be used to identify pest and disease areas from agricultural images, enabling timely response to pest and disease problems and improving agricultural cultivation efficiency. In industrial quality inspection, it can be applied to defect detection of parts, identifying defect areas in part images, improving the efficiency and accuracy of industrial quality inspection. It is worth noting that the above application areas are merely illustrative examples, and the embodiments of this application do not limit the scope of the application.

[0236] The following explanation uses image semantic segmentation in the context of image segmentation of two-dimensional images.

[0237] Step 1, obtain image samples;

[0238] The image samples include labeled samples and unlabeled samples. Labeled samples are image samples with ground truth labels, while unlabeled samples are image samples without ground truth labels.

[0239] In related technologies, in traditional semantic segmentation tasks, the input to a semantic segmentation model typically consists of batches of multiple image samples. These batches contain P categories, and each sample is an image of shape R. B×C×H×W Let represent an image batch with B samples, C channels, H height, and W width. During the training phase, a neural network extracts the feature representation of each image sample, performs feature map encoding and decoding, and finally outputs the class label for each pixel. The class label for each pixel is predicted using softmax or other activation functions, resulting in S∈R. B×P×H×W , representing the probability distribution of the image sample output.

[0240] The corresponding truth label G usually has shape R B×H×WFor standard semantic segmentation tasks, ground truth labels are typically transformed into a single-channel class index map (using sparse integer values ​​to represent class indices). For example, to determine whether a pixel in an image sample belongs to the first class, a value of 1 is assigned if the pixel belongs to the first class, and a value of 0 is assigned if the pixel does not belong to the first class. These ground truth labels are used for supervised training and to calculate cross-entropy loss or other relevant loss functions.

[0241] In the first stage of training, this application uses labeled loss (i.e., supervised loss) for labeled samples and unlabeled loss (i.e., consistency loss) for unlabeled samples. The loss value of the semantic segmentation model is the sum of the labeled loss and the unlabeled loss. Various random augmentation operations are applied to the unlabeled samples using a contrastive learning method to generate augmented unlabeled samples from different perspectives. Then, the semantic segmentation model predicts these augmented unlabeled samples and calculates the consistency loss between different unlabeled prediction maps to constrain the learning process of the semantic segmentation model. This application uses RWKV-UNet as an example for illustration. The loss in the first stage is a combination of labeled loss (i.e., supervised loss) and unlabeled loss (i.e., consistency loss), which allows the semantic segmentation model to learn from both labeled and unlabeled samples simultaneously.

[0242] Step 2: In the first phase of training, calculate the labeled loss for labeled samples;

[0243] The following section introduces labeled loss.

[0244] In some embodiments, labeled samples are input into a semantic segmentation model to obtain a labeled prediction map corresponding to the labeled samples, and a labeled loss is obtained based on the labeled prediction map and ground truth labels. The labeled prediction map contains the semantic prediction category for each pixel in the labeled samples.

[0245] In some embodiments, labeled samples are input into a semantic segmentation model to obtain the image feature representations corresponding to the labeled samples. Based on the semantic segmentation model, the image feature representations corresponding to the labeled samples are predicted to obtain the labeled prediction maps corresponding to the labeled samples.

[0246] Optionally, a labeled loss is derived based on the labeled prediction map and the ground truth labels. The labeled loss indicates the error between the labeled prediction map predicted by the semantic segmentation model (an untrained image semantic segmentation model, or a general semantic segmentation model) and the ground truth labels. It can also be understood as the error between the semantic prediction class of each pixel in the labeled samples and the semantic class (true semantic class) of each pixel in the ground truth labels. The labeled loss is also known as supervised loss or supervised error.

[0247] Optionally, the labeled loss includes two parts: similarity coefficient loss and cross-entropy loss. Taking the example where the first weight corresponding to the similarity coefficient loss and the second weight corresponding to the cross-entropy loss are equal, the labeled loss is expressed as:

[0248] L supervised =L dice +L ce

[0249] Among them, L supervised This indicates the presence of labeling loss, or in other words, the presence of supervision loss (or monitoring loss). L dice L represents the loss due to the similarity coefficient. ce This represents the cross-entropy loss.

[0250] • For similarity coefficient loss

[0251] In some embodiments, the labeled loss includes a similarity coefficient loss. The similarity coefficient loss measures the degree of overlap between the labeled predicted graph and the ground truth labels. For example, the similarity coefficient loss is the Di ce loss. The similarity coefficient loss is expressed as:

[0252]

[0253] Among them, L dice Let G represent the similarity coefficient loss, also known as the Dice loss. P represents the pixel class in the image sample (here referring to labeled samples), p represents the p-th class, i represents the pixel index in the image sample (labeled samples), that is, represents the i-th pixel in the image sample (labeled samples), and G represents the similarity coefficient loss. ip S represents the true class of the i-th pixel. ip Let represent the probability of the i-th pixel in the p-th category, N represent the number of pixels in the image sample (labeled sample), and α represent the smoothing factor to prevent the denominator from being zero.

[0254] • Regarding cross-entropy loss

[0255] In some embodiments, the labeled loss includes cross-entropy loss. The similarity coefficient loss measures the overlap between the labeled predicted graph and the ground truth labels. The cross-entropy loss measures the difference between the labeled predicted graph and the ground truth labels. The cross-entropy loss is expressed as:

[0256]

[0257] Where N is the number of pixels in the image sample (labeled sample), i represents the pixel index in the image sample (labeled sample), that is, represents the i-th pixel in the image sample (labeled sample), P is the number of categories to be classified in the image sample (labeled sample), and p represents the p-th category. G ip S represents the true class of the i-th pixel. ip This represents the probability of the i-th pixel in the p-th category.

[0258] Step 3: In the first phase of training, calculate the unlabeled loss for unlabeled samples;

[0259] The following section introduces unlabeled loss.

[0260] Unlabeled loss (also known as consistency loss) is the core of the first stage, used for training on unlabeled samples. Its core idea is that the semantic segmentation model should maintain consistency in its predictions (i.e., unlabeled prediction maps) for different augmented versions of the same unlabeled sample. Specifically, multiple random augmentation operations are applied to the unlabeled sample to obtain different augmented unlabeled samples. These augmented unlabeled samples are then input into the semantic segmentation model to obtain corresponding augmented outputs, i.e., the unlabeled prediction maps corresponding to each augmented unlabeled sample. To ensure the comparability of different augmented outputs (unlabeled prediction maps), an inverse transform is applied to the augmented outputs (unlabeled prediction maps).

[0261] In some embodiments, an augmentation operation is performed on the unlabeled samples to generate at least two augmented unlabeled samples. The augmentation operation is used to perturb the unlabeled samples from at least two perspectives. The unlabeled prediction map contains a semantic prediction category for each pixel in the unlabeled sample. Optionally, the at least two augmented unlabeled samples are input into a semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two augmented unlabeled samples, and an unlabeled loss is obtained based on the at least two unlabeled prediction maps.

[0262] Optionally, the enhancement operation includes, but is not limited to, at least one of the following operations: a geometric transformation operation based on a first probability; a color transformation operation based on a second probability; a noise injection operation based on a third probability; and a mixing operation based on a fourth probability.

[0263] In some embodiments, the at least two enhanced unlabeled samples include a first enhanced unlabeled sample and a second enhanced unlabeled sample. The first enhanced unlabeled sample is the sample obtained by performing a first enhancement operation on the unlabeled sample. The second enhanced unlabeled sample is the sample obtained by performing a second enhancement operation on the unlabeled sample.

[0264] In some embodiments, a first augmentation operation is performed on the unlabeled sample to generate a first augmented unlabeled sample. The first augmentation operation is used to perturb the unlabeled sample from a first viewpoint. Optionally, the first augmented unlabeled sample is input into a semantic segmentation model, which predicts the first augmented unlabeled sample to obtain a first unlabeled prediction map corresponding to the first augmented unlabeled sample. The first unlabeled prediction map contains the semantic prediction category for each pixel in the first augmented unlabeled sample.

[0265] In some embodiments, a second augmentation operation is performed on the unlabeled sample to generate a second augmented unlabeled sample. The second augmentation operation is used to perturb the unlabeled sample from a first viewpoint. Optionally, the second augmented unlabeled sample is input into a semantic segmentation model, which predicts the second augmented unlabeled sample to obtain a second unlabeled prediction map corresponding to the second augmented unlabeled sample. The second unlabeled prediction map contains the semantic prediction category for each pixel in the second augmented unlabeled sample.

[0266] In some embodiments, a first inverse transform operation is performed on the first unlabeled prediction map to obtain an inversely transformed first unlabeled prediction map; and a second inverse transform operation is performed on the second unlabeled prediction map to obtain an inversely transformed second unlabeled prediction map. The first inverse transform operation refers to the operation of reversing the processing (inverse processing) of the first unlabeled prediction map corresponding to the unlabeled sample after the first augmentation operation (i.e., the first augmented unlabeled sample). The second inverse transform operation refers to the operation of reversing the processing (inverse processing) of the second unlabeled prediction map corresponding to the unlabeled sample after the second augmentation operation (i.e., the second augmented unlabeled sample). Optionally, an unlabeled loss is obtained based on the inversely transformed first unlabeled prediction map and the inversely transformed second unlabeled prediction map.

[0267] For example, the first augmentation operation is a horizontal flip operation, where the first augmented unlabeled sample is flipped along the x-axis compared to the unlabeled sample. The second augmentation operation is a rotation flip operation, where the second augmented unlabeled sample is rotated 90 degrees compared to the unlabeled sample. After predicting the first and second augmented unlabeled samples based on the semantic segmentation model to obtain the first and second unlabeled prediction maps, a first inverse transformation operation is performed on the first unlabeled prediction map, which is an inverse flip along the x-axis. Similarly, a second inverse transformation operation is performed on the second unlabeled prediction map, which is a 90-degree rotation flip.

[0268] In some embodiments, the unlabeled loss (i.e., consistency loss) is defined as the mean squared error between different augmentation outputs (the unlabeled prediction maps corresponding to different augmented unlabeled samples). Taking the different augmentation outputs including the first unlabeled prediction map and the second unlabeled prediction map as an example, the unlabeled loss can be expressed as:

[0269]

[0270] Among them, L consistency This represents the unlabeled loss, also known as the consistency loss. N is the number of pixels in the unlabeled prediction map, j represents the j-th pixel in the unlabeled prediction map, and O 1j O represents the j-th pixel in the first unlabeled prediction map. 2j Let I represent the j-th pixel in the second unlabeled prediction map, where I1 represents the first inverse transformation operation and I2 represents the second inverse transformation operation.

[0271] It should be noted that, in general, inverse transformation operations are mainly for geometric transformations. This application does not limit this.

[0272] Step 4: In the first stage of training, the labeled loss and unlabeled loss are determined as the loss values ​​of the semantic segmentation model. The semantic segmentation model is trained based on the loss values ​​to obtain the pre-trained semantic segmentation model.

[0273] In some embodiments, a labeled loss is obtained based on the labeled prediction map corresponding to the labeled sample and the ground truth label. An unlabeled loss is obtained based on at least two augmented unlabeled samples. The labeled loss indicates the error between the labeled prediction map predicted by the semantic segmentation model and the ground truth label. The unlabeled loss indicates the error between at least two unlabeled prediction maps predicted by the semantic segmentation model.

[0274] Optionally, the semantic segmentation model can be trained by combining labeled and unlabeled losses to obtain a pre-trained semantic segmentation model. The pre-trained semantic segmentation model learns the features of unlabeled samples from different perspectives, enabling a more comprehensive understanding of the distribution and semantic information of unlabeled samples. Therefore, the pre-trained semantic segmentation model exhibits better image semantic segmentation capabilities.

[0275] Taking the example that the third weight corresponding to the labeled loss and the fourth weight corresponding to the unlabeled loss are equal, the loss value of the semantic segmentation model determined based on the labeled loss and the unlabeled loss can be expressed as:

[0276] L1 = L supervised +L consistency

[0277] Where L1 represents the loss value of the semantic segmentation model, that is, the loss value of the semantic segmentation model in the first stage of training. supervised This indicates the presence of labeling loss, also known as supervision loss (or monitoring loss), L. consistency This represents the unlabeled loss, also known as the consistency loss.

[0278] Step 5: In the second stage of training, the pre-trained semantic segmentation model is used to predict unlabeled samples to obtain pseudo-labels.

[0279] Optionally, the semantic segmentation models for the first and second stages are the same; or, the semantic segmentation models for the first and second stages are different, and this application does not limit this. The embodiments of this application mainly use the example of the first and second stages using the same semantic segmentation model, but this does not constitute a limitation on the model.

[0280] Based on the pre-trained semantic segmentation model obtained in the first stage, pseudo-labels are generated by predicting unlabeled samples, and a large number of unlabeled samples are used to expand the training dataset. Multiple training iterations are performed to optimize the pseudo-labels and improve the performance of the pre-trained semantic segmentation model. With each training iteration, the pre-trained semantic segmentation model regenerates pseudo-labels based on the updated network parameters, thereby improving its quality.

[0281] In some embodiments, the semantic segmentation model for the second stage is denoted as f, the network parameters are denoted as θ, and the unlabeled sample set in the second stage is denoted as D. u ={x j},j=1,…,N u Among them, x j Let N represent the j-th unlabeled sample, where the maximum number of unlabeled samples is N. The pseudo-labels generated during the n-th training iteration. It can be represented as:

[0282]

[0283] in, Let θ represent the pseudo-label generated in the nth training iteration (during the nth training iteration in the second phase). (n) These are the network parameters at the start of the nth training iteration.

[0284] Step 6: In the second stage of training, pseudo-labeled samples are obtained by combining pseudo-labeled and unlabeled samples, and expanded samples are obtained by combining pseudo-labeled samples with labeled samples.

[0285] In some embodiments, unlabeled samples are input into a pre-trained semantic segmentation model, which then predicts pseudo-labels from the unlabeled samples. The unlabeled samples and pseudo-labels are combined to obtain pseudo-labeled samples. These pseudo-labeled samples are image samples annotated with pseudo-labels. During the second stage of training, the pseudo-labeled samples can be used to expand the training set, utilizing the pseudo-labels generated by the pre-trained semantic segmentation model obtained in the first stage to assist in the second stage of training.

[0286] In some embodiments, the labeled sample set in the second phase is represented as x i Let y represent the i-th labeled sample in the set of labeled samples. i This represents the truth label corresponding to the i-th labeled sample, which is the same as the truth label G and N mentioned above. l This indicates the number of labeled samples in the labeled sample set.

[0287] In some embodiments, after obtaining pseudo-labeled samples based on unlabeled samples and pseudo-labels, expanded samples are obtained based on the pseudo-labeled samples and labeled samples. The expanded sample set in the second stage is represented as follows: x k This represents the k-th expanded sample in the expanded sample set. N represents the expanded label (i.e., the expanded label prediction map) corresponding to the k-th expanded sample. l N represents the number of labeled samples. u N represents the number of unlabeled samples. l +N u This indicates the number of expanded samples.

[0288] Step 7: In the second phase of training, calculate the expanded label loss for the expanded samples;

[0289] In some embodiments, the extended samples are input into a pre-trained semantic segmentation model, and predictions are made on the extended samples based on the pre-trained semantic segmentation model to obtain an extended label prediction map corresponding to the extended samples. The extended label prediction map contains the semantic prediction category for each pixel in the extended samples.

[0290] Optionally, an expanded label loss is obtained based on the expanded label prediction map and the ground truth labels. This expanded label loss indicates the error between the expanded label prediction map and the ground truth labels predicted by the pre-trained semantic segmentation model. Alternatively, it can be understood as the error between the semantic prediction category of each pixel in the expanded samples and the semantic category (true semantic category) of each pixel in the ground truth labels predicted by the pre-trained semantic segmentation model. Optionally, the pre-trained semantic segmentation model is trained using the expanded label loss to optimize its network parameters, resulting in a trained semantic segmentation model.

[0291] During the nth training iteration, a uniform loss function is used to calculate the expanded label loss for the expanded samples:

[0292]

[0293] Among them, L (m) N represents the expanded label loss. l +N u θ represents the number of expanded samples. (n) These are the network parameters at the start of the nth training iteration, x. k This represents the k-th expanded sample in the expanded sample set. This represents the expanded label (or expanded label prediction map) generated in the nth training iteration (the nth training iteration in the second phase).

[0294] In some embodiments, the expanded label loss, like the initial labeled loss (supervised loss), is a combination of the similarity coefficient loss (Dice loss) and the cross-entropy loss. The expanded label loss can also be expressed as:

[0295] L (n) =L supervised =L dice +L ce

[0296] Step 8: In the second stage of training, the extended label loss is determined as the loss value of the pre-trained semantic segmentation model. After multiple rounds of iterative training, the trained semantic segmentation model is obtained.

[0297] In some embodiments, the training process of the pre-trained semantic segmentation model in the second stage includes multiple rounds of iterative training. Optionally, unlabeled samples are input into the semantic segmentation model pre-trained in the i-th round in the second stage to obtain the i-th round pseudo-labels corresponding to the unlabeled samples; i-th round pseudo-labeled samples are obtained based on the unlabeled samples and the i-th round pseudo-labels, and i-th round expanded samples are obtained based on the i-th round pseudo-labeled samples and labeled samples; the i-th round expanded samples are input into the semantic segmentation model pre-trained in the i-th round to obtain the i-th round expanded label prediction map corresponding to the i-th round expanded samples, and the i-th round expanded label loss is obtained based on the i-th round expanded label prediction map and ground truth labels; the semantic segmentation model pre-trained in the i-th round is trained based on the i-th round expanded label loss, and the network parameters of the i-th round training are updated to obtain the semantic segmentation model for the (i+1)-th round training, where i is a positive integer. The above steps are repeated, and the iterative training process is stopped in response to reaching the maximum number of training iterations or reaching the convergence condition, resulting in a trained semantic segmentation model.

[0298] Beneficial effects of this plan

[0299] The training method for the semantic segmentation model proposed in this application has at least the following beneficial effects:

[0300] (1) Effective generation and optimization of pseudo-tags

[0301] The pre-trained semantic segmentation model generates pseudo-labels, which are then continuously optimized in subsequent training iterations, gradually improving their quality. These high-quality pseudo-labels not only expand the training dataset but also provide richer and more accurate information for the semantic segmentation model's learning, further driving its performance improvement.

[0302] (2) Improve data utilization efficiency

[0303] In the first stage (or initial stage), the labeled loss (or supervised loss) for labeled samples and the unlabeled loss (or consistency loss) for unlabeled samples are combined, enabling the semantic segmentation model to make full use of a large amount of unlabeled data information. That is, it makes full use of the data information in unlabeled samples, which greatly improves the efficiency of data use, avoids the limitations of relying on only a small number of labeled samples for training, and expands the breadth and depth of semantic segmentation model learning.

[0304] (3) Enhance the generalization ability of the model

[0305] Contrastive learning methods provide diverse random augmentations to unlabeled samples, enabling the semantic segmentation model to learn data features from unlabeled samples from different perspectives. This enhances the semantic segmentation model's adaptability to various complex scenarios and data variations, significantly improves its generalization performance on unseen data, and reduces the risk of overfitting.

[0306] (4) Improve training stability

[0307] During training, a unified loss function is used to calculate the merged dataset, maintaining the consistency and stability of the training process and avoiding training fluctuations caused by differences in data processing methods. This helps the semantic segmentation model converge more smoothly to a better parameter state, thereby improving training efficiency and the performance of the trained semantic segmentation model.

[0308] Application Scenario Description

[0309] The training methods for semantic segmentation models can be applied to multiple domains. Examples from several domains are provided below.

[0310] Application Scenario 1

[0311] The trained semantic segmentation model is used as an example in the field of autonomous driving.

[0312] Step 1: Obtain labeled road image samples and unlabeled road image samples;

[0313] Labeled road image samples are those with truth labels, while unlabeled road image samples are those without truth labels. Road truth labels can also be called road truth label maps. When the road image sample is a two-dimensional image sample, the road truth label is used to indicate the semantic category of each pixel in the road image sample.

[0314] For example, the pixels of road image sample A include four categories. Road image sample A is an image containing roads, pedestrians, traffic lights, and vehicles. The road truth label is used to indicate the semantic category of each pixel in road image sample A. For example, the road truth label indicates that the first part of the pixels in road image sample A belongs to "road", the second part belongs to "pedestrians", the third part belongs to "traffic lights", and the fourth part belongs to "vehicles".

[0315] Step 2: Input the labeled road image samples into the semantic segmentation model to obtain the labeled prediction map corresponding to the labeled road image samples, and obtain the labeled loss based on the labeled prediction map and the ground truth road labels;

[0316] In some embodiments, labeled road image samples are input into a semantic segmentation model, and predictions are made based on the semantic segmentation model to obtain a labeled prediction map corresponding to the labeled road image samples. The labeled prediction map contains the semantic prediction category for each pixel in the labeled road image sample.

[0317] In some embodiments, labeled road image samples are input into a semantic segmentation model to obtain image feature representations corresponding to the labeled road image samples. Based on these semantic segmentation models, predictions are made on the road image feature representations corresponding to the labeled road image samples to obtain labeled prediction maps. Optionally, a transformer in the semantic segmentation model converts the labeled road image samples into corresponding road image feature representations. These road image feature representations indicate the content features of the labeled road image samples; that is, they indicate the content features extracted from the labeled road image samples.

[0318] Optionally, a labeled loss is derived based on the labeled prediction map and the ground truth road labels. The labeled loss is used to indicate the error between the labeled prediction map predicted by the semantic segmentation model (the untrained image semantic segmentation model, or the general semantic segmentation model) and the ground truth road labels. Alternatively, the labeled loss can be understood as indicating the error between the semantic prediction category of each pixel in the labeled road image sample predicted by the semantic segmentation model and the semantic category (semantic true category) of each pixel in the ground truth road labels.

[0319] In some embodiments, the label loss includes similarity coefficient loss and cross-entropy loss.

[0320] Optionally, labeled road image samples are input into a semantic segmentation model to obtain labeled prediction maps corresponding to the labeled road image samples. A first operation is performed on the labeled prediction maps and ground truth road labels to obtain a similarity coefficient loss, which measures the degree of overlap between the labeled prediction maps and ground truth road labels. A second operation is performed on the labeled prediction maps and ground truth road labels to obtain a cross-entropy loss, which measures the difference between the labeled prediction maps and ground truth road labels. Based on the similarity coefficient loss and the cross-entropy loss, the labeled loss is calculated.

[0321] In some embodiments, a first weight corresponding to the similarity coefficient loss and a second weight corresponding to the cross-entropy loss are obtained; a first product of the similarity coefficient loss and the first weight is calculated, and a second product of the cross-entropy loss and the second weight is calculated; the sum of the first product and the second product is determined as the labeled loss.

[0322] For details on calculating labeled loss, please refer to the relevant sections on "Calculating Labeled Loss" mentioned above. These details will not be repeated here.

[0323] Step 3: Perform enhancement operations on the unlabeled road image samples to generate at least two enhanced unlabeled road image samples;

[0324] The enhancement operation is used to perturb unlabeled road image samples from at least two perspectives. At least two perspectives means processing the unlabeled road image samples from at least two angles or in at least two ways.

[0325] In some embodiments, an enhancement operation is performed on unlabeled road image samples, perturbing the unlabeled road image samples from at least two perspectives to generate at least two enhanced unlabeled road image samples, that is, generating enhanced unlabeled road image samples from at least two perspectives. Here, enhanced unlabeled road image samples refer to unlabeled road image samples obtained after performing a random enhancement operation on unlabeled road image samples, or can be understood as unlabeled road image samples obtained after perturbing unlabeled road image samples.

[0326] Optionally, the enhancement operation includes, but is not limited to, at least one of the following operations:

[0327] • Geometric transformation operations based on the first probability;

[0328] • Color transformation operation based on the second probability;

[0329] • Noise injection operation based on third probability;

[0330] • Hybrid operations based on the fourth probability.

[0331] The geometric transformation operations described above include, but are not limited to: horizontal flipping, vertical flipping, rotation, cropping, scaling, or at least one of other geometric transformation operations. The color transformation operations described above include, but are not limited to: brightness transformation, contrast transformation, saturation transformation, hue transformation, or at least one of other color transformation operations.

[0332] For details on the enhanced operation, please refer to the relevant content in step 230 above, which will not be repeated here.

[0333] In some embodiments, a geometric transformation operation based on a first probability is performed on unlabeled road image samples to generate geometrically transformed enhanced unlabeled road image samples, which are unlabeled road image samples after the geometric transformation operation. In some embodiments, a color transformation operation based on a second probability is performed on unlabeled road image samples to generate color-transformed enhanced unlabeled road image samples, which are unlabeled road image samples after the color transformation operation. In some embodiments, a noise injection operation based on a third probability is performed on unlabeled road image samples to generate noise-enhanced unlabeled road image samples, which are unlabeled road image samples after the noise injection operation. In some embodiments, a mixing operation based on a fourth probability is performed on unlabeled road image samples to generate mixed-enhanced unlabeled road image samples, which are unlabeled road image samples after the mixing operation.

[0334] In this embodiment of the application, by perturbing the unlabeled road image samples from multiple angles, the semantic segmentation model's adaptability to different conditions can be enhanced.

[0335] Step 4: Input at least two enhanced unlabeled road image samples into the semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two enhanced unlabeled road image samples, and obtain unlabeled loss based on the at least two unlabeled prediction maps;

[0336] Among them, augmented unlabeled road image samples refer to unlabeled road image samples obtained after performing random augmentation operations on unlabeled road image samples.

[0337] In some embodiments, at least two enhanced unlabeled road image samples are input into a semantic segmentation model. Based on the semantic segmentation model, predictions are made for each of the at least two enhanced unlabeled road image samples to obtain unlabeled prediction maps corresponding to each of the at least two enhanced unlabeled road image samples. The unlabeled prediction map contains the semantic prediction category for each pixel in the enhanced unlabeled road image sample.

[0338] The following explanation uses at least two enhanced unlabeled road image samples, including a first enhanced unlabeled road image sample and a second enhanced unlabeled road image sample, as an example. Optionally, a first enhancement operation is performed on the unlabeled road image sample to generate a first enhanced unlabeled road image sample based on the first enhancement operation; a second enhancement operation is performed on the unlabeled road image sample to generate a second enhanced unlabeled road image sample based on the second enhancement operation.

[0339] In some embodiments, at least two enhanced unlabeled road image samples are input into a semantic segmentation model to obtain unlabeled prediction maps corresponding to at least two enhanced unlabeled road image samples. Optionally, a first enhanced unlabeled sample is input into the semantic segmentation model, and the semantic segmentation model predicts the first enhanced unlabeled road image sample to obtain a first unlabeled prediction map corresponding to the first enhanced unlabeled road image sample. The first unlabeled prediction map contains a semantic prediction category for each pixel in the first enhanced unlabeled road image sample. Optionally, a second enhanced unlabeled road image sample is input into the semantic segmentation model, and the semantic segmentation model predicts the second enhanced unlabeled road image sample to obtain a second unlabeled prediction map corresponding to the second enhanced unlabeled road image sample. The second unlabeled prediction map contains a semantic prediction category for each pixel in the second enhanced unlabeled road image sample.

[0340] In some embodiments, when calculating the unlabeled loss based on at least two unlabeled prediction maps, an inverse transformation operation is performed on the at least two unlabeled prediction maps to obtain at least two inversely transformed unlabeled prediction maps, and the unlabeled loss is obtained based on the at least two inversely transformed unlabeled prediction maps. Optionally, a first inverse transformation operation is performed on the first unlabeled prediction map to obtain an inversely transformed first unlabeled prediction map, the first inverse transformation operation corresponding to a first enhancement operation. Optionally, a second inverse transformation operation is performed on the second unlabeled prediction map to obtain an inversely transformed second unlabeled prediction map, the second inverse transformation operation corresponding to a second enhancement operation. The unlabeled loss is obtained based on the inversely transformed first unlabeled prediction map and the inversely transformed second unlabeled prediction map. The unlabeled loss is used to indicate the error between the first and second unlabeled prediction maps predicted by the semantic segmentation model. Further, the unlabeled loss is used to indicate the error between the inversely transformed first and second unlabeled prediction maps.

[0341] For example, the first enhancement operation is a horizontal flip operation, where the first enhanced unlabeled road image sample is flipped along the x-axis compared to the original unlabeled road image sample. The second enhancement operation is a rotation flip operation, where the second enhanced unlabeled road image sample is rotated 90 degrees compared to the original unlabeled road image sample. After predicting the first and second enhanced unlabeled road image samples based on the semantic segmentation model, and obtaining the first and second unlabeled prediction maps, a first inverse transformation operation is performed on the first unlabeled prediction map, which is an inverse x-axis flip operation. Similarly, a second inverse transformation operation is performed on the second unlabeled prediction map, which is a 90-degree rotation flip operation.

[0342] For details on calculating unlabeled loss, please refer to the relevant sections on "Calculating Unlabeled Loss" mentioned above. These details will not be repeated here.

[0343] In this embodiment, the first and second unlabeled prediction maps after inverse transformation are prediction results recovered from enhanced unlabeled road image samples after different enhancement operations and then after inverse transformation. The differences between them reflect the adaptability and accuracy of the semantic segmentation model to road image sample transformations. Calculating the unlabeled loss based on the inverse-transformed unlabeled prediction maps means recovering the original features of the unlabeled samples through inverse transformation operations, and then calculating the differences between different versions of the unlabeled samples to optimize the semantic segmentation model and improve its robustness to data changes.

[0344] Step 5: Train the semantic segmentation model based on labeled loss and unlabeled loss to obtain the pre-trained semantic segmentation model.

[0345] In some embodiments, a third weight corresponding to the labeled loss and a fourth weight corresponding to the unlabeled loss are obtained; a third product of the labeled loss and the third weight is calculated, and a fourth product of the unlabeled loss and the fourth weight is calculated; the sum of the third product and the fourth product is determined as the loss value of the semantic segmentation model, and the semantic segmentation model is trained based on the loss value to obtain a pre-trained semantic segmentation model.

[0346] In some embodiments, a pre-trained semantic segmentation model can be obtained by performing semi-supervised training (combining labeled and unlabeled road image samples) on the semantic segmentation model in the first stage. The pre-trained semantic segmentation model has good image semantic segmentation capabilities for unlabeled samples. In the second stage, the performance of the semantic segmentation model is further improved through road pseudo-label generation and multiple training optimizations.

[0347] In some embodiments, unlabeled road image samples are input into a pre-trained semantic segmentation model to obtain road pseudo-labels corresponding to the unlabeled road image samples; pseudo-labeled road image samples are obtained based on the unlabeled road image samples and the road pseudo-labels, and extended road image samples are obtained based on the pseudo-labeled road image samples and labeled road image samples; the extended road image samples are input into the pre-trained semantic segmentation model to obtain an extended label prediction map corresponding to the extended road image samples, and an extended label loss is obtained based on the extended label prediction map and the ground truth road labels; the pre-trained semantic segmentation model is trained based on the extended label loss to obtain a trained semantic segmentation model.

[0348] In some embodiments, the pre-trained semantic segmentation model's training process in the second stage includes multiple rounds of iterative training. Optionally, based on the extended label loss of the i-th round, the network parameters of the pre-trained semantic segmentation model trained in the i-th round are updated to obtain the semantic segmentation model for the (i+1)-th round of training. The semantic segmentation model trained in the (i+1)-th round generates pseudo-labels for the unlabeled road image samples corresponding to the (i+1)-th round of road pseudo-labels, where i is a positive integer.

[0349] Specifically, unlabeled road image samples are input into the semantic segmentation model pre-trained in the second stage to obtain the i-th round pseudo-labels for the unlabeled road image samples. Based on the unlabeled road image samples and the i-th round pseudo-labels, i-th round pseudo-labeled road image samples are obtained, along with i-th round extended road image samples based on the i-th round pseudo-labeled road image samples and labeled road image samples. The i-th round extended road image samples are input into the i-th round pre-trained semantic segmentation model to obtain the i-th round extended label prediction map, along with the i-th round extended label loss based on the i-th round extended label prediction map and ground truth road labels. The i-th round pre-trained semantic segmentation model is trained based on the i-th round extended label loss, and the network parameters from the i-th round training are updated to obtain the semantic segmentation model for the (i+1)-th round training, where i is a positive integer. These steps are repeated until the maximum number of training iterations is reached or the convergence condition is met, at which point the iterative training process stops, resulting in a trained semantic segmentation model.

[0350] In summary, by combining labeled and unlabeled road image samples for semi-supervised training of the semantic segmentation model, the process first performs random augmentation on the unlabeled road image samples, perturbing them from at least two different perspectives to obtain at least two augmented unlabeled road image samples. An unlabeled loss is then derived based on these two augmented samples, and the semantic segmentation model is trained by combining this unlabeled loss with the labeled loss corresponding to the labeled samples. The unlabeled loss obtained from at least two augmented unlabeled road image samples allows the semantic segmentation model to learn the features of unlabeled road image samples from different perspectives, enabling it to more comprehensively understand the semantic information within the samples. This allows for more efficient utilization of the latent information in the unlabeled road image samples, improving the learning effect and performance of the semantic segmentation model. For example, in the field of autonomous driving, the trained semantic segmentation model can be used to identify road elements such as obstacles in road images, thereby improving the accuracy of target recognition in autonomous driving and ensuring its safety.

[0351] Application Scenario 2

[0352] The trained semantic segmentation model is used as an example in the field of medical image segmentation.

[0353] Step 1: Obtain labeled and unlabeled medical image samples;

[0354] Labeled medical image samples are those with ground truth labels, while unlabeled medical image samples are those without ground truth labels. Medical ground truth labels are also called medical ground truth map. When the medical image sample is a two-dimensional image sample, the medical ground truth label is used to indicate the semantic category of each pixel in the medical image sample.

[0355] For example, the pixels of medical image sample A include three categories. Medical image sample A is an image containing a heart region, a blood region, and a bone region. The medical truth value label is used to indicate the semantic category of each pixel in medical image sample A. For example, the medical truth value label indicates that the first part of the pixels in medical image sample A belongs to "heart", the second part of the pixels belongs to "blood", and the third part of the pixels belongs to "bone".

[0356] Step 2: Input the labeled medical image samples into the semantic segmentation model to obtain the labeled prediction map corresponding to the labeled medical image samples, and obtain the labeled loss based on the labeled prediction map and the medical ground truth label.

[0357] In some embodiments, labeled medical image samples are input into a semantic segmentation model, and predictions are made based on the semantic segmentation model to obtain a labeled prediction map corresponding to the labeled medical image sample. The labeled prediction map contains the semantic prediction category for each pixel in the labeled medical image sample.

[0358] In some embodiments, labeled medical image samples are input into a semantic segmentation model to obtain image feature representations corresponding to the labeled medical image samples. Based on the semantic segmentation model, the medical image feature representations corresponding to the labeled medical image samples are predicted to obtain labeled prediction maps corresponding to the labeled medical image samples. Optionally, a transformer in the semantic segmentation model converts the labeled medical image samples into corresponding medical image feature representations. The medical image feature representations are used to indicate the content features of the labeled medical image samples; that is, the medical image feature representations are used to indicate the content features extracted from the labeled medical image samples.

[0359] Optionally, a labeled loss is derived based on the labeled prediction map and the medical ground truth labels. The labeled loss is used to indicate the error between the labeled prediction map predicted by the semantic segmentation model (the untrained image semantic segmentation model, or the general semantic segmentation model) and the medical ground truth labels. Alternatively, the labeled loss can be understood as indicating the error between the semantic prediction category of each pixel in the labeled medical image sample predicted by the semantic segmentation model and the semantic category (semantic true category) of each pixel in the medical ground truth labels.

[0360] In some embodiments, the label loss includes similarity coefficient loss and cross-entropy loss.

[0361] Optionally, labeled medical image samples are input into a semantic segmentation model to obtain labeled prediction maps corresponding to the labeled medical image samples. A first operation is performed on the labeled prediction maps and ground truth medical labels to obtain a similarity coefficient loss, which measures the degree of overlap between the labeled prediction maps and ground truth medical labels. A second operation is performed on the labeled prediction maps and ground truth medical labels to obtain a cross-entropy loss, which measures the difference between the labeled prediction maps and ground truth medical labels. Based on the similarity coefficient loss and the cross-entropy loss, the labeled loss is calculated.

[0362] In some embodiments, a first weight corresponding to the similarity coefficient loss and a second weight corresponding to the cross-entropy loss are obtained; a first product of the similarity coefficient loss and the first weight is calculated, and a second product of the cross-entropy loss and the second weight is calculated; the sum of the first product and the second product is determined as the labeled loss.

[0363] For details on calculating labeled loss, please refer to the relevant sections on "Calculating Labeled Loss" mentioned above. These details will not be repeated here.

[0364] Step 3: Perform enhancement operations on the unlabeled medical image samples to generate at least two enhanced unlabeled medical image samples;

[0365] The enhancement operation is used to perturb unlabeled medical image samples from at least two perspectives. At least two perspectives means processing the unlabeled medical image samples from at least two angles or in at least two ways.

[0366] In some embodiments, an enhancement operation is performed on unlabeled medical image samples, perturbing the unlabeled medical image samples from at least two perspectives to generate at least two enhanced unlabeled medical image samples, that is, generating enhanced unlabeled medical image samples from at least two perspectives. Here, enhanced unlabeled medical image samples refer to unlabeled medical image samples obtained after performing a random enhancement operation on unlabeled medical image samples, or can be understood as unlabeled medical image samples obtained after perturbing unlabeled medical image samples.

[0367] Optionally, the enhancement operation includes, but is not limited to, at least one of the following operations:

[0368] • Geometric transformation operations based on the first probability;

[0369] • Color transformation operation based on the second probability;

[0370] • Noise injection operation based on third probability;

[0371] • Hybrid operations based on the fourth probability.

[0372] The geometric transformation operations described above include, but are not limited to: horizontal flipping, vertical flipping, rotation, cropping, scaling, or at least one of other geometric transformation operations. The color transformation operations described above include, but are not limited to: brightness transformation, contrast transformation, saturation transformation, hue transformation, or at least one of other color transformation operations.

[0373] For details on the enhanced operation, please refer to the relevant content in step 230 above, which will not be repeated here.

[0374] In some embodiments, a geometric transformation operation based on a first probability is performed on an unlabeled medical image sample, and a geometrically enhanced unlabeled medical image sample is generated according to the first probability. The geometrically enhanced unlabeled medical image sample is an unlabeled medical image sample after undergoing the geometric transformation operation. In some embodiments, a color transformation operation based on a second probability is performed on an unlabeled medical image sample, and a color transformation enhanced unlabeled medical image sample is generated according to the second probability. The color transformation enhanced unlabeled medical image sample is an unlabeled medical image sample after undergoing the color transformation operation. In some embodiments, a noise injection operation based on a third probability is performed on an unlabeled medical image sample, and a noise-enhanced unlabeled medical image sample is generated according to the third probability. The noise-enhanced unlabeled medical image sample is an unlabeled medical image sample after undergoing the noise injection operation. In some embodiments, a mixing operation based on a fourth probability is performed on an unlabeled medical image sample, and a mixed enhanced unlabeled medical image sample is generated according to the fourth probability. The mixed enhanced unlabeled medical image sample is an unlabeled medical image sample after undergoing the mixing operation.

[0375] In this embodiment of the application, by perturbing the unlabeled medical image sample from multiple angles, the semantic segmentation model's adaptability to different conditions can be enhanced.

[0376] Step 4: Input at least two enhanced unlabeled medical image samples into the semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two enhanced unlabeled medical image samples, and obtain unlabeled loss based on the at least two unlabeled prediction maps;

[0377] Among them, augmented unlabeled medical image samples refer to unlabeled medical image samples obtained after performing random augmentation operations on unlabeled medical image samples.

[0378] In some embodiments, at least two enhanced unlabeled medical image samples are input into a semantic segmentation model. Based on the semantic segmentation model, predictions are made for each of the at least two enhanced unlabeled medical image samples to obtain unlabeled prediction maps corresponding to each of the at least two enhanced unlabeled medical image samples. The unlabeled prediction map contains the semantic prediction category for each pixel in the enhanced unlabeled medical image sample.

[0379] The following description uses at least two enhanced unlabeled medical image samples, including a first enhanced unlabeled medical image sample and a second enhanced unlabeled medical image sample, as an example. Optionally, a first enhancement operation is performed on the unlabeled medical image sample to generate a first enhanced unlabeled medical image sample based on the first enhancement operation; a second enhancement operation is performed on the unlabeled medical image sample to generate a second enhanced unlabeled medical image sample based on the second enhancement operation.

[0380] In some embodiments, at least two enhanced unlabeled medical image samples are input into a semantic segmentation model to obtain unlabeled prediction maps corresponding to at least two enhanced unlabeled medical image samples, respectively. Optionally, a first enhanced unlabeled sample is input into the semantic segmentation model, and the semantic segmentation model predicts the first enhanced unlabeled medical image sample to obtain a first unlabeled prediction map corresponding to the first enhanced unlabeled medical image sample. The first unlabeled prediction map contains a semantic prediction category for each pixel in the first enhanced unlabeled medical image sample. Optionally, a second enhanced unlabeled medical image sample is input into the semantic segmentation model, and the semantic segmentation model predicts the second enhanced unlabeled medical image sample to obtain a second unlabeled prediction map corresponding to the second enhanced unlabeled medical image sample. The second unlabeled prediction map contains a semantic prediction category for each pixel in the second enhanced unlabeled medical image sample.

[0381] In some embodiments, when calculating the unlabeled loss based on at least two unlabeled prediction maps, an inverse transformation operation is performed on the at least two unlabeled prediction maps to obtain at least two inversely transformed unlabeled prediction maps, and the unlabeled loss is obtained based on the at least two inversely transformed unlabeled prediction maps. Optionally, a first inverse transformation operation is performed on the first unlabeled prediction map to obtain an inversely transformed first unlabeled prediction map, the first inverse transformation operation corresponding to a first enhancement operation. Optionally, a second inverse transformation operation is performed on the second unlabeled prediction map to obtain an inversely transformed second unlabeled prediction map, the second inverse transformation operation corresponding to a second enhancement operation. The unlabeled loss is obtained based on the inversely transformed first unlabeled prediction map and the inversely transformed second unlabeled prediction map. The unlabeled loss is used to indicate the error between the first and second unlabeled prediction maps predicted by the semantic segmentation model. Further, the unlabeled loss is used to indicate the error between the inversely transformed first and second unlabeled prediction maps.

[0382] For example, the first enhancement operation is a horizontal flip operation, where the first enhanced unlabeled medical image sample is flipped along the x-axis compared to the original unlabeled medical image sample. The second enhancement operation is a rotation flip operation, where the second enhanced unlabeled medical image sample is rotated 90 degrees compared to the original unlabeled medical image sample. After predicting the first and second enhanced unlabeled medical image samples based on the semantic segmentation model, and obtaining the first and second unlabeled prediction maps, a first inverse transformation operation is performed on the first unlabeled prediction map, which is an inverse x-axis flip operation. Similarly, a second inverse transformation operation is performed on the second unlabeled prediction map, which is a 90-degree rotation flip operation.

[0383] For details on calculating unlabeled loss, please refer to the relevant sections on "Calculating Unlabeled Loss" mentioned above. These details will not be repeated here.

[0384] In this embodiment, the first and second unlabeled prediction maps after inverse transformation are prediction results of enhanced unlabeled medical image samples after different enhancement operations, restored by inverse transformation. The differences between them reflect the adaptability and accuracy of the semantic segmentation model to the transformation of medical image samples. Calculating the unlabeled loss based on the inverse-transformed unlabeled prediction maps means restoring the original features of the unlabeled samples through inverse transformation operations, and then calculating the differences between different versions of the unlabeled samples to optimize the semantic segmentation model and improve its robustness to data changes.

[0385] Step 5: Train the semantic segmentation model based on labeled loss and unlabeled loss to obtain the pre-trained semantic segmentation model.

[0386] In some embodiments, a third weight corresponding to the labeled loss and a fourth weight corresponding to the unlabeled loss are obtained; a third product of the labeled loss and the third weight is calculated, and a fourth product of the unlabeled loss and the fourth weight is calculated; the sum of the third product and the fourth product is determined as the loss value of the semantic segmentation model, and the semantic segmentation model is trained based on the loss value to obtain a pre-trained semantic segmentation model.

[0387] In some embodiments, a pre-trained semantic segmentation model can be obtained by performing semi-supervised training (combining labeled and unlabeled medical image samples) on the semantic segmentation model in the first stage. The pre-trained semantic segmentation model has good image semantic segmentation capabilities for unlabeled samples. In the second stage, the performance of the semantic segmentation model is further improved through medical pseudo-label generation and multiple training optimizations.

[0388] In some embodiments, unlabeled medical image samples are input into a pre-trained semantic segmentation model to obtain medical pseudo-labels corresponding to the unlabeled medical image samples; pseudo-labeled medical image samples are obtained based on the unlabeled medical image samples and medical pseudo-labels, and extended medical image samples are obtained based on the pseudo-labeled medical image samples and labeled medical image samples; the extended medical image samples are input into the pre-trained semantic segmentation model to obtain an extended label prediction map corresponding to the extended medical image samples, and an extended label loss is obtained based on the extended label prediction map and medical ground truth labels; the pre-trained semantic segmentation model is trained based on the extended label loss to obtain a trained semantic segmentation model.

[0389] In some embodiments, the pre-trained semantic segmentation model's training process in the second stage includes multiple rounds of iterative training. Optionally, based on the extended label loss of the i-th round, the network parameters of the pre-trained semantic segmentation model trained in the i-th round are updated to obtain the semantic segmentation model for the (i+1)-th round of training. The semantic segmentation model trained in the (i+1)-th round generates medical pseudo-labels for the unlabeled medical image samples in the (i+1)-th round. The medical pseudo-labels in the (i+1)-th round are pseudo-labels optimized from the medical pseudo-labels in the i-th round, where i is a positive integer.

[0390] Specifically, unlabeled medical image samples are input into the semantic segmentation model pre-trained in the second stage to obtain the i-th round of medical pseudo-labels corresponding to the unlabeled medical image samples. Based on the unlabeled medical image samples and the i-th round of medical pseudo-labels, i-th round of pseudo-labeled medical image samples are obtained, along with i-th round of expanded medical image samples based on the i-th round of pseudo-labeled medical image samples and labeled medical image samples. The i-th round of expanded medical image samples are input into the semantic segmentation model pre-trained in the i-th round to obtain the i-th round of expanded label prediction map corresponding to the i-th round of expanded medical image samples, along with the i-th round of expanded label loss based on the i-th round of expanded label prediction map and the ground truth medical labels. The semantic segmentation model pre-trained in the i-th round is trained based on the i-th round of expanded label loss, and the network parameters from the i-th round of training are updated to obtain the semantic segmentation model for the (i+1)-th round of training, where i is a positive integer. The above steps are repeated until the maximum number of training iterations is reached or the convergence condition is met, at which point the iterative training process stops, and the trained semantic segmentation model is obtained.

[0391] In summary, by combining labeled and unlabeled medical image samples for semi-supervised training of the semantic segmentation model, when processing unlabeled medical image samples, a random augmentation operation is first performed on the unlabeled medical image samples, perturbing them from at least two different perspectives to obtain at least two augmented unlabeled medical image samples. An unlabeled loss is then obtained based on these at least two augmented unlabeled medical image samples. This unlabeled loss, combined with the labeled loss corresponding to the labeled samples, is used to train the semantic segmentation model. The unlabeled loss obtained from at least two augmented unlabeled medical image samples allows the semantic segmentation model to learn the features of unlabeled medical image samples from different perspectives, enabling the model to more comprehensively understand the semantic information in the unlabeled medical image samples. This allows for more full utilization of the potential information in the unlabeled medical image samples, improving the learning effect and performance of the semantic segmentation model. For example, in the field of medical image processing, the trained semantic segmentation model can be used to identify lesion regions from medical images, helping doctors make more accurate diagnoses.

[0392] Application Scenario 3

[0393] The application of a trained semantic segmentation model to the field of agricultural scene analysis is illustrated.

[0394] Step 1: Obtain labeled and unlabeled agricultural image samples;

[0395] Labeled agricultural image samples are those labeled with ground truth values, while unlabeled agricultural image samples are those without ground truth values. Agricultural ground truth labels can also be called agricultural ground truth map. When the agricultural image sample is a two-dimensional image sample, the agricultural ground truth label is used to indicate the semantic category of each pixel in the agricultural image sample.

[0396] For example, the pixels of agricultural image sample A include three categories. Agricultural image sample A is an image containing crop leaves, crop roots and stems, and pest and disease areas. The agricultural truth value label is used to indicate the semantic category of each pixel in agricultural image sample A. For example, the agricultural truth value label indicates that the first part of the pixels in agricultural image sample A belongs to "crop leaves", the second part of the pixels belongs to "crop roots and stems", and the third part of the pixels belongs to "pest and disease areas".

[0397] Step 2: Input the labeled agricultural image samples into the semantic segmentation model to obtain the labeled prediction map corresponding to the labeled agricultural image samples, and obtain the labeled loss based on the labeled prediction map and the ground truth agricultural labels.

[0398] In some embodiments, labeled agricultural image samples are input into a semantic segmentation model, and predictions are made based on the semantic segmentation model to obtain a labeled prediction map corresponding to the labeled agricultural image sample. The labeled prediction map contains the semantic prediction category for each pixel in the labeled agricultural image sample.

[0399] In some embodiments, labeled agricultural image samples are input into a semantic segmentation model to obtain image feature representations corresponding to the labeled agricultural image samples. Based on the semantic segmentation model, predictions are made on the agricultural image feature representations corresponding to the labeled agricultural image samples to obtain labeled prediction maps corresponding to the labeled agricultural image samples. Optionally, a transformer in the semantic segmentation model converts the labeled agricultural image samples into corresponding agricultural image feature representations. The agricultural image feature representations are used to indicate the content features of the labeled agricultural image samples; that is, the agricultural image feature representations are used to indicate the content features extracted from the labeled agricultural image samples.

[0400] Optionally, a labeled loss is derived based on the labeled prediction map and the agricultural ground truth label. The labeled loss is used to indicate the error between the labeled prediction map predicted by the semantic segmentation model (the untrained image semantic segmentation model, or the general semantic segmentation model) and the agricultural ground truth label. Alternatively, the labeled loss can be understood as indicating the error between the semantic prediction category of each pixel in the labeled agricultural image sample predicted by the semantic segmentation model and the semantic category (semantic true category) of each pixel in the agricultural ground truth label.

[0401] In some embodiments, the label loss includes similarity coefficient loss and cross-entropy loss.

[0402] Optionally, labeled agricultural image samples are input into a semantic segmentation model to obtain labeled prediction maps corresponding to the labeled agricultural image samples. A first operation is performed on the labeled prediction maps and the ground truth agricultural labels to obtain a similarity coefficient loss, which measures the degree of overlap between the labeled prediction maps and the ground truth agricultural labels. A second operation is performed on the labeled prediction maps and the ground truth agricultural labels to obtain a cross-entropy loss, which measures the difference between the labeled prediction maps and the ground truth agricultural labels. Based on the similarity coefficient loss and the cross-entropy loss, the labeled loss is calculated.

[0403] In some embodiments, a first weight corresponding to the similarity coefficient loss and a second weight corresponding to the cross-entropy loss are obtained; a first product of the similarity coefficient loss and the first weight is calculated, and a second product of the cross-entropy loss and the second weight is calculated; the sum of the first product and the second product is determined as the labeled loss.

[0404] For details on calculating labeled loss, please refer to the relevant sections on "Calculating Labeled Loss" mentioned above. These details will not be repeated here.

[0405] Step 3: Perform enhancement operations on the unlabeled agricultural image samples to generate at least two enhanced unlabeled agricultural image samples;

[0406] The enhancement operation is used to perturb unlabeled agricultural image samples from at least two perspectives. At least two perspectives means processing the unlabeled agricultural image samples from at least two angles or in at least two ways.

[0407] In some embodiments, an enhancement operation is performed on unlabeled agricultural image samples, perturbing the unlabeled agricultural image samples from at least two perspectives to generate at least two enhanced unlabeled agricultural image samples, that is, generating enhanced unlabeled agricultural image samples from at least two perspectives. Here, enhanced unlabeled agricultural image samples refer to unlabeled agricultural image samples obtained after performing a random enhancement operation on unlabeled agricultural image samples, or can be understood as unlabeled agricultural image samples obtained after perturbing unlabeled agricultural image samples.

[0408] Optionally, the enhancement operation includes, but is not limited to, at least one of the following operations:

[0409] • Geometric transformation operations based on the first probability;

[0410] • Color transformation operation based on the second probability;

[0411] • Noise injection operation based on third probability;

[0412] • Hybrid operations based on the fourth probability.

[0413] The geometric transformation operations described above include, but are not limited to: horizontal flipping, vertical flipping, rotation, cropping, scaling, or at least one of other geometric transformation operations. The color transformation operations described above include, but are not limited to: brightness transformation, contrast transformation, saturation transformation, hue transformation, or at least one of other color transformation operations.

[0414] For details on the enhanced operation, please refer to the relevant content in step 230 above, which will not be repeated here.

[0415] In some embodiments, a geometric transformation operation based on a first probability is performed on unlabeled agricultural image samples to generate geometrically enhanced unlabeled agricultural image samples, which are the unlabeled agricultural image samples after the geometric transformation operation. In some embodiments, a color transformation operation based on a second probability is performed on unlabeled agricultural image samples to generate color-enhanced unlabeled agricultural image samples, which are the unlabeled agricultural image samples after the color transformation operation. In some embodiments, a noise injection operation based on a third probability is performed on unlabeled agricultural image samples to generate noise-enhanced unlabeled agricultural image samples, which are the unlabeled agricultural image samples after the noise injection operation. In some embodiments, a mixing operation based on a fourth probability is performed on unlabeled agricultural image samples to generate mixed-enhanced unlabeled agricultural image samples, which are the unlabeled agricultural image samples after the mixing operation.

[0416] In this embodiment of the application, by perturbing the unlabeled agricultural image samples from multiple angles, the semantic segmentation model's adaptability to different conditions can be enhanced.

[0417] Step 4: Input at least two augmented unlabeled agricultural image samples into the semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two augmented unlabeled agricultural image samples, and obtain unlabeled loss based on the at least two unlabeled prediction maps;

[0418] Among them, augmented unlabeled agricultural image samples refer to unlabeled agricultural image samples obtained after performing random augmentation operations on unlabeled agricultural image samples.

[0419] In some embodiments, at least two augmented unlabeled agricultural image samples are input into a semantic segmentation model. Based on the semantic segmentation model, predictions are made for each of the at least two augmented unlabeled agricultural image samples to obtain unlabeled prediction maps corresponding to each of the at least two augmented unlabeled agricultural image samples. The unlabeled prediction map contains the semantic prediction category for each pixel in the augmented unlabeled agricultural image sample.

[0420] The following explanation uses at least two enhanced unlabeled agricultural image samples, including a first enhanced unlabeled agricultural image sample and a second enhanced unlabeled agricultural image sample, as an example. Optionally, a first enhancement operation is performed on the unlabeled agricultural image sample to generate a first enhanced unlabeled agricultural image sample based on the first enhancement operation; a second enhancement operation is performed on the unlabeled agricultural image sample to generate a second enhanced unlabeled agricultural image sample based on the second enhancement operation.

[0421] In some embodiments, at least two enhanced unlabeled agricultural image samples are input into a semantic segmentation model to obtain unlabeled prediction maps corresponding to at least two enhanced unlabeled agricultural image samples, respectively. Optionally, a first enhanced unlabeled sample is input into the semantic segmentation model, and the semantic segmentation model predicts the first enhanced unlabeled agricultural image sample to obtain a first unlabeled prediction map corresponding to the first enhanced unlabeled agricultural image sample. The first unlabeled prediction map contains a semantic prediction category for each pixel in the first enhanced unlabeled agricultural image sample. Optionally, a second enhanced unlabeled agricultural image sample is input into the semantic segmentation model, and the semantic segmentation model predicts the second enhanced unlabeled agricultural image sample to obtain a second unlabeled prediction map corresponding to the second enhanced unlabeled agricultural image sample. The second unlabeled prediction map contains a semantic prediction category for each pixel in the second enhanced unlabeled agricultural image sample.

[0422] In some embodiments, when calculating the unlabeled loss based on at least two unlabeled prediction maps, an inverse transformation operation is performed on the at least two unlabeled prediction maps to obtain at least two inversely transformed unlabeled prediction maps, and the unlabeled loss is obtained based on the at least two inversely transformed unlabeled prediction maps. Optionally, a first inverse transformation operation is performed on the first unlabeled prediction map to obtain an inversely transformed first unlabeled prediction map, the first inverse transformation operation corresponding to a first enhancement operation. Optionally, a second inverse transformation operation is performed on the second unlabeled prediction map to obtain an inversely transformed second unlabeled prediction map, the second inverse transformation operation corresponding to a second enhancement operation. The unlabeled loss is obtained based on the inversely transformed first unlabeled prediction map and the inversely transformed second unlabeled prediction map. The unlabeled loss is used to indicate the error between the first and second unlabeled prediction maps predicted by the semantic segmentation model. Further, the unlabeled loss is used to indicate the error between the inversely transformed first and second unlabeled prediction maps.

[0423] For example, the first enhancement operation is a horizontal flip operation, where the first enhanced unlabeled agricultural image sample is flipped along the x-axis compared to the original unlabeled agricultural image sample. The second enhancement operation is a rotation flip operation, where the second enhanced unlabeled agricultural image sample is rotated 90 degrees compared to the original unlabeled agricultural image sample. After predicting the first and second enhanced unlabeled agricultural image samples based on the semantic segmentation model, and obtaining the first and second unlabeled prediction maps, a first inverse transformation operation is performed on the first unlabeled prediction map, which is an inverse x-axis flip operation. Similarly, a second inverse transformation operation is performed on the second unlabeled prediction map, which is a 90-degree rotation flip operation.

[0424] For details on calculating unlabeled loss, please refer to the relevant sections on "Calculating Unlabeled Loss" mentioned above. These details will not be repeated here.

[0425] In this embodiment, the first and second unlabeled prediction maps after inverse transformation are prediction results of enhanced unlabeled agricultural image samples after different enhancement operations, restored by inverse transformation. The differences between them reflect the adaptability and accuracy of the semantic segmentation model to the transformation of agricultural image samples. Calculating the unlabeled loss based on the inverse-transformed unlabeled prediction maps means restoring the original features of the unlabeled samples through inverse transformation operations, and then calculating the differences between different versions of the unlabeled samples to optimize the semantic segmentation model and improve its robustness to data changes.

[0426] Step 5: Train the semantic segmentation model based on labeled loss and unlabeled loss to obtain the pre-trained semantic segmentation model.

[0427] In some embodiments, a third weight corresponding to the labeled loss and a fourth weight corresponding to the unlabeled loss are obtained; a third product of the labeled loss and the third weight is calculated, and a fourth product of the unlabeled loss and the fourth weight is calculated; the sum of the third product and the fourth product is determined as the loss value of the semantic segmentation model, and the semantic segmentation model is trained based on the loss value to obtain a pre-trained semantic segmentation model.

[0428] In some embodiments, a pre-trained semantic segmentation model can be obtained by semi-supervised training of the semantic segmentation model in the first stage (combining labeled and unlabeled agricultural image samples). The pre-trained semantic segmentation model has good image semantic segmentation capabilities for unlabeled samples. In the second stage, the performance of the semantic segmentation model is further improved through agricultural pseudo-label generation and multiple training optimizations.

[0429] In some embodiments, unlabeled agricultural image samples are input into a pre-trained semantic segmentation model to obtain agricultural pseudo-labels corresponding to the unlabeled sample agricultural images; pseudo-labeled agricultural image samples are obtained based on the unlabeled agricultural image samples and agricultural pseudo-labels, and extended agricultural image samples are obtained based on the pseudo-labeled agricultural image samples and labeled agricultural image samples; the extended agricultural image samples are input into the pre-trained semantic segmentation model to obtain an extended label prediction map corresponding to the extended agricultural image samples, and an extended label loss is obtained based on the extended label prediction map and agricultural ground truth labels; the pre-trained semantic segmentation model is trained based on the extended label loss to obtain a trained semantic segmentation model.

[0430] In some embodiments, the pre-trained semantic segmentation model's training process in the second stage includes multiple rounds of iterative training. Optionally, based on the extended label loss of the i-th round, the network parameters of the pre-trained semantic segmentation model trained in the i-th round are updated to obtain the semantic segmentation model for the (i+1)-th round of training. The semantic segmentation model trained in the (i+1)-th round generates agricultural pseudo-labels corresponding to the unlabeled agricultural image samples in the (i+1)-th round. The agricultural pseudo-labels in the (i+1)-th round are pseudo-labels optimized from the agricultural pseudo-labels in the i-th round, where i is a positive integer.

[0431] Specifically, unlabeled agricultural image samples are input into the semantic segmentation model pre-trained in the second stage to obtain the i-th round of agricultural pseudo-labels corresponding to the unlabeled agricultural image samples. Based on the unlabeled agricultural image samples and the i-th round of agricultural pseudo-labels, i-th round of pseudo-labeled agricultural image samples are obtained, along with i-th round of expanded agricultural image samples based on the i-th round of pseudo-labeled agricultural image samples and labeled agricultural image samples. The i-th round of expanded agricultural image samples are input into the i-th round of pre-trained semantic segmentation model to obtain the i-th round of expanded label prediction map corresponding to the i-th round of expanded agricultural image samples, along with the i-th round of expanded label loss based on the i-th round of expanded label prediction map and the ground truth agricultural labels. The i-th round of pre-trained semantic segmentation model is trained based on the i-th round of expanded label loss, and the network parameters from the i-th round of training are updated to obtain the semantic segmentation model for the (i+1)-th round of training, where i is a positive integer. The above steps are repeated until the maximum number of training iterations is reached or the convergence condition is met, at which point the iterative training process stops, and the trained semantic segmentation model is obtained.

[0432] In summary, by combining labeled and unlabeled agricultural image samples for semi-supervised training of the semantic segmentation model, the process first performs random augmentation on the unlabeled agricultural image samples, perturbing them from at least two different perspectives to obtain at least two augmented unlabeled agricultural image samples. An unlabeled loss is then derived based on these two augmented samples, and the semantic segmentation model is trained by combining this unlabeled loss with the labeled loss corresponding to the labeled samples. The unlabeled loss obtained from at least two augmented unlabeled agricultural image samples allows the semantic segmentation model to learn the features of unlabeled agricultural image samples from different perspectives, enabling the model to more comprehensively understand the semantic information within the samples. This allows for more efficient utilization of the latent information in the unlabeled agricultural image samples, improving the learning effect and performance of the semantic segmentation model. For example, in the agricultural field, the trained semantic segmentation model can be used to identify pest and disease areas from agricultural images, thereby addressing pest and disease problems promptly and improving agricultural cultivation efficiency.

[0433] Application Scenario 4

[0434] The trained semantic segmentation model is used as an example in the field of industrial quality inspection analysis.

[0435] Step 1: Obtain image samples of labeled parts and unlabeled parts;

[0436] Among them, labeled part image samples are part image samples with truth value labels, while unlabeled part image samples are part image samples without truth value labels. The part truth value labels can also be called part truth value label maps. When the part image sample is a two-dimensional image sample, the part truth value labels are used to indicate the semantic category of each pixel in the part image sample.

[0437] For example, the pixels of part image sample A include three categories. Part image sample A is a region containing normal areas and defective areas. The part truth label is used to indicate the semantic category of each pixel in part image sample A. For example, the part truth label indicates that the first part of the pixels in part image sample A belongs to the "normal area" and the second part of the pixels belongs to the "defective area".

[0438] Step 2: Input the labeled part image samples into the semantic segmentation model to obtain the labeled prediction map corresponding to the labeled part image samples, and obtain the labeled loss based on the labeled prediction map and the ground truth label of the part.

[0439] In some embodiments, labeled part image samples are input into a semantic segmentation model, and predictions are made on the labeled part image samples based on the semantic segmentation model to obtain a labeled prediction map corresponding to the labeled part image samples. The labeled prediction map contains the semantic prediction category for each pixel in the labeled part image sample.

[0440] In some embodiments, labeled part image samples are input into a semantic segmentation model to obtain image feature representations corresponding to the labeled part image samples. Based on the semantic segmentation model, the part image feature representations corresponding to the labeled part image samples are predicted to obtain labeled prediction maps corresponding to the labeled part image samples. Optionally, a transformer in the semantic segmentation model converts the labeled part image samples into corresponding part image feature representations. The part image feature representations are used to indicate the content features of the labeled part image samples; that is, the part image feature representations are used to indicate the content features extracted from the labeled part image samples.

[0441] Optionally, a labeled loss is derived based on the labeled prediction map and the ground truth labels of the parts. The labeled loss is used to indicate the error between the labeled prediction map predicted by the semantic segmentation model (the untrained image semantic segmentation model, or the general semantic segmentation model) and the ground truth labels of the parts. Alternatively, the labeled loss can be understood as the error between the semantic prediction category of each pixel in the labeled part image sample predicted by the semantic segmentation model and the semantic category (semantic true category) of each pixel in the ground truth labels of the parts.

[0442] In some embodiments, the label loss includes similarity coefficient loss and cross-entropy loss.

[0443] Optionally, labeled part image samples are input into the semantic segmentation model to obtain labeled prediction maps corresponding to the labeled part image samples; a first operation is performed on the labeled prediction maps and ground truth labels of the parts to obtain a similarity coefficient loss, which is used to measure the degree of overlap between the labeled prediction maps and the ground truth labels of the parts. A second operation is performed on the labeled prediction maps and ground truth labels of the parts to obtain a cross-entropy loss, which is used to measure the difference between the labeled prediction maps and the ground truth labels of the parts; based on the similarity coefficient loss and the cross-entropy loss, the labeled loss is calculated.

[0444] In some embodiments, a first weight corresponding to the similarity coefficient loss and a second weight corresponding to the cross-entropy loss are obtained; a first product of the similarity coefficient loss and the first weight is calculated, and a second product of the cross-entropy loss and the second weight is calculated; the sum of the first product and the second product is determined as the labeled loss.

[0445] For details on calculating labeled loss, please refer to the relevant sections on "Calculating Labeled Loss" mentioned above. These details will not be repeated here.

[0446] Step 3: Perform enhancement operations on the unlabeled part image samples to generate at least two enhanced unlabeled part image samples;

[0447] The enhancement operation is used to perturb unlabeled part image samples from at least two perspectives. At least two perspectives means processing the unlabeled part image samples from at least two angles or in at least two ways.

[0448] In some embodiments, an enhancement operation is performed on unlabeled part image samples, perturbing the unlabeled part image samples from at least two perspectives to generate at least two enhanced unlabeled part image samples, that is, generating enhanced unlabeled part image samples from at least two perspectives. Here, enhanced unlabeled part image samples refer to unlabeled part image samples obtained after performing a random enhancement operation on unlabeled part image samples, or can be understood as unlabeled part image samples obtained after perturbing unlabeled part image samples.

[0449] Optionally, the enhancement operation includes, but is not limited to, at least one of the following operations:

[0450] • Geometric transformation operations based on the first probability;

[0451] • Color transformation operation based on the second probability;

[0452] • Noise injection operation based on third probability;

[0453] • Hybrid operations based on the fourth probability.

[0454] The geometric transformation operations described above include, but are not limited to: horizontal flipping, vertical flipping, rotation, cropping, scaling, or at least one of other geometric transformation operations. The color transformation operations described above include, but are not limited to: brightness transformation, contrast transformation, saturation transformation, hue transformation, or at least one of other color transformation operations.

[0455] For details on the enhanced operation, please refer to the relevant content in step 230 above, which will not be repeated here.

[0456] In some embodiments, a geometric transformation operation based on a first probability is performed on an unlabeled part image sample, and a geometrically transformed enhanced unlabeled part image sample is generated according to the first probability. The geometrically transformed enhanced unlabeled part image sample is the unlabeled part image sample after the geometric transformation operation. In some embodiments, a color transformation operation based on a second probability is performed on the unlabeled part image sample, and a color transformation enhanced unlabeled part image sample is generated according to the second probability. The color transformation enhanced unlabeled part image sample is the unlabeled part image sample after the color transformation operation. In some embodiments, a noise injection operation based on a third probability is performed on the unlabeled part image sample, and a noise-enhanced unlabeled part image sample is generated according to the third probability. The noise-enhanced unlabeled part image sample is the unlabeled part image sample after the noise injection operation. In some embodiments, a mixing operation based on a fourth probability is performed on the unlabeled part image sample, and a mixed enhanced unlabeled part image sample is generated according to the fourth probability. The mixed enhanced unlabeled part image sample is the unlabeled part image sample after the mixing operation.

[0457] In this embodiment of the application, by perturbing the unlabeled part image sample from multiple angles, the semantic segmentation model's adaptability to different conditions can be enhanced.

[0458] Step 4: Input at least two enhanced unlabeled part image samples into the semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two enhanced unlabeled part image samples, and obtain the unlabeled loss based on the at least two unlabeled prediction maps;

[0459] Among them, enhanced unlabeled part image samples refer to unlabeled part image samples obtained after performing random enhancement operations on unlabeled part image samples.

[0460] In some embodiments, at least two enhanced unlabeled part image samples are input into a semantic segmentation model. Based on the semantic segmentation model, predictions are made for each of the at least two enhanced unlabeled part image samples to obtain unlabeled prediction maps corresponding to each of the at least two enhanced unlabeled part image samples. The unlabeled prediction map contains the semantic prediction category for each pixel in the enhanced unlabeled part image sample.

[0461] The following explanation uses at least two enhanced unlabeled part image samples, including a first enhanced unlabeled part image sample and a second enhanced unlabeled part image sample, as an example. Optionally, a first enhancement operation is performed on the unlabeled part image sample to generate a first enhanced unlabeled part image sample based on the first enhancement operation; a second enhancement operation is performed on the unlabeled part image sample to generate a second enhanced unlabeled part image sample based on the second enhancement operation.

[0462] In some embodiments, at least two enhanced unlabeled part image samples are input into a semantic segmentation model to obtain unlabeled prediction maps corresponding to at least two enhanced unlabeled part image samples. Optionally, a first enhanced unlabeled sample is input into the semantic segmentation model, and the semantic segmentation model predicts the first enhanced unlabeled part image sample to obtain a first unlabeled prediction map corresponding to the first enhanced unlabeled part image sample. The first unlabeled prediction map contains a semantic prediction category for each pixel in the first enhanced unlabeled part image sample. Optionally, a second enhanced unlabeled part image sample is input into the semantic segmentation model, and the semantic segmentation model predicts the second enhanced unlabeled part image sample to obtain a second unlabeled prediction map corresponding to the second enhanced unlabeled part image sample. The second unlabeled prediction map contains a semantic prediction category for each pixel in the second enhanced unlabeled part image sample.

[0463] In some embodiments, when calculating the unlabeled loss based on at least two unlabeled prediction maps, an inverse transformation operation is performed on the at least two unlabeled prediction maps to obtain at least two inversely transformed unlabeled prediction maps, and the unlabeled loss is obtained based on the at least two inversely transformed unlabeled prediction maps. Optionally, a first inverse transformation operation is performed on the first unlabeled prediction map to obtain an inversely transformed first unlabeled prediction map, the first inverse transformation operation corresponding to a first enhancement operation. Optionally, a second inverse transformation operation is performed on the second unlabeled prediction map to obtain an inversely transformed second unlabeled prediction map, the second inverse transformation operation corresponding to a second enhancement operation. The unlabeled loss is obtained based on the inversely transformed first unlabeled prediction map and the inversely transformed second unlabeled prediction map. The unlabeled loss is used to indicate the error between the first and second unlabeled prediction maps predicted by the semantic segmentation model. Further, the unlabeled loss is used to indicate the error between the inversely transformed first and second unlabeled prediction maps.

[0464] For example, the first enhancement operation is a horizontal flip operation, where the first enhanced unlabeled part image sample is flipped along the x-axis compared to the previous unlabeled part image sample. The second enhancement operation is a rotation flip operation, where the second enhanced unlabeled part image sample is rotated 90 degrees compared to the previous unlabeled part image sample. After predicting the first and second enhanced unlabeled part image samples based on the semantic segmentation model, and obtaining the first and second unlabeled prediction images, a first inverse transformation operation is performed on the first unlabeled prediction image, which is an inverse x-axis flip operation. Similarly, a second inverse transformation operation is performed on the second unlabeled prediction image, which is a 90-degree rotation flip operation.

[0465] For details on calculating unlabeled loss, please refer to the relevant sections on "Calculating Unlabeled Loss" mentioned above. These details will not be repeated here.

[0466] In this embodiment, the first and second unlabeled prediction maps after inverse transformation are prediction results recovered from enhanced unlabeled part image samples after different enhancement operations and then after inverse transformation. The differences between them reflect the adaptability and accuracy of the semantic segmentation model to the transformation of part image samples. Calculating the unlabeled loss based on the inverse-transformed unlabeled prediction maps means recovering the original features of the unlabeled samples through inverse transformation operations, and then calculating the differences between different versions of unlabeled samples to optimize the semantic segmentation model and improve its robustness to data changes.

[0467] Step 5: Train the semantic segmentation model based on labeled loss and unlabeled loss to obtain the pre-trained semantic segmentation model.

[0468] In some embodiments, a third weight corresponding to the labeled loss and a fourth weight corresponding to the unlabeled loss are obtained; a third product of the labeled loss and the third weight is calculated, and a fourth product of the unlabeled loss and the fourth weight is calculated; the sum of the third product and the fourth product is determined as the loss value of the semantic segmentation model, and the semantic segmentation model is trained based on the loss value to obtain a pre-trained semantic segmentation model.

[0469] In some embodiments, a pre-trained semantic segmentation model can be obtained by performing semi-supervised training (combining labeled and unlabeled part image samples) on the semantic segmentation model in the first stage. The pre-trained semantic segmentation model has good image semantic segmentation capabilities for unlabeled samples. In the second stage, the performance of the semantic segmentation model is further improved through the generation of pseudo-labels for parts and multiple training optimizations.

[0470] In some embodiments, unlabeled part image samples are input into a pre-trained semantic segmentation model to obtain pseudo-labels for the unlabeled part images; pseudo-labeled part image samples are obtained based on the unlabeled part image samples and the pseudo-labels, and extended part image samples are obtained based on the pseudo-labeled part image samples and labeled part image samples; the extended part image samples are input into the pre-trained semantic segmentation model to obtain an extended label prediction map corresponding to the extended part image samples, and an extended label loss is obtained based on the extended label prediction map and the ground truth labels of the parts; the pre-trained semantic segmentation model is trained based on the extended label loss to obtain a trained semantic segmentation model.

[0471] In some embodiments, the pre-trained semantic segmentation model's training process in the second stage includes multiple rounds of iterative training. Optionally, based on the expanded label loss of the i-th round, the network parameters of the pre-trained semantic segmentation model trained in the i-th round are updated to obtain the semantic segmentation model for the (i+1)-th round of training. The semantic segmentation model trained in the (i+1)-th round generates pseudo-labels for the unlabeled part image samples corresponding to the (i+1)-th round parts. The (i+1)-th round parts pseudo-labels are pseudo-labels optimized from the (i-th round parts pseudo-labels), where i is a positive integer.

[0472] Specifically, the unlabeled part image samples are input into the semantic segmentation model pre-trained in the second stage to obtain the pseudo-labels for the unlabeled part images in the i-th round. Based on the unlabeled part image samples and the pseudo-labels in the i-th round, pseudo-labeled part image samples for the i-th round are obtained, along with extended part image samples for the i-th round based on the pseudo-labeled and labeled part image samples. The extended part image samples for the i-th round are input into the semantic segmentation model pre-trained in the i-th round to obtain the extended label prediction map for the i-th round, along with the extended label loss for the i-th round based on the extended label prediction map and the ground truth labels for the parts. The semantic segmentation model pre-trained in the i-th round is trained based on the extended label loss, and the network parameters from the i-th round of training are updated to obtain the semantic segmentation model for the (i+1)-th round of training, where i is a positive integer. These steps are repeated until the maximum number of training iterations is reached or the convergence condition is met, at which point the iterative training process stops, resulting in a trained semantic segmentation model.

[0473] In summary, by combining labeled and unlabeled part image samples for semi-supervised training of the semantic segmentation model, when processing unlabeled part image samples, a random augmentation operation is first performed on the unlabeled part image samples, perturbing them from at least two different perspectives to obtain at least two augmented unlabeled part image samples. An unlabeled loss is then obtained based on these at least two augmented unlabeled part image samples. This unlabeled loss, combined with the labeled loss corresponding to the labeled samples, is used to train the semantic segmentation model. The unlabeled loss obtained from at least two augmented unlabeled part image samples allows the semantic segmentation model to learn the features of unlabeled part image samples from different perspectives, enabling the model to more comprehensively understand the semantic information in the unlabeled part image samples. This allows for more full utilization of the potential information in the unlabeled part image samples, improving the learning effect and performance of the semantic segmentation model. For example, in the field of industrial quality inspection, the semantic segmentation model can be applied to part defect detection, identifying defective regions in part images, improving the efficiency and accuracy of industrial quality inspection.

[0474] Application Scenario 5

[0475] The application of a trained semantic segmentation model to the field of satellite remote sensing image analysis is used as an example.

[0476] Step 1: Obtain labeled and unlabeled remote sensing image samples;

[0477] Labeled remote sensing image samples are those labeled with truth values, while unlabeled remote sensing image samples are those without truth values. The remote sensing truth value label can also be called a remote sensing truth value label map. When the remote sensing image sample is a two-dimensional image sample, the remote sensing truth value label is used to indicate the semantic category of each pixel in the remote sensing image sample.

[0478] For example, the pixels of remote sensing image sample A are categorized into three classes. Remote sensing image sample A is an image containing forest areas, ocean areas, and land areas. The remote sensing ground truth label is used to indicate the semantic category of each pixel in remote sensing image sample A. For example, the remote sensing ground truth label indicates that the first part of the pixels in remote sensing image sample A belongs to "forest", the second part belongs to "ocean", and the third part belongs to "land".

[0479] Step 2: Input the labeled remote sensing image samples into the semantic segmentation model to obtain the labeled prediction map corresponding to the labeled remote sensing image samples, and obtain the labeled loss based on the labeled prediction map and the ground truth labels of remote sensing images.

[0480] In some embodiments, labeled remote sensing image samples are input into a semantic segmentation model, and predictions are made on the labeled remote sensing image samples based on the semantic segmentation model to obtain a labeled prediction map corresponding to the labeled remote sensing image samples. The labeled prediction map contains the semantic prediction category for each pixel in the labeled remote sensing image sample.

[0481] In some embodiments, labeled remote sensing image samples are input into a semantic segmentation model to obtain image feature representations corresponding to the labeled remote sensing image samples. Based on the semantic segmentation model, the remote sensing image feature representations corresponding to the labeled remote sensing image samples are predicted to obtain labeled prediction maps corresponding to the labeled remote sensing image samples. Optionally, a transformer in the semantic segmentation model converts the labeled remote sensing image samples into corresponding remote sensing image feature representations. The remote sensing image feature representations are used to indicate the content features of the labeled remote sensing image samples; that is, the remote sensing image feature representations are used to indicate the content features extracted from the labeled remote sensing image samples.

[0482] Optionally, a labeled loss is derived based on the labeled prediction map and the ground truth remote sensing labels. The labeled loss is used to indicate the error between the labeled prediction map predicted by the semantic segmentation model (the untrained image semantic segmentation model, or the general semantic segmentation model) and the ground truth remote sensing labels. Alternatively, the labeled loss can be understood as indicating the error between the semantic prediction category of each pixel in the labeled remote sensing image sample predicted by the semantic segmentation model and the semantic category (semantic true category) of each pixel in the ground truth remote sensing labels.

[0483] In some embodiments, the label loss includes similarity coefficient loss and cross-entropy loss.

[0484] Optionally, labeled remote sensing image samples are input into a semantic segmentation model to obtain labeled prediction maps corresponding to the labeled remote sensing image samples. A first operation is performed on the labeled prediction maps and ground truth remote sensing labels to obtain a similarity coefficient loss, which measures the degree of overlap between the labeled prediction maps and the ground truth remote sensing labels. A second operation is performed on the labeled prediction maps and the ground truth remote sensing labels to obtain a cross-entropy loss, which measures the difference between the labeled prediction maps and the ground truth remote sensing labels. Based on the similarity coefficient loss and the cross-entropy loss, the labeled loss is calculated.

[0485] In some embodiments, a first weight corresponding to the similarity coefficient loss and a second weight corresponding to the cross-entropy loss are obtained; a first product of the similarity coefficient loss and the first weight is calculated, and a second product of the cross-entropy loss and the second weight is calculated; the sum of the first product and the second product is determined as the labeled loss.

[0486] For details on calculating labeled loss, please refer to the relevant sections on "Calculating Labeled Loss" mentioned above. These details will not be repeated here.

[0487] Step 3: Perform enhancement operations on the unlabeled remote sensing image samples to generate at least two enhanced unlabeled remote sensing image samples;

[0488] The enhancement operation is used to perturb unlabeled remote sensing image samples from at least two perspectives. At least two perspectives means processing the unlabeled remote sensing image samples from at least two angles or in at least two ways.

[0489] In some embodiments, an enhancement operation is performed on unlabeled remote sensing image samples, perturbing the unlabeled remote sensing image samples from at least two perspectives to generate at least two enhanced unlabeled remote sensing image samples, that is, generating enhanced unlabeled remote sensing image samples from at least two perspectives. Here, enhanced unlabeled remote sensing image samples refer to unlabeled remote sensing image samples obtained after performing random enhancement operations on unlabeled remote sensing image samples, or can be understood as unlabeled remote sensing image samples obtained after perturbing unlabeled remote sensing image samples.

[0490] Optionally, the enhancement operation includes, but is not limited to, at least one of the following operations:

[0491] • Geometric transformation operations based on the first probability;

[0492] • Color transformation operation based on the second probability;

[0493] • Noise injection operation based on third probability;

[0494] • Hybrid operations based on the fourth probability.

[0495] The geometric transformation operations described above include, but are not limited to: horizontal flipping, vertical flipping, rotation, cropping, scaling, or at least one of other geometric transformation operations. The color transformation operations described above include, but are not limited to: brightness transformation, contrast transformation, saturation transformation, hue transformation, or at least one of other color transformation operations.

[0496] For details on the enhanced operation, please refer to the relevant content in step 230 above, which will not be repeated here.

[0497] In some embodiments, a geometric transformation operation based on a first probability is performed on unlabeled remote sensing image samples to generate geometrically transformed enhanced unlabeled remote sensing image samples, which are unlabeled remote sensing image samples after the geometric transformation operation. In some embodiments, a color transformation operation based on a second probability is performed on unlabeled remote sensing image samples to generate color-transformed enhanced unlabeled remote sensing image samples, which are unlabeled remote sensing image samples after the color transformation operation. In some embodiments, a noise injection operation based on a third probability is performed on unlabeled remote sensing image samples to generate noise-enhanced unlabeled remote sensing image samples, which are unlabeled remote sensing image samples after the noise injection operation. In some embodiments, a mixing operation based on a fourth probability is performed on unlabeled remote sensing image samples to generate mixed-enhanced unlabeled remote sensing image samples, which are unlabeled remote sensing image samples after the mixing operation.

[0498] In this embodiment of the application, by perturbing the unlabeled remote sensing image sample from multiple angles, the semantic segmentation model's adaptability to different conditions can be enhanced.

[0499] Step 4: Input at least two enhanced unlabeled remote sensing image samples into the semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two enhanced unlabeled remote sensing image samples respectively, and obtain unlabeled loss based on the at least two unlabeled prediction maps;

[0500] Among them, augmented unlabeled remote sensing image samples refer to unlabeled remote sensing image samples obtained after performing random augmentation operations on unlabeled remote sensing image samples.

[0501] In some embodiments, at least two augmented unlabeled remote sensing image samples are input into a semantic segmentation model. Based on the semantic segmentation model, predictions are made for each of the at least two augmented unlabeled remote sensing image samples to obtain unlabeled prediction maps corresponding to each of the at least two augmented unlabeled remote sensing image samples. The unlabeled prediction map contains the semantic prediction category for each pixel in the augmented unlabeled remote sensing image sample.

[0502] The following explanation uses at least two enhanced unlabeled remote sensing image samples, including a first enhanced unlabeled remote sensing image sample and a second enhanced unlabeled remote sensing image sample, as an example. Optionally, a first enhancement operation is performed on the unlabeled remote sensing image sample to generate a first enhanced unlabeled remote sensing image sample based on the first enhancement operation; a second enhancement operation is performed on the unlabeled remote sensing image sample to generate a second enhanced unlabeled remote sensing image sample based on the second enhancement operation.

[0503] In some embodiments, at least two enhanced unlabeled remote sensing image samples are input into a semantic segmentation model to obtain unlabeled prediction maps corresponding to at least two enhanced unlabeled remote sensing image samples. Optionally, a first enhanced unlabeled sample is input into the semantic segmentation model, and the semantic segmentation model predicts the first enhanced unlabeled remote sensing image sample to obtain a first unlabeled prediction map corresponding to the first enhanced unlabeled remote sensing image sample. The first unlabeled prediction map contains a semantic prediction category for each pixel in the first enhanced unlabeled remote sensing image sample. Optionally, a second enhanced unlabeled remote sensing image sample is input into the semantic segmentation model, and the semantic segmentation model predicts the second enhanced unlabeled remote sensing image sample to obtain a second unlabeled prediction map corresponding to the second enhanced unlabeled remote sensing image sample. The second unlabeled prediction map contains a semantic prediction category for each pixel in the second enhanced unlabeled remote sensing image sample.

[0504] In some embodiments, when calculating the unlabeled loss based on at least two unlabeled prediction maps, an inverse transformation operation is performed on the at least two unlabeled prediction maps to obtain at least two inversely transformed unlabeled prediction maps, and the unlabeled loss is obtained based on the at least two inversely transformed unlabeled prediction maps. Optionally, a first inverse transformation operation is performed on the first unlabeled prediction map to obtain an inversely transformed first unlabeled prediction map, the first inverse transformation operation corresponding to a first enhancement operation. Optionally, a second inverse transformation operation is performed on the second unlabeled prediction map to obtain an inversely transformed second unlabeled prediction map, the second inverse transformation operation corresponding to a second enhancement operation. The unlabeled loss is obtained based on the inversely transformed first unlabeled prediction map and the inversely transformed second unlabeled prediction map. The unlabeled loss is used to indicate the error between the first and second unlabeled prediction maps predicted by the semantic segmentation model. Further, the unlabeled loss is used to indicate the error between the inversely transformed first and second unlabeled prediction maps.

[0505] For example, the first enhancement operation is a horizontal flip operation, where the first enhanced unlabeled remote sensing image sample is flipped along the x-axis compared to the original unlabeled remote sensing image sample. The second enhancement operation is a rotation flip operation, where the second enhanced unlabeled remote sensing image sample is rotated 90 degrees compared to the original unlabeled remote sensing image sample. After predicting the first and second enhanced unlabeled remote sensing image samples based on the semantic segmentation model, and obtaining the first and second unlabeled prediction images, a first inverse transformation operation is performed on the first unlabeled prediction image, which is an inverse x-axis flip operation. Similarly, a second inverse transformation operation is performed on the second unlabeled prediction image, which is a 90-degree rotation flip operation.

[0506] For details on calculating unlabeled loss, please refer to the relevant sections on "Calculating Unlabeled Loss" mentioned above. These details will not be repeated here.

[0507] In this embodiment, the first and second unlabeled prediction maps after inverse transformation are prediction results recovered from enhanced unlabeled remote sensing image samples after different enhancement operations and then after inverse transformation. The differences between them reflect the adaptability and accuracy of the semantic segmentation model to changes in remote sensing image samples. Calculating the unlabeled loss based on the inverse-transformed unlabeled prediction maps means recovering the original features of the unlabeled samples through inverse transformation operations, and then calculating the differences between different versions of the unlabeled samples to optimize the semantic segmentation model and improve its robustness to data changes.

[0508] Step 5: Train the semantic segmentation model based on labeled loss and unlabeled loss to obtain the pre-trained semantic segmentation model.

[0509] In some embodiments, a third weight corresponding to the labeled loss and a fourth weight corresponding to the unlabeled loss are obtained; a third product of the labeled loss and the third weight is calculated, and a fourth product of the unlabeled loss and the fourth weight is calculated; the sum of the third product and the fourth product is determined as the loss value of the semantic segmentation model, and the semantic segmentation model is trained based on the loss value to obtain a pre-trained semantic segmentation model.

[0510] In some embodiments, a pre-trained semantic segmentation model can be obtained by performing semi-supervised training (combining labeled and unlabeled remote sensing image samples) on the semantic segmentation model in the first stage. The pre-trained semantic segmentation model has good image semantic segmentation capabilities for unlabeled samples. In the second stage, the performance of the semantic segmentation model is further improved through remote sensing pseudo-label generation and multiple training optimizations.

[0511] In some embodiments, unlabeled remote sensing image samples are input into a pre-trained semantic segmentation model to obtain remote sensing pseudo-labels corresponding to the unlabeled sample remote sensing images; pseudo-labeled remote sensing image samples are obtained based on the unlabeled remote sensing image samples and remote sensing pseudo-labels, and extended remote sensing image samples are obtained based on the pseudo-labeled remote sensing image samples and labeled remote sensing image samples; the extended remote sensing image samples are input into the pre-trained semantic segmentation model to obtain an extended label prediction map corresponding to the extended remote sensing image samples, and an extended label loss is obtained based on the extended label prediction map and the ground truth remote sensing labels; the pre-trained semantic segmentation model is trained based on the extended label loss to obtain a trained semantic segmentation model.

[0512] In some embodiments, the pre-trained semantic segmentation model's training process in the second stage includes multiple rounds of iterative training. Optionally, based on the extended label loss of the i-th round, the network parameters of the pre-trained semantic segmentation model trained in the i-th round are updated to obtain the semantic segmentation model for the (i+1)-th round of training. The semantic segmentation model trained in the (i+1)-th round generates remote sensing pseudo-labels for the unlabeled remote sensing image samples in the (i+1)-th round. The remote sensing pseudo-labels in the (i+1)-th round are pseudo-labels optimized from the remote sensing pseudo-labels in the i-th round, where i is a positive integer.

[0513] Specifically, unlabeled remote sensing image samples are input into the semantic segmentation model pre-trained in the second stage to obtain the i-th round of remote sensing pseudo-labels corresponding to the unlabeled remote sensing image samples; based on the unlabeled remote sensing image samples and the i-th round of remote sensing pseudo-labels, i-th round of pseudo-labeled remote sensing image samples are obtained, and i-th round of extended remote sensing image samples are obtained based on the i-th round of pseudo-labeled remote sensing image samples and labeled remote sensing image samples; the i-th round of extended remote sensing image samples are input into the semantic segmentation model pre-trained in the i-th round to obtain the i-th round of extended label prediction map corresponding to the i-th round of extended remote sensing image samples, and the i-th round of extended label loss is obtained based on the i-th round of extended label prediction map and the ground truth remote sensing labels; the semantic segmentation model pre-trained in the i-th round is trained based on the i-th round of extended label loss, and the network parameters trained in the i-th round are updated to obtain the semantic segmentation model used for training in the (i+1)-th round, where i is a positive integer. Repeat the above steps until the maximum number of training iterations is reached or the convergence condition is met, then stop the iterative training process to obtain the trained semantic segmentation model.

[0514] In summary, by combining labeled and unlabeled remote sensing image samples for semi-supervised training of the semantic segmentation model, the process first performs random augmentation on the unlabeled remote sensing image samples, perturbing them from at least two different perspectives to obtain at least two augmented unlabeled remote sensing image samples. An unlabeled loss is then obtained based on these at least two augmented unlabeled remote sensing image samples. This unlabeled loss is combined with the labeled loss corresponding to the labeled samples to train the semantic segmentation model. The unlabeled loss obtained from at least two augmented unlabeled remote sensing image samples allows the semantic segmentation model to learn the features of unlabeled remote sensing image samples from different perspectives. This enables the semantic segmentation model to more comprehensively understand the semantic information in the unlabeled remote sensing image samples, thereby making fuller use of the potential information in the unlabeled remote sensing image samples and improving the learning effect and performance of the semantic segmentation model. For example, in the field of satellite remote sensing image processing, the semantic segmentation model can more accurately identify various information about the Earth's surface, such as forest cover areas, from remote sensing images.

[0515] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0516] Figure 10 This diagram illustrates a structural block diagram of a training apparatus for a semantic segmentation model according to an embodiment of this application. The apparatus has the functionality to implement the training method example of the aforementioned semantic segmentation model; this functionality can be implemented in hardware or by hardware executing corresponding software. Figure 10 As shown, the device may include: a sample acquisition module 910, a sample prediction module 920, an enhancement generation module 930, and a model training module 940.

[0517] The sample acquisition module 910 is used to acquire labeled samples and unlabeled samples, wherein the labeled samples are image samples labeled with truth value labels, and the unlabeled samples are image samples without the truth value labels.

[0518] The sample prediction module 920 is used to input the labeled sample into the semantic segmentation model to obtain the labeled prediction map corresponding to the labeled sample, and to obtain the labeled loss based on the labeled prediction map and the ground truth label.

[0519] The enhancement generation module 930 is used to perform an enhancement operation on the unlabeled sample to generate at least two enhanced unlabeled samples, wherein the enhancement operation is used to perturb the unlabeled sample from at least two perspectives.

[0520] The sample prediction module 920 is used to input the at least two enhanced unlabeled samples into the semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two enhanced unlabeled samples respectively, and to obtain unlabeled loss based on the at least two unlabeled prediction maps;

[0521] The model training module 940 is used to train the semantic segmentation model based on the labeled loss and the unlabeled loss to obtain the pre-trained semantic segmentation model.

[0522] In some embodiments, the sample prediction module 920 includes an inverse transformation submodule and a generation submodule.

[0523] In an optional example, the inverse transform submodule is used to perform an inverse transform operation on the at least two unlabeled prediction maps to obtain at least two inversely transformed unlabeled prediction maps;

[0524] A generation submodule is used to obtain the unlabeled loss based on the at least two inversely transformed unlabeled prediction maps.

[0525] In some embodiments, the inverse transform submodule includes an inverse transform unit, and the generation submodule includes a generation unit.

[0526] In an optional example, the at least two enhanced unlabeled samples include: a first enhanced unlabeled sample generated after a first enhancement operation and a second enhanced unlabeled sample generated after a second enhancement operation, wherein the first enhanced unlabeled sample corresponds to a first unlabeled prediction map and the second enhanced unlabeled sample corresponds to a second unlabeled prediction map;

[0527] The inverse transformation unit is used to perform a first inverse transformation operation on the first unlabeled prediction map to obtain the inversely transformed first unlabeled prediction map; and to perform a second inverse transformation operation on the second unlabeled prediction map to obtain the inversely transformed second unlabeled prediction map.

[0528] A generation unit is used to obtain the unlabeled loss based on the first unlabeled prediction map after the inverse transformation and the second unlabeled prediction map after the inverse transformation;

[0529] Wherein, the first inverse transformation operation corresponds to the first enhancement operation, and the second inverse transformation operation corresponds to the second enhancement operation.

[0530] In an optional example, the first enhancement operation and the second enhancement operation are enhancement operations from different perspectives, and the first enhancement operation and the second enhancement operation include at least one of the following operations:

[0531] Geometric transformation operations based on the first probability;

[0532] Color transformation operation based on the second probability;

[0533] Noise injection operation based on third probability;

[0534] Hybrid operations based on the fourth probability.

[0535] In an optional example, the geometric transformation operation includes at least one of the following operations: horizontal flip operation; vertical flip operation; rotation operation; cropping operation; scaling operation;

[0536] The color transformation operation includes at least one of the following operations: brightness change operation; contrast change operation; saturation change operation; hue change operation.

[0537] In some embodiments, the sample prediction module 920 includes an execution submodule and a calculation submodule.

[0538] In an optional example, an execution submodule is used to perform a first operation on the labeled prediction graph and the ground truth labels to obtain the similarity coefficient loss, which is used to measure the degree of overlap between the labeled prediction graph and the ground truth labels;

[0539] An execution submodule is used to perform a second operation on the labeled prediction graph and the ground truth labels to obtain the cross-entropy loss, which is used to measure the difference between the labeled prediction graph and the ground truth labels;

[0540] The calculation submodule is used to calculate the labeled loss based on the similarity coefficient loss and the cross-entropy loss.

[0541] In some embodiments, the calculation submodule includes an acquisition unit, a calculation unit, and a determination unit.

[0542] In an optional example, the acquisition unit is used to acquire the first weight corresponding to the similarity coefficient loss and the second weight corresponding to the cross-entropy loss;

[0543] The calculation unit is used to calculate the first product of the similarity coefficient loss and the first weight, and to calculate the second product of the cross-entropy loss and the second weight;

[0544] A determining unit is configured to determine the sum of the first product and the second product as the labeled loss.

[0545] In some embodiments, the model training module 940 includes an acquisition submodule, a calculation submodule, and a determination submodule.

[0546] In an optional example, a submodule is used to obtain the third weight corresponding to the labeled loss and the fourth weight corresponding to the unlabeled loss.

[0547] The calculation submodule is used to calculate the third product of the labeled loss and the third weight, and to calculate the fourth product of the unlabeled loss and the fourth weight;

[0548] A submodule is defined to determine the sum of the third product and the fourth product as the loss value of the semantic segmentation model, and to train the semantic segmentation model based on the loss value to obtain the pre-trained semantic segmentation model.

[0549] In some embodiments, the apparatus further includes a pseudo-label generation module and a sample combination module.

[0550] In an optional example, the pseudo-label generation module is used to input the unlabeled sample into the pre-trained semantic segmentation model to obtain the pseudo-label corresponding to the unlabeled sample;

[0551] The sample combination module is used to obtain pseudo-labeled samples based on the unlabeled samples and the pseudo-labels, and to obtain extended samples based on the pseudo-labeled samples and the labeled samples;

[0552] The sample prediction module 920 is used to input the extended sample into the pre-trained semantic segmentation model to obtain the extended label prediction map corresponding to the extended sample, and to obtain the extended label loss based on the extended label prediction map and the ground truth label.

[0553] The model training module 940 is used to train the pre-trained semantic segmentation model based on the extended label loss to obtain the trained semantic segmentation model.

[0554] In some embodiments, the model training module 940 includes an update submodule and a stop training submodule.

[0555] In an optional example, the training process of the pre-trained semantic segmentation model includes multiple rounds of iterative training;

[0556] The update submodule is used to update the network parameters of the pre-trained semantic segmentation model trained in the i-th round based on the expanded label loss in the i-th round, so as to obtain the semantic segmentation model for training in the (i+1)-th round, where i is a positive integer;

[0557] The Stop Training submodule is used to repeat the above steps until the maximum number of training iterations is reached or the convergence condition is met. Then, the iterative training process is stopped, and the trained semantic segmentation model is obtained.

[0558] It should be noted that the specific limitations of the embodiments of the training devices for one or more semantic segmentation models provided above can be found in the limitations of the training methods for semantic segmentation models above, and will not be repeated here. Each module of the above device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in the processor of the computer device in hardware form or independent of the processor, or it can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0559] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0560] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the training method of the semantic segmentation model provided in the above-described method embodiments.

[0561] For example, Figure 11 This is a structural block diagram of a computer device 1000 provided in an exemplary embodiment of this application. The computer device 1000 may be... Figure 2 The terminal device shown is used to implement the training method of the semantic segmentation model provided in the above embodiments.

[0562] The computer device 1000 includes a central processing unit (CPU) 1001, a system memory 1004 including random access memory (RAM) 1002 and read-only memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. The computer device 1000 also includes a basic input / output system (I / O system) 1006 to facilitate information transfer between various components within the computer device, and a mass storage device 1007 for storing the operating system 1013, application programs 1014, and other program modules 1015.

[0563] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009 for user input, such as a mouse or keyboard. Both the display 1008 and the input device 1009 are connected to the central processing unit 1001 via an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include the input / output controller 1010 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, printer, or other types of output devices.

[0564] The mass storage device 1007 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer-readable storage media provide non-volatile storage for the computer device 1000. That is, the mass storage device 1007 may include computer-readable storage media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0565] Without loss of generality, the computer-readable storage medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable storage instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage devices, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 1004 and mass storage device 1007 described above can be collectively referred to as memory.

[0566] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1001. The one or more programs contain instructions for implementing the above method embodiments. The central processing unit 1001 executes the one or more programs to implement the training method of the semantic segmentation model provided by the above method embodiments.

[0567] According to various embodiments of this application, the computer device 1000 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1000 can be connected to the network 1012 via the network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 can be used to connect to other types of networks or remote computer device systems (not shown).

[0568] The memory further includes one or more programs stored in the memory, and the one or more programs include steps executed by a computer device in the training method for the semantic segmentation model provided in the embodiments of this application.

[0569] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set. When the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor of a computer device, the training method of the semantic segmentation model provided in the above-described method embodiments is implemented.

[0570] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the training method for the semantic segmentation model provided in the above-described method embodiments.

[0571] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0572] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent switching, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for a semantic segmentation model, characterized in that, The method includes: Acquire labeled samples and unlabeled samples, wherein the labeled samples are image samples labeled with ground truth labels, and the unlabeled samples are image samples without ground truth labels; The labeled sample is input into the semantic segmentation model to obtain the labeled prediction map corresponding to the labeled sample, and the labeled loss is obtained based on the labeled prediction map and the ground truth label; An enhancement operation is performed on the unlabeled sample to generate at least two enhanced unlabeled samples, wherein the enhancement operation is used to perturb the unlabeled sample from at least two perspectives; The at least two enhanced unlabeled samples are input into the semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two enhanced unlabeled samples, and unlabeled loss is obtained based on the at least two unlabeled prediction maps; The semantic segmentation model is trained based on the labeled loss and the unlabeled loss to obtain a pre-trained semantic segmentation model.

2. The method according to claim 1, characterized in that, The unlabeled loss is derived based on at least two unlabeled prediction maps, including: Perform an inverse transformation operation on the at least two unlabeled prediction maps to obtain at least two inversely transformed unlabeled prediction maps, and obtain the unlabeled loss based on the at least two inversely transformed unlabeled prediction maps.

3. The method according to claim 1 or 2, characterized in that, The at least two enhanced unlabeled samples include: a first enhanced unlabeled sample generated after the first enhancement operation and a second enhanced unlabeled sample generated after the second enhancement operation, wherein the first enhanced unlabeled sample corresponds to a first unlabeled prediction map and the second enhanced unlabeled sample corresponds to a second unlabeled prediction map. The step of performing an inverse transformation operation on the at least two unlabeled prediction maps to obtain at least two inversely transformed unlabeled prediction maps, and obtaining the unlabeled loss based on the at least two inversely transformed unlabeled prediction maps, includes: Perform a first inverse transformation operation on the first unlabeled prediction map to obtain an inversely transformed first unlabeled prediction map; and perform a second inverse transformation operation on the second unlabeled prediction map to obtain an inversely transformed second unlabeled prediction map; The unlabeled loss is obtained based on the first unlabeled prediction map after the inverse transformation and the second unlabeled prediction map after the inverse transformation. Wherein, the first inverse transformation operation corresponds to the first enhancement operation, and the second inverse transformation operation corresponds to the second enhancement operation.

4. The method according to claim 3, characterized in that, The first enhancement operation and the second enhancement operation are enhancement operations from different perspectives, and the first enhancement operation and the second enhancement operation include at least one of the following operations: Geometric transformation operations based on the first probability; Color transformation operation based on the second probability; Noise injection operation based on third probability; Hybrid operations based on the fourth probability.

5. The method according to claim 4, characterized in that, The geometric transformation operation includes at least one of the following operations: horizontal flip operation; vertical flip operation; rotation operation; cropping operation; scaling operation; The color transformation operation includes at least one of the following operations: brightness change operation; contrast change operation; saturation change operation; hue change operation.

6. The method according to any one of claims 1 to 5, characterized in that, The labeled loss includes similarity coefficient loss and cross-entropy loss; The process of obtaining the labeled loss based on the labeled prediction graph and the ground truth labels includes: A first operation is performed on the labeled prediction map and the ground truth labels to obtain the similarity coefficient loss, which is used to measure the degree of overlap between the labeled prediction map and the ground truth labels. A second operation is performed on the labeled prediction graph and the ground truth labels to obtain the cross-entropy loss, which is used to measure the difference between the labeled prediction graph and the ground truth labels. The labeled loss is calculated based on the similarity coefficient loss and the cross-entropy loss.

7. The method according to claim 6, characterized in that, The calculation of the labeled loss based on the similarity coefficient loss and the cross-entropy loss includes: Obtain the first weight corresponding to the similarity coefficient loss and the second weight corresponding to the cross-entropy loss; Calculate the first product of the similarity coefficient loss and the first weight, and calculate the second product of the cross-entropy loss and the second weight; The sum of the first product and the second product is determined as the labeled loss.

8. The method according to any one of claims 1 to 7, characterized in that, The semantic segmentation model is trained based on the labeled loss and the unlabeled loss to obtain a pre-trained semantic segmentation model; Obtain the third weight corresponding to the labeled loss, and obtain the fourth weight corresponding to the unlabeled loss; Calculate the third product of the labeled loss and the third weight, and calculate the fourth product of the unlabeled loss and the fourth weight; The sum of the third product and the fourth product is determined as the loss value of the semantic segmentation model. The semantic segmentation model is trained based on the loss value to obtain the pre-trained semantic segmentation model.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: The unlabeled sample is input into the pre-trained semantic segmentation model to obtain the pseudo-label corresponding to the unlabeled sample; A pseudo-labeled sample is obtained based on the unlabeled sample and the pseudo-label, and an extended sample is obtained based on the pseudo-labeled sample and the labeled sample; The expanded sample is input into the pre-trained semantic segmentation model to obtain the expanded label prediction map corresponding to the expanded sample, and the expanded label loss is obtained based on the expanded label prediction map and the ground truth label. The pre-trained semantic segmentation model is trained based on the extended label loss to obtain the trained semantic segmentation model.

10. The method according to claim 9, characterized in that, The training process of the pre-trained semantic segmentation model includes multiple rounds of iterative training. The process of training the pre-trained semantic segmentation model based on the extended label loss to obtain the trained semantic segmentation model includes: Based on the expanded label loss of the i-th round, the network parameters of the pre-trained semantic segmentation model trained in the i-th round are updated to obtain the semantic segmentation model for training in the (i+1)-th round. The semantic segmentation model trained in the (i+1)-th round generates pseudo-labels for the unlabeled samples in the (i+1)-th round. The pseudo-labels in the (i+1)-th round are pseudo-labels optimized from the pseudo-labels in the i-th round, where i is a positive integer. Repeat the above steps until the maximum number of training iterations is reached or the convergence condition is met, then stop the iterative training process to obtain the trained semantic segmentation model.

11. The method according to any one of claims 1 to 10, characterized in that, The trained semantic segmentation model is used for autonomous driving. The labeled sample represents a labeled road image sample, the unlabeled sample represents an unlabeled road image sample, and the at least two enhanced unlabeled samples represent at least two enhanced unlabeled road image samples. The trained semantic segmentation model is used for medical image segmentation. The labeled sample represents a labeled medical image sample, the unlabeled sample represents an unlabeled medical image sample, and the at least two enhanced unlabeled samples represent at least two enhanced unlabeled medical image samples. The trained semantic segmentation model is used for agricultural scene analysis. The labeled sample represents a labeled agricultural image sample, the unlabeled sample represents an unlabeled agricultural image sample, and the at least two enhanced unlabeled samples represent at least two enhanced unlabeled agricultural image samples. The trained semantic segmentation model is used for industrial quality inspection analysis. The labeled sample represents a labeled part image sample, the unlabeled sample represents an unlabeled part image sample, and the at least two enhanced unlabeled samples represent at least two enhanced unlabeled part image samples. The trained semantic segmentation model is used for satellite remote sensing image analysis. The labeled sample represents a labeled remote sensing image sample, the unlabeled sample represents an unlabeled remote sensing image sample, and the at least two enhanced unlabeled samples represent at least two enhanced unlabeled remote sensing image samples.

12. A semantic segmentation device, characterized in that, The device includes: The sample acquisition module is used to acquire labeled samples and unlabeled samples, wherein the labeled samples are image samples labeled with ground truth labels, and the unlabeled samples are image samples without the ground truth labels. The sample prediction module is used to input the labeled sample into the semantic segmentation model to obtain the labeled prediction map corresponding to the labeled sample, and to obtain the labeled loss based on the labeled prediction map and the ground truth label; An enhancement generation module is used to perform enhancement operations on the unlabeled samples to generate at least two enhanced unlabeled samples, wherein the enhancement operations are used to perturb the unlabeled samples from at least two perspectives. The sample prediction module is used to input the at least two enhanced unlabeled samples into the semantic segmentation model to obtain unlabeled prediction maps corresponding to the at least two enhanced unlabeled samples respectively, and to obtain unlabeled loss based on the at least two unlabeled prediction maps; The model training module is used to train the semantic segmentation model based on the labeled loss and the unlabeled loss to obtain a pre-trained semantic segmentation model.

13. A computer device, characterized in that, The computer device includes a processor and a memory, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the training method of the semantic segmentation model as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the training method of the semantic segmentation model as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium; the processor reads from the computer-readable storage medium and executes the computer program to implement the training method of the semantic segmentation model as described in any one of claims 1 to 11.

Images