A Domain-Adaptive Image Semantic Segmentation Method and System with Two-Level Interaction

Abstract

This invention provides a two-level interactive domain-adaptive image semantic segmentation method and system, belonging to the field of computer vision technology. The method includes: constructing an initial image instance library for a source domain image set; training and determining an initial segmentation model; generating semantic pseudo-labels for a target domain image set based on semantic-level domain adaptation using the initial segmentation model; determining pixel instance features and corresponding ground truth labels in the source domain image set using class equalization, and updating the initial image instance library using these features and ground truth labels; generating instance-level pseudo-labels; and optimizing the updated initial image instance library using backpropagation based on these pseudo-labels to obtain the image semantic segmentation result. This invention introduces instance-level discrimination on top of the semantic level, assisting in calibrating noisy pseudo-labels that the semantic classifier cannot correctly classify. This enables faster training speed for image semantic segmentation and features low cost and high classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a two-level interactive domain-adaptive image semantic segmentation method and system. Background Technology

[0002] In the field of autonomous driving, semantic segmentation is a core and fundamental image processing method. The purpose of semantic segmentation is to assign a label to each pixel in a given image, which is a basic task in computer vision.

[0003] Existing semantic segmentation models are trained on labeled real-world datasets, resulting in relatively objective training results. However, this pixel-level image annotation is extremely labor-intensive and time-consuming. Some solutions employ synthetic data, such as data from virtual 3D simulations or open-world games, to train semantic segmentation models, thereby saving annotation time and costs. However, rendered synthetic data suffers from domain shift issues compared to real-world distributions, severely degrading model performance. To address this problem, unsupervised domain adaptation (UDA) has been explored. Its goal is to allow models trained on labeled source domain (synthetic) data to use unlabeled target (real) data, thereby improving model performance on real-world datasets.

[0004] Building upon this foundation, domain adaptation algorithms are proposed, primarily falling into two categories: one aligns the distributions of the source and target domains at different levels (i.e., input, feature, and output levels) through adversarial training; the other is self-training, which generates pseudo-labels for the target domain based on the semantic-level category predictions of the network trained in the source domain, and then applies these pseudo-labels as if they were real labels during the self-training process in the target domain. Most state-of-the-art and effective training methods are based on self-training, but semantic classifiers inherently struggle to clearly classify pixels located at class boundaries, resulting in erroneous and noisy semantic pseudo-labels. Therefore, existing methods still struggle to clearly distinguish noisy pseudo-labels near classification boundaries, leading to a decline in model adaptation performance. Furthermore, the performance of existing models in handling unevenly distributed "long-tailed" categories and easily confused (similar or adjacent) categories needs improvement.

[0005] Therefore, for the application of domain-adaptive semantic segmentation in images, there is an urgent need to propose a new image semantic segmentation method that can effectively correct noisy semantic pseudo-labels and handle long-tailed distributions and easily confused categories. Summary of the Invention

[0006] This invention provides a two-level interactive domain-adaptive image semantic segmentation method and system to address the shortcomings of existing domain-adaptive algorithms in image semantic segmentation, such as the occurrence of noisy semantic pseudo-labels.

[0007] In a first aspect, the present invention provides a domain-adaptive image semantic segmentation method with two-level interaction, comprising:

[0008] Construct an initial image instance library for the source domain image set;

[0009] Training determines the initial segmentation model;

[0010] Based on semantic hierarchical domain adaptation, semantic pseudo-labels for the target domain image set are generated using the initial segmentation model;

[0011] In the source domain image set, class equalization is used to determine pixel instance features and corresponding real labels, and the initial image instance library is updated using the pixel instance features and the corresponding real labels.

[0012] Instance-level pseudo-labels are generated. Based on the instance-level pseudo-labels, the optimized and updated initial image instance library is backpropagated using reverse propagation to obtain the image semantic segmentation result.

[0013] According to a two-level interactive domain-adaptive image semantic segmentation method provided by the present invention, the initial image instance library for constructing the source domain image set includes:

[0014] Determine the size of the image instance feature library, the dimension of the extracted feature vector, the number of source domain segmentation categories, and the number of target domain segmentation categories, wherein the number of source domain segmentation categories is equal to the number of target domain segmentation categories;

[0015] The image instance feature library is constructed based on the size of the image instance feature library and the dimension of the extracted feature vector;

[0016] The size of the image instance feature library is divided by the number of source domain segmentation categories to obtain category marker placeholders, and an instance label library is constructed based on the category marker placeholders.

[0017] According to a two-level interactive domain-adaptive image semantic segmentation method provided by the present invention, the training to determine the initial segmentation model includes:

[0018] In the source domain image set, a single source domain image sample and the number of source domain image samples are determined respectively; in the source domain label set, a single source domain label sample and the number of source domain label samples are determined respectively; in the target domain image set, a single target domain image sample and the number of target domain image samples are determined respectively; and in the target domain label set, a single target domain label sample and the number of target domain label samples are determined respectively.

[0019] A feature map is obtained by extracting the single source domain image sample using a feature extractor based on an encoder-decoder structure. A fully connected pixel-level classifier is then used to map the feature map to semantic prediction to obtain semantic-level prediction probabilities.

[0020] The classification cross-entropy loss function is adopted, which is used to obtain the source domain sample loss function by considering the feature map length, feature map width, probability that any pixel in the feature map belongs to any class, and probability that any label in the feature map belongs to any class.

[0021] The initial segmentation model is obtained by optimizing the source domain samples based on the source domain sample loss function.

[0022] According to the present invention, a two-level interactive domain-adaptive image semantic segmentation method is provided, wherein the semantic-level domain adaptation, based on the initial segmentation model, generates semantic pseudo-labels for a target domain image set, comprising:

[0023] Determine the semantic pseudo-label category identifier, and determine the probability that any target domain pixel belongs to any category based on the semantic pseudo-label category identifier;

[0024] The initial semantic classification pseudo-label of the target domain is obtained from the probability that any pixel in the target domain belongs to any category and the initial segmentation model;

[0025] A confidence threshold is determined, and the pseudo-labels of the primary semantic classification of the target domain are filtered based on the confidence threshold to obtain the semantic loss function of the target domain.

[0026] Based on the target domain semantic-level loss function, semantic pseudo-labels are obtained for the target domain image set.

[0027] According to a two-level interactive domain-adaptive image semantic segmentation method provided by the present invention, the step of determining pixel instance features and corresponding ground truth labels in the source domain image set using class equalization, and updating the initial image instance library using the pixel instance features and the corresponding ground truth labels, includes:

[0028] A boundary mask for any source domain image in the source domain image set is generated using a mask generation algorithm, and a boundary pixel map is obtained by multiplying the source domain image label by the boundary mask.

[0029] The average value of pixel feature embedding for any class in the boundary pixel image is obtained, and the average value of pixel feature embedding for any cluster is determined in non-edge entities using a preset clustering method.

[0030] The average instance features of the embedding average of any class of pixel features and the embedding average of any cluster of pixel features are used as the embedding features of the image instance feature library;

[0031] Based on the preset instance library update interval and momentum coefficient, the embedded features of the image instance feature library are updated using exponential moving average to obtain the updated image instance feature library.

[0032] According to a two-level interactive domain-adaptive image semantic segmentation method provided by the present invention, the step of generating a boundary mask for any source domain image in the source domain image set through a mask generation algorithm includes:

[0033] Obtain the true label, height, and width of any source domain image;

[0034] Initialize an all-zero matrix constructed using the height and width of any source domain image, and determine the preset receptive field and the preset number of pixel categories;

[0035] If it is determined that there are different pixels within the preset receptive field of the preset dimension around any pixel in any source domain image, which are greater than or equal to the number of preset pixel categories, then any pixel is determined to be a boundary pixel, and the boundary mask position corresponding to any pixel is set to 1; otherwise, it is set to 0.

[0036] The boundary mask is obtained by traversing all pixels in any source domain image.

[0037] According to the present invention, a two-level interactive domain-adaptive image semantic segmentation method is provided, wherein the generation of instance-level pseudo-labels, and the backpropagation of the optimized and updated initial image instance library based on the instance-level pseudo-labels, to obtain the image semantic segmentation result, includes:

[0038] Obtain feature maps, pixel embedding representations from image instance libraries, preset strong enhancement target sample instance embeddings, preset weak enhancement target sample instance embeddings, and any instance from the instance library to determine the temperature parameter for distribution flatness.

[0039] Using cosine similarity, based on the matrix transpose of the preset strongly enhanced target sample instance embedding, any instance in the instance library, the pixel embedding representation of the image instance library, and the temperature parameter of the distribution flatness, the probability that any preset strongly enhanced target sample instance embedding is identified as any instance in the instance library is obtained.

[0040] Using cosine similarity, based on the matrix transpose of the preset weakly enhanced target sample instance embedding, any instance in the instance library, the pixel embedding representation of the image instance library, and the temperature parameter of the distribution flatness, the probability that any preset weakly enhanced target sample instance embedding is identified as any instance in the instance library is obtained.

[0041] The cross-entropy loss function is used to minimize the difference between the probability that any preset strongly enhanced target sample instance embedding is identified as any instance in the instance library and the probability that any preset weakly enhanced target sample instance embedding is identified as any instance in the instance library, thus obtaining the instance-level loss function;

[0042] The domain adaptive overall loss function is obtained by summing the products of the source domain sample loss function, the target domain semantic level loss function, and the instance level loss function with the instance level loss weight.

[0043] Repeat the above training process for a preset number of iterations, and output the image semantic segmentation result.

[0044] Secondly, the present invention also provides a two-level interactive domain-adaptive image semantic segmentation system, comprising:

[0045] The building module is used to construct an initial image instance library for the source domain image set;

[0046] The training module is used to train and determine the initial segmentation model;

[0047] The generation module is used to generate semantic pseudo-labels for the target domain image set based on semantic hierarchical domain adaptation and using the initial segmentation model;

[0048] The update module is used to determine pixel instance features and corresponding real labels in the source domain image set by class equalization, and update the initial image instance library using the pixel instance features and the corresponding real labels;

[0049] The processing module is used to generate instance-level pseudo-labels, and based on the instance-level pseudo-labels, it uses reverse propagation to back-pass the optimized and updated initial image instance library to obtain the image semantic segmentation result.

[0050] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the domain-adaptive image semantic segmentation method with two-level interaction as described above.

[0051] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the domain-adaptive image semantic segmentation method with two-level interaction as described above.

[0052] The present invention provides a two-level interactive domain-adaptive image semantic segmentation method and system that addresses the common problems of classification errors and semantic pseudo-labels with various noises during image semantic segmentation. By introducing instance-level discrimination on the basis of semantic level, it assists in calibrating noisy pseudo-labels that the semantic classifier cannot classify correctly. This enables faster training speed for image semantic segmentation and features low cost and high classification accuracy. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0054] Figure 1 This is one of the flowcharts of the domain-adaptive image semantic segmentation method with two-level interaction provided by the present invention;

[0055] Figure 2 This is the second flowchart of the domain-adaptive image semantic segmentation method with two-level interaction provided by the present invention;

[0056] Figure 3 This is a schematic diagram of instance-level information calibration semantic-level noise pseudo-labels provided by the present invention;

[0057] Figure 4 This is a diagram illustrating the hierarchical interaction details provided by the present invention;

[0058] Figure 5 This is an overview diagram of the overall domain adaptive training method provided by the present invention;

[0059] Figure 6 This is a schematic diagram of the structure of the domain-adaptive image semantic segmentation system with two-level interaction provided by the present invention;

[0060] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0062] To address the issue of semantic noise and pseudo-labels in image semantic segmentation within the autonomous driving domain, which severely impacts the accuracy of classification results, this invention proposes a two-level interactive domain-adaptive image semantic segmentation method. Figure 1 This is one of the flowcharts illustrating the domain-adaptive image semantic segmentation method with two-level interaction provided in this embodiment of the invention, such as... Figure 1 As shown, it includes:

[0063] Step 100: Construct an initial image instance library for the source domain image set;

[0064] Step 200: Train and determine the initial segmentation model;

[0065] Step 300: Based on semantic hierarchical domain adaptation, generate semantic pseudo-labels for the target domain image set using the initial segmentation model;

[0066] Step 400: In the source domain image set, class equalization is used to determine pixel instance features and corresponding ground truth labels, and the initial image instance library is updated using the pixel instance features and the corresponding ground truth labels;

[0067] Step 500: Generate instance-level pseudo-labels. Based on the instance-level pseudo-labels, use reverse propagation to backpropagate and optimize the updated initial image instance library to obtain the image semantic segmentation result.

[0068] Specifically, in this embodiment of the invention, an instance feature library and an instance label library of initial size are first constructed. During one iteration, a segmentation model is trained on the source domain using a conventional self-trained domain-adaptive semantic segmentation process. Then, semantic pseudo-labels are generated and filtered based on the semantic classifier prediction of the model. Next, class balance is used to dynamically select features and labels, and moving average is used to update the instance library. Instance-level information is introduced to perform instance similarity graph discrimination, and instance pseudo-labels are generated. The parameters of the segmentation model are optimized by backpropagating the loss function. After completing a preset number of iterations (e.g., 40,000), testing is completed and the model parameters are exported. After the model training and optimization are completed, the image semantic segmentation results are output. The process is summarized as follows: Figure 2 As shown.

[0069] It should be noted that semantic segmentation is a very important core technology in the field of autonomous driving. For images of road scenes, it can perform pixel-level segmentation of objects in the image, thereby providing basic data support for driverless or intelligent assisted driving.

[0070] To address the shortcomings of existing domain-adaptive semantic segmentation algorithms, this invention introduces instance-level information and an additional instance-level loss function to assist in calibrating noisy pseudo-labels that the semantic classifier cannot correctly classify. Furthermore, considering the specific requirements of semantic segmentation tasks, this embodiment designs a limited-size instance library and a dynamic update strategy. This avoids consuming excessive storage space while selecting more representative pixels for instance discrimination calculations. After 40,000 iterations, accuracy is tested, and the final model parameters are derived. This fully utilizes instance-level information for feature discrimination, class equalization, and explores factors affecting segmentation performance, keeping GPU memory usage within the limits of a single 3090 graphics card throughout the training process.

[0071] This invention addresses the common issues of classification errors and noisy pseudo-labels in image semantic segmentation. By introducing instance-level discrimination on top of the semantic level, it assists in calibrating noisy pseudo-labels that the semantic classifier cannot correctly classify. This enables faster training speed for image semantic segmentation and features low cost and high classification accuracy.

[0072] Based on the above embodiments, step 100 includes:

[0073] Determine the size of the image instance feature library, the dimension of the extracted feature vector, the number of source domain segmentation categories, and the number of target domain segmentation categories, wherein the number of source domain segmentation categories is equal to the number of target domain segmentation categories;

[0074] The image instance feature library is constructed based on the size of the image instance feature library and the dimension of the extracted feature vector;

[0075] The size of the image instance feature library is divided by the number of source domain segmentation categories to obtain category marker placeholders, and an instance label library is constructed based on the category marker placeholders.

[0076] Specifically, in this embodiment of the invention, an instance feature library is first constructed. and an instance tag library This is used to store instance information from the source domain. The instance library here has a fixed size of K, and the source and target domains, along with the number of categories to be segmented, are C. For each category, a placeholder marker is set in the same proportion, denoted as . D represents the dimension of the extracted feature vector, and a matching label is attached to the corresponding position in the label library. The instance feature library is initialized by filling with random vectors.

[0077] Based on the above embodiments, step 200 includes:

[0078] In the source domain image set, a single source domain image sample and the number of source domain image samples are determined respectively; in the source domain label set, a single source domain label sample and the number of source domain label samples are determined respectively; in the target domain image set, a single target domain image sample and the number of target domain image samples are determined respectively; and in the target domain label set, a single target domain label sample and the number of target domain label samples are determined respectively.

[0079] A feature map is obtained by extracting the single source domain image sample using a feature extractor based on an encoder-decoder structure. A fully connected pixel-level classifier is then used to map the feature map to semantic prediction to obtain semantic-level prediction probabilities.

[0080] The classification cross-entropy loss function is adopted, which is used to obtain the source domain sample loss function by considering the feature map length, feature map width, probability that any pixel in the feature map belongs to any class, and probability that any label in the feature map belongs to any class.

[0081] The initial segmentation model is obtained by optimizing the source domain samples based on the source domain sample loss function.

[0082] Specifically, given a source domain image and single hot label Train a neural network to acquire useful knowledge from the source domain and expect it to be applied to the target image. Without accessing the target ground truth label γ t Here, j represents the number of image samples in each domain, and x s y represents a single sample in the source domain image set. s n represents a single sample in the source domain label set. s n represents the number of samples in the source domain image. t x represents the number of samples in the target domain image. t This represents a single sample in the target domain image set.

[0083] In a typical UDA process, an encoder-decoder based feature extractor f(·) is used to extract the feature map m = f(x). s And use a fully connected pixel-level classifier g(·) to map m to semantic prediction, written as p s = g(m), where m represents the entire feature map extracted by the feature extractor, containing the feature embedding of each pixel, and p s This represents the probability of semantic-level predictions obtained after passing a complete feature map m through a classifier. Then, the classification cross-entropy (CE) loss is used to directly optimize the source domain samples. in Represents pixels The softmax probability belonging to class c is similarly defined and applies to... Represents pixels The softmax probability of belonging to class c. This represents the label of the corresponding pixel in the source domain, where H represents the length of the image and W represents the width of the image. This represents the source domain sample loss function.

[0084] Based on the above embodiments, step 300 includes:

[0085] Determine the semantic pseudo-label category identifier, and determine the probability that any target domain pixel belongs to any category based on the semantic pseudo-label category identifier;

[0086] The initial semantic classification pseudo-label of the target domain is obtained from the probability that any pixel in the target domain belongs to any category and the initial segmentation model;

[0087] A confidence threshold is determined, and the pseudo-labels of the primary semantic classification of the target domain are filtered based on the confidence threshold to obtain the semantic loss function of the target domain.

[0088] Based on the target domain semantic-level loss function, semantic pseudo-labels are obtained for the target domain image set.

[0089] Specifically, semantic pseudo-labels are directly generated using a classifier trained on the source domain. Notation:

[0090]

[0091] in, This represents the initial semantic classification pseudo-label obtained after applying a classifier trained in the source domain to the target domain; c ′ The category identifier representing the semantic pseudo-tag is the same as the specified number of categories; This represents the probability that a pixel in the target domain is predicted to belong to any category after passing through the image classifier; i represents any pixel in the target domain.

[0092] Then, a confidence threshold τ is set to filter target sample pixels in the pseudo-labels whose maximum class probability is greater than τ. The unsupervised semantic classification loss on the target domain can be defined as:

[0093]

[0094] Here L represents the target sample pixel whose maximum class probability is greater than τ in the pseudo-label. tgt It is the semantic loss function of the target domain.

[0095] Based on the above embodiments, step 400 includes:

[0096] A boundary mask for any source domain image in the source domain image set is generated using a mask generation algorithm, and a boundary pixel map is obtained by multiplying the source domain image label by the boundary mask.

[0097] The average value of pixel feature embedding for any class in the boundary pixel image is obtained, and the average value of pixel feature embedding for any cluster is determined in non-edge entities using a preset clustering method.

[0098] The average instance features of the embedding average of any class of pixel features and the embedding average of any cluster of pixel features are used as the embedding features of the image instance feature library;

[0099] Based on the preset instance library update interval and momentum coefficient, the embedded features of the image instance feature library are updated using exponential moving average to obtain the updated image instance feature library.

[0100] The step of generating a boundary mask for any source domain image in the source domain image set using a mask generation algorithm includes:

[0101] Obtain the true label, height, and width of any source domain image;

[0102] Initialize an all-zero matrix constructed using the height and width of any source domain image, and determine the preset receptive field and the preset number of pixel categories;

[0103] If it is determined that there are different pixels within the preset receptive field of the preset dimension around any pixel in any source domain image, which are greater than or equal to the number of preset pixel categories, then any pixel is determined to be a boundary pixel, and the boundary mask position corresponding to any pixel is set to 1; otherwise, it is set to 0.

[0104] The boundary mask is obtained by traversing all pixels in any source domain image.

[0105] Specifically, for each iteration, the source domain image with its own true label is input, and a boundary mask is generated using a mask generation algorithm, as follows:

[0106] Algorithm: Boundary Mask Generation

[0107] Input: Real tags The dimensions are H×W

[0108] Output: Boundary mask M of image l l

[0109] Initialization: A zero matrix M of size H×W l

[0110] A receptive field R of size 3×3, threshold σ←2

[0111] For i←0 to H do

[0112] Forj←0toWdo

[0113] Initialize ClassCount ← 0

[0114] Current ← pixel γ[i,j]

[0115] ClassCount ← Number of different categories within the Current Sensing Field (R)

[0116] If ClassCount > σthenM[i,j] ← 1

[0117] Furthermore, the boundary pixel map ε s It is easy to be Calculations show that This represents the image label in the source domain. For the c-th category, the average value of the pixel feature embeddings belonging to the c-th category on the boundary pixel map is taken. To balance the selection, k-means clustering is used on non-marginal entities, and the center embedding of the c-th cluster is selected. Finally, e b and e θ Average instance feature embedding between e avg ∈R 1×C As a feature instance library updated each time The required feature embedding, e b e represents the average feature value of pixels belonging to class c on the boundary pixel map; θ This represents the average feature value of pixels belonging to the c-th cluster on non-edge entities.

[0118] Set the instance library update interval to u, and the update method to Exponential Moving Average (EMA):

[0119] e u ←ωe u-1 +(1-ω)e avg

[0120] Here, ω∈[0,1) is the momentum coefficient. If the current image does not contain an instance of a certain class, an update of all embeddings belonging to that class will be skipped.

[0121] Based on the above embodiments, step 500 includes:

[0122] Obtain feature maps, pixel embedding representations from image instance libraries, preset strong enhancement target sample instance embeddings, preset weak enhancement target sample instance embeddings, and any instance from the instance library to determine the temperature parameter for distribution flatness.

[0123] Using cosine similarity, based on the matrix transpose of the preset strongly enhanced target sample instance embedding, any instance in the instance library, the pixel embedding representation of the image instance library, and the temperature parameter of the distribution flatness, the probability that any preset strongly enhanced target sample instance embedding is identified as any instance in the instance library is obtained.

[0124] Using cosine similarity, based on the matrix transpose of the preset weakly enhanced target sample instance embedding, any instance in the instance library, the pixel embedding representation of the image instance library, and the temperature parameter of the distribution flatness, the probability that any preset weakly enhanced target sample instance embedding is identified as any instance in the instance library is obtained.

[0125] The cross-entropy loss function is used to minimize the difference between the probability that any preset strongly enhanced target sample instance embedding is identified as any instance in the instance library and the probability that any preset weakly enhanced target sample instance embedding is identified as any instance in the instance library, thus obtaining the instance-level loss function;

[0126] The domain adaptive overall loss function is obtained by summing the products of the source domain sample loss function, the target domain semantic level loss function, and the instance level loss function with the instance level loss weight.

[0127] Repeat the above training process for a preset number of iterations, and output the image semantic segmentation result.

[0128] Specifically, with Figure 3 The example shown is a schematic diagram of instance-level information calibration semantic-level noise pseudo-labels, illustrated using the classification of motorcycles and bicycles as an example:

[0129] (a) Semantic pseudo-labels: A semantic classifier trained on the source domain can be viewed as a class feature center. It has an inherent weakness in classifying pixels that are close to or cross class boundaries, thus producing erroneous and noisy semantic pseudo-labels.

[0130] (b) The method proposed in this invention: Since instance-level discrimination is not affected by the semantic classifier boundary, the "instance loss" instance-level discrimination proposed in this embodiment adjusts the noisy pseudo-labels. It brings similar pixels closer in distribution and excludes out-of-class pixels. By simultaneously considering semantic-level and instance-level information with cross-level interaction, the classification boundary is reset to obtain more robust pseudo-labels.

[0131] Based on the above principles, when considering instance-level differentiation, each pixel instance is treated as a unique class, and the model is adjusted to distinguish between "individual" instance classes. Before classifying each pixel, a feature embedding mapping m is obtained. This is obtained from the source domain image and stored in the instance library. The pixel embedding is represented as {ek :k∈(1,…,K)}. Similarly, e t ∈m t and Instance embeddings are used to represent strongly and weakly enhanced target samples. Embodiments of this invention use a method similar to cosine similarity to calculate the similarity between pixels in the target sample and source domain pixels stored in the instance library:

[0132] For those with embedded e t Strongly enhanced target pixel instance x t The probability that it is identified as the i-th instance in the instance library can be defined as follows: Where t is a temperature parameter that controls the flatness of the distribution.

[0133] Weakly enhanced samples The similarity is then used as instance-level pseudo-tags.

[0134]

[0135] Here, T represents the matrix transpose, and α represents the weakly augmented sample label from the target domain. and The definitions are the same, but here, in order to match the semantic level tags, an additional instance level pseudo-tag, e, is defined. i This represents the i-th element in the instance library;

[0136] Then, an additional instance-level cross-entropy loss function is introduced. To minimize q i and Differences between them:

[0137]

[0138] Figure 4 The process of two-level self-training is shown. The left side shows the training process at the semantic level, and the right side shows the training process at the instance level, which uses source samples, strongly enhanced target images, and weakly enhanced target images, respectively.

[0139] Finally, as Figure 5 As shown, the overall domain loss function is derived from the training objective. for:

[0140]

[0141] Where λ ins Weights that control instance-level loss.

[0142] This invention fully utilizes instance-level information for feature discrimination, class balancing, and the exploration of factors influencing segmentation performance. Throughout the training process, GPU memory usage is kept within the limits of a single 3090 graphics card, resulting in fast training speed, low cost, and high segmentation accuracy. It exhibits particularly good segmentation performance for long-tailed and easily confused categories. Furthermore, the invention's two-level interactive approach can be extended to similar self-training tasks, providing an effective training algorithm framework for domain adaptation and semi-supervised learning.

[0143] The following describes the domain-adaptive image semantic segmentation system with two-level interaction provided by the present invention. The domain-adaptive image semantic segmentation system with two-level interaction described below can be referred to in correspondence with the domain-adaptive image semantic segmentation method with two-level interaction described above.

[0144] Figure 6 This is a schematic diagram of the structure of the two-level interactive domain-adaptive image semantic segmentation system provided in an embodiment of the present invention, as shown below. Figure 6 As shown, it includes: a construction module 61, a training module 62, a generation module 63, an update module 64, and a processing module 65, wherein:

[0145] The construction module 61 is used to construct an initial image instance library for the source domain image set; the training module 62 is used to train and determine the initial segmentation model; the generation module 63 is used to generate semantic pseudo-labels for the target domain image set based on semantic hierarchical domain adaptation using the initial segmentation model; the update module 64 is used to determine pixel instance features and corresponding ground truth labels in the source domain image set using class equalization, and update the initial image instance library using the pixel instance features and the corresponding ground truth labels; the processing module 65 is used to generate instance-level pseudo-labels, and based on the instance-level pseudo-labels, backpropagation is used to optimize and update the initial image instance library to obtain the image semantic segmentation result.

[0146] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a two-level interactive domain-adaptive image semantic segmentation method. This method includes: constructing an initial image instance library for a source domain image set; training and determining an initial segmentation model; generating semantic pseudo-labels for a target domain image set based on semantic hierarchical domain adaptation using the initial segmentation model; determining pixel instance features and corresponding ground truth labels in the source domain image set using class equalization, and updating the initial image instance library using the pixel instance features and corresponding ground truth labels; generating instance-level pseudo-labels; and optimizing the updated initial image instance library using backpropagation based on the instance-level pseudo-labels to obtain the image semantic segmentation result.

[0147] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a two-level interactive domain-adaptive image semantic segmentation method provided by the methods described above. The method includes: constructing an initial image instance library of a source domain image set; training and determining an initial segmentation model; generating semantic pseudo-labels for a target domain image set using the initial segmentation model based on semantic hierarchical domain adaptation; determining pixel instance features and corresponding real labels in the source domain image set using class equalization, and updating the initial image instance library using the pixel instance features and the corresponding real labels; generating instance-level pseudo-labels; and optimizing the updated initial image instance library using backpropagation based on the instance-level pseudo-labels to obtain the image semantic segmentation result.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A domain-adaptive image semantic segmentation method with two-level interaction, characterized in that, include: Construct an initial image instance library for the source domain image set; Training determines the initial segmentation model; Based on semantic hierarchical domain adaptation, semantic pseudo-labels for the target domain image set are generated using the initial segmentation model; In the source domain image set, class equalization is used to determine pixel instance features and corresponding real labels, and the initial image instance library is updated using the pixel instance features and the corresponding real labels. Instance-level pseudo-labels are generated, and based on the instance-level pseudo-labels, the optimized and updated initial image instance library is backpropagated using reverse propagation to obtain the image semantic segmentation result. The semantic hierarchical domain adaptation method, which utilizes the initial segmentation model to generate semantic pseudo-labels for the target domain image set, includes: Determine the semantic pseudo-label category identifier, and determine the probability that any target domain pixel belongs to any category based on the semantic pseudo-label category identifier; The initial semantic classification pseudo-label of the target domain is obtained from the probability that any pixel in the target domain belongs to any category and the initial segmentation model; A confidence threshold is determined, and the pseudo-labels of the primary semantic classification of the target domain are filtered based on the confidence threshold to obtain the semantic loss function of the target domain. Based on the target domain semantic-level loss function, semantic pseudo-labels are obtained for the target domain image set; The generated instance-level pseudo-labels, based on which the optimized and updated initial image instance library is backpropagated using reverse propagation, yield image semantic segmentation results, including: Obtain feature maps, pixel embedding representations from image instance libraries, preset strong enhancement target sample instance embeddings, preset weak enhancement target sample instance embeddings, and any instance from the instance library to determine the temperature parameter for distribution flatness. Using cosine similarity, based on the matrix transpose of the preset strongly enhanced target sample instance embedding, any instance in the instance library, the pixel embedding representation of the image instance library, and the temperature parameter of the distribution flatness, the probability that any preset strongly enhanced target sample instance embedding is identified as any instance in the instance library is obtained. Using cosine similarity, based on the matrix transpose of the preset weakly enhanced target sample instance embedding, any instance in the instance library, the pixel embedding representation of the image instance library, and the temperature parameter of the distribution flatness, the probability that any preset weakly enhanced target sample instance embedding is identified as any instance in the instance library is obtained. The cross-entropy loss function is used to minimize the difference between the probability that any preset strongly enhanced target sample instance embedding is identified as any instance in the instance library and the probability that any preset weakly enhanced target sample instance embedding is identified as any instance in the instance library, thus obtaining the instance-level loss function; The domain adaptive overall loss function is obtained by summing the products of the source domain sample loss function, the target domain semantic level loss function, and the instance level loss function with the instance level loss weight. Repeat the above training process for a preset number of iterations, and output the image semantic segmentation result.

2. The domain-adaptive image semantic segmentation method with two-level interaction according to claim 1, characterized in that, The initial image instance library for constructing the source domain image set includes: Determine the size of the image instance feature library, the dimension of the extracted feature vector, the number of source domain segmentation categories, and the number of target domain segmentation categories, wherein the number of source domain segmentation categories is equal to the number of target domain segmentation categories; The image instance feature library is constructed based on the size of the image instance feature library and the dimension of the extracted feature vector; The size of the image instance feature library is divided by the number of source domain segmentation categories to obtain category marker placeholders, and an instance label library is constructed based on the category marker placeholders.

3. The domain-adaptive image semantic segmentation method with two-level interaction according to claim 1, characterized in that, The training determines the initial segmentation model, including: In the source domain image set, a single source domain image sample and the number of source domain image samples are determined respectively; in the source domain label set, a single source domain label sample and the number of source domain label samples are determined respectively; in the target domain image set, a single target domain image sample and the number of target domain image samples are determined respectively; and in the target domain label set, a single target domain label sample and the number of target domain label samples are determined respectively. A feature map is obtained by extracting the single source domain image sample using a feature extractor based on an encoder-decoder structure. A fully connected pixel-level classifier is then used to map the feature map to semantic prediction to obtain semantic-level prediction probabilities. The classification cross-entropy loss function is adopted, which is used to obtain the source domain sample loss function by considering the feature map length, feature map width, probability that any pixel in the feature map belongs to any class, and probability that any label in the feature map belongs to any class. The initial segmentation model is obtained by optimizing the source domain samples based on the source domain sample loss function.

4. The domain-adaptive image semantic segmentation method with two-level interaction according to claim 1, characterized in that, The step of determining pixel instance features and corresponding ground truth labels by using class equalization in the source domain image set, and updating the initial image instance library using the pixel instance features and the corresponding ground truth labels, includes: A boundary mask for any source domain image in the source domain image set is generated using a mask generation algorithm, and a boundary pixel map is obtained by multiplying the source domain image label by the boundary mask. The average value of pixel feature embedding for any class in the boundary pixel image is obtained, and the average value of pixel feature embedding for any cluster is determined in non-edge entities using a preset clustering method. The average instance features of the embedding average of any class of pixel features and the embedding average of any cluster of pixel features are used as the embedding features of the image instance feature library; Based on the preset instance library update interval and momentum coefficient, the embedded features of the image instance feature library are updated using exponential moving average to obtain the updated image instance feature library.

5. The domain-adaptive image semantic segmentation method with two-level interaction according to claim 4, characterized in that, The step of generating a boundary mask for any source domain image in the source domain image set using a mask generation algorithm includes: Obtain the true label, height, and width of any source domain image; Initialize an all-zero matrix constructed using the height and width of any source domain image, and determine the preset receptive field and the preset number of pixel categories; If it is determined that there are different pixels within the preset receptive field of the preset dimension around any pixel in any source domain image, which are greater than or equal to the number of preset pixel categories, then any pixel is determined to be a boundary pixel, and the boundary mask position corresponding to any pixel is set to 1; otherwise, it is set to 0. The boundary mask is obtained by traversing all pixels in any source domain image.

6. A two-level interactive domain-adaptive image semantic segmentation system, based on the two-level interactive domain-adaptive image semantic segmentation method according to any one of claims 1 to 5, characterized in that, include: The building module is used to construct an initial image instance library for the source domain image set; The training module is used to train and determine the initial segmentation model; The generation module is used to generate semantic pseudo-labels for the target domain image set based on semantic hierarchical domain adaptation and using the initial segmentation model; The update module is used to determine pixel instance features and corresponding real labels in the source domain image set by class equalization, and update the initial image instance library using the pixel instance features and the corresponding real labels; The processing module is used to generate instance-level pseudo-labels, and based on the instance-level pseudo-labels, it uses reverse propagation to back-pass the optimized and updated initial image instance library to obtain the image semantic segmentation result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the domain-adaptive image semantic segmentation method with two-level interaction as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the domain-adaptive image semantic segmentation method with two-level interaction as described in any one of claims 1 to 5.

Images