Lightweight Unsupervised Adaptive Image Semantic Segmentation Method and System

Abstract

This invention discloses a lightweight unsupervised adaptive image semantic segmentation method and system, belonging to the field of image processing technology. The technical problem it addresses is how to achieve image semantic segmentation in high-resolution scenes while reducing the need for labeled data. The method includes the following steps: acquiring source and target domain images as raw images and performing image preprocessing on the raw images; constructing an image semantic segmentation model based on the Transformer model, and performing lightweight processing on the image semantic segmentation model based on sparse coding and SmoothQuant quantization methods; using the encoder in the lightweight semantic image segmentation model as a shared feature extractor, the decoder in the lightweight semantic image segmentation model as a generator, and introducing a discriminator and generator as a GAN adversarial mechanism. The shared feature extractor is used to achieve domain transfer through feature alignment and domain adaptation in the target domain, and the adversarial mechanism is used to achieve similar feature distributions between the source and target domains.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a lightweight unsupervised adaptive image semantic segmentation method and system. Background Technology

[0002] Traditional Transformer models typically perform well in natural language processing, but their high complexity can lead to significant computational and memory consumption issues when used in computer vision tasks. In image semantic segmentation, processing high-resolution or large-scale images can result in substantial computational overhead. Lightweight Transformer models reduce the number of parameters and computational complexity, accelerating image semantic segmentation and making them easier to apply to real-time high-resolution scene image semantic segmentation tasks. Furthermore, training a high-performance image semantic segmentation model for high-resolution scenes requires a large amount of labeled data; however, acquiring labeled data is a time-consuming and labor-intensive task. Unsupervised domain adaptation techniques are an effective knowledge transfer method in image semantic segmentation, reducing the need for labeled data, improving model generalization performance, and making the model more adaptable to diverse scenarios, thus making it more practical and scalable in real-world high-resolution scene image applications.

[0003] How to achieve semantic segmentation of images in high-resolution scenarios and reduce the need for labeled data is a technical problem that needs to be solved. Summary of the Invention

[0004] The technical objective of this invention is to address the above-mentioned shortcomings by providing a lightweight unsupervised adaptive image semantic segmentation method and system, thereby solving the problem of how to achieve image semantic segmentation in high-resolution scenarios and reduce the need for labeled data.

[0005] In a first aspect, the present invention provides a lightweight unsupervised adaptive image semantic segmentation method, comprising the following steps:

[0006] Image preprocessing: Obtain source and target domain images as raw images. The source domain image contains labeled data, and the target domain image contains unlabeled or limited labeled data. Perform image preprocessing on the raw images to obtain preprocessed images, and use the preprocessed images as sample images.

[0007] Model Construction: An image semantic segmentation model is constructed based on the Transformer model, and SmoothQuant is introduced into the Transformer model. The image semantic segmentation model is lightweighted based on sparse coding and SmoothQuant quantization methods to obtain a lightweight image semantic segmentation model. The image semantic segmentation model includes an encoder and a decoder, which are used to extract features from the input image and output feature maps.

[0008] Unsupervised Adaptation: The encoder in the lightweight semantic image segmentation model is used as a shared feature extractor, and the decoder in the lightweight semantic image segmentation model is used as a generator. A discriminator and a generator are introduced as GAN adversaries. The shared feature extractor and generator are trained on sample images corresponding to the source domain image and the target domain image. The shared feature extractor is used to achieve domain transfer by performing feature alignment and domain adaptation on the target domain. The adversaries are used to achieve adversarial learning of feature distribution to achieve similarity of feature distribution between the source domain and the target domain.

[0009] Preferably, when preprocessing the original image, the source and target images are resized, and data augmentation operations are performed on the source image, including random flipping, rotation, scaling, and cropping.

[0010] The annotation data corresponding to the source and target domain images includes category labels and location information, and the category labels between the source and target domain images are consistent.

[0011] Preferably, the encoder and decoder in the Transformer model both include a global-local Transformer network structure. The global-local Transformer network structure includes an input module, a global-local Transformer block, and an output module. The input module is used to map the input image to the dimensional space of the network structure and add positional encoding to take into account the positional information in the sequence.

[0012] The global-local Transformer block includes a local branch, a global branch, and a feature fusion part. The local branch is used to extract local context information based on convolutional operations or windowed attention mechanisms, and outputs a local feature map. The global branch is used to extract global uplink information based on a linear multi-head attention mechanism, and outputs a global feature map. The output of the global branch is deployed with a trainable scale factor. For the extracted global context information, the global context information at different locations is fused by calculating a weighted sum to obtain a feature map. A cross-shaped window context interaction module is used to capture cross-window relationships, and the two feature maps generated by the horizontal average pooling layer and the vertical average pooling layer are fused to obtain global features. The feature fusion part is used to fuse local features and global features, and performs convolutional operations and batch normalization operations on the fused image to obtain a fused feature map.

[0013] The output module includes one or more linear layers and activation functions to map the feature map output by the GLTB block back to the target dimension of the model to generate the final prediction or feature representation.

[0014] As a preferred approach, the image semantic segmentation model is lightweighted based on sparse coding and SmoothQuant quantization methods, including the following steps:

[0015] The weights in the image semantic segmentation model are represented by a sparse coding method, and the weights are decomposed into sparse weights and dense weights.

[0016] The dense weights are quantized and then reconstructed using sparse weights.

[0017] Preferably, the shared feature extractor is used to extract semantic features from the input image and output the corresponding feature map by taking the sample images corresponding to the source domain image and the target domain image as input;

[0018] The GAN adversarial generator includes a generator and a discriminator. The generator is used to map the features corresponding to the target domain image to the source domain feature space and generate a disguised source domain feature representation.

[0019] The discriminator is used to distinguish between source domain and target domain features by taking the fake source domain features, real source domain features and real target domain features output by the generator as input, and outputting a binary classification result.

[0020] Secondly, the present invention provides a lightweight unsupervised adaptive image semantic segmentation system, characterized in that it includes an image preprocessing module, a model building module, and an unsupervised adaptive module.

[0021] The image preprocessing module is used to perform the following: acquire source domain image and target domain image as original images. The source domain image contains labeled data, and the target domain image contains unlabeled or limited labeled data. The original images are preprocessed to obtain preprocessed images, and the preprocessed images are used as sample images.

[0022] The model building module is used to perform the following: build an image semantic segmentation model based on the Transformer model, introduce SmoothQuant into the Transformer model, and perform model lightweighting processing on the image semantic segmentation model based on sparse coding and SmoothQuant quantization method to obtain a lightweight image semantic segmentation model. The image semantic segmentation model includes an encoder and a decoder, which are used to extract features from the input image and output feature maps.

[0023] The unsupervised adaptive module is used to perform the following: the encoder in the lightweight semantic image segmentation model is used as a shared feature extractor, the decoder in the lightweight semantic image segmentation model is used as a generator, and a discriminator and generator are introduced as GAN adversaries. The shared feature extractor and generator are trained on sample images corresponding to the source domain image and the target domain image. The shared feature extractor is used to achieve domain transfer by performing feature alignment and domain adaptation on the target domain. The adversaries are used to achieve adversarial learning of feature distribution to achieve similarity of feature distribution between the source domain and the target domain.

[0024] Preferably, when preprocessing the original image, the source and target images are resized, and data augmentation operations are performed on the source image, including random flipping, rotation, scaling, and cropping.

[0025] The annotation data corresponding to the source and target domain images includes category labels and location information, and the category labels between the source and target domain images are consistent.

[0026] Preferably, the encoder and decoder in the Transformer model both include a global-local Transformer network structure. The global-local Transformer network structure includes an input module, a global-local Transformer block, and an output module. The input module is used to map the input image to the dimensional space of the network structure and add positional encoding to take into account the positional information in the sequence.

[0027] The global-local Transformer block includes a local branch, a global branch, and a feature fusion part. The local branch is used to extract local context information based on convolutional operations or windowed attention mechanisms, and outputs a local feature map. The global branch is used to extract global uplink information based on a linear multi-head attention mechanism, and outputs a global feature map. The output of the global branch is deployed with a trainable scale factor. For the extracted global context information, the global context information at different locations is fused by calculating a weighted sum to obtain a feature map. A cross-shaped window context interaction module is used to capture cross-window relationships, and the two feature maps generated by the horizontal average pooling layer and the vertical average pooling layer are fused to obtain global features. The feature fusion part is used to fuse local features and global features, and performs convolutional operations and batch normalization operations on the fused image to obtain a fused feature map.

[0028] The output module includes one or more linear layers and activation functions to map the feature map output by the GLTB block back to the target dimension of the model to generate the final prediction or feature representation.

[0029] As a preferred approach, the image semantic segmentation model is lightweighted based on sparse coding and SmoothQuant quantization methods, including the following steps:

[0030] The weights in the image semantic segmentation model are represented by a sparse coding method, and the weights are decomposed into sparse weights and dense weights.

[0031] The dense weights are quantized and then reconstructed using sparse weights.

[0032] Preferably, the shared feature extractor is used to extract semantic features from the input image and output the corresponding feature map by taking the sample images corresponding to the source domain image and the target domain image as input;

[0033] The GAN adversarial generator includes a generator and a discriminator. The generator is used to map the features corresponding to the target domain image to the source domain feature space and generate a disguised source domain feature representation.

[0034] The discriminator is used to distinguish between source domain and target domain features by taking the fake source domain features, real source domain features and real target domain features output by the generator as input, and outputting a binary classification result.

[0035] The lightweight unsupervised adaptive image semantic segmentation method and system of the present invention have the following advantages: based on the Transformer model, it effectively captures the global and local contextual information of the image, adds unsupervised domain adaptive technology, and performs lightweight operations to be suitable for real-time tasks, thereby achieving semantic segmentation of high-resolution scene images. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] The invention will be further described below with reference to the accompanying drawings.

[0038] Figure 1 This is a schematic diagram of the quantization module structure in the Transformer model of the lightweight unsupervised adaptive image semantic segmentation method in Example 1;

[0039] Figure 2 This is a schematic diagram of the GLTB module in the Transformer model of the lightweight unsupervised adaptive image semantic segmentation method in Example 1;

[0040] Figure 3 This is a schematic diagram of the cross-shaped window context interaction module in the GTLB module of the lightweight unsupervised adaptive image semantic segmentation method in Example 1;

[0041] Figure 4 This is a schematic diagram of the quantization module in the Transformer model of the lightweight unsupervised adaptive image semantic segmentation method in Example 1. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0043] This invention provides a lightweight unsupervised adaptive image semantic segmentation method and system to address the technical problem of how to achieve image semantic segmentation in high-resolution scenarios and reduce the need for labeled data.

[0044] Example 1:

[0045] This invention provides a lightweight unsupervised adaptive image semantic segmentation method, which includes three steps: image preprocessing, model building, and unsupervised adaptation.

[0046] Step S100 Image Preprocessing: Obtain source domain image and target domain image as original images. The source domain image contains labeled data, and the target domain image contains unlabeled or limited labeled data. Perform image preprocessing on the original images to obtain preprocessed images, and use the preprocessed images as sample images.

[0047] As a specific implementation of image preprocessing, when preprocessing the original image, the source domain image and the target domain image are resized, and data augmentation operations are performed on the source domain image, including random flipping, rotation, scaling, and cropping.

[0048] In this embodiment, the annotation data corresponding to the source domain image and the target domain image includes category labels and location information, and the category labels between the source domain image and the target domain image are consistent.

[0049] The labels include information about the object's category, such as cats, dogs, and cars in the image. These labels help the model identify different objects in the image.

[0050] The location information of an object is usually provided in the form of a bounding box. This helps the model not only to recognize the object, but also to locate its position.

[0051] In this embodiment, a category label is assigned to each pixel of the image to identify different objects or parts of objects in the image. The purpose of the labeling is to enable machine learning models to automatically recognize and understand new and unseen images by learning these labels.

[0052] Limited labeled data typically refers to datasets within the target domain that contain relatively little or no labeled data. The main difference between limited labeled data and labeled data in the source domain is the smaller quantity or lower quality. This may be because obtaining high-quality labeled data in the target domain can be more difficult, expensive, or time-consuming, or the target domain may be a new field with insufficient labeled data available. Therefore, limited labeled data may not provide the same training signals as labeled data in the source domain, which can negatively impact the performance of models trained in the target domain.

[0053] In transfer learning or domain adaptation tasks, the typical goal is to train a model using labeled data from the source domain, and then fine-tune the model using a limited set of labeled data from the target domain to enable it to perform well in the target domain. This is because labeled data from the source domain provides more information to initialize the model's weights and parameters, while the limited labeled data is used to fine-tune the model to adapt it to the specific data distribution and features of the target domain.

[0054] In this embodiment, the labeled data is used to enable the machine learning model to learn the relationship between images and labels. The limited labeled data is less numerous or of lower quality compared to the source domain labeled data, but it is still used to adjust the model to adapt to the target domain data.

[0055] The typical goal is to train the model using labeled data from the source domain, and then fine-tune the model using a limited set of labeled data from the target domain to ensure good performance in the target domain. This is because labeled data from the source domain provides more information to initialize the model's weights and parameters, while the limited labeled data is used to fine-tune the model to fit the specific data distribution and features of the target domain.

[0056] Step S200 Model Construction: Construct an image semantic segmentation model based on the Transformer model, and introduce SmoothQuant into the Transformer model. Perform model lightweighting processing on the image semantic segmentation model based on sparse coding and SmoothQuant quantization method to obtain a lightweight image semantic segmentation model. The image semantic segmentation model includes an encoder and a decoder, which are used to extract features from the input image and output feature maps.

[0057] In this embodiment, both the encoder and decoder in the Transformer model include a global-local Transformer network structure, which includes an input module, a global-local Transformer block, and an output module.

[0058] The input module maps the input image to the dimensional space of the network structure and adds positional encoding to take into account positional information in the sequence.

[0059] The global-local Transformer block includes a local branch, a global branch, and a feature fusion section. The local branch extracts local context information based on convolutional operations or windowed attention mechanisms, outputting a local feature map. The global branch extracts global uplink information based on a linear multi-head attention mechanism, outputting a global feature map. The output of the global branch is equipped with a trainable scale factor. For the extracted global context information, the global context information at different locations is fused by calculating a weighted sum to obtain a feature map. A cross-shaped window context interaction module is used to capture cross-window relationships, fusing the two feature maps generated by the horizontal average pooling layer and the vertical average pooling layer to obtain global features. The feature fusion section fuses the local and global features and performs convolutional operations and batch normalization operations on the fused image to obtain the fused feature map.

[0060] In this embodiment, the global-local Transformer block is used to perform the following:

[0061] (1) Global attention: Learn the correlation between global features through self-attention mechanism to capture the global dependencies at different positions in the sequence;

[0062] (2) Local attention: By using local convolution or windowing attention mechanisms, we can focus on the features of certain local regions in the sequence in order to capture more fine-grained local information;

[0063] (3) Global-local fusion: By combining global features and local features, a comprehensive feature representation is created.

[0064] The output module is used to map the output features of the GLTB block back to the target dimension of the model. It typically includes one or more linear layers and activation functions to generate the final prediction or feature representation.

[0065] As a specific implementation of this global-local Transformer network structure, the following explanation is provided:

[0066] 1. Input Module:

[0067] Input: Raw sequence data or feature representation;

[0068] Function: It takes raw sequence data or feature representation as input, prepares the data for processing by GLTB blocks, which typically includes an embedding layer to map the input data to the model’s dimensional space and adds positional encoding to take positional information in the sequence into account.

[0069] 2. Global-Local Transformer Block (GLTB):

[0070] Input: Feature representations obtained from the input module;

[0071] Function: GLTB is a core component of the model, performing the following main functions:

[0072] (1) Global attention: Through the self-attention mechanism (multi-head attention in Transformer), GLTB can learn the correlation between global features and capture the global dependencies at different positions in the sequence.

[0073] (2) Local attention: GLTB also includes a local attention mechanism, which uses local convolution or windowed attention mechanism to focus on the features of certain local regions in the sequence in order to capture more fine-grained local information.

[0074] (3) Global-Local Fusion: GLTB creates a comprehensive feature representation by combining global and local features, aiming to balance global and local information and thus improve model performance.

[0075] 3. Output Module:

[0076] Input: Output feature representation of the GLTB block;

[0077] Function: The output module is responsible for mapping the output features of the GLTB block back to the target dimension of the model. It typically includes one or more linear layers and activation functions to generate the final prediction or feature representation.

[0078] The overall workflow of this global-local Transformer network structure is as follows:

[0079] (1) The input module accepts raw sequence data or feature representations and prepares the input data through embedding and positional encoding;

[0080] (2) The GLTB block receives the input feature representation and performs global and local attention operations simultaneously to capture global and local information and generate a comprehensive feature representation;

[0081] (3) The output module receives the output feature representation of the GLTB block, maps it to the target dimension of the model, and may perform the final prediction or feature representation.

[0082] As a specific implementation of the GLTB block, the GLTB block consists of global-local attention, MLP (Multilayer Perception), two batchnorm layers, and two summation operations.

[0083] The local branch (which is the local part of the Global-LocalTransformerBlock, GLTB) uses two parallel convolutional layers (Conv3×3, Conv1×1) to extract local context information. Batch normalization is applied after the convolutional layers, and finally, a summation operation is performed.

[0084] The global branch (here, "global" refers to the global branch in GLTB) uses linear multi-head self-attention. (Linear multi-head self-attention is an attention mechanism typically used to process sequential data, such as text in natural language processing. In self-attention, each input element (e.g., a word or a position) interacts with other elements and is assigned a weight to determine which elements should receive more attention when modeling the context. Linear multi-head self-attention is an extended form of self-attention that uses multiple attention heads (independent sets of weights) to capture different contextual relationships.)

[0085] In this embodiment, the GLTB block deploys a trainable scaling factor on its output: during self-attention computation, attention scores (weights) are typically multiplied by the input values ​​to calculate a weighted combination of each element. The trainable scaling factor is a learnable parameter used to scale these attention scores. This helps control the range and stability of the attention distribution. By learning an appropriate scaling factor, the model can better adapt to different input data and tasks.

[0086] This embodiment of the GLTB block implements stable context aggregation: context aggregation refers to combining information from different locations or elements to better understand the context of the entire sequence or image. In self-attention, context aggregation is achieved by calculating a weighted sum, where the weight of each element is determined by the attention mechanism. By using a trainable scaling factor, the attention mechanism can be made more stable, helping the model to better capture the relationships between input data and aggregate contextual information.

[0087] The process for implementing the aggregation context is as follows: First, expand the number of channels in the input feature map; then, apply a window partitioning operation to divide the feature sequence into Q, K, and V. The normalization function is softmax, and the result generated by self-attention is:

[0088]

[0089] Where Vi,j, represents the j-th feature in the i-th row of the resulting matrix, a trainable scaling factor is deployed on the output of the linear multi-head self-attention to achieve stable context aggregation.

[0090] Specifically, the cross-shaped window context interaction module is used to capture cross-window relationships and fuse the two feature maps generated by the horizontal average pooling layer and the vertical average pooling layer to obtain the global context.

[0091] Horizontal Average Pooling Layer: Horizontal average pooling is a pooling operation typically applied to feature maps (or images). In horizontal average pooling, the average pooling operation is performed on each row (horizontally) of the feature map, merging the pixel values ​​of each row into a single value. This means that each row is compressed into a single number that reflects the average features of that row. This helps capture the horizontal global information of the image.

[0092] Based on cross-window pixel-level dependencies, the horizontal average pooling layer establishes horizontal relationships between windows, such as the horizontal relationship between Win1 and Win2: Win1 = H(Win2). Similarly, the vertical average pooling layer establishes horizontal relationships between windows, such as the vertical relationship between windows Win1 and Win3: Win1 = V(Win3); for Win4, Win1 = V(H(Win4)) + H(V(Win4)). For an M×M input (where M represents the number of windows), long-term dependencies between any two windows are established by connecting more intermediate windows.

[0093] After the generated global and local context information is summed, depthwise convolution (DWConv3×3), batch normalization (BatchNorm), and convolution (Conv 1×1) are performed.

[0094] In this embodiment, the image semantic segmentation model is lightweighted based on sparse coding and SmoothQuant quantization methods, including the following steps:

[0095] (1) The weights in the image semantic segmentation model are represented by sparse coding method, and the weights are decomposed into sparse weights and dense weights.

[0096] (2) Quantize the dense weights and reconstruct them using sparse weights.

[0097] Quantization is integrated throughout the entire transformer model; specifically, it's used for things like quantizing model weights and quantizing activation.

[0098] Quantizing model weights reduces weight precision: In standard deep learning models, weights are typically represented as floating-point numbers (32-bit or 16-bit), which requires significant storage and computational resources. Quantizing weights means converting these floating-point parameters to a lower-precision representation, such as 8-bit integers or even lower. This significantly reduces the model's storage requirements and computational complexity.

[0099] Specifically, first, the weights of the neural network are represented using a sparse coding method, decomposing the weights into sparse weights and dense weights. Then, the dense weights are quantized and reconstructed using the sparse weights.

[0100] Specifically, a smoothing factor s is used to smooth the input activations and the weights are scaled inversely.

[0101]

[0102] At the same time, a hyperparameter, transfer strength α, is introduced to control the difficulty of transferring from activation to weight:

[0103] s j =max(|X j |) α / max(|W j |) 1-α

[0104] j = 1, 2, ..., Ci, where j corresponds to the j-th input channel; the same quantizer is used for weights and activation selection α = 0.5, and the quantization difficulty is evenly distributed. The weights and activations of the corresponding channels share the same quantization difficulty.

[0105] In this embodiment, SmoothQuant performs scaled smoothing on the input activations of all linear layers in the Transformer and uses W8A8 to quantize the linear layers. The BMM operator (a special tensor multiplication operation that keeps the batch size of the first dimension unchanged and performs dot product quantization on the two subsequent dimensions) is enabled in the attention calculation.

[0106] The transformer model consists of multiple network layers, such as linear layers, and is a basic layer type in deep learning neural networks. It is one of the core components of a neural network, typically used to perform matrix multiplication of input data with weights and add biases to produce an output. In deep neural networks, multiple linear layers are usually stacked on top of each other, and these layers may interleave with each other through activation functions, forming part of the deep neural network.

[0107] In this embodiment, when quantizing the transformer model, the inputs and weights of computationally intensive operators such as BMM in the linear layer and attention layer are first quantized using INT8, while the activation of other lightweight elements, such as Softmax and LayerNorm, is kept at FP16.

[0108] Step S300 Unsupervised Adaptation: The encoder in the lightweight semantic image segmentation model is used as a shared feature extractor, and the decoder in the lightweight semantic image segmentation model is used as a generator. A discriminator and a generator are introduced as GAN adversaries. The shared feature extractor and generator are trained on sample images corresponding to the source domain image and the target domain image. The shared feature extractor is used to achieve domain transfer by performing feature alignment and domain adaptation on the target domain. The adversaries are used to achieve adversarial learning of feature distribution to achieve similarity of feature distribution between the source domain and the target domain.

[0109] In this embodiment, the shared feature extractor is used to extract semantic features from the input image and output the corresponding feature map by taking the sample images corresponding to the source domain image and the target domain image as input.

[0110] GAN adversarial processors consist of a generator and a discriminator. The generator maps the features corresponding to the target domain image to the source domain feature space and generates a disguised source domain feature representation.

[0111] The discriminator is used to distinguish between source domain and target domain features by taking the fake source domain features, real source domain features and real target domain features output by the generator as input, and outputting a binary classification result.

[0112] For GAN adversarial mechanisms, the details are as follows:

[0113] 1. Generator:

[0114] Input: Feature representation of the target domain.

[0115] Output: A camouflaged representation of the source domain features.

[0116] Function / Workflow: The generator's task is to map feature representations from the target domain to feature representations from the source domain. It uses a Transformer decoder architecture, potentially including multiple layers of decoder stacks for complex feature mapping. The generator is trained to generate masquerading source domain features, making them similar to the source and target domains.

[0117] 2. Discriminator:

[0118] Input: Feature representations of the source and target domains.

[0119] Output: The binary classification result of the discriminator (source domain or target domain).

[0120] Function / Workflow: The discriminator's task is to identify whether the input feature representation belongs to the source domain or the target domain. It accepts masquerading features generated by the generator, as well as genuine source and target domain features. The discriminator is trained to classify the source domain of the features as accurately as possible.

[0121] The workflow is as follows:

[0122] (1) The generator receives the feature representation of the target domain as input;

[0123] (2) The generator maps the target domain features to the source domain feature space through the structure of the Transformer decoder and generates a disguised source domain feature representation;

[0124] (3) The discriminator receives the fake source domain features, real source domain features and real target domain features from the generator as input;

[0125] (4) The discriminator attempts to classify these features into source domain or target domain and outputs a binary classification result.

[0126] The generator and discriminator are trained adversarially. The generator aims to generate spoofed features to deceive the discriminator, making it unable to accurately classify the source domain of the features, while the discriminator aims to classify the features as accurately as possible.

[0127] Through adversarial training, the generator is forced to generate masquerading features that are more similar to those in the source domain, thereby achieving adversarial learning of feature distributions. This makes the feature distributions of the source and target domains as similar as possible, which helps improve the performance of domain transfer. This adversarial training method is often used in domain adaptation tasks, especially when labeled data is lacking.

[0128] Example 2:

[0129] This invention provides a lightweight unsupervised adaptive image semantic segmentation system, comprising an image preprocessing module, a model building module, and an unsupervised adaptive module. The system in this embodiment can execute the method common to Embodiment 1.

[0130] The image preprocessing module is used to perform the following: acquire source domain image and target domain image as original images. The source domain image contains labeled data, and the target domain image contains unlabeled or limited labeled data. Perform image preprocessing on the original images to obtain preprocessed images, and use the preprocessed images as sample images.

[0131] As a specific implementation of the image preprocessing module, when preprocessing the original image, this module is used to resize the source domain image and the target domain image, and to perform data augmentation operations on the source domain image, including random flipping, rotation, scaling and cropping.

[0132] In this embodiment, the annotation data corresponding to the source domain image and the target domain image includes category labels and location information, and the category labels between the source domain image and the target domain image are consistent.

[0133] The labels include information about the object's category, such as cats, dogs, and cars in the image. These labels help the model identify different objects in the image.

[0134] The location information of an object is usually provided in the form of a bounding box. This helps the model not only to recognize the object, but also to locate its position.

[0135] Limited labeled data typically refers to datasets within the target domain that contain relatively little or no labeled data. The main difference between limited labeled data and labeled data in the source domain is the smaller quantity or lower quality. This may be because obtaining high-quality labeled data in the target domain can be more difficult, expensive, or time-consuming, or the target domain may be a new field with insufficient labeled data available. Therefore, limited labeled data may not provide the same training signals as labeled data in the source domain, which can negatively impact the performance of models trained in the target domain.

[0136] The model building module is used to perform the following: build an image semantic segmentation model based on the Transformer model, and introduce SmoothQuant into the Transformer model. Based on sparse coding and SmoothQuant quantization methods, the image semantic segmentation model is lightweighted to obtain a lightweight image semantic segmentation model. The image semantic segmentation model includes an encoder and a decoder, which are used to extract features from the input image and output a feature map.

[0137] In this embodiment, both the encoder and decoder in the Transformer model include a global-local Transformer network structure, which includes an input module, a global-local Transformer block, and an output module.

[0138] The input module maps the input image to the dimensional space of the network structure and adds positional encoding to take into account positional information in the sequence.

[0139] The global-local Transformer block includes a local branch, a global branch, and a feature fusion section. The local branch extracts local context information based on convolutional operations or windowed attention mechanisms, outputting a local feature map. The global branch extracts global uplink information based on a linear multi-head attention mechanism, outputting a global feature map. The output of the global branch is equipped with a trainable scale factor. For the extracted global context information, the global context information at different locations is fused by calculating a weighted sum to obtain a feature map. A cross-shaped window context interaction module is used to capture cross-window relationships, fusing the two feature maps generated by the horizontal average pooling layer and the vertical average pooling layer to obtain global features. The feature fusion section fuses the local and global features and performs convolutional operations and batch normalization operations on the fused image to obtain the fused feature map.

[0140] In this embodiment, the global-local Transformer block is used to perform the following:

[0141] (1) Global attention: Learn the correlation between global features through self-attention mechanism to capture the global dependencies at different positions in the sequence;

[0142] (2) Local attention: By using local convolution or windowing attention mechanisms, we can focus on the features of certain local regions in the sequence in order to capture more fine-grained local information;

[0143] (3) Global-local fusion: By combining global features and local features, a comprehensive feature representation is created.

[0144] The output module is used to map the output features of the GLTB block back to the target dimension of the model. It typically includes one or more linear layers and activation functions to generate the final prediction or feature representation.

[0145] As a specific implementation of this global-local Transformer network structure, the following explanation is provided:

[0146] 1. Input Module:

[0147] Input: Raw sequence data or feature representation;

[0148] Function: It takes raw sequence data or feature representation as input, prepares the data for processing by GLTB blocks, which typically includes an embedding layer to map the input data to the model’s dimensional space and adds positional encoding to take positional information in the sequence into account.

[0149] 2. Global-Local Transformer Block (GLTB):

[0150] Input: Feature representations obtained from the input module;

[0151] Function: GLTB is a core component of the model, performing the following main functions:

[0152] (1) Global attention: Through the self-attention mechanism (multi-head attention in Transformer), GLTB can learn the correlation between global features and capture the global dependencies at different positions in the sequence.

[0153] (2) Local attention: GLTB also includes a local attention mechanism, which uses local convolution or windowed attention mechanism to focus on the features of certain local regions in the sequence in order to capture more fine-grained local information.

[0154] (3) Global-Local Fusion: GLTB creates a comprehensive feature representation by combining global and local features, aiming to balance global and local information and thus improve model performance.

[0155] 3. Output Module:

[0156] Input: Output feature representation of the GLTB block;

[0157] Function: The output module is responsible for mapping the output features of the GLTB block back to the target dimension of the model. It typically includes one or more linear layers and activation functions to generate the final prediction or feature representation.

[0158] The overall workflow of this global-local Transformer network structure is as follows:

[0159] (1) The input module accepts raw sequence data or feature representations and prepares the input data through embedding and positional encoding;

[0160] (2) The GLTB block receives the input feature representation and performs global and local attention operations simultaneously to capture global and local information and generate a comprehensive feature representation;

[0161] (3) The output module receives the output feature representation of the GLTB block, maps it to the target dimension of the model, and may perform the final prediction or feature representation.

[0162] As a specific implementation of the GLTB block, the GLTB block consists of global-local attention, MLP (Multilayer Perception), two batchnorm layers, and two summation operations.

[0163] The local branch (which is the local part of the Global-LocalTransformerBlock, GLTB) uses two parallel convolutional layers (Conv3×3, Conv1×1) to extract local context information. Batch normalization is applied after the convolutional layers, and finally, a summation operation is performed.

[0164] The global branch (here, "global" refers to the global branch in GLTB) uses linear multi-head self-attention. (Linear multi-head self-attention is an attention mechanism typically used to process sequential data, such as text in natural language processing. In self-attention, each input element (e.g., a word or a position) interacts with other elements and is assigned a weight to determine which elements should receive more attention when modeling the context. Linear multi-head self-attention is an extended form of self-attention that uses multiple attention heads (independent sets of weights) to capture different contextual relationships.)

[0165] In this embodiment, the GLTB block deploys a trainable scaling factor on its output: during self-attention computation, attention scores (weights) are typically multiplied by the input values ​​to calculate a weighted combination of each element. The trainable scaling factor is a learnable parameter used to scale these attention scores. This helps control the range and stability of the attention distribution. By learning an appropriate scaling factor, the model can better adapt to different input data and tasks.

[0166] This embodiment of the GLTB block implements stable context aggregation: context aggregation refers to combining information from different locations or elements to better understand the context of the entire sequence or image. In self-attention, context aggregation is achieved by calculating a weighted sum, where the weight of each element is determined by the attention mechanism. By using a trainable scaling factor, the attention mechanism can be made more stable, helping the model to better capture the relationships between input data and aggregate contextual information.

[0167] The process for implementing the aggregation context is as follows: First, expand the number of channels in the input feature map; then, apply a window partitioning operation to divide the feature sequence into Q, K, and V. The normalization function is softmax, and the result generated by self-attention is:

[0168]

[0169] Where Vi,j, represents the j-th feature in the i-th row of the resulting matrix, a trainable scaling factor is deployed on the output of the linear multi-head self-attention to achieve stable context aggregation.

[0170] Specifically, the cross-shaped window context interaction module is used to capture cross-window relationships and fuse the two feature maps generated by the horizontal average pooling layer and the vertical average pooling layer to obtain the global context.

[0171] Horizontal Average Pooling Layer: Horizontal average pooling is a pooling operation typically applied to feature maps (or images). In horizontal average pooling, the average pooling operation is performed on each row (horizontally) of the feature map, merging the pixel values ​​of each row into a single value. This means that each row is compressed into a single number that reflects the average features of that row. This helps capture the horizontal global information of the image.

[0172] Based on cross-window pixel-level dependencies, the horizontal average pooling layer establishes horizontal relationships between windows, such as the horizontal relationship between Win1 and Win2: Win1 = H(Win2). Similarly, the vertical average pooling layer establishes horizontal relationships between windows, such as the vertical relationship between windows Win1 and Win3: Win1 = V(Win3); for Win4, Win1 = V(H(Win4)) + H(V(Win4)). For an M×M input (where M represents the number of windows), long-term dependencies between any two windows are established by connecting more intermediate windows.

[0173] After the generated global and local context information is summed, depthwise convolution (DWConv3×3), batch normalization (BatchNorm), and convolution (Conv 1×1) are performed.

[0174] In this embodiment, the image semantic segmentation model is lightweighted based on sparse coding and SmoothQuant quantization methods, including the following steps:

[0175] (1) The weights in the image semantic segmentation model are represented by sparse coding method, and the weights are decomposed into sparse weights and dense weights.

[0176] (2) Quantize the dense weights and reconstruct them using sparse weights.

[0177] Quantization is integrated throughout the entire transformer model; specifically, it's used for things like quantizing model weights and quantizing activation.

[0178] Quantizing model weights reduces weight precision: In standard deep learning models, weights are typically represented as floating-point numbers (32-bit or 16-bit), which requires significant storage and computational resources. Quantizing weights means converting these floating-point parameters to a lower-precision representation, such as 8-bit integers or even lower. This significantly reduces the model's storage requirements and computational complexity.

[0179] Specifically, first, the weights of the neural network are represented using a sparse coding method, decomposing the weights into sparse weights and dense weights. Then, the dense weights are quantized and reconstructed using the sparse weights.

[0180] Specifically, a smoothing factor s is used to smooth the input activations and the weights are scaled inversely.

[0181]

[0182] At the same time, a hyperparameter, transfer strength α, is introduced to control the difficulty of transferring from activation to weight:

[0183] s j =max(|X j |) α / max(|W j |) 1-α

[0184] j = 1, 2, ..., Ci, where j corresponds to the j-th input channel; the same quantizer is used for weights and activation selection α = 0.5, and the quantization difficulty is evenly distributed. The weights and activations of the corresponding channels share the same quantization difficulty.

[0185] In this embodiment, SmoothQuant performs scaled smoothing on the input activations of all linear layers in the Transformer and uses W8A8 to quantize the linear layers. The BMM operator (a special tensor multiplication operation that keeps the batch size of the first dimension unchanged and performs dot product quantization on the two subsequent dimensions) is enabled in the attention calculation.

[0186] The unsupervised adaptive module is used to perform the following: the encoder in the lightweight semantic image segmentation model is used as a shared feature extractor, the decoder in the lightweight semantic image segmentation model is used as a generator, and a discriminator and generator are introduced as GAN adversaries. The shared feature extractor and generator are trained on sample images corresponding to the source domain image and the target domain image. The shared feature extractor is used to achieve domain transfer by performing feature alignment and domain adaptation on the target domain. The adversaries are used to achieve adversarial learning of feature distribution to achieve similarity of feature distribution between the source domain and the target domain.

[0187] In this embodiment, the shared feature extractor is used to extract semantic features from the input image and output the corresponding feature map by taking the sample images corresponding to the source domain image and the target domain image as input.

[0188] GAN adversarial processors consist of a generator and a discriminator. The generator maps the features corresponding to the target domain image to the source domain feature space and generates a disguised source domain feature representation.

[0189] The discriminator is used to distinguish between source domain and target domain features by taking the fake source domain features, real source domain features and real target domain features output by the generator as input, and outputting a binary classification result.

[0190] For GAN adversarial mechanisms, the details are as follows:

[0191] 1. Generator:

[0192] Input: Feature representation of the target domain.

[0193] Output: A camouflaged representation of the source domain features.

[0194] Function / Workflow: The generator's task is to map feature representations from the target domain to feature representations from the source domain. It uses a Transformer decoder architecture, potentially including multiple layers of decoder stacks for complex feature mapping. The generator is trained to generate masquerading source domain features, making them similar to the source and target domains.

[0195] 2. Discriminator:

[0196] Input: Feature representations of the source and target domains.

[0197] Output: The binary classification result of the discriminator (source domain or target domain).

[0198] Function / Workflow: The discriminator's task is to identify whether the input feature representation belongs to the source domain or the target domain. It accepts masquerading features generated by the generator, as well as genuine source and target domain features. The discriminator is trained to classify the source domain of the features as accurately as possible.

[0199] The workflow is as follows:

[0200] (1) The generator receives the feature representation of the target domain as input;

[0201] (2) The generator maps the target domain features to the source domain feature space through the structure of the Transformer decoder and generates a disguised source domain feature representation;

[0202] (3) The discriminator receives the fake source domain features, real source domain features and real target domain features from the generator as input;

[0203] (4) The discriminator attempts to classify these features into source domain or target domain and outputs a binary classification result.

[0204] The generator and discriminator are trained adversarially. The generator aims to generate spoofed features to deceive the discriminator, making it unable to accurately classify the source domain of the features, while the discriminator aims to classify the features as accurately as possible.

[0205] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the code review methods in the different embodiments. These embodiments are also within the protection scope of the present invention.

Claims

1. A lightweight unsupervised adaptive image semantic segmentation method, characterized in that, Includes the following steps: Image preprocessing: Obtain source and target domain images as raw images. The source domain image contains labeled data, and the target domain image contains unlabeled or limited labeled data. Perform image preprocessing on the raw images to obtain preprocessed images, and use the preprocessed images as sample images. Model Construction: An image semantic segmentation model is constructed based on the Transformer model, and SmoothQuant is introduced into the Transformer model. The image semantic segmentation model is lightweighted based on sparse coding and SmoothQuant quantization methods to obtain a lightweight image semantic segmentation model. The image semantic segmentation model includes an encoder and a decoder, which are used to extract features from the input image and output feature maps. Unsupervised Adaptation: The encoder in the lightweight semantic image segmentation model is used as a shared feature extractor, and the decoder in the lightweight semantic image segmentation model is used as a generator. A discriminator and a generator are introduced as GAN adversaries. The shared feature extractor and generator are trained on sample images corresponding to the source domain image and the target domain image. The shared feature extractor is used to achieve domain transfer by performing feature alignment and domain adaptation on the target domain. The adversaries are used to achieve adversarial learning of feature distribution to achieve similarity of feature distribution between the source domain and the target domain.

2. The lightweight unsupervised adaptive image semantic segmentation method according to claim 1, characterized in that, When preprocessing the original image, the source and target images are resized, and data augmentation operations are performed on the source image, including random flipping, rotation, scaling, and cropping. The annotation data corresponding to the source and target domain images includes category labels and location information, and the category labels between the source and target domain images are consistent.

3. The lightweight unsupervised adaptive image semantic segmentation method according to claim 1, characterized in that, The Transformer model includes a global-local Transformer network structure for both the encoder and decoder. The global-local Transformer network structure includes an input module, a global-local Transformer block, and an output module. The input module is used to map the input image to the dimensional space of the network structure and add positional encoding to take into account the positional information in the sequence. The global-local Transformer block includes a local branch, a global branch, and a feature fusion part. The local branch is used to extract local context information based on convolutional operations or windowed attention mechanisms, and outputs a local feature map. The global branch is used to extract global uplink information based on a linear multi-head attention mechanism, and outputs a global feature map. The output of the global branch is deployed with a trainable scale factor. For the extracted global context information, the global context information at different locations is fused by calculating a weighted sum to obtain a feature map. A cross-shaped window context interaction module is used to capture cross-window relationships, and the two feature maps generated by the horizontal average pooling layer and the vertical average pooling layer are fused to obtain global features. The feature fusion part is used to fuse local features and global features, and performs convolutional operations and batch normalization operations on the fused image to obtain a fused feature map. The output module includes one or more linear layers and activation functions to map the feature map output by the GLTB block back to the target dimension of the model to generate the final prediction or feature representation.

4. The lightweight unsupervised adaptive image semantic segmentation method according to claim 1, characterized in that, The image semantic segmentation model is lightweighted based on sparse coding and SmoothQuant quantization, including the following steps: The weights in the image semantic segmentation model are represented by a sparse coding method, and the weights are decomposed into sparse weights and dense weights. The dense weights are quantized and then reconstructed using sparse weights.

5. The lightweight unsupervised adaptive image semantic segmentation method according to claim 1, characterized in that, The shared feature extractor is used to extract semantic features from the input image and output the corresponding feature map by taking the sample images corresponding to the source domain image and the target domain image as input. The GAN adversarial generator includes a generator and a discriminator. The generator is used to map the features corresponding to the target domain image to the source domain feature space and generate a disguised source domain feature representation. The discriminator is used to distinguish between source domain and target domain features by taking the fake source domain features, real source domain features and real target domain features output by the generator as input, and outputting a binary classification result.

6. A lightweight unsupervised adaptive image semantic segmentation system, characterized in that, It includes an image preprocessing module, a model building module, and an unsupervised adaptive module. The image preprocessing module is used to perform the following: acquire source domain image and target domain image as original images. The source domain image contains labeled data, and the target domain image contains unlabeled or limited labeled data. The original images are preprocessed to obtain preprocessed images, and the preprocessed images are used as sample images. The model building module is used to perform the following: build an image semantic segmentation model based on the Transformer model, introduce SmoothQuant into the Transformer model, and perform model lightweighting processing on the image semantic segmentation model based on sparse coding and SmoothQuant quantization method to obtain a lightweight image semantic segmentation model. The image semantic segmentation model includes an encoder and a decoder, which are used to extract features from the input image and output feature maps. The unsupervised adaptive module is used to perform the following: the encoder in the lightweight semantic image segmentation model is used as a shared feature extractor, the decoder in the lightweight semantic image segmentation model is used as a generator, and a discriminator and generator are introduced as GAN adversaries. The shared feature extractor and generator are trained on sample images corresponding to the source domain image and the target domain image. The shared feature extractor is used to achieve domain transfer by performing feature alignment and domain adaptation on the target domain. The adversaries are used to achieve adversarial learning of feature distribution to achieve similarity of feature distribution between the source domain and the target domain.

7. The lightweight unsupervised adaptive image semantic segmentation system according to claim 6, characterized in that, When preprocessing the original image, the source and target images are resized, and data augmentation operations are performed on the source image, including random flipping, rotation, scaling, and cropping. The annotation data corresponding to the source and target domain images includes category labels and location information, and the category labels between the source and target domain images are consistent.

8. The lightweight unsupervised adaptive image semantic segmentation system according to claim 6, characterized in that, The Transformer model includes a global-local Transformer network structure for both the encoder and decoder. The global-local Transformer network structure includes an input module, a global-local Transformer block, and an output module. The input module is used to map the input image to the dimensional space of the network structure and add positional encoding to take into account the positional information in the sequence. The global-local Transformer block includes a local branch, a global branch, and a feature fusion part. The local branch is used to extract local context information based on convolutional operations or windowed attention mechanisms, and outputs a local feature map. The global branch is used to extract global uplink information based on a linear multi-head attention mechanism, and outputs a global feature map. The output of the global branch is deployed with a trainable scale factor. For the extracted global context information, the global context information at different locations is fused by calculating a weighted sum to obtain a feature map. A cross-shaped window context interaction module is used to capture cross-window relationships, and the two feature maps generated by the horizontal average pooling layer and the vertical average pooling layer are fused to obtain global features. The feature fusion part is used to fuse local features and global features, and performs convolutional operations and batch normalization operations on the fused image to obtain a fused feature map. The output module includes one or more linear layers and activation functions to map the feature map output by the GLTB block back to the target dimension of the model to generate the final prediction or feature representation.

9. The lightweight unsupervised adaptive image semantic segmentation system according to claim 6, characterized in that, The image semantic segmentation model is lightweighted based on sparse coding and SmoothQuant quantization, including the following steps: The weights in the image semantic segmentation model are represented by a sparse coding method, and the weights are decomposed into sparse weights and dense weights. The dense weights are quantized and then reconstructed using sparse weights.

10. The lightweight unsupervised adaptive image semantic segmentation system according to claim 6, characterized in that, The shared feature extractor is used to extract semantic features from the input image and output the corresponding feature map by taking the sample images corresponding to the source domain image and the target domain image as input. The GAN adversarial generator includes a generator and a discriminator. The generator is used to map the features corresponding to the target domain image to the source domain feature space and generate a disguised source domain feature representation. The discriminator is used to distinguish between source domain and target domain features by taking the fake source domain features, real source domain features and real target domain features output by the generator as input, and outputting a binary classification result.

Images