A semi-supervised semantic segmentation system, method, device and storage medium

Abstract

The application provides a kind of semi-supervised semantic segmentation system, method, equipment and storage medium, belong to the image segmentation technical field based on computer vision;The application proposes a frequency auxiliary consistency disturbance method FreqMatch, constructs joint disturbance space IFFAP by image feature disturbance flow IFPS and frequency domain auxiliary disturbance flow FAPS, dynamically balances disturbance intensity and direction to optimize quality-efficiency trade-off;At the same time, introduce multi-feature channel class memory mechanism (FCCM), by integrating the multi-channel information of original features, enhanced features and disturbance features, alleviate the class imbalance problem and suppress pseudo-label noise. This method significantly improves the segmentation accuracy and model robustness in the limited labeling scenario, and achieves the current optimal performance on Pascal VOC and COCO datasets.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology based on computer vision, and particularly relates to a semi-supervised semantic segmentation system, method, device and storage medium. Background Technology

[0002] Semi-supervised semantic segmentation technology aims to improve model performance by combining a small amount of labeled data with a large amount of unlabeled data, and it has significant application value in fields such as medical image analysis and autonomous driving. However, existing methods generally face two core challenges: First, consistency regularization-based methods (such as Mean Teacher and FixMatch) typically employ random perturbation strategies, lacking a global balancing mechanism for the intensity and direction of perturbations, making the model susceptible to false label noise and generating confirmation bias; Second, the prevalent long-tail distribution of categories in datasets results in insufficient learning of semantic information from the tail categories by the model, further exacerbating the uncertainty of prediction results.

[0003] Traditional fully supervised methods (such as FCN and DeepLab) rely on large amounts of pixel-level labeled data, resulting in high annotation costs and difficulty in scaling to complex scenes. Semi-supervised methods alleviate the annotation pressure through pseudo-label generation and consistency training, but mainstream methods have significant limitations: perturbation strategies based on random image enhancement (such as rotation and cropping) can improve the robustness of local features, but they neglect the optimization of perturbation of global features in the frequency domain, making it difficult for the model to capture image texture details and boundary information. For example, FixMatch generates pseudo-labels through weak enhancement and imposes consistency constraints on strongly enhanced data, but its fixed threshold setting cannot adapt to the dynamic needs of multi-class scenes; SemiCVT and other studies have attempted to enhance global feature perception using Fourier transform, but they have not designed an effective frequency domain masking mechanism to suppress aliasing effects, and high-frequency detail information is easily lost during sampling.

[0004] In recent years, Transformer-based methods (such as AllSpark) have improved global modeling capabilities by expanding the receptive field. However, their high computational complexity and over-reliance on low-frequency background features weaken the representation of high-frequency textures. Furthermore, existing methods lack semantic compensation mechanisms for long-tailed class distributions. The scarcity of samples in tail classes leads to low-quality pseudo-labels, further impacting model generalization performance. For example, U²PL utilizes unreliable predictions through contrastive learning, but it lacks a multi-source feature storage and callback mechanism, making it difficult to dynamically balance class semantic information during training. Summary of the Invention

[0005] To address the above issues, the following problems are to be solved: (1) A collaborative optimization strategy that takes into account both image features and frequency domain perturbation needs to be designed to balance the perturbation intensity and enhance the expression of detailed features; (2) A multi-source feature storage and semantic callback mechanism needs to be constructed to alleviate the pseudo-label bias caused by the long-tail distribution of categories; (3) A dynamic scoring strategy needs to be developed to suppress the accumulation of noise features during training.

[0006] The first aspect of this invention provides a semi-supervised semantic segmentation system, including a Transformer encoder feature extraction module, an image feature and frequency-assisted consistency perturbation flow network (IFFAP), a multi-feature (FCCM) channel category memory mechanism network, and a decoder module;

[0007] The Transformer encoder feature extraction module takes the original images after weak enhancement and strong enhancement as inputs and feeds them into the structurally shared backbone network for feature extraction. The backbone network is based on a multi-scale structure and outputs four sets of semantic feature maps at different levels in sequence during forward propagation, denoted as c1, c2, c3, and c4, which represent the layer-by-layer semantic expression from shallow to deep layers.

[0008] The image feature and frequency-assisted consistency perturbation flow network includes a frequency-assisted perturbation flow (FAPS) module and an image feature perturbation flow (IFPS) module. The FAPS module sequentially performs Fourier transform (FFT) on the input c1, c2, c3, and c4 feature maps to obtain a frequency domain representation, retaining low-frequency components and suppressing high-frequency noise. Then, it performs adaptive weighted fusion of low-frequency and high-frequency components and finally reconstructs the spatial domain perturbation features through inverse Fourier transform (IFFT). The IFPS module applies random deactivation perturbation to the feature map output by the backbone network and generates perturbation branch features through feature splicing and block operations.

[0009] The multi-feature channel category memory mechanism network takes the deepest feature map c4 as input and fuses multi-source features from weak and strong enhancement paths, FAPS frequency-assisted perturbation path, and IFPS image feature perturbation path. The input features are converted into block embedding sequences and separated into labeled embeddings and unlabeled embeddings. After processing by the cross attention mechanism, the output is an enhanced feature map with fused embedding features. Upsampling is used to keep the shape of the output consistent.

[0010] The decoder module adopts the SegFormerHead structure, taking the multi-scale feature map processed by the self-independent network and the multi-feature channel category memory mechanism network as input, and uniformly mapping the features of each level to the embedding_dim dimension. After feature fusion and classification prediction, the semantic segmentation result is output.

[0011] Preferably, the input 513×513 original image is subjected to weak enhancement and strong enhancement processing respectively; the weak enhancement operation includes normalization, random scaling, random cropping, random horizontal flipping, color dithering, random grayscale conversion and Gaussian blur processing of the image; the strong enhancement operation further increases the probability or amplitude of color dithering, grayscale conversion and blurring on the basis of weak enhancement to enhance the diversity of the image; the enhanced images are used as inputs for labeled and unlabeled branches respectively for subsequent feature extraction and semi-supervised training.

[0012] Preferably, the Transformer encoder feature extraction module takes the enhanced image as input, and the input format is a tensor with shape [B, 3, 513, 513], where B is the batch size and 3 is the number of channels; the backbone network adopts a multi-scale hybrid visual Transformer structure, and goes through four stages of feature extraction in sequence. Each stage outputs a set of feature maps, which are denoted as c1, c2, c3, and c4, respectively. The output formats of each feature map are [B, 64, H1, W1], [B, 128, H2, W2], [B, 320, H3, W3], and [B, 512, H4, W4].

[0013] Preferably, the frequency-assisted perturbation flow (FAPS) module takes a feature map with shape [B, C, H, W] as input, first performs a Fourier transform and applies a low-pass mask with a frequency threshold of 0.25, retains the low-frequency region in the center and filters out the high-frequency components at the edges, then separates the low-frequency and high-frequency representations of the Fourier transform frequency domain, calculates the channel attention weights of the low-frequency mask and the high-frequency mask respectively, reassembles them after adaptive weighting, and outputs the spatial domain perturbation features through inverse Fourier transform;

[0014] The Image Feature Perturbation Stream (IFPS) module takes the multi-scale feature maps c1, c2, c3, and c4 output by the backbone network as input, applies a Dropout2d random deactivation operation with a probability of 0.5 to each feature map, generates a perturbation feature version, splices the original feature and the perturbation feature to form a dual-branch input, and after processing by the shared decoder, separates the prediction result of the perturbation branch, which is used to perform consistency constraint calculation with the original branch.

[0015] Preferably, the multi-feature channel category memory mechanism network includes a channel semantic grouping unit, a semantic memory queue update unit, and a cross-attention calculation unit;

[0016] The channel semantic grouping unit performs bilinear interpolation on the input pseudo-probability map to a size that matches the number of feature patches, calculates the similarity matrix between the normalized pseudo-probability map and the normalized unlabeled embedding emb_u through matrix multiplication, and uses the argmax operation to determine the semantic category label corresponding to each feature channel.

[0017] The semantic memory queue is updated by a key-value pair column cache of shape [num_class, SMem_size, patch_num], and is updated by category: for each category, the unlabeled embedding features of the corresponding channel are extracted, and the queue pointer is updated in a first-in-first-out manner. When the queue is full, the queue is overwritten in a loop to ensure the continuous accumulation of semantic information of each category.

[0018] The cross-attention calculation unit processes labeled embeddings emb_l and unlabeled embeddings emb_u respectively: for unlabeled embeddings, it performs self-attention calculation to generate query, key, and value, and obtains self-attention output through a multi-head attention mechanism; for labeled embeddings, it extracts historical semantic memory from the key-value pair column as key and value when semantic memory is activated, otherwise it uses the current unlabeled embedding, calculates the cross-attention weight to obtain the output out_l2u, and finally concatenates the output features of the two branches to form a complete enhanced embedding.

[0019] Preferably, the decoder module adopts a SegFormerHead structure, taking the multi-scale feature maps processed by the self-intercepting network and FCCM module as input, including feature maps of four levels: c1, c2, c3, and c4, with input formats of [B, 64, H1, W1], [B, 128, H2, W2], [B, 320, H3, W3], and [B, 512, H4, W4], respectively. The decoder module includes multi-level linear embedding units, feature fusion units, and classification prediction units. After MLP linear embedding, the features of each level are uniformly mapped to the embedding_dim dimension, upsampled to the highest resolution through bilinear interpolation, and then concatenated by torch.cat channels and fused by ConvModule features. The final output is a semantic segmentation prediction map with shape [B, num_class, H, W], which is restored to the original input image size through bilinear interpolation.

[0020] Preferably, the multi-level linear embedding unit includes four MLP modules: linear_c1, linear_c2, linear_c3, and linear_c4. Each MLP module performs a flatten(2).transpose(1,2) operation on the input feature map to flatten the spatial dimensions and transpose them to the [B, H×W, C] format. Then, the channel dimensions are uniformly mapped to embedding_dim=768 dimensions through the nn.Linear linear projection layer.

[0021] The feature fusion unit performs the permute(0,2,1).reshape operation on the processed four-level features _c1, c2, _c3, and _c4 to restore them to the spatial feature map format. Then, it upsamples _c2, _c3, and _c4 to the same spatial resolution as _c1 using F.interpolate bilinear interpolation. Finally, it uses torch.cat([_c4, _c3, _c2, _c1], dim=1) to concatenate them according to the channel dimension to form a fused feature with 4×embedding_dim channels.

[0022] The classification prediction unit reduces the fused features from 4×embedding_dim dimension to embedding_dim dimension through the linear_fuse convolution module. After being regularized by Dropout2d(0.1), the linear_pred convolution layer maps the features to the number of num_class categories, and outputs the final semantic segmentation logits.

[0023] Preferably, the total loss function is implemented by a weighted combination of supervised cross-entropy loss and unsupervised consistency loss, specifically including the following implementation details:

[0024] Supervised Cross-Entropy Loss (CE Loss): On labeled data, the cross-entropy loss between the segmentation prediction output by the model and the true label is calculated to ensure that the model learns accurate semantic segmentation capabilities.

[0025] Unsupervised Consistency Loss: Based on the outputs of the three feedforward streams—Frequency-Assisted Perturbation Stream (FAPS), Image Feature Perturbation Stream (IFPS), and Basic Image Stream—the consistency loss between the prediction results of each stream is calculated (e.g., pseudo-label consistency loss based on cross-entropy). The stability of the model's output under perturbation is measured by comparing the segmentation prediction probability maps corresponding to different perturbation streams.

[0026] Dynamic weighting strategy: For the sub-losses corresponding to different perturbation flows (FAPS flow, IFPS flow and basic flow) in the unsupervised consistency loss, dynamic weight coefficients are used for weighted combination. The weights are automatically adjusted according to the importance of the perturbation flow in the training process, ensuring that the model focuses on optimizing more effective perturbation paths and improving the stability and effect of unsupervised training.

[0027] The final total loss function is defined as:

[0028]

[0029] in: Supervised cross-entropy loss; The unsupervised consistency loss corresponds to the iiith perturbation flow; is the dynamic weighting coefficient for the iii-th perturbation flow; The overall weight hyperparameter for unsupervised loss.

[0030] The model training adopts a poly learning rate scheduling strategy, with the learning rate ratio of encoder to decoder being 1:5, and the training cycle is set to 10-80 rounds depending on the size of the dataset.

[0031] A second aspect of the present invention provides a semi-supervised semantic segmentation method, using the semi-supervised semantic segmentation system as described in the first aspect, and comprising the following processes:

[0032] S1, Data Preprocessing: Strong enhancement processing is performed on labeled image data, and weak enhancement processing is performed on unlabeled image data to generate training samples;

[0033] S2, Pseudo-label generation: The weakly enhanced unlabeled image is input into the semi-supervised semantic segmentation system, and pseudo-labels and pseudo-probability maps are generated by the teacher model;

[0034] S3, Multi-path Feature Processing: Labeled and unlabeled data are processed by the IFFAP module to perform frequency-assisted perturbation and image feature perturbation, respectively, to generate multi-source enhanced features;

[0035] S4, Feature Fusion and Memory Update: The FCCM module performs channel semantic grouping on multi-path features, updates the semantic memory queue, and performs cross-attention calculation.

[0036] S5, Decoding and Prediction: The fused features are used by the decoder module to generate pixel-level semantic segmentation results;

[0037] S6, Loss Calculation and Model Update: Calculate supervised loss and unsupervised consistency loss, update model parameters, and complete one round of training iteration.

[0038] A third aspect of the present invention provides a semi-supervised semantic segmentation device, the device comprising at least one processor and at least one memory coupled together; the memory storing a computer-executable program of the semi-supervised semantic segmentation system as described in the first aspect; when the processor executes the computer-executable program stored in the memory, the processor performs a semi-supervised semantic segmentation method.

[0039] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer-executable program of a semi-supervised semantic segmentation system as described in the first aspect, wherein when the computer-executable program is executed by a processor, the processor performs a semi-supervised semantic segmentation method.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] This invention significantly improves the performance and robustness of semi-supervised semantic segmentation models by fusing image features with Frequency-Assisted Consistency Perturbation Flow (IFFAP) and a multi-feature channel and category memory mechanism (FCCM). Specifically, this invention mainly achieves the following technical advantages:

[0042] 1. Perturbation strategy optimization: By combining the frequency-domain assisted perturbation flow (FAP) and the image feature perturbation flow (IFP), an efficient balance between global and local features is achieved in the frequency and spatial domains, suppressing the confirmation bias caused by traditional random perturbations and enhancing the model's adaptability to complex scenarios.

[0043] 2. Category Balance Enhancement: The FCCM mechanism effectively alleviates the problem of missing category information under long-tail distribution by dynamically storing and recalling multi-source features and combining channel attention weighting strategy, which significantly improves the segmentation accuracy of small targets and edge regions.

[0044] 3. Improved computational efficiency: The IFFAP module processes features in parallel in the frequency and spatial domains, avoiding the introduction of complex network structures. It also achieves state-of-the-art (SOTA) performance on both the Pascal VOC and COCO datasets, with significantly better model training and inference efficiency than existing methods.

[0045] 4. Wide applicability: The model does not require multi-stage training or additional annotation costs and can be directly applied to semi-supervised segmentation tasks in highly complex scenarios such as remote sensing images and medical images, which has high practical value and universality.

[0046] The combination of the above-mentioned technical advantages enables the present invention to achieve a significant improvement in segmentation accuracy while maintaining efficient utilization of computing resources, providing a new solution for the field of semi-supervised semantic segmentation. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the following description is only one embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a block diagram of the overall structure of the image segmentation system of the present invention.

[0049] Figure 2 This is a schematic diagram of the structure of the IFFAP module of the present invention.

[0050] Figure 3The figure shows the experimental results of this invention on Pascal Original.

[0051] Figure 4 The figure shows the experimental results of this invention on Pascal augmentation.

[0052] Figure 5 This is a visualization of the invention on Pascal Originals.

[0053] Figure 6 This is a simplified structural diagram of the semi-supervised semantic segmentation device of the present invention. Detailed Implementation

[0054] This invention primarily provides a semi-supervised semantic segmentation system based on frequency-assisted and image feature consistency, along with its corresponding segmentation method. It constructs a FreqMatch system model comprising a Transformer encoder feature extraction module, an image feature and frequency-assisted consistency perturbation flow (IFFAP), a multi-feature channel class memory mechanism (FCCM), and a decoder module. Its overall structure is as follows: Figure 1 As shown.

[0055] The Transformer encoder feature extraction module takes a 513×513 pixel original image as input, processes it through weak and strong enhancement operations, and then feeds it into a structurally shared backbone network for feature extraction. This backbone network is based on a multi-scale structure and sequentially outputs four sets of semantic feature maps at different levels during forward propagation, denoted as c1, c2, c3, and c4, representing layer-by-layer semantic representation from shallow to deep layers, used for subsequent semantic segmentation decoding and consistency loss calculation.

[0056] Image features and frequency-aided consistent perturbation flow (IFFAP) such as Figure 2 As shown, the network includes a Frequency-Assisted Perturbation Stream (FAPS) module and an Image Feature Perturbation Stream (IFPS) module. The FAPS module takes the input multi-scale feature maps as input and performs FFT transformations on feature maps c1, c2, c3, and c4 sequentially to obtain a frequency domain representation, preserving low-frequency components and suppressing high-frequency noise. It then performs adaptive weighted fusion of low-frequency and high-frequency components and finally reconstructs the spatial domain perturbation features using IFFT (Inverse Fourier Transform). The IFPS module applies Dropout2d random deactivation perturbation to the feature maps output by the backbone network. It generates perturbation branches through feature concatenation and block operations, which, along with the original feature branches, participate in subsequent decoding and consistency loss calculations. Both modules ultimately ensure input and output dimensionality consistency through sampling operations.

[0057] The Multi-Feature Channel Class Memory (FCCM) module includes a channel semantic grouping unit, a semantic memory queue update unit, and a cross-attention computation unit. This module takes the deepest feature map c4 of shape [B, C, H, W] as input and can fuse multi-source features from the enhancement path, the FAPS frequency-assisted perturbation path, the IFPS image feature perturbation path, and others. The input features are converted into a patch embedding sequence of shape [2B, N, C], which is separated into labeled embedding emb_l and unlabeled embedding emb_u, each with shape [B, N, C], through a split operation. After processing by the CrossAttnMem cross-attention memory module, the fused embedding features are output and reconstructed into an enhancement feature map of shape [B, C, H, W] for subsequent decoder processing.

[0058] The invention will be further described below with reference to specific embodiments.

[0059] Implementation 1:

[0060] This example demonstrates experiments conducted on the Pascal VOC 2012 dataset:

[0061] The Pascal VOC 2012 dataset consists of approximately 4000 samples, divided into a training set (1464 images), a validation set (1449 images), and a test set (1456 images), covering 21 categories (including background). To augment the training data, 9118 coarsely labeled images from the SBD dataset were introduced as supplementary data.

[0062] Data preprocessing stage:

[0063] In the data preprocessing stage, differentiated processing strategies were adopted based on whether the data had labeled information. For labeled data, strong augmentation operations were performed to fully explore the potential features of the data, expand its diversity, and thus improve the model's generalization ability. For unlabeled data, considering the lack of accurate labeling information, and to avoid introducing too much noise to interfere with model training, only weak augmentation operations similar to regularization were performed.

[0064] For labeled data, a variety of powerful enhancement techniques were employed. In addition to conventional operations such as random horizontal flipping (probability set to 0.5), color dithering (brightness, contrast, and saturation adjusted within a range of 0.5, hue adjusted within a range of 0.25), random grayscale conversion (probability 0.2), and Gaussian blur (probability 0.5, blur radius range of 0.1-2.0), random scaling and cropping were also added. During random scaling, the image was scaled within a ratio range of [0.5, 2.0] and then cropped to a fixed size of 321×321, thus enabling the model to learn feature representations at different image scales. Through these powerful enhancement operations, the labeled data, while maintaining labeling accuracy, was greatly enriched in its representation, providing the model with more diverse learning samples.

[0065] For unlabeled data, weak enhancement operations similar to regularization are performed. These mainly include basic image transformations, such as random horizontal flipping (with a probability of 0.5), random scaling, and cropping (with scaling ratios ranging from [0.5, 2.0]), but do not include strong transformation operations such as color jittering, grayscale conversion, and blurring. This weak enhancement method increases the diversity of the data without introducing intractable noise due to excessive transformation, thus ensuring the stability of unlabeled data when participating in model training.

[0066] Model initialization and parameter setting:

[0067] When initializing the FreqMatch model, a pre-trained MIT-B5 Transformer backbone network was selected. This network outputs four levels of multi-scale feature maps c1, c2, c3, and c4, with channel dimensions of 64, 128, 320, and 512, respectively, and spatial resolution decreasing sequentially. The pre-trained backbone network has learned rich image features on large-scale image datasets, providing good initialization weights for the model and accelerating model convergence. The IFFAP module processes the multi-scale features: the FAPS frequency-assisted perturbation stream enhances the frequency domain of c1, c2, c3, and c4; the IFPS image feature perturbation stream performs spatial perturbation on the features at each level through Dropout2d(0.5) random deactivation, generating perturbation feature versions. The total model loss is supervised cross-entropy loss. loss of consistency with unsupervised The weighted average, i.e. This weighted averaging method can balance the contributions of labeled and unlabeled data in model training, making full use of the advantages of both types of data.

[0068] Supervised cross-entropy loss The calculation formula is:

[0069]

[0070] in The batch size represents the number of labeled images input into the model during each training iteration. The choice of batch size affects the model's training efficiency and convergence speed. Generally, a larger batch size can make the model more stable during training, but it also requires more computational resources. It is the model's prediction of the labeled data; it is the category probability distribution generated by the model based on the input labeled image. It is the true label, that is, the true category label of each pixel in the image. This is the standard cross-entropy loss function, used to measure the difference between the model's predictions and the ground truth labels. By minimizing this loss function, the model can continuously adjust its parameters to make the predictions closer to the ground truth labels. Unsupervised consistency loss. The weighted average of the cross-entropy loss from the three feedforward streams of the IFFAP module is given by the following formula:

[0071]

[0072] in, This indicates the batch size of unlabeled data. Similar to the batch size of labeled data, it determines the number of unlabeled images input into the model during each training session. These are pseudo-labels, which are predicted labels generated by the model based on unlabeled images. Since unlabeled images do not have real labels, pseudo-labels are used to replace real labels for unsupervised learning. It is the model's prediction of weakly augmented data. It is the model's prediction of data processed by the IFP template. It is the model's prediction of the data processed by the FAP module. , , These are weighting coefficients dynamically assigned based on the importance of the perturbation flows. In the initial training phase, they can be set to 0.3, 0.3, and 0.4 respectively. This is because, in the early stages of training, it is desirable for each perturbation flow to contribute to the model's training. By setting similar weighting coefficients, the model can comprehensively learn features under different perturbation flows. As training progresses, these weighting coefficients can be dynamically adjusted based on performance on the validation set. For example, if it is found that the perturbation features generated by the IFP module significantly improve model performance, these weighting coefficients can be appropriately increased. The value of is increased to increase the weight of this perturbation flow in the unsupervised loss.

[0073] The model training uses a poly learning rate scheduling strategy, the formula is as follows: Initial learning rate This value was determined after multiple experimental verifications. At this initial learning rate, the model can converge quickly in the early stages of training. It represents the current iteration number, indicating how many training iterations the model has undergone. This represents the maximum number of iterations. For the Pascal VOC 2012 dataset, the training cycle is set to 80 rounds, with each round containing multiple training iterations. The encoder-decoder learning rate ratio is set to 1:5. This is because the encoder is mainly responsible for extracting image features, while the decoder is responsible for generating semantic segmentation results based on these features. The decoder needs more fine-tuning of parameters to improve segmentation accuracy; therefore, a higher learning rate is given to the decoder to enable faster convergence.

[0074] Model training process:

[0075] During training, each round simultaneously inputs a batch of labeled data and a batch of unlabeled data. The labeled data is used to calculate the supervised cross-entropy loss. Through this loss function, the model can learn explicit category information from the labeled data, guiding the model to adjust parameters in the correct direction.

[0076] For unlabeled data, perturbation features are first generated using the IFFAP module. The FAP module performs a Fast Fourier Transform (FFT) on the input image, as shown in the formula:

[0077]

[0078] This formula describes the transformation process from the spatial domain to the frequency domain. By performing a Fourier transform on an image, it can be decomposed into components of different frequencies. Among them, F c (u,v) x represents the complex value at (u,v) in the frequency domain after the Fourier transform. It is the result of the transform and represents the information of the original image at the corresponding frequency components. c (h, w) represents the pixel values ​​of the original image in the spatial domain, where h is the horizontal (row) index of the image, ranging from 0 to H−1; w is the vertical (column) index of the image, ranging from 0 to W−1. H and W are the height and width of the image, respectively, i.e., the size of the image in the spatial domain. −2πi( (h / H)u + (w / W)v )This is the kernel function part of the Fourier transform, where i is the imaginary unit. This kernel function is used to perform a weighted summation on each pixel in the spatial domain to achieve the transformation from the spatial domain to the frequency domain. Here, (h, H) and (w, W) are the normalized coordinates in the horizontal and vertical directions, respectively, and u and v are the frequency variables in the frequency domain, corresponding to the frequencies in the horizontal and vertical directions, respectively. In the frequency domain, high-frequency components correspond to the detailed information of the image, while low-frequency components correspond to the general outline and background information of the image. Based on the Nyquist sampling theorem, the frequency domain mask weights are adaptively calculated, and a directional perturbation is applied to the high-frequency components. This is because high-frequency components contain important detailed information of the image; perturbing the high-frequency components can increase the feature diversity of the image while preserving the key information. Then, the perturbed frequency domain features are reconstructed through the inverse Fourier transform (IFFT) and transformed back to the spatial domain to obtain the perturbed image. The relevant formulas are as follows:

[0079] 1. Formula for filtering high-frequency components in the preprocessing stage: This formula indicates that during the preprocessing stage, the Fourier transform of the original image is combined with a frequency domain mask. Multiply the components, then perform an inverse Fourier transform to obtain the image after filtering out high-frequency components. .

[0080] 2. Formula for balancing high and low frequency components: Here, the image after filtering high-frequency components is used. Fourier transform and low-frequency and high-frequency masks Multiply them separately, then perform an inverse Fourier transform to obtain the balanced low-frequency and high-frequency components. .

[0081] 3. Final characteristic formula:

[0082]

[0083] This formula will filter the image after filtering high-frequency components. High-frequency components and low frequency components Fusion is performed through an adaptive function. The high- and low-frequency components are weighted to obtain the final disturbance characteristics. .

[0084] The IFP module applies random masking to the feature map in the spatial domain. For example, it might randomly select a region for masking with a probability of 0.2, or inject Gaussian noise with a mean of 0 and a variance of 0.1. Random masking simulates occlusion scenarios that might occur in real-world applications, allowing the model to learn the contextual information of the occluded portion. Injecting Gaussian noise increases the noise level in the image, improving the model's robustness to noise.

[0085] The IFFAP module achieves consistent training through three feedforward streams:

[0086] 1. Basic Stream: For weakly enhanced images Feature extraction , here The feature extraction function, typically composed of a series of convolutional and pooling layers, is used to extract abstract feature representations from the input image. Then, pseudo-labels are generated. These pseudo-labels are generated based on the model's predictions of the weakly enhanced image, and are converted into pseudo-labels using a certain threshold or strategy.

[0087] 2. IFPS stream: Applies mask encoding or random noise perturbation to image features. .in, This indicates a perturbation operation, such as the random masking or noise injection mentioned earlier. The decoder represents the process of transforming perturbed features into predictions. .

[0088] 3. FAPS stream: Performs dealiasing and high / low frequency weighting on the Fourier spectrum. In this stream, the Fourier spectrum is processed to remove aliasing and the high and low frequency components are weighted, enabling the model to better learn the importance of different frequency features.

[0089] Meanwhile, the FCCM module plays a crucial role. This module takes the deepest feature c4 as input and converts it into a patch embedding sequence format [B, N, C] using the AllSpark wrapper, where N is the number of patches, calculated as (image_size / / 32+1)². The CrossAttnMem cross-attention memory module separates the labeled embedding emb_l and the unlabeled embedding emb_u through a split operation, maintaining a semantic memory queue of key-value pairs with the shape [num_class, SMem_size, patch_num]. The channel semantic grouping unit calculates the similarity matrix between the pseudo-probability map and the feature embedding to determine the category of each channel. The semantic memory queue dynamically stores historical features of each category using a first-in-first-out (FIFO) mechanism, recalling representative features of long-tail categories based on semantic similarity.

[0090] The decoder adopts the SegFormerHead structure, which uses four MLP modules, linear_c1, linear_c2, linear_c3, and linear_c4, to uniformly map multi-scale features to the embedding_dim=768 dimension. After feature fusion and classification prediction, the output is a semantic segmentation result with shape [B, num_class, H, W].

[0091] In terms of training strategy, a low learning rate (0.001) was used to warm up the model for 5 epochs, resulting in a total of 82 training rounds. The total model loss was supervised cross-entropy loss. loss of consistency with unsupervised The weighted average, i.e. The using_SMem state and feature recall threshold of the FCCM module are dynamically adjusted based on the mIoU of the validation set. After warmup_epoch=5, the semantic memory mechanism is activated to suppress the pollution of the memory queue by noisy pseudo-labels.

[0092] Model evaluation phase:

[0093] During training, model performance is evaluated on the validation set every 5 rounds, monitoring metrics such as mean Intersection over Union (mIoU) and accuracy. MIoU is a commonly used evaluation metric in semantic segmentation tasks; it measures the model's segmentation accuracy by calculating the ratio of the intersection to the union between the model's predictions and the ground truth annotations. Accuracy refers to the proportion of correctly predicted pixels out of the total number of pixels. After training, a final evaluation is performed on the test set. Experimental results show that the FreqMatch model of this invention achieves excellent results on the Pascal VOC 2012 dataset, improving mIoU by 3-5 percentage points compared to other state-of-the-art methods. Figure 3 As shown, this demonstrates that the model exhibits higher accuracy and robustness in semantic segmentation tasks.

[0094] Example 2:

[0095] This example demonstrates experiments conducted on the COCO dataset:

[0096] The COCO dataset is a large-scale object detection and semantic segmentation dataset containing 118,000 training images and 5,000 validation images, covering 80 object categories and including a rich variety of indoor and outdoor scenes. The diversity and complexity of this dataset provide a more challenging environment for model training and evaluation.

[0097] Parameter adjustment:

[0098] Given the large size, diversity, and complexity of the COCO dataset, adjustments to the training parameters are necessary. The initial learning rate remains at 0.001 because a smaller initial learning rate on a large dataset allows for greater model stability in the early stages of training, preventing rapid parameter updates that could lead to convergence issues. However, the training cycle is adjusted to 10 epochs. This is because the COCO dataset's large volume allows the model to learn sufficient features even within a shorter training period. Furthermore, considering data diversity, the random cropping ratio is increased to 0.9 and the rotation angle range is expanded to [-30°, 30°] during data augmentation. A larger random cropping ratio and rotation angle range further increase data diversity, enabling the model to learn features from more different perspectives and local regions, thus improving its generalization ability.

[0099] Training and Assessment:

[0100] The model was trained following the training procedure in Example 1, with close monitoring of its performance on the validation set during training. Because the COCO dataset features more complex scenes and a greater number of object categories, the model needs to learn more features and patterns during training. During training, supervised cross-entropy loss and unsupervised consistency loss were used to optimize the model. Perturbation features were generated using the IFFAP module, and the FCCM module was used to optimize class balance training.

[0101] After training, the model was evaluated on the validation set, and the results showed that the FreqMatch model also performed excellently on the COCO dataset. Under different annotation ratios, compared to other comparative methods, the mean intersection-union ratio (mIoU) was improved by 4-6 percentage points, such as... Figure 4 As shown in the figure. This further verifies the effectiveness and generalization ability of the model of the present invention. Even on complex large-scale datasets, the model can accurately perform semantic segmentation of images and identify different object categories. For example, in some complex indoor scene images, the model can accurately segment different objects such as furniture, people, and appliances, providing strong support for subsequent image analysis and applications. An example segmentation image is shown below. Figure 5 As shown.

[0102] like Figure 6As shown, this invention also provides a semi-supervised semantic segmentation device. The device includes at least one processor and at least one memory, as well as a communication interface and an internal bus. The memory stores a computer-executable program for the semi-supervised semantic segmentation system described above. When the processor executes the computer-executable program stored in the memory, it can execute a semi-supervised semantic segmentation method. The internal bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus in the accompanying drawings is not limited to only one bus or one type of bus. The memory may include high-speed RAM, and may also include non-volatile memory (NVM), such as at least one disk storage device, or a USB flash drive, portable hard drive, read-only memory, disk, or optical disk, etc.

[0103] The device may be provided as a terminal, server, or other type of device. In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0104] The present invention also provides a computer-readable storage medium storing a computer executable program of the semi-supervised semantic segmentation system as described above. When the computer executable program is executed by a processor, it enables the processor to perform a semi-supervised semantic segmentation method.

[0105] Specifically, a system, apparatus, or device may be provided equipped with a readable storage medium on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system, apparatus, or device reads and executes the instructions stored in the readable storage medium. In this case, the program code read from the readable medium itself can implement the functions of any of the embodiments described above, therefore, the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present invention.

[0106] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0107] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A semi-supervised semantic segmentation system, characterized in that: It includes a Transformer encoder feature extraction module, an image feature and frequency-assisted consistency perturbation flow network, a multi-feature channel category memory mechanism network, and a decoder module; The Transformer encoder feature extraction module takes the original images after weak enhancement and strong enhancement as inputs and feeds them into the structurally shared backbone network for feature extraction. The backbone network is based on a multi-scale structure and outputs four sets of semantic feature maps at different levels in sequence during forward propagation, denoted as c1, c2, c3, and c4, which represent the layer-by-layer semantic expression from shallow to deep layers. The image feature and frequency-assisted consistency perturbation flow network includes a frequency-assisted perturbation flow (FAPS) module and an image feature perturbation flow (IFPS) module. The FAPS module sequentially performs Fourier transform on the input c1, c2, c3, and c4 feature maps to obtain a frequency domain representation, retaining low-frequency components and suppressing high-frequency noise. Then, it performs adaptive weighted fusion of low-frequency and high-frequency components and finally reconstructs the spatial domain perturbation features through inverse Fourier transform. The IFPS module applies random deactivation perturbation to the feature map output by the backbone network and generates perturbation branch features through feature splicing and block operations. The multi-feature channel category memory mechanism network takes the deepest feature map c4 as input and fuses multi-source features from weak and strong enhancement paths, FAPS frequency-assisted perturbation path, and IFPS image feature perturbation path. The input features are converted into a block embedding sequence, which is then separated into labeled and unlabeled embeddings. After processing by a cross-attention mechanism, the output is an enhanced feature map that fuses the embedded features. Upsampling is used to ensure that the shape of the output remains consistent. The decoder module adopts the SegFormerHead structure, taking the multi-scale feature map processed by the self-independent network and the multi-feature channel category memory mechanism network as input, and uniformly mapping the features of each level to the embedding_dim dimension. After feature fusion and classification prediction, the semantic segmentation result is output.

2. The semi-supervised semantic segmentation system as described in claim 1, characterized in that: The input 513×513 original image is subjected to weak enhancement and strong enhancement processing respectively. The weak enhancement operation includes normalization, random scaling, random cropping, random horizontal flipping, color dithering, random grayscale conversion and Gaussian blur processing of the image. The strong enhancement operation further increases the probability or magnitude of color dithering, grayscale conversion and blurring on the basis of weak enhancement to enhance the diversity of the image. The enhanced images are used as inputs for labeled and unlabeled branches respectively for subsequent feature extraction and semi-supervised training.

3. The semi-supervised semantic segmentation system as described in claim 1, characterized in that: The Transformer encoder feature extraction module takes the enhanced image as input, and the input format is a tensor with shape [B, 3, 513, 513], where B is the batch size and 3 is the number of channels. The backbone network adopts a multi-scale hybrid visual Transformer structure, which sequentially goes through four stages of feature extraction. Each stage outputs a set of feature maps, denoted as c1, c2, c3, and c4, respectively. The output formats of each feature map are [B, 64, H1, W1], [B, 128, H2, W2], [B, 320, H3, W3], and [B, 512, H4, W4].

4. The semi-supervised semantic segmentation system as described in claim 1, characterized in that: The Frequency-Assisted Perturbation Flow (FAPS) module takes a feature map of shape [B, C, H, W] as input. First, it performs a Fourier transform and applies a low-pass mask with a frequency threshold of 0.25 to retain the low-frequency region in the center and filter out the high-frequency components at the edges. Then, it separates the low-frequency and high-frequency representations of the Fourier transform frequency domain and calculates the channel attention weights of the low-frequency mask and the high-frequency mask respectively. After adaptive weighting, the components are recombined and the spatial domain perturbation features are output through inverse Fourier transform. The Image Feature Perturbation Stream (IFPS) module takes the multi-scale feature maps c1, c2, c3, and c4 output by the backbone network as input, applies a Dropout2d random deactivation operation with a probability of 0.5 to each feature map, generates a perturbation feature version, splices the original feature and the perturbation feature to form a dual-branch input, and after processing by the shared decoder, separates the prediction result of the perturbation branch, which is used to perform consistency constraint calculation with the original branch.

5. A semi-supervised semantic segmentation system as described in claim 1, characterized in that: The multi-feature channel category memory mechanism network includes a channel semantic grouping unit, a semantic memory queue update unit, and a cross-attention calculation unit; The channel semantic grouping unit performs bilinear interpolation on the input pseudo-probability map to a size that matches the number of feature blocks (patch), calculates the similarity matrix between the normalized pseudo-probability map and the normalized unlabeled embedding (emb_u) through matrix multiplication, and uses the argmax operation to determine the semantic category label corresponding to each feature channel. The semantic memory queue is updated by a key-value pair column cache of shape [num_class, SMem_size, patch_num], and is updated by category: for each occurrence of category cls_i, the unlabeled embedding features of the corresponding channel are extracted, and the queue pointer is updated in a first-in-first-out manner. When the queue is full, the queue is overwritten in a loop to ensure the continuous accumulation of semantic information of each category. The cross-attention calculation unit processes the labeled embedding emb_l and the unlabeled embedding emb_u respectively: for the unlabeled embedding, it performs self-attention calculation to generate a query, key, and value, and obtains the self-attention output out_u through a multi-head attention mechanism; for the labeled embedding, when using_SMem is activated, it extracts historical semantic memory from the key-value pair column as the key and value, otherwise it uses the current unlabeled embedding, calculates the cross-attention weight to obtain the output out_l2u, and finally concatenates the output features of the two branches through torch.cat to form a complete enhanced embedding.

6. The semi-supervised semantic segmentation system as described in claim 1, characterized in that: The decoder module adopts the SegFormerHead structure, taking the multi-scale feature maps processed by the self-intercepting network and FCCM module as input, including feature maps of four levels: c1, c2, c3, and c4. The decoder module contains multi-level linear embedding units, feature fusion units, and classification prediction units. After MLP linear embedding, the features of each level are uniformly mapped to the embedding_dim dimension. The highest resolution is upsampled by bilinear interpolation, and then the torch.cat channel concatenation and ConvModule feature fusion are performed. The final output is a semantic segmentation prediction map with shape [B, num_class, H, W], which is restored to the original input image size by bilinear interpolation.

7. A semi-supervised semantic segmentation system as described in claim 6, characterized in that: The multi-level linear embedding unit contains four MLP modules. Each MLP module flattens and transposes the spatial dimension into the [B, H×W, C] format, and then maps the channel dimension to 768 dimensions through a linear projection layer. The feature fusion unit restores the processed four-level features _c1, c2, _c3, and _c4 into a spatial feature map format. It then upsamples _c2, _c3, and _c4 to the same spatial resolution as _c1 using bilinear interpolation and then stitches them together according to the channel dimension to form a fusion feature with 4×embedding_dim channels. The classification prediction unit reduces the fused features from 4×embedding_dim dimension to embedding_dim dimension through a convolution module. After being regularized by Dropout2d(0.1), the features are mapped to the number of classes num_class using a linear_pred convolutional layer, and the final semantic segmentation logits are output.

8. A semi-supervised semantic segmentation method, characterized in that: Using the semi-supervised semantic segmentation system as described in any one of claims 1 to 7, and comprising the following process: S1, preprocess the unlabeled image data to generate a weakly enhanced view and pseudo-labels; S2, input the weakly enhanced view and pseudo-labels into the semi-supervised semantic segmentation system; S5 outputs pixel-level semantic segmentation results.

9. A semi-supervised semantic segmentation device, characterized in that: The device includes at least one processor and at least one memory, the processor and the memory being coupled together; the memory stores a computer-executable program for a semi-supervised semantic segmentation system as described in any one of claims 1 to 7; when the processor executes the computer-executable program stored in the memory, the processor performs a semi-supervised semantic segmentation method.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer executable program for the semi-supervised semantic segmentation system as described in any one of claims 1 to 7, wherein when the computer executable program is executed by a processor, the processor performs a semi-supervised semantic segmentation method.

Images