Training method and device of image segmentation model, equipment and storage medium

Abstract

The application provides a training method and device of an image segmentation model, equipment and a storage medium, comprising: adding a feature fusion neural network to a SAM model, which extracts high-frequency feature information in early feature maps of an image and low-frequency feature information in late feature maps and fuses them. In training, a sample image and a degraded image of the sample image are input into a model image encoder, the early feature maps and the late feature maps of the sample image and the degraded image are processed respectively through the feature fusion neural network, the fused feature maps generated after processing are added to the feature maps embedded finally output by the image encoder respectively, the fused embedded feature maps of the two branches of the sample image and the degraded image are generated, and the frequency domain distance loss and the consistency loss of the fused embedded feature maps of the two branches are calculated. By extracting and fusing the high-frequency and low-frequency feature information and calculating the distance loss and the consistency loss of the sample image in the frequency domain, the segmentation accuracy of the SAM model for the degraded image can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a training method, apparatus, device, and storage medium for an image segmentation model. Background Technology

[0002] The SAM (Segment Anything Model) is a general-purpose large-scale visual model in the field of image segmentation. The design of the SAM model enables it to perform efficient segmentation in various scenarios, especially in zero-shot settings, without the need for additional training for specific tasks.

[0003] The SAM model architecture primarily consists of three core components: an image encoder, a cue encoder, and a mask decoder. The image encoder's main task is to convert the input image into high-dimensional feature maps. These feature maps preserve the spatial and semantic information of the image and form the basis for subsequent segmentation tasks. The cue encoder converts user-provided cues (such as points, boxes, and text) into cue vectors that the model can process. The mask decoder combines the feature maps generated by the image encoder and the cue vectors generated by the cue encoder to generate the final image segmentation mask. Through the collaborative work of these three components, the SAM model achieves powerful image segmentation capabilities.

[0004] However, while the SAM model has strong zero-shot capability and good segmentation accuracy for natural images, its segmentation quality drops significantly when faced with degraded images (such as those with rain, snow, fog, low light, or overexposure), making it difficult to accurately identify the segmented target. Furthermore, although some degraded image restoration and segmentation methods can effectively remove image noise and obtain sample images from degraded images, the segmentation results of these methods heavily depend on the quality of image restoration. When the restored image lacks fine texture or boundary features, the segmentation accuracy of the target region will also be affected. Summary of the Invention

[0005] This application provides a training method, apparatus, device, and storage medium for an image segmentation model to address the problem of decreased segmentation accuracy of the SAM model for degraded images in the prior art.

[0006] In a first aspect, this application provides a method for training an image segmentation model, the method comprising:

[0007] The sample image, and the degraded image of the sample image, are input into the image encoder of the image segmentation model;

[0008] High-frequency feature information of the early feature map of the sample image and low-frequency feature information of the late feature map of the sample image are extracted, as are high-frequency feature information of the early feature map of the degraded image and low-frequency feature information of the late feature map of the degraded image. The early feature map is the feature map output by the first global attention module in the neural network of the image encoder, and the late feature map is the feature map output by the last layer in the neural network of the image encoder.

[0009] The high-frequency feature information of the early feature map of the sample image and the low-frequency feature information of the late feature map of the sample image are fused to obtain the fused feature map of the sample image. Furthermore, the high-frequency feature information of the early feature map of the degraded image and the low-frequency feature information of the late feature map of the degraded image are fused through the feature fusion neural network to obtain the fused feature map of the degraded image.

[0010] The fused feature map of the sample image is embedded and added to the first final feature map to obtain a first fused embedded feature map. The fused feature map of the degraded image is embedded and added to the second final feature map to obtain a second fused embedded feature map. The first final feature map is a feature map converted by the image encoder based on the sample image and output to the mask decoder of the image segmentation model. The second final feature map is a feature map converted by the image encoder based on the degraded image and output to the mask decoder.

[0011] Calculate the distance loss and consistency loss between the first fused embedded feature map and the second fused embedded feature map in the frequency domain;

[0012] Based on the distance loss and the consistency loss, update the weight parameters of the feature fusion neural network and the multilayer perceptron of the mask decoder.

[0013] In one possible design, calculating the distance loss and consistency loss between the first fused embedded feature map and the second fused embedded feature map in the frequency domain includes:

[0014] Fast Fourier transform is performed on the first fused embedding feature map and the second fused embedding feature map respectively to obtain the first amplitude component and the first phase component of the first fused embedding feature map in the frequency domain, and the second amplitude component and the second phase component of the second fused embedding feature map in the frequency domain.

[0015] Based on the first phase component and the second phase component, calculate the Earth movement distance loss of the first phase component and the second phase component;

[0016] The consistency loss is calculated based on the first amplitude component and the second amplitude component.

[0017] In one possible design, calculating the consistency loss based on the first amplitude component and the second amplitude component includes:

[0018] A centering transformation is performed on the first amplitude component and the second amplitude component, placing the low-frequency feature information of the first amplitude component and the low-frequency feature information of the second amplitude component in the central region of the amplitude spectrum.

[0019] The low-frequency feature information of the first amplitude component and the second amplitude component are locally mixed to obtain a first mixed amplitude component, and the low-frequency feature information of the second amplitude component and the first amplitude component are locally mixed to obtain a second mixed amplitude component.

[0020] The first mixed amplitude component and the first phase component are subjected to inverse Fourier transform to obtain a first spatial domain feature map, and the second mixed amplitude component and the second phase component are subjected to inverse Fourier transform to obtain a second spatial domain feature map.

[0021] Calculate the consistency loss between the first spatial feature map and the second spatial feature map.

[0022] In one possible design, the step of locally mixing the low-frequency characteristic information of the first amplitude component and the second amplitude component to obtain a first mixed amplitude component, and locally mixing the low-frequency characteristic information of the second amplitude component and the first amplitude component to obtain a second mixed amplitude component, includes:

[0023] The low-frequency feature information of the first amplitude component and the low-frequency feature information of the second amplitude component are linearly interpolated and then fused with the high-frequency feature information of the first amplitude component to obtain the first mixed amplitude component. The low-frequency feature information of the second amplitude component and the low-frequency feature information of the first amplitude component are linearly interpolated and then fused with the high-frequency feature information of the first amplitude component to obtain the second mixed amplitude component.

[0024] In one possible design, the method further includes:

[0025] The first predictive segmentation mask feature map is obtained by performing matrix multiplication on the fused feature map of the sample image and the attention weight vectors of each channel of the first final feature map output by the multilayer perceptron. The second predictive segmentation mask feature map is obtained by performing matrix multiplication on the fused feature map of the degraded image and the attention weight vectors of each channel of the second final feature map output by the multilayer perceptron.

[0026] Based on the first predicted segmentation mask feature map and the second predicted segmentation mask feature map, calculate the semantic consistency comparison loss between the first predicted segmentation mask feature map, the second predicted segmentation mask feature map and the ground truth annotations of the sample image and the degraded image, and calculate the segmentation loss between the second predicted segmentation mask feature map and the ground truth annotations;

[0027] The weight parameters of the feature fusion neural network and the multilayer perceptron are updated based on the distance loss, the consistency loss, the semantic consistency comparison loss, and the segmentation loss.

[0028] In one possible design, the extraction of high-frequency feature information from the early feature map of the degraded image and low-frequency feature information from the late feature map of the degraded image includes:

[0029] Perform convolution and pixel recombination operations on the late feature map of the degraded image, calculate the weight of each pixel in the late feature map of the degraded image, generate a filter kernel based on the weight of each pixel in the late feature map of the degraded image, and generate an adaptive low-pass filter based on the filter kernel.

[0030] Subtract the filter core from the unit core to generate an adaptive high-pass filter;

[0031] The adaptive low-pass filter extracts low-frequency feature information from the late feature map of the degraded image, and the adaptive high-pass filter extracts high-frequency feature information from the early feature map of the degraded image.

[0032] In one possible design, the step of extracting low-frequency feature information of the late feature map of the degraded image through the adaptive low-pass filter and extracting high-frequency feature information of the early feature map of the degraded image through the adaptive high-pass filter includes:

[0033] The first low-frequency feature information of the late feature map of the degraded image is extracted by the adaptive low-pass filter, and the first high-frequency feature information of the early feature map of the degraded image is extracted by the adaptive high-pass filter. The first low-frequency feature information and the first high-frequency feature information are then fused to generate a first fused feature map.

[0034] Based on the first fused feature map, a spatially location-related low-pass filter and a spatially location-related high-pass filter are generated. The weights of the spatially location-related low-pass filter and the spatially location-related high-pass filter change with the pixel position of the first fused feature map.

[0035] The second low-frequency feature information of the late feature map of the degraded image is extracted by the spatially location-correlated low-pass filter, and the second low-frequency feature information is used as the low-frequency feature information of the late feature map of the degraded image. The second high-frequency feature information of the early feature map of the degraded image is extracted by the spatially location-correlated high-pass filter, and the second high-frequency feature information is used as the high-frequency feature information of the early feature map of the degraded image.

[0036] Secondly, this application provides a training apparatus for an image segmentation model, the apparatus comprising:

[0037] The feature extraction module is used for:

[0038] The sample image, and the degraded image of the sample image, are input into the image encoder of the image segmentation model;

[0039] High-frequency feature information of the early feature map of the sample image and low-frequency feature information of the late feature map of the sample image are extracted, as are high-frequency feature information of the early feature map of the degraded image and low-frequency feature information of the late feature map of the degraded image. The early feature map is the feature map output by the first global attention module in the neural network of the image encoder, and the late feature map is the feature map output by the last layer in the neural network of the image encoder.

[0040] The feature fusion module is used to: fuse high-frequency feature information of the early feature map of the sample image and low-frequency feature information of the late feature map of the sample image to obtain a fused feature map of the sample image; and to fuse high-frequency feature information of the early feature map of the degraded image and low-frequency feature information of the late feature map of the degraded image through the feature fusion neural network to obtain a fused feature map of the degraded image.

[0041] The fused feature map of the sample image is embedded and added to the first final feature map to obtain a first fused embedded feature map. The fused feature map of the degraded image is embedded and added to the second final feature map to obtain a second fused embedded feature map. The first final feature map is a feature map converted by the image encoder based on the sample image and output to the mask decoder of the image segmentation model. The second final feature map is a feature map converted by the image encoder based on the degraded image and output to the mask decoder.

[0042] The loss calculation module is used to: calculate the distance loss and consistency loss between the first fused embedded feature map and the second fused embedded feature map in the frequency domain;

[0043] The weight update module is used to update the weight parameters of the feature fusion neural network and the multilayer perceptron of the mask decoder based on the distance loss and the consistency loss.

[0044] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0045] The memory stores computer-executed instructions;

[0046] The processor executes computer execution instructions stored in the memory to implement the method as described in the first aspect.

[0047] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect.

[0048] The image segmentation model training method, apparatus, device, and storage medium provided in this application have the following technical effects:

[0049] 1. By extracting and fusing high-frequency and low-frequency feature information, the model can better capture the details and overall structure in the image, thereby retaining more useful information in degraded images to improve segmentation accuracy;

[0050] 2. Generating more representative fused feature maps through feature fusion neural networks can improve the model's ability to understand complex images;

[0051] 3. By adding the fused feature map to the final feature map, the model can combine feature information from different stages, enhancing the expressive power of the features. This combination enables the model to capture more contextual information and detailed features, thereby improving the accuracy of image segmentation.

[0052] 4. By comparing the feature maps of the sample images and the degraded images, and calculating their distance loss and consistency loss in the frequency domain, the model can better adapt to different degrees of image degradation. This robustness allows the model to maintain high segmentation performance when facing noise, blur, or other degradation factors. Attached Figure Description

[0053] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0054] Figure 1 A schematic diagram illustrating a training method for an image segmentation model provided in an embodiment of this application;

[0055] Figure 2A schematic diagram illustrating the frequency domain distance loss and consistency loss calculation method provided in this application embodiment;

[0056] Figure 3 This is a schematic diagram of the SAM model architecture and training process provided in the embodiments of this application;

[0057] Figure 4 A schematic diagram illustrating a method for enhancing the fusion of high-frequency and low-frequency features in a mask decoder using high-pass and low-pass filters, as provided in an embodiment of this application.

[0058] Figure 5 A schematic diagram of a training device for an image segmentation model provided in an embodiment of this application;

[0059] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0060] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0061] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0062] The SAM model architecture mainly consists of three core components: an image encoder, a cue encoder, and a mask decoder. The main task of the image encoder is to convert the input image into high-dimensional feature maps. These feature maps preserve the spatial and semantic information in the image and form the basis for subsequent segmentation tasks. The cue encoder is used to convert user-provided cues (such as points, boxes, and text) into cue vectors that the model can process. The task of the mask decoder is to combine the feature maps generated by the image encoder and the cue vectors generated by the cue encoder to generate the final image segmentation mask.

[0063] The mask decoder mainly includes the following components:

[0064] 1. Transformer Decoder:

[0065] Function: The Transformer decoder processes input features and generates an output sequence. It typically consists of multiple layers of self-attention mechanisms and feedforward neural networks.

[0066] In SAM, the Transformer decoder extracts information from features received from the encoder, helping to generate more accurate masks. It captures relationships between features through a self-attention mechanism, thereby improving the model's ability to understand complex image content.

[0067] 2. Token to Image Attention:

[0068] Function: This attention mechanism is used to align token-level features with image-level features.

[0069] In SAM, token-to-image attention helps the model combine textual or other input features with image features during mask generation, resulting in masks that better match the input conditions. This mechanism can improve the model's performance in multimodal tasks.

[0070] 3. Transposed Convolution (also known as deconvolution or upsampling convolution):

[0071] Function: Transposed Convolution is used to increase the resolution of feature maps, i.e., to perform upsampling.

[0072] Its role in SAM: In the mask decoder, transposed convolution can be used to upsample low-resolution feature maps to the same resolution as the input image, thereby generating a high-resolution mask. This is crucial for refining mask edges and details.

[0073] 4. MLP (Multilayer Perceptron):

[0074] Function: MLP is a type of feedforward neural network, typically used for feature transformation and nonlinear mapping.

[0075] In SAM (Sample Amplifier), the MLP (Multi-Level Processor) can be used to further process and transform features in the mask decoder to generate the final mask output. It helps the model perform complex nonlinear transformations in the feature space, improving the accuracy of the mask.

[0076] The workflow of the existing SAM model will be explained below.

[0077] The SAM model's prompt encoder receives prompts from the user, such as dots, boxes, or text. The prompt encoder converts these prompts into prompt tokens (prompt vectors), which are used to guide the model's attention mechanism.

[0078] The SAM model's image encoder is responsible for converting the input image into a feature map. The extracted feature map preserves the spatial and semantic information of the image and is the foundation for subsequent segmentation tasks.

[0079] The Transformer decoder of the mask decoder combines image features and cue tokens to generate feature representations for segmentation. Specifically, it uses an attention mechanism to combine cue tokens with image features, focusing on image regions related to the cue, and integrates cue information and image features through multi-layered self-attention and cross-attention mechanisms.

[0080] The Token to Image Attention mechanism in the mask decoder propagates the influence of the cue token to image features to better guide segmentation. Specifically, through the attention mechanism, the correlation between the cue token and image features is calculated, and the image features are adjusted to focus more on regions related to the cue.

[0081] The mask decoder's Transposed Convolution progressively upsamples the low-resolution feature map to restore details. Through a series of transposed convolutional layers, it generates a feature map that matches the size of the input image.

[0082] The MLP layer of the mask decoder performs further nonlinear transformations on the feature map upsampled by the Transposed Convolution. Specifically, the MLP layer performs independent nonlinear transformations on each pixel in the feature map and adjusts the feature representation of the feature map through a series of fully connected layers.

[0083] After the above steps, the SAM model generates a feature map, which is then processed by an activation function (such as softmax) to generate the final segmentation mask. The segmentation mask is an image of the same size as the input image, where the value of each pixel represents its category or object.

[0084] In existing technologies, the SAM model's low segmentation accuracy for degraded images is mainly due to the difficulty in accurately extracting fine texture and boundary features from degraded images, and the lack of a suitable loss function during training. Therefore, to address the problem of decreased segmentation accuracy for degraded images in existing SAM models, this application proposes adding a feature fusion neural network to the SAM model. This network extracts high-frequency features from early feature maps and low-frequency features from late feature maps, and then fuses them. The early feature maps are those extracted after the first global attention module in the image encoder, while the late feature maps are the global feature maps output from the last layer of the image encoder. During training, a clear image and a degraded image of the clear image are input into the SAM model's image encoder. The feature fusion neural network processes the early and late feature maps of both the clear and degraded images separately. The resulting fused feature maps are then embedded and added to the final feature map output from the image encoder, generating fused embedded feature maps for both the clear and degraded images. The frequency domain distance loss and consistency loss of the fused embedded feature maps for the two branches are then calculated.

[0085] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0086] This embodiment provides a training method for an image segmentation model, which can be applied to a server. Figure 1 This is a schematic diagram of a training method for an image segmentation model provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0087] S101. Input the sample image and the degraded image of the sample image into the image encoder of the image segmentation model;

[0088] Specifically, the sample image is the clear image, which is composed of clear image I. c After image processing operations such as noise addition, blurring, elastic transformation, and color transformation, the degraded image I is obtained. dDegraded and sharp images share ground truth masking, forming the training dataset. It's important to note that in image processing and computer vision, ground truth masking refers to the precise labeling of regions of interest (ROIs) in an image. Ground truth masking is the precise identification of the classification result for each pixel in the image; for each pixel, the mask label specifies which category it belongs to (e.g., foreground, background, or a specific object). Ground truth masks are typically generated manually, with annotators manually drawing the boundaries of the ROI based on the image content. During the training phase, one degraded image is randomly selected from N images and paired with a sharp image to form an image pair (I0). c ,I d ) is used as the input to the SAM model image encoder.

[0089] S102. Extract high-frequency feature information from the early feature map of the sample image and low-frequency feature information from the late feature map of the sample image, and extract high-frequency feature information from the early feature map of the degraded image and low-frequency feature information from the late feature map of the degraded image.

[0090] In this model, the early feature map is the feature map output by the first global attention module in the neural network of the image encoder, while the late feature map is the feature map output by the last layer in the neural network of the image encoder. It should be noted that the early feature map extracted by the image encoder contains a large amount of high-frequency feature information such as texture and boundaries, while the late feature map contains low-frequency feature information such as global context information. By extracting high-frequency feature information from the early feature map and low-frequency feature information from the late feature map, and then fusing the high-frequency and low-frequency feature information, the semantic information of the input image can be effectively extracted, thus enhancing feature extraction. Specifically, a high-pass filter can be used to process the early feature map to obtain high-frequency feature information, and a low-pass filter can be used to process the late feature map.

[0091] S103. The high-frequency feature information of the early feature map of the sample image and the low-frequency feature information of the late feature map of the sample image are fused to obtain the fused feature map of the sample image. And, through the feature fusion neural network, the high-frequency feature information of the early feature map of the degraded image and the low-frequency feature information of the late feature map of the degraded image are fused to obtain the fused feature map of the degraded image.

[0092] Specifically, high-frequency and low-frequency feature information can be concatenated along the channel dimension and then further processed through convolutional layers to fuse information of different frequencies, or an attention mechanism can be used to dynamically adjust the fusion weights of high-frequency and low-frequency feature information for fusion processing.

[0093] S104. The fusion feature map of the sample image is embedded and added to the first final feature map to obtain the first fusion embedded feature map; and the fusion feature map of the degraded image is embedded and added to the second final feature map to obtain the second fusion embedded feature map.

[0094] Among them, the first final feature map F c f The first feature map, F, is the feature map generated by the image encoder from the sample image and output to the mask decoder of the image segmentation model. d f The first and second final feature maps are generated by the image encoder based on the degraded image and output to the mask decoder. Specifically, after the first and second final feature maps are input to the mask decoder, they first pass through the mask decoder's transformer decoder. The transformer decoder uses an attention mechanism to process the input features. Through self-attention and cross-attention, the transformer decoder can aggregate information from the feature maps and combine it with the cue vector output by the cue encoder for feature enhancement. The low-resolution feature map output by the transformer decoder is upsampled to a higher resolution by the mask decoder's Transposed Convolution. Finally, after the above processing, the first final feature map is embedded and added to the fused feature map of the sample image; the second final feature map, after the above processing, is embedded and added to the fused feature map of the degraded image. Specifically, the embedding and addition operation refers to adding two or more feature tensors at the element-wise level. The specific steps are as follows:

[0095] Before performing feature addition, ensure that all feature tensors that need to be added are aligned in spatial dimensions (such as height and width) and channel dimensions;

[0096] Once the feature tensors are aligned, element-wise addition can be performed. After the addition operation, some post-processing steps are usually performed, such as non-linear transformation using activation functions (e.g., ReLU) or normalization using batch normalization, to ensure that the fused features have good performance.

[0097] It should be noted that the purpose of the embedding addition operation is to combine feature information from different sources, thereby retaining richer feature information in the feature map.

[0098] S105. Calculate the distance loss and consistency loss between the first fused embedded feature map and the second fused embedded feature map in the frequency domain;

[0099] Specifically, firstly, the first and second fused embedded feature maps are transformed into the frequency domain. This can be achieved by applying a two-dimensional discrete Fourier transform (DFT) to each feature map, or by using a fast Fourier transform (FFT) algorithm to improve computational efficiency. After performing the Fourier transform, the spectral representation is obtained. The spectrum can be divided into the amplitude spectrum and the phase spectrum.

[0100] Distance loss can be achieved by calculating the difference between two spectra. Common methods include:

[0101] Euclidean distance: Calculates the Euclidean distance between two spectral amplitudes.

[0102] L1 / L2 norm: Calculates the L1 or L2 norm of the difference in amplitude between two spectra.

[0103] Cosine similarity: Calculate the cosine similarity between two spectral vectors and convert it into a distance metric.

[0104] Consistency Loss: Consistency loss measures the similarity between two feature maps in the frequency domain. It can be calculated as follows:

[0105] Phase consistency: Compare the phase spectra of two feature maps and calculate the consistency of the phase difference.

[0106] Structural Similarity (SSIM): Calculates the structural similarity index in the frequency domain to evaluate the structural similarity between two feature maps.

[0107] Cross-entropy or KL divergence: If the spectrum is viewed as a probability distribution, cross-entropy or Kullback-Leibler divergence can be used to measure consistency.

[0108] Then, the distance loss and consistency loss can be combined into a total loss function to optimize the model. The combination can be a simple weighted sum or a complex combination adjusted based on experiments.

[0109] During the training phase, calculating the distance loss and consistency loss between the first and second fused embedding feature maps in the frequency domain significantly improves the segmentation accuracy of the SAM model on degraded images. This is because frequency domain analysis captures both global and local features in the image, enhancing the model's understanding of details such as edges and textures. Furthermore, this method improves the model's robustness to degradation factors such as noise and blur, promoting the alignment of features between clear and degraded images and ensuring the model maintains high accuracy when processing degraded images. By capturing features at multiple scales and reducing overfitting, the model can learn more robust and semantically rich feature embeddings, thereby improving the overall performance of the segmentation task.

[0110] S106. Update the weight parameters of the multilayer perceptron of the feature fusion neural network and the mask decoder based on the distance loss and consistency loss.

[0111] Specifically, in this step, the total loss value calculated using the overall loss function described above is used to update the weight parameters of the feature fusion neural network and the multilayer perceptron of the mask decoder through backpropagation. It should be noted that during the training phase, the pre-trained weights of the SAM model need to be loaded, and the pre-trained parameters of the SAM model need to be frozen, updating only the weight parameters of the feature fusion neural network and the multilayer perceptron. Since the SAM model is trained on a large-scale dataset and has already learned rich feature representations, loading these pre-trained weights can effectively utilize this existing knowledge, thereby improving the performance of new tasks. Freezing most of the parameters of the pre-trained model means that these parameters do not need to be updated during training, which can greatly reduce training time and required computational resources, because only a small number of parameters need to be updated, and by freezing most parameters and training only a small number of parameters, the risk of overfitting can be reduced.

[0112] Since the feature fusion module and MLP layer of the decoder mask are often task-specific parts, by training only these modules, the model can be better tuned to adapt to a specific segmentation task without affecting the general features already learned in the pre-trained model.

[0113] The image segmentation model training method provided in this embodiment has the following technical effects:

[0114] 1. By extracting and fusing high-frequency and low-frequency feature information, the model can better capture the details and overall structure in the image, thereby retaining more useful information in degraded images to improve segmentation accuracy;

[0115] 2. Generating more representative fused feature maps through feature fusion neural networks can improve the model's ability to understand complex images;

[0116] 3. By adding the fused feature map to the final feature map, the model can combine feature information from different stages, enhancing the expressive power of the features. This combination enables the model to capture more contextual information and detailed features, thereby improving the accuracy of image segmentation.

[0117] 4. By comparing the feature maps of the sample images and the degraded images, and calculating their distance loss and consistency loss in the frequency domain, the model can better adapt to different degrees of image degradation. This robustness allows the model to maintain high segmentation performance when facing noise, blur, or other degradation factors.

[0118] Figure 2 This is a schematic diagram illustrating the frequency domain distance loss and consistency loss calculation method provided in the embodiments of this application, as shown below. Figure 2 As shown, optionally, the distance loss and consistency loss in the frequency domain between the first fused embedded feature map and the second fused embedded feature map are calculated, including:

[0119] Fast Fourier transform is performed on the first fused embedded feature map and the second fused embedded feature map respectively to obtain the first amplitude component and the first phase component of the first fused embedded feature map in the frequency domain, and the second amplitude component and the second phase component of the second fused embedded feature map in the frequency domain.

[0120] Calculate the Earth's distance loss based on the first phase component and the second phase component;

[0121] The consistency loss is calculated based on the first amplitude component and the second amplitude component.

[0122] Specifically, for the first fused embedded feature map Second fusion embedding feature map Performing a Fast Fourier Transform yields two sets of amplitude components in the frequency domain. and phase components For the feature map F, its frequency domain and amplitude components Phase components It can be expressed as a formula:

[0123]

[0124] Where u and v are the coordinates in the frequency domain, H and W represent the height and width of the feature map, h and w are the coordinates of the feature map in the spatial domain, and R and I represent the real and imaginary parts of the frequency domain, respectively.

[0125] Since the phase components of the feature map in the frequency domain retain a large amount of semantic information, the Wasserstein distance can be used to quantify the similarity between probability distributions, and the frequency domain distance loss L can be used. fdlTo minimize and The semantic differences between them can be expressed by the following formula:

[0126]

[0127] Here, WD represents the Wasserstein distance between the phase components. It's important to note that the distance loss in the frequency domain between the first and second fused embedded feature maps is used to minimize the semantic difference between them. The Wasserstein distance, also known as Earth Mover's Distance (EMD), is a metric used to measure the distance between two probability distributions. In this formula, WD is used to calculate the distance between the frequency domain phase components of the two feature embeddings. The Wasserstein distance of the phase components can capture the semantic differences between two features because phase information is closely related to the structure and shape of the image.

[0128] like Figure 2 As shown, optionally, the consistency loss is calculated based on the first amplitude component and the second amplitude component, including:

[0129] A centering transformation is performed on the first amplitude component and the second amplitude component, placing the low-frequency characteristic information of the first amplitude component and the low-frequency characteristic information of the second amplitude component in the central region of the amplitude spectrum.

[0130] The low-frequency feature information of the first amplitude component and the second amplitude component are locally mixed to obtain the first mixed amplitude component, and the low-frequency feature information of the second amplitude component and the first amplitude component are locally mixed to obtain the second mixed amplitude component.

[0131] The first mixed amplitude component and the first phase component are subjected to inverse Fourier transform to obtain a first spatial domain feature map, and the second mixed amplitude component and the second phase component are subjected to inverse Fourier transform to obtain a second spatial domain feature map.

[0132] Calculate the consistency loss between the first spatial feature map and the second spatial feature map.

[0133] Specifically, to further enhance the semantic consistency between feature maps, a centering transformation is performed on the amplitude component spectrum in the frequency domain, that is, the low-frequency feature information is placed in the central region of the amplitude spectrum, thereby affecting the first fused embedded feature map. Second fusion embedding feature map Local mixing of low-frequency features is achieved through centering transformation. Centering transformation is a signal processing technique whose main purpose is to concentrate the low-frequency information of a signal in the central region of the spectrum for better analysis or processing. In the frequency domain, the spectrum of a signal is usually symmetrical, with low-frequency information located at the four corners. Centering transformation moves this low-frequency information to the center of the spectrum. This transformation typically involves a translation operation of the spectrum, concentrating low-frequency components in the center and moving high-frequency components to the edges. By placing low-frequency information in the central region of the amplitude spectrum, local mixing of low-frequency features in the feature map can be performed more effectively. This method helps achieve better information fusion between feature maps while avoiding excessive information mixing that leads to excessive feature variation. In image processing and computer vision, centering transformation is often used to enhance the semantic consistency between feature maps. To avoid excessive information mixing leading to large feature variations and unlearnable semantic gaps, linear interpolation can be used as a soft mixing strategy. Specifically, the low-frequency feature information of the first amplitude component and the low-frequency feature information of the second amplitude component are linearly interpolated and then fused with the high-frequency feature information of the first amplitude component to obtain the first mixed amplitude component. Similarly, the low-frequency feature information of the second amplitude component and the low-frequency feature information of the first amplitude component are linearly interpolated and then fused with the high-frequency feature information of the second amplitude component to obtain the second mixed amplitude component. The amplitude component exchange process can be represented as follows:

[0134]

[0135]

[0136] in Indicates the first mixed amplitude component. Let λ represent the second mixed amplitude component, λ~U(0,η) represent the linear interpolation coefficients, and η is used to control the mixing intensity of the amplitude components. M represents the mixing window located in the low-frequency region of the amplitude components, used to control the amount of low-frequency information mixed. The linear interpolation strategy is used to control the intensity and range of the mixing to ensure that the mixed feature map still retains learnable semantic information.

[0137] After mixing the low-frequency amplitude components, the first mixed amplitude component and the first phase component are subjected to inverse Fourier transform to obtain the first spatial domain feature map. Similarly, the second mixed amplitude component and the second phase component are subjected to inverse Fourier transform to obtain the second spatial domain feature map. This process can be represented as follows:

[0138]

[0139]

[0140] and These are the first and second spatial domain feature maps, respectively. It's important to note that a spatial domain feature map refers to a feature map represented in the spatial domain (i.e., the pixel space of the image). Unlike frequency domain feature maps, spatial domain feature maps directly correspond to the pixel values ​​and spatial structure of the image. In image processing, feature maps are typically extracted from the original image to represent certain features or attributes of the image, such as edges, textures, or other patterns. Spatial domain feature maps preserve the location and distribution information of these features in the image space. Spatial domain feature maps can be used to compare and optimize image features in the spatial domain by calculating a consistency loss to minimize the distribution differences at the feature level between sharp and degraded images.

[0141] Specifically, consistency loss L con It can be represented as:

[0142]

[0143] During training, the SAM model provided in this application embodiment can also calculate semantic consistency contrast loss and segmentation loss to update the weight parameters of the feature fusion neural network and the multilayer perceptron. Figure 3 This is a schematic diagram of the SAM model architecture and training process provided in the embodiments of this application, such as... Figure 3 As shown, training methods for image segmentation models also include:

[0144] The first predicted segmentation mask feature map is obtained by performing matrix multiplication between the fused feature map of the sample image and the attention weight vectors of each channel of the first final feature map output by the multilayer perceptron. The second predicted segmentation mask feature map is obtained by performing matrix multiplication between the fused feature map of the degraded image and the attention weight vectors of each channel of the second final feature map output by the multilayer perceptron.

[0145] Based on the first predicted segmentation mask feature map and the second predicted segmentation mask feature map, calculate the semantic consistency comparison loss between the first predicted segmentation mask feature map, the second predicted segmentation mask feature map and the ground annotation of the sample image and the degraded image, and calculate the segmentation loss between the second predicted segmentation mask feature map and the ground annotation.

[0146] The weight parameters of the feature fusion neural network and multilayer perceptron are updated based on distance loss, consistency loss, semantic consistency comparison loss, and segmentation loss.

[0147] Specifically, such as Figure 3As shown, the Transformer decoder of the mask decoder processes the cue vector output by the cue encoder and the final feature map output by the image encoder, and then outputs the feature map to Token to ImageAttention for further processing. The feature map output by Token to ImageAttention is then output to a Multilayer Perceptron (MLP) for processing. The MLP layer outputs the attention weight vectors for each channel of the final feature map, i.e., the tokens. The working principles of the Transformer decoder, Token to ImageAttention, and Multilayer Perceptron can be found in the existing SAM model provided in the embodiments of this application, and will not be elaborated here. After processing the first final feature map and the second final feature map through the above steps, the MLP layer outputs the attention weight vectors for each channel of the first final feature map and the attention weight vectors for each channel of the second final feature map. Figure 3 In this process, Clear Feature Fusion and Degrade Feature Fusion complete the work of the feature fusion neural network. They fuse the feature maps of the sample images. Matrix multiplication is performed on the attention weight vectors of each channel of the first final feature map output by the multilayer perceptron to obtain the first predicted segmentation mask feature map y. c fusion feature maps of degraded images Matrix multiplication is performed on the attention weight vectors of each channel of the second final feature map output by the multilayer perceptron to obtain the second predicted segmentation mask feature map y. d To reduce the pixel-level misclassification rate of degraded images and improve the segmentation performance of the SAM model, a contrastive learning strategy can be used to enhance the semantic consistency between degraded and clear images through InfoNCE loss. The InfoNCE loss can be expressed as:

[0148]

[0149] Where a is the anchor point, p is the positive sample point, n is the negative sample point, and N is the negative sample point. -This is the set of negative sample points. Positive sample points come from the predicted segmentation mask feature map of the clear image, which is the same as the ground truth label at the same position as the anchor point. Negative sample points come from the predicted segmentation mask feature map of the clear image, which is different from the ground truth label at the same position as the anchor point, but is the same as the ground truth label in the predicted segmentation mask feature map of the degraded image. Since the resolution of the generated first and second predicted segmentation mask feature maps is lower than that of the original input image, in order to compare the first and second predicted segmentation mask feature maps generated by the model with the ground truth label of the original input image, it is usually necessary to upsample the segmentation mask to the same resolution as the original input image. This ensures that the predicted value at each pixel position corresponds one-to-one with the ground truth label. Specifically, the anchor point comes from the second predicted segmentation mask feature map y of the degraded image. d Set the anchor point to Positive and negative sample points are derived from the first predicted segmentation mask feature map y of the sharp image. c The semantic consistency contrast loss can be obtained as follows:

[0150]

[0151] In the above formula, (m,n) is the location index of the anchor point, and the predicted value at this location is used as a reference when calculating the loss; (p,q) is the location index of the negative sample point, where

[0152] Furthermore, the second predicted segmentation mask feature map y of the degraded image is calculated. d Dice loss L between the actual labeled y and the true labeled y dice and Focal loss L focal As the total segmentation loss L seg , can be represented as:

[0153] L seg =L dice (y d ,y)+L focal (y d ,y).

[0154] Specifically, Dice loss is a loss function based on the Dice coefficient, specifically designed to measure the overlap between two sets. It is particularly effective in handling class imbalance because it directly optimizes the proportion of overlapping regions. Focal loss is an improvement on cross-entropy loss, designed to address class imbalance, especially in object detection tasks. It focuses on difficult samples by reducing the loss weight of easily classified samples. By combining Dice loss and Focal loss, a comprehensive segmentation loss function can be formed to leverage the advantages of both. Dice loss helps the model improve segmentation accuracy in cases of class imbalance, while Focal loss focuses on difficult samples, further enhancing the model's robustness and accuracy. Through this combined strategy, the model can achieve better segmentation performance in various complex scenarios.

[0155] Therefore, based on the distance loss, consistency loss, semantic consistency comparison loss, and segmentation loss, the weight parameters of the feature fusion neural network and the multilayer perceptron are updated, and the total loss value can be calculated using the following formula:

[0156] L total =λ1L seg +λ2L fdl +λ3L con +λ4L sccl ,

[0157] λ1, λ2, λ3, and λ4 are the weights of the loss function. Furthermore, based on the total loss value L... total The weight parameters of the feature fusion neural network and the multilayer perceptron are updated through backpropagation.

[0158] Figure 4 A schematic diagram illustrating a method for enhancing high-frequency and low-frequency feature fusion in a mask decoder using high-pass and low-pass filters, as provided in this application embodiment, is shown below. Figure 4 As shown, in this embodiment, optionally, extracting high-frequency feature information from the early feature map of the degraded image and low-frequency feature information from the late feature map of the degraded image includes:

[0159] Perform convolution and pixel reassembly operations on the late feature map of the degraded image, calculate the weight of each pixel in the late feature map of the degraded image, generate a filter kernel based on the weight of each pixel in the late feature map of the degraded image, and generate an adaptive low-pass filter based on the filter kernel.

[0160] Subtract the filter kernel from the unit kernel to generate an adaptive high-pass filter;

[0161] Low-frequency feature information of the late feature map of the degraded image is extracted by an adaptive low-pass filter, and high-frequency feature information of the early feature map of the degraded image is extracted by an adaptive high-pass filter.

[0162] It should be noted that although the early feature maps extracted by the image encoder contain a wealth of texture and boundary information, and the late feature maps contain global contextual information, simple interpolation or convolutional fusion can smooth the fused features and only capture intermediate features of a fixed frequency, losing the high-frequency and low-frequency feature information of the original features. Therefore, in order to effectively extract semantic information from degraded images, such as... Figure 4 As shown, high-pass and low-pass filters are used to extract high-frequency and low-frequency feature information, respectively. Specifically, for the late feature map F... f This includes low-frequency feature information such as global context. An adaptive low-pass filter is used to preserve the low-frequency feature information in the feature map. For the early feature map F... e This map contains a significant amount of high-frequency feature information, such as boundaries and textures. An adaptive high-pass filter is used to preserve this high-frequency feature information. A low-pass filter is then applied. and high-pass filter It can be expressed as a formula:

[0163]

[0164] In the above formula, φ represents the filter weight generation process, which consists of convolution operation and pixel recombination, and is used to calculate the weight of each position, i.e., each pixel coordinate, in the feature map; The filter kernel size is represented by (i,j); (i,j) represents a point on the feature map in the height and width dimensions; (p,q) represents points in the neighborhood of (i,j); and E is the unit kernel. After the Softmax operation, the weights are constrained to positive and summed to 1, ensuring that the filter output is a weighted average, thus preserving low-frequency information and ensuring a low-pass filter is obtained. To generate the corresponding high-pass filter, the low-pass filter kernel is subtracted from the unit kernel to obtain the high-pass filter. In this embodiment, the convolution operation can be achieved by sliding the convolution kernel on the feature map and generating a new feature representation through dot product with local regions; pixel recombination can be implemented in various ways, such as pooling operations (max pooling, average pooling), upsampling (e.g., deconvolution, interpolation), or other spatial transformations (e.g., spatial attention mechanisms). These operations can adjust the resolution of the feature map or emphasize features in specific regions.

[0165] Optionally, low-frequency feature information of the late feature map of the degraded image is extracted using an adaptive low-pass filter, and high-frequency feature information of the early feature map of the degraded image is extracted using an adaptive high-pass filter, including:

[0166] The first low-frequency feature information of the late feature map of the degraded image is extracted by an adaptive low-pass filter, and the first high-frequency feature information of the early feature map of the degraded image is extracted by an adaptive high-pass filter. The first low-frequency feature information and the first high-frequency feature information are then fused to generate a first fused feature map.

[0167] Based on the first fused feature map, a spatially location-related low-pass filter and a spatially location-related high-pass filter are generated. The weights of the spatially location-related low-pass filter and the spatially location-related high-pass filter change with the pixel position of the first fused feature map.

[0168] The second low-frequency feature information of the late feature map of the degraded image is extracted by a spatially location-correlated low-pass filter, and the second low-frequency feature information is used as the low-frequency feature information of the late feature map of the degraded image. The second high-frequency feature information of the early feature map of the degraded image is extracted by a spatially location-correlated high-pass filter, and the second high-frequency feature information is used as the high-frequency feature information of the early feature map of the degraded image.

[0169] Specifically, the steps for generating the first fused feature map can be expressed by the following formula:

[0170]

[0171] In the above formula, This indicates the use of high-pass or low-pass filters in the convolution operation. In this embodiment, the high-pass filter extracts high-frequency feature information, and the low-pass filter extracts low-frequency feature information; both are accomplished through convolution operations. It should also be noted that... The original early feature maps are added as residuals to the convolution result. This residual connection helps preserve the original feature information and prevents information loss. Based on the same principle, These are the original late-stage features, which are also added as residuals to the convolution result.

[0172] Furthermore, in order to... e and F f To extract higher quality low-frequency and high-frequency feature information, it is possible to start from the initial first fused feature map. Generate spatially correlated low-pass filters Spatial location-dependent high-pass filter Specifically, spatial location dependence of a filter means that the filter's properties (such as weights or responses) change depending on the location of the input signal or image. This location dependence can be achieved in various ways, especially in deep learning and computer vision. Here are some common methods:

[0173] 1. In Convolutional Neural Networks (CNNs), position-dependent filters can be designed with learnable weights that are adjusted based on training data. This approach allows filters to learn different feature responses at different locations.

[0174] 2. Attention Mechanism: The attention mechanism dynamically adjusts the filter's response at different spatial locations. By calculating attention weights, the network can enhance or suppress certain features at specific locations. This mechanism makes the filter more sensitive to features at different locations.

[0175] 3. Conditional Convolution: Conditional convolution (or dynamic convolution) is a method that dynamically generates convolution kernels based on input features or external conditions. This method allows the convolution kernel to have different weights at different positions, thereby achieving positional relevance.

[0176] 4. Location Encoding: Spatial location information can be introduced by explicitly adding location encoding. This encoding can be combined with input features to influence the filter's response, making it more sensitive to different locations.

[0177] 5. Local Adaptive Filtering: Local adaptive filters can be used, whose parameters are adjusted based on local image features (such as gradients and textures). This method allows different filtering properties to be applied to different regions of the image.

[0178] These methods allow filters to exhibit different behaviors in different regions of an image, thus better adapting to and processing local image features. This locational relevance is particularly useful when processing complex image data because it allows models to more flexibly capture and utilize spatial information.

[0179] Therefore, the method of extracting the second low-frequency feature information of the late feature map of the degraded image through a spatially location-dependent low-pass filter, extracting the second high-frequency feature information of the early feature map of the degraded image through a spatially location-dependent high-pass filter, and generating a fused feature map of the degraded image can be expressed by the following formula:

[0180]

[0181] in, This is the fused feature map of the degraded image. This indicates a convolution operation using a high-pass or low-pass filter.

[0182] It should be noted that after training the SAM model using the above method, during model application, only the degraded image needs to be input, and the SAM model can generate a predicted segmentation mask for the degraded image.

[0183] Figure 5This application provides a schematic diagram of a training device for an image segmentation model, the device comprising:

[0184] Feature extraction module 501 is used for:

[0185] Input the sample image, and the degraded image of the sample image, into the image encoder of the image segmentation model;

[0186] High-frequency feature information of early feature maps and low-frequency feature information of late feature maps of sample images are extracted, as are high-frequency feature information of early feature maps and low-frequency feature information of late feature maps of degraded images. The early feature map is the feature map output by the first global attention module in the neural network of the image encoder, and the late feature map is the feature map output by the last layer in the neural network of the image encoder.

[0187] The feature fusion module 502 is used to: fuse the high-frequency feature information of the early feature map of the sample image and the low-frequency feature information of the late feature map of the sample image to obtain a fused feature map of the sample image; and to fuse the high-frequency feature information of the early feature map of the degraded image and the low-frequency feature information of the late feature map of the degraded image through a feature fusion neural network to obtain a fused feature map of the degraded image.

[0188] The fusion feature map of the sample image is embedded and added to the first final feature map to obtain the first fusion embedded feature map. The fusion feature map of the degraded image is embedded and added to the second final feature map to obtain the second fusion embedded feature map. The first final feature map is the feature map converted by the image encoder based on the sample image and output to the mask decoder of the image segmentation model. The second final feature map is the feature map converted by the image encoder based on the degraded image and output to the mask decoder.

[0189] The loss calculation module 503 is used to: calculate the distance loss and consistency loss between the first fused embedded feature map and the second fused embedded feature map in the frequency domain;

[0190] The weight update module 504 is used to update the weight parameters of the multilayer perceptron of the feature fusion neural network and the mask decoder based on the distance loss and consistency loss.

[0191] The vehicle high-voltage electronic fan control device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0192] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6As shown, the electronic device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.

[0193] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.

[0194] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0195] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0196] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0197] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0198] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0199] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0200] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0201] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0202] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0203] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0204] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0205] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0206] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0207] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A training method for an image segmentation model, characterized in that, The method includes: The sample image, and the degraded image of the sample image, are input into the image encoder of the image segmentation model; High-frequency feature information of the early feature map of the sample image and low-frequency feature information of the late feature map of the sample image are extracted, as are high-frequency feature information of the early feature map of the degraded image and low-frequency feature information of the late feature map of the degraded image. The early feature map is the feature map output by the first global attention module in the neural network of the image encoder, and the late feature map is the feature map output by the last layer in the neural network of the image encoder. The high-frequency feature information of the early feature map of the sample image and the low-frequency feature information of the late feature map of the sample image are fused to obtain the fused feature map of the sample image. Furthermore, the high-frequency feature information of the early feature map of the degraded image and the low-frequency feature information of the late feature map of the degraded image are fused through the feature fusion neural network to obtain the fused feature map of the degraded image. The fused feature map of the sample image is embedded and added to the first final feature map to obtain a first fused embedded feature map. The fused feature map of the degraded image is embedded and added to the second final feature map to obtain a second fused embedded feature map. The first final feature map is a feature map converted by the image encoder based on the sample image and output to the mask decoder of the image segmentation model. The second final feature map is a feature map converted by the image encoder based on the degraded image and output to the mask decoder. Calculate the distance loss and consistency loss between the first fused embedded feature map and the second fused embedded feature map in the frequency domain; Based on the distance loss and the consistency loss, update the weight parameters of the feature fusion neural network and the multilayer perceptron of the mask decoder.

2. The method according to claim 1, characterized in that, The calculation of the distance loss and consistency loss between the first fused embedded feature map and the second fused embedded feature map in the frequency domain includes: Fast Fourier transform is performed on the first fused embedding feature map and the second fused embedding feature map respectively to obtain the first amplitude component and the first phase component of the first fused embedding feature map in the frequency domain, and the second amplitude component and the second phase component of the second fused embedding feature map in the frequency domain. Based on the first phase component and the second phase component, calculate the Earth movement distance loss of the first phase component and the second phase component; The consistency loss is calculated based on the first amplitude component and the second amplitude component.

3. The method according to claim 2, characterized in that, The step of calculating the consistency loss based on the first amplitude component and the second amplitude component includes: A centering transformation is performed on the first amplitude component and the second amplitude component, placing the low-frequency feature information of the first amplitude component and the low-frequency feature information of the second amplitude component in the central region of the amplitude spectrum. The low-frequency feature information of the first amplitude component and the second amplitude component are locally mixed to obtain a first mixed amplitude component, and the low-frequency feature information of the second amplitude component and the first amplitude component are locally mixed to obtain a second mixed amplitude component. The first mixed amplitude component and the first phase component are subjected to inverse Fourier transform to obtain a first spatial domain feature map, and the second mixed amplitude component and the second phase component are subjected to inverse Fourier transform to obtain a second spatial domain feature map. Calculate the consistency loss between the first spatial feature map and the second spatial feature map.

4. The method according to claim 3, characterized in that, The step of locally mixing the low-frequency feature information of the first amplitude component and the second amplitude component to obtain a first mixed amplitude component, and locally mixing the low-frequency feature information of the second amplitude component and the first amplitude component to obtain a second mixed amplitude component, includes: The low-frequency feature information of the first amplitude component and the low-frequency feature information of the second amplitude component are linearly interpolated and then fused with the high-frequency feature information of the first amplitude component to obtain the first mixed amplitude component. The low-frequency feature information of the second amplitude component and the low-frequency feature information of the first amplitude component are linearly interpolated and then fused with the high-frequency feature information of the second amplitude component to obtain the second mixed amplitude component.

5. The method according to claim 1, characterized in that, The method further includes: The first predictive segmentation mask feature map is obtained by performing matrix multiplication on the fused feature map of the sample image and the attention weight vectors of each channel of the first final feature map output by the multilayer perceptron. The second predictive segmentation mask feature map is obtained by performing matrix multiplication on the fused feature map of the degraded image and the attention weight vectors of each channel of the second final feature map output by the multilayer perceptron. Based on the first predicted segmentation mask feature map and the second predicted segmentation mask feature map, calculate the semantic consistency comparison loss between the first predicted segmentation mask feature map, the second predicted segmentation mask feature map and the ground truth annotations of the sample image and the degraded image, and calculate the segmentation loss between the second predicted segmentation mask feature map and the ground truth annotations; The weight parameters of the feature fusion neural network and the multilayer perceptron are updated based on the distance loss, the consistency loss, the semantic consistency comparison loss, and the segmentation loss.

6. The method according to claim 1, wherein extracting the high-frequency feature information of the early feature map of the degraded image and the low-frequency feature information of the late feature map of the degraded image comprises: Perform convolution and pixel recombination operations on the late feature map of the degraded image, calculate the weight of each pixel in the late feature map of the degraded image, generate a filter kernel based on the weight of each pixel in the late feature map of the degraded image, and generate an adaptive low-pass filter based on the filter kernel. Subtract the filter core from the unit core to generate an adaptive high-pass filter; The adaptive low-pass filter extracts low-frequency feature information from the late feature map of the degraded image, and the adaptive high-pass filter extracts high-frequency feature information from the early feature map of the degraded image.

7. The method according to claim 6, characterized in that, The step of extracting low-frequency feature information of the late feature map of the degraded image through the adaptive low-pass filter and extracting high-frequency feature information of the early feature map of the degraded image through the adaptive high-pass filter includes: The first low-frequency feature information of the late feature map of the degraded image is extracted by the adaptive low-pass filter, and the first high-frequency feature information of the early feature map of the degraded image is extracted by the adaptive high-pass filter. The first low-frequency feature information and the first high-frequency feature information are then fused to generate a first fused feature map. Based on the first fused feature map, a spatially location-related low-pass filter and a spatially location-related high-pass filter are generated. The weights of the spatially location-related low-pass filter and the spatially location-related high-pass filter change with the pixel position of the first fused feature map. The second low-frequency feature information of the late feature map of the degraded image is extracted by the spatially location-correlated low-pass filter, and the second low-frequency feature information is used as the low-frequency feature information of the late feature map of the degraded image. The second high-frequency feature information of the early feature map of the degraded image is extracted by the spatially location-correlated high-pass filter, and the second high-frequency feature information is used as the high-frequency feature information of the early feature map of the degraded image.

8. A training device for an image segmentation model, characterized in that, The device includes: The feature extraction module is used for: The sample image, and the degraded image of the sample image, are input into the image encoder of the image segmentation model; High-frequency feature information of the early feature map of the sample image and low-frequency feature information of the late feature map of the sample image are extracted, as are high-frequency feature information of the early feature map of the degraded image and low-frequency feature information of the late feature map of the degraded image. The early feature map is the feature map output by the first global attention module in the neural network of the image encoder, and the late feature map is the feature map output by the last layer in the neural network of the image encoder. The feature fusion module is used to: fuse high-frequency feature information of the early feature map of the sample image and low-frequency feature information of the late feature map of the sample image to obtain a fused feature map of the sample image; and to fuse high-frequency feature information of the early feature map of the degraded image and low-frequency feature information of the late feature map of the degraded image through the feature fusion neural network to obtain a fused feature map of the degraded image. The fused feature map of the sample image is embedded and added to the first final feature map to obtain a first fused embedded feature map. The fused feature map of the degraded image is embedded and added to the second final feature map to obtain a second fused embedded feature map. The first final feature map is a feature map converted by the image encoder based on the sample image and output to the mask decoder of the image segmentation model. The second final feature map is a feature map converted by the image encoder based on the degraded image and output to the mask decoder. The loss calculation module is used to: calculate the distance loss and consistency loss between the first fused embedded feature map and the second fused embedded feature map in the frequency domain; The weight update module is used to update the weight parameters of the feature fusion neural network and the multilayer perceptron of the mask decoder based on the distance loss and the consistency loss.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images