A cross-domain semantic segmentation system and method based on a diffusion model

Abstract

The application discloses a cross-domain semantic segmentation system based on a diffusion model, which comprises a backbone network, an FPN network, an encoding disturbance module, a style projector, a mean adjustment module, a variance adjustment module, a feature fusion module and a mask decoder; the backbone network and the encoding disturbance module input a target image; the encoding disturbance module outputs an initial noise feature map; the backbone network is connected with the FPN network; the FPN network outputs multi-scale features of the target image; the backbone network and / or the FPN network output the lth layer image feature map to the i th style projector; the i th style projector outputs the lth layer image style feature map to the mean and variance adjustment module; the mean and variance adjustment module correspondingly outputs the mean and variance of the style feature; the feature fusion module respectively inputs signals from the encoding disturbance module, the FPN network, the mean and variance adjustment module, generates a mixed feature map to the mask decoder, and obtains a predicted mask. The application solves the problem of low target semantic segmentation accuracy caused by domain deviation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine learning, and in particular to a cross-domain semantic segmentation system and method based on a diffusion model. BACKGROUND

[0002] Currently, the task of semantic segmentation is to classify each pixel of an image to achieve accurate segmentation of different objects. The accuracy of this task depends on the model classifying each pixel to generate accurate semantic labels for each part of the image. Compared with classification or detection tasks, semantic segmentation requires more fine-grained analysis because it not only identifies objects but also determines their precise boundaries. Due to the complexity of images in real-world scenarios, such as changes in lighting, different perspectives, occlusions, and other factors, the generalization ability of semantic segmentation models faces higher requirements. The model must be able to accurately recognize and segment target objects under different scenarios and conditions. This puts higher requirements on the training data, data augmentation methods, and network architecture of the model.

[0003] Existing methods often rely on domain adaptation techniques to address domain shift issues. While this approach can alleviate the distribution difference between source and target domains to some extent, it often relies on a large amount of labeled data and performs poorly in unlabeled scenarios. By adopting a domain generalization strategy, the model's generalization ability can be improved without target domain data, effectively addressing the distribution difference between different domains, especially in unsupervised or weakly supervised scenarios. However, existing methods mostly focus on global feature alignment or adversarial training, failing to fully consider the differences in fine-grained features, resulting in poor performance when dealing with complex scenarios or significant local structure changes, leading to poor detail capture ability of the model in practical applications. Most semantic segmentation models assume that the test data and training data have the same distribution, but in practical applications, this assumption often does not hold. Especially in open environments, models often face domain shift problems. This shift can be caused by semantic differences (such as unseen classes) or environmental differences (such as samples from different scenarios or styles). This makes semantic segmentation more complex, and the model must have good generalization ability to adapt to changing scenarios and challenges.

[0004] In summary, to achieve more extensive application scenarios, the model must have stronger adaptability. Not only does it need to perform well on standard segmentation tasks, but it also needs to be flexible in dealing with different visual styles and environmental conditions. Therefore, for the cross-domain semantic segmentation task, in addition to considering the model's generalization ability across domains, it is also important to focus on the handling of domain differences, such as changes in visual style and environmental diversity. Only in this way can the model ensure its robustness and efficiency in practical applications, meeting the needs of complex and variable scenarios. SUMMARY

[0005] The present invention provides a cross-domain semantic segmentation system and method based on a diffusion model to solve the technical problems existing in the known technology.

[0006] The technical solution adopted by the present invention to solve the technical problems existing in the known technology is:

[0007] A cross-domain semantic segmentation system based on a diffusion model, including a backbone network, an FPN network, a coding perturbation module, a style projector, a mean adjustment module, a variance adjustment module, a feature fusion module, and a mask decoder;

[0008] The backbone network is used to extract the feature map of the target image from bottom to top;

[0009] The FPN network is used to fuse the low-level feature maps and high-level feature maps in the feature maps extracted by the backbone network to generate multi-scale feature maps;

[0010] The encoding perturbation module is used to add Gaussian noise to the target image to generate an initial noise feature map;

[0011] The style projector is used to extract the style features of each layer of image from the multi-layer image features to obtain the multi-layer style features of the image;

[0012] The mean adjustment module is used to extract the mean information of each channel in each layer from the multi-layer style features and expand the size to make it correspond to the size of the initial noise feature;

[0013] The variance adjustment module is used to extract the variance information of each layer and channel from the multi-layer style features and expand the size to make it correspond to the size of the initial noise feature;

[0014] The feature fusion module is used to combine the mean and variance of the image style features of each layer with the initial noise feature map of the image; and connect it with the multi-scale feature map generated by the FPN network to obtain a hybrid feature map;

[0015] The backbone network and the encoding perturbation module input the target image respectively; the encoding perturbation module outputs the initial noise feature map; the backbone network outputs the N-layer image feature map to the FPN network; the FPN network outputs the multi-scale features of the target image; the backbone network and / or the FPN network outputs the l-th layer image feature map to the i-th style projector, i=l≤N; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the mean adjustment module and the variance adjustment module; the mean adjustment module outputs the mean of the style features of the first to N-th layer images; the variance adjustment module outputs the variance of the style features of the first to N-th layer images; the feature fusion module inputs the output signals from the encoding perturbation module, the FPN network, the mean adjustment module and the variance adjustment module respectively to generate a mixed feature map; the mask decoder inputs the mixed feature map generated by the feature fusion module and outputs a predicted mask.

[0016] Furthermore, the FPN network outputs the l-th layer image feature map to the i-th style projector, and the i-th style projector includes a mean calculation module, a first linear layer, a first activation layer, and a second linear layer connected in sequence: the mean calculation module calculates the average value of the input data; the first and second linear layers are used to map the input data to the output through a linear transformation, and the first activation layer is used to process the input data through a nonlinear activation function and output it; the FPN network outputs the l-th layer image feature map to the mean calculation module of the i-th style projector, and the second linear layer of the i-th style projector outputs the style feature map of the l-th layer image.

[0017] Furthermore, the mean adjustment module includes a second activation layer and a feature map mean module connected in sequence; the second activation layer is used to process and output the input data through a nonlinear activation function, and the feature map mean module is used to calculate the mean of the feature map; the variance adjustment module includes a third activation layer and a feature map variance module connected in sequence; the third activation layer is used to process and output the input data through a nonlinear activation function, and the feature map variance module is used to calculate the variance of the feature map; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the second activation layer of the mean adjustment module and the third activation layer of the variance adjustment module; the feature map mean module of the mean adjustment module outputs the style feature mean of the first to N-th layer images; the feature map variance module of the variance adjustment module outputs the style feature variance of the first to N-th layer images.

[0018] Furthermore, the feature fusion module includes a first matrix multiplier and a first matrix adder connected in sequence, the first matrix multiplier is used to multiply two matrices; the first matrix adder is used to add two matrices; the first matrix multiplier inputs the variance matrix of the image style features of each layer of images and the initial noise feature map, multiplies the two and outputs them to the first input end of the first matrix adder, the second input end of the first matrix adder inputs the mean matrix of the image style features of each layer of images, and the first matrix adder outputs the combined feature map.

[0019] The present invention also provides a cross-domain semantic segmentation method based on a diffusion model, which constructs a cross-domain semantic segmentation model; the cross-domain semantic segmentation model is provided with a backbone network, an FPN network, a coding perturbation module, a style projector, a mean adjustment module, a variance adjustment module, a feature fusion module and a mask decoder;

[0020] The backbone network is used to extract the feature map of the target image from bottom to top;

[0021] The FPN network is used to fuse the low-level feature maps and high-level feature maps in the feature maps extracted by the backbone network to generate multi-scale feature maps;

[0022] The encoding perturbation module is used to add Gaussian noise to the target image to generate an initial noise feature map;

[0023] The style projector is used to extract the style features of each layer of image from the multi-layer image features to obtain the multi-layer style features of the image;

[0024] The mean adjustment module is used to extract the mean information of each channel in each layer from the multi-layer style features and expand the size to make it correspond to the size of the initial noise feature;

[0025] The variance adjustment module is used to extract the variance information of each layer and channel from the multi-layer style features and expand the size to make it correspond to the size of the initial noise feature;

[0026] The feature fusion module is used to combine the mean and variance of the image style features of each layer with the initial noise feature map of the image; and connect it with the multi-scale feature map generated by the FPN network to obtain a hybrid feature map;

[0027] The backbone network and the coding perturbation module are respectively input to the target image; the coding perturbation module outputs the initial noise feature map; the backbone network outputs the N-layer image feature map to the FPN network; the FPN network outputs the multi-scale features of the target image; the backbone network and / or the FPN network outputs the l-th layer image feature map to the i-th style projector, i=l≤N; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the mean adjustment module and the variance adjustment module; the mean adjustment module outputs the style feature mean of the first to N-th layer images; the variance adjustment module outputs the style feature variance of the first to N-th layer images; the feature fusion module respectively inputs the output signals from the coding perturbation module, the FPN network, the mean adjustment module and the variance adjustment module to generate a mixed feature map; the mask decoder inputs the mixed feature map generated by the feature fusion module and outputs a predicted mask.

[0028] Furthermore, the method for training a cross-domain semantic segmentation model includes the following steps: the input data needs to be properly preprocessed before being input into the network. The preprocessing methods include data enhancement strategies such as rotation, cropping, adjusting sharpness and brightness, Fourier transform, motion blur, and grayscale adjustment; 40,000 rounds of training are performed using the AdamW optimizer; the initial learning rate is set to 0.00006, and a poly learning rate scheduler with a default factor of 1.0 is used.

[0029] Furthermore, the coding perturbation module adds Gaussian noise to the target image in T steps to generate an initial noise feature map of the target image.

[0030] Furthermore, the FPN network outputs the l-th layer image feature map to the i-th style projector. The calculation formula of the i-th style projector is as follows:

[0031] F style =Linear2(ReLU(Linear1(Avg(F l ))));

[0032] Where:

[0033] F style is the style feature map of the l-th layer image output by the i-th style projector;

[0034] F l It is the lth layer image feature data output by the FPN network;

[0035] Avg() represents the mean calculation function;

[0036] Linear1() represents the first linear transformation function;

[0037] ReLU() represents the activation function;

[0038] Linear2() represents the second linear transformation function.

[0039] Furthermore, the calculation formula for calculating the mean value of the style feature by the mean adjustment module is as follows:

[0040] μ adjustment =Linear μ (ReLU(F style ));

[0041] The formula for calculating the style feature variance in the variance adjustment module is as follows:

[0042] σ adjustment =Linear σ (ReLU(F style ));

[0043] Where:

[0044] F style It is a stylistic feature;

[0045] Linear μ () represents the mean calculation function of feature map data;

[0046] Linear σ () represents the variance calculation function of feature map data;

[0047] ReLU() represents the activation function;

[0048] μ adjustment is the mean of the image style features;

[0049] σ adjustment is the variance of the image style feature.

[0050] Furthermore, when the feature fusion module combines the mean and variance of the image style features of each layer with the initial noise feature map of the image, the noise image calculation formula of the feature fusion module is as follows:

[0051] F adjusted =μ adjustment +F noise ×σ adjustment ;

[0052] Where:

[0053] μ adjustment is the mean of the image style features;

[0054] σ adjustment is the variance of the image style feature;

[0055] F noise is the initial noise characteristic;

[0056] F adjusted is the adjusted noise characteristic.

[0057] The advantages and positive effects of the present invention are:

[0058] 1. This paper proposes a semantic segmentation method based on a diffusion model. This method leverages the advantages of the conditional diffusion model in controllable image generation and, by incorporating its properties, achieves more efficient semantic segmentation. The trained model demonstrates strong versatility and robustness across data from different domains, adapting to the needs of a variety of complex scenarios.

[0059] 2. This paper proposes a cross-domain feature enhancement method that effectively addresses the domain distribution discrepancy between training data and target images. By perturbing the channel statistics of source domain image features while preserving the internal structure of the image, it flexibly captures and expresses various latent patterns, improving the model's adaptability and performance in semantic segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a schematic diagram of the workflow of a cross-domain semantic segmentation system based on a diffusion model of the present invention.

[0061] In the figure: FPN represents FPN network; T-steps represents T steps; P represents style projector. DETAILED DESCRIPTION

[0062] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0063] The Chinese meanings of the following English words, phrases and abbreviations are as follows:

[0064] The FPN (Feature Pyramid Network) is a deep learning network structure used for computer vision tasks such as object detection and semantic segmentation. It solves multi-scale problems by constructing a feature pyramid, improving the model's ability to detect objects of different scales.

[0065] AdamW: A variant of the Adam optimizer, primarily used for training deep learning models. AdamW stands for "Decoupled Weight Decay Regularization." AdamW adds weight decay to Adam to prevent overfitting.

[0066] See Figure 1, a cross-domain semantic segmentation system based on the diffusion model, including a backbone network, an FPN network, a coding perturbation module, a style projector, a mean adjustment module, a variance adjustment module, a feature fusion module and a mask decoder;

[0067] The backbone network is used to extract the feature map of the target image from bottom to top;

[0068] The FPN network is used to fuse the low-level feature maps and high-level feature maps in the feature maps extracted by the backbone network to generate multi-scale feature maps;

[0069] The encoding perturbation module is used to add Gaussian noise to the target image to generate an initial noise feature map;

[0070] The style projector is used to extract the style features of each layer of image from the multi-layer image features to obtain the multi-layer style features of the image;

[0071] The mean adjustment module is used to extract the mean information of each channel in each layer from the multi-layer style features and expand the size to make it correspond to the size of the initial noise feature;

[0072] The variance adjustment module is used to extract the variance information of each layer and channel from the multi-layer style features and expand the size to make it correspond to the size of the initial noise feature;

[0073] The feature fusion module is used to combine the mean and variance of the image style features of each layer with the initial noise feature map of the image; and connect it with the multi-scale feature map generated by the FPN network to obtain a hybrid feature map;

[0074] The backbone network and the encoding perturbation module input the target image respectively; the encoding perturbation module outputs the initial noise feature map; the backbone network outputs the N-layer image feature map to the FPN network; the FPN network outputs the multi-scale features of the target image; the backbone network and / or the FPN network outputs the l-th layer image feature map to the i-th style projector, i=l≤N; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the mean adjustment module and the variance adjustment module; the mean adjustment module outputs the mean of the style features of the first to N-th layer images; the variance adjustment module outputs the variance of the style features of the first to N-th layer images; the feature fusion module inputs the output signals from the encoding perturbation module, the FPN network, the mean adjustment module and the variance adjustment module respectively to generate a mixed feature map; the mask decoder inputs the mixed feature map generated by the feature fusion module and outputs a predicted mask.

[0075] Preferably, the FPN network outputs the l-th layer image feature map to the i-th style projector, and the i-th style projector may include a mean calculation module, a first linear layer, a first activation layer, and a second linear layer connected in sequence: the mean calculation module calculates the average value of the input data; the first and second linear layers are used to map the input data to the output through a linear transformation, and the first activation layer is used to process the input data through a nonlinear activation function and output it; the FPN network outputs the l-th layer image feature map to the mean calculation module of the i-th style projector, and the second linear layer of the i-th style projector outputs the style feature map of the l-th layer image.

[0076] Preferably, the mean adjustment module may include a second activation layer and a feature map mean module connected in sequence; the second activation layer is used to process and output the input data through a nonlinear activation function, and the feature map mean module is used to calculate the mean of the feature map; the variance adjustment module may include a third activation layer and a feature map variance module connected in sequence; the third activation layer is used to process and output the input data through a nonlinear activation function, and the feature map variance module is used to calculate the variance of the feature map; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the second activation layer of the mean adjustment module and the third activation layer of the variance adjustment module; the feature map mean module of the mean adjustment module outputs the style feature mean of the first to N-th layer images; the feature map variance module of the variance adjustment module outputs the style feature variance of the first to N-th layer images.

[0077] Preferably, the feature fusion module may include a first matrix multiplier and a first matrix adder connected in sequence, the first matrix multiplier is used to multiply two matrices; the first matrix adder is used to add two matrices; the first matrix multiplier inputs the variance matrix of the image style features of each layer of images and the initial noise feature map, multiplies the two and outputs them to the first input end of the first matrix adder, the second input end of the first matrix adder inputs the mean matrix of the image style features of each layer of images, and the first matrix adder outputs the combined feature map.

[0078] The present invention also provides a cross-domain semantic segmentation method based on a diffusion model, which constructs a cross-domain semantic segmentation model; the cross-domain semantic segmentation model is provided with a backbone network, an FPN network, a coding perturbation module, a style projector, a mean adjustment module, a variance adjustment module, a feature fusion module and a mask decoder;

[0079] The backbone network is used to extract the feature map of the target image from bottom to top;

[0080] The FPN network is used to fuse the low-level feature maps and high-level feature maps in the feature maps extracted by the backbone network to generate multi-scale feature maps;

[0081] The encoding perturbation module is used to add Gaussian noise to the target image to generate an initial noise feature map;

[0082] The style projector is used to extract the style features of each layer of image from the multi-layer image features to obtain the multi-layer style features of the image;

[0083] The mean adjustment module is used to extract the mean information of each channel in each layer from the multi-layer style features and expand the size to make it correspond to the size of the initial noise feature;

[0084] The variance adjustment module is used to extract the variance information of each layer and channel from the multi-layer style features and expand the size to make it correspond to the size of the initial noise feature;

[0085] The feature fusion module is used to combine the mean and variance of the image style features of each layer with the initial noise feature map of the image; and connect it with the multi-scale feature map generated by the FPN network to obtain a hybrid feature map;

[0086] The backbone network and the coding perturbation module are respectively input to the target image; the coding perturbation module outputs the initial noise feature map; the backbone network outputs the N-layer image feature map to the FPN network; the FPN network outputs the multi-scale features of the target image; the backbone network and / or the FPN network outputs the l-th layer image feature map to the i-th style projector, i=l≤N; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the mean adjustment module and the variance adjustment module; the mean adjustment module outputs the style feature mean of the first to N-th layer images; the variance adjustment module outputs the style feature variance of the first to N-th layer images; the feature fusion module respectively inputs the output signals from the coding perturbation module, the FPN network, the mean adjustment module and the variance adjustment module to generate a mixed feature map; the mask decoder inputs the mixed feature map generated by the feature fusion module and outputs a predicted mask.

[0087] The trained cross-domain semantic segmentation model can be used as input for the target image set. The mask can be predicted by retrieving the target on the target image set and classifying it pixel by pixel to obtain the final semantic segmentation result.

[0088] Preferably, the method for training a cross-domain semantic segmentation model may include the following method steps: the input data needs to be appropriately preprocessed before being input into the network, and the preprocessing method may include data enhancement strategies such as rotation, cropping, adjusting sharpness and brightness, Fourier transform, motion blur, and grayscale adjustment; the AdamW optimizer can be used for training for 40,000 rounds; the initial learning rate can be set to 0.00006, and a poly learning rate scheduler with a default factor of 1.0 is used.

[0089] Preferably, the coding perturbation module may be configured to add Gaussian noise to the target image in T steps to generate an initial noise feature map of the target image.

[0090] Preferably, the FPN network outputs the l-th layer image feature map to the i-th style projector, and the calculation formula of the i-th style projector can be as follows:

[0091] F style =Linear2(ReLU(Linear1(Avg(F l ))));

[0092] Where:

[0093] F style is the style feature map of the l-th layer image output by the i-th style projector;

[0094] F l It is the lth layer image feature data output by the FPN network;

[0095] Avg() represents the mean calculation function;

[0096] Linear1() represents the first linear transformation function;

[0097] ReLU() represents the activation function;

[0098] Linear2() represents the second linear transformation function.

[0099] Preferably, the calculation formula for the mean value adjustment module to calculate the mean value of the style features may be as follows:

[0100] μ adjustment =Linea r μ(ReLU(F style ));

[0101] The formula for calculating the style feature variance in the variance adjustment module can be as follows:

[0102] σ adjustment =Linear σ (ReLU(F style ));

[0103] Where:

[0104] F style It is a stylistic feature;

[0105] Linear μ () represents the mean calculation function of feature map data;

[0106] Linear σ () represents the variance calculation function of feature map data;

[0107] ReLU() represents the activation function;

[0108] μ adjustment is the mean of the image style features;

[0109] σ adjustment is the variance of the image style feature.

[0110] Preferably, when the input target image passes through the FPN network, the vector is defined Capture the output feature map of the lth layer feature extraction backbone network The channel-by-channel mean and variance of , calculate the feature map The calculation formulas for the mean and variance of each channel data can be as follows:

[0111]

[0112] Where:

[0113] l is the FPN network feature layer number;

[0114] c is the channel number of the feature map;

[0115] is the mean of the feature map data of the cth channel in the lth layer;

[0116] is the variance of the feature map data of the cth channel in the lth layer;

[0117] W is the feature map of the lth layer width;

[0118] H is the feature map of the lth layer height;

[0119] is the feature map data of the cth channel of the lth layer;

[0120] x is the input target image;

[0121] w is the feature map The width unit;

[0122] h is the feature map The height unit.

[0123] Preferably, when the feature fusion module combines the mean and variance of the image style features of each layer of images with the initial noise feature map of the image, the noise image calculation formula of the feature fusion module can be as follows:

[0124] F adjusted =μ adjustment +F noise ×σ adjustment ;

[0125] Where:

[0126] μ adjustment is the mean of the image style features;

[0127] σ adjustment is the variance of the image style feature;

[0128] F noise is the initial noise characteristic;

[0129] F adjusted is the adjusted noise characteristic.

[0130] The noise characteristic F after the above adjustment adjusted The signal output by the FPN network is input to the mask decoder. The mask decoder generates a mask based on the structure of noise + condition. The signal output by the FPN network is used as the conditional signal in the process of restoring the mask, assisting the mask decoder to obtain the adjusted noise feature F. adjusted Generates the output prediction mask.

[0131] The following is a preferred embodiment of the present invention to further illustrate the workflow and working principle of the present invention:

[0132] The semantic segmentation model is trained using the following steps:

[0133] (1) The backbone network is used to extract features from the target image, and the encoding perturbation module generates initial noise features from the target image mask.

[0134] (1) The input data needs to be properly preprocessed before being input into the network. Preprocessing methods include rotation, cropping, adjusting sharpness and brightness, Fourier transform, motion blur, grayscale adjustment and other data enhancement strategies;

[0135] (2) The encoding perturbation module adds Gaussian noise to the input image in T steps until it becomes a noisy image.

[0136] (3) The backbone network processes the input target image and outputs a feature map containing important visual and spatial information.

[0137] (2) Use FPN network to generate multi-scale features.

[0138] The FPN network inputs N layers of image features and generates a feature map of the target image The mean and variance of each channel data are calculated as follows:

[0139]

[0140] Where:

[0141] l is the FPN network feature layer number;

[0142] c is the channel number of the feature map;

[0143] is the mean of the feature map data of the cth channel in the lth layer;

[0144] is the variance of the feature map data of the cth channel in the lth layer;

[0145] W is the feature map of the lth layer width;

[0146] H is the feature map of the lth layer height;

[0147] is the feature map data of the cth channel of the lth layer;

[0148] x is the input target image;

[0149] w is the feature map The width unit;

[0150] h is the feature map The height unit.

[0151] (3) Use style projector to extract style features from multi-scale features.

[0152] The style projector is used to extract the style features of the input image from the multi-level image features. The style feature calculation formula is as follows:

[0153] F style =Linear2(ReLU(Linear(Avg(F l ))));

[0154] Where:

[0155] F style is the style feature map of the l-th layer image output by the i-th style projector;

[0156] F l It is the lth layer image feature data output by the FPN network;

[0157] Avg() represents the mean calculation function;

[0158] Linear1() represents the first linear layer calculation function;

[0159] ReLU() represents the activation layer calculation function;

[0160] Linear2() represents the second linear layer calculation function.

[0161] (4) Use the mean adjustment layer and variance adjustment layer to calculate the mean and variance of the image style features.

[0162] In step 4, the mean adjustment layer and variance adjustment layer calculate the mean and variance from the style features. The calculation formula is as follows:

[0163] μ adjustment =Linear μ (ReLU(F style ));

[0164] σ adjustment =Linear σ (ReLU(F style ));

[0165] Where:

[0166] F style It is a stylistic feature;

[0167] Linear μ () represents the mean calculation function of feature map data;

[0168] Linear σ () represents the variance calculation function of feature map data;

[0169] ReLU() represents the activation layer calculation function;

[0170] μ adjustment is the mean of the image style features;

[0171] σ adjustment is the variance of the image style feature.

[0172] The mean calculation function of the feature map data can be calculated using the following formula:

[0173]

[0174] The variance calculation function of the feature map data can be calculated using the following formula:

[0175]

[0176] Where:

[0177] m is the layer number of the feature map;

[0178] e is the channel number of the feature map;

[0179] is the mean of the c-th channel data of the m-th layer feature map;

[0180] is the variance of the c-th channel data of the m-th layer feature map;

[0181] A is the feature map of the mth layer width;

[0182] B is the feature map of the mth layer height;

[0183] is the e-th channel data of the m-th layer feature map;

[0184] z is the input feature map;

[0185] a is the feature map The width unit;

[0186] b is the feature map The height unit.

[0187] is the feature map.

[0188] (5) The initial noise features output by the coding perturbation module are adjusted using mean and variance, and are input into the mask decoder together with the target image features. The mask decoder outputs the predicted mask.

[0189] In order to narrow the style or domain differences between the target image and the source domain, and between the training and test datasets, and to improve the network's ability to perform semantic segmentation in images of various styles, the present invention adopts a mean and variance adjustment method to narrow such differences.

[0190] Since the statistics of image feature channels (mean and standard deviation) are closely related to image style, changing these channel statistics can be seen as an implicit way to change the style of the input image. The adjusted mean and variance are used to adjust the noisy image output by the encoding perturbation module.

[0191] When the feature fusion module combines the mean and variance of the image style features of each layer with the initial noise feature map of the image, the noise image calculation formula of the feature fusion module is as follows:

[0192] F adjusted =μ adjustment +F noise ×σ adjustment ;

[0193] Where:

[0194] μ adjustment is the mean of the image style features;

[0195] σ adjustment is the variance of the image style feature.

[0196] F noise is the initial noise characteristic;

[0197] F adjusted is the adjusted noise characteristic.

[0198] (6) Obtain the optimal model and use it for semantic segmentation.

[0199] Specifically, training is performed using data from the visible domain. After training, the network can generalize to achieve good semantic segmentation results even on unseen style features. This significantly increases the difficulty and brings it closer to real-world scenarios.

[0200] (1) The present invention uses the AdamW optimizer for training for 40,000 rounds.

[0201] (2) The initial learning rate is set to 0.00006, and the poly learning rate scheduler with a default factor of 1.0 is used.

[0202] (3) Repeat the above steps until the training is completed. Use the trained model to perform semantic segmentation, that is, input the target image set, the model retrieves the target on the target image set and classifies it pixel by pixel to obtain the final semantic segmentation result.

[0203] The above-mentioned neural networks or functional modules such as the backbone network, FPN network, coding perturbation module, style projector, mean adjustment layer, variance adjustment layer, feature fusion module and mask decoder can all adopt applicable neural networks or functional modules in the existing technology, or adopt neural networks or functional modules in the existing technology and construct them using conventional technical means.

[0204] The embodiments described above are only used to illustrate the technical ideas and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of the patent of the present invention cannot be limited by these embodiments alone. That is, any equivalent changes or modifications made to the spirit disclosed by the present invention still fall within the scope of the patent of the present invention.

Claims

1. A cross-domain semantic segmentation system based on a diffusion model, characterized by: Including backbone network, FPN network, encoding perturbation module, style projector, mean adjustment module, variance adjustment module, feature fusion module and mask decoder; The backbone network is used to extract the feature map of the target image from bottom to top; The FPN network is used to fuse the low-level feature maps and high-level feature maps in the feature maps extracted by the backbone network to generate multi-scale feature maps; The encoding perturbation module is used to add Gaussian noise to the target image to generate an initial noise feature map; The style projector is used to extract the style features of each layer of image from the multi-layer image features to obtain the multi-layer style features of the image; The mean adjustment module is used to extract the mean information of each channel in each layer from the multi-layer style features and expand the size to make it correspond to the size of the initial noise feature; The variance adjustment module is used to extract the variance information of each layer and channel from the multi-layer style features and expand the size to make it correspond to the size of the initial noise feature; The feature fusion module is used to combine the mean and variance of the image style features of each layer with the initial noise feature map of the image; and connect it with the multi-scale feature map generated by the FPN network to obtain a hybrid feature map; The backbone network and the encoding perturbation module input the target image respectively; the encoding perturbation module outputs the initial noise feature map; the backbone network outputs the N-layer image feature map to the FPN network; the FPN network outputs the multi-scale features of the target image; The backbone network outputs the l-th layer image feature map to the i-th style projector, i=l≤N; the i-th style projector outputs the style feature map of the l-th layer image to the mean adjustment module and the variance adjustment module respectively; the mean adjustment module outputs the mean of the style features of the first to N-th layer images; The variance adjustment module outputs the style feature variance of the first to Nth layer images; the feature fusion module inputs the output signals from the encoding perturbation module, FPN network, mean adjustment module and variance adjustment module respectively to generate a mixed feature map; the mask decoder inputs the mixed feature map generated by the feature fusion module and outputs a predicted mask; The mean adjustment module includes a second activation layer and a feature map mean module connected in sequence; the second activation layer is used to process the input data through a nonlinear activation function and output it, and the feature map mean module is used to calculate the mean of the feature map; the variance adjustment module includes a third activation layer and a feature map variance module connected in sequence; the third activation layer is used to process the input data through a nonlinear activation function and output it, and the feature map variance module is used to calculate the variance of the feature map; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the second activation layer of the mean adjustment module and the third activation layer of the variance adjustment module; the feature map mean module of the mean adjustment module outputs the mean of the style features of the first to N-th layers of images; The feature map variance module of the variance adjustment module outputs the style feature variance of the first layer to the Nth layer image.

2. The cross-domain semantic segmentation system based on the diffusion model according to claim 1 is characterized in that The FPN network outputs the l-th layer image feature map to the i-th style projector. The i-th style projector includes a mean calculation module, a first linear layer, a first activation layer, and a second linear layer connected in sequence: the mean calculation module calculates the average value of the input data; the first and second linear layers are used to map the input data to the output through a linear transformation, and the first activation layer is used to process the input data through a nonlinear activation function and output it; the FPN network outputs the l-th layer image feature map to the mean calculation module of the i-th style projector, and the second linear layer of the i-th style projector outputs the style feature map of the l-th layer image.

3. The cross-domain semantic segmentation system based on the diffusion model according to claim 1 is characterized in that The feature fusion module includes a first matrix multiplier and a first matrix adder connected in sequence. The first matrix multiplier is used to multiply two matrices; the first matrix adder is used to add two matrices; the first matrix multiplier inputs the variance matrix of the image style features of each layer of images and the initial noise feature map, multiplies the two and outputs them to the first input end of the first matrix adder, the second input end of the first matrix adder inputs the mean matrix of the image style features of each layer of images, and the first matrix adder outputs the combined feature map.

4. A cross-domain semantic segmentation method based on a diffusion model, characterized in that: The method constructs a cross-domain semantic segmentation model; the cross-domain semantic segmentation model is equipped with a backbone network, an FPN network, a coding perturbation module, a style projector, a mean adjustment module, a variance adjustment module, a feature fusion module and a mask decoder; The backbone network is used to extract the feature map of the target image from bottom to top; The FPN network is used to fuse the low-level feature maps and high-level feature maps in the feature maps extracted by the backbone network to generate multi-scale feature maps; The encoding perturbation module is used to add Gaussian noise to the target image to generate an initial noise feature map; The style projector is used to extract the style features of each layer of image from the multi-layer image features to obtain the multi-layer style features of the image; The mean adjustment module is used to extract the mean information of each channel in each layer from the multi-layer style features and expand the size to make it correspond to the size of the initial noise feature; The variance adjustment module is used to extract the variance information of each layer and channel from the multi-layer style features and expand the size to make it correspond to the size of the initial noise feature; The feature fusion module is used to combine the mean and variance of the image style features of each layer with the initial noise feature map of the image; and connect it with the multi-scale feature map generated by the FPN network to obtain a hybrid feature map; The backbone network and the encoding perturbation module are respectively input with the target image; the encoding perturbation module outputs the initial noise feature map; the backbone network outputs the N-layer image feature map to the FPN network; the FPN network outputs the multi-scale features of the target image; The backbone network outputs the l-th layer image feature map to the i-th style projector, i=l≤N; the i-th style projector outputs the style feature map of the l-th layer image to the mean adjustment module and the variance adjustment module respectively; Make the mean adjustment module output the mean of the style features of the first to Nth layer images; The variance adjustment module outputs the style feature variance of the first to Nth layer images; The feature fusion module is respectively input with the output signals from the encoding perturbation module, the FPN network, the mean adjustment module and the variance adjustment module to generate a mixed feature map; the mask decoder is input with the mixed feature map generated by the feature fusion module to output a predicted mask; The mean adjustment module includes a second activation layer and a feature map mean module connected in sequence; the second activation layer is used to process the input data through a nonlinear activation function and output it, and the feature map mean module is used to calculate the mean of the feature map; the variance adjustment module includes a third activation layer and a feature map variance module connected in sequence; the third activation layer is used to process the input data through a nonlinear activation function and output it, and the feature map variance module is used to calculate the variance of the feature map; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the second activation layer of the mean adjustment module and the third activation layer of the variance adjustment module; the feature map mean module of the mean adjustment module outputs the mean of the style features of the first to N-th layers of images; The feature map variance module of the variance adjustment module outputs the style feature variance of the first layer to the Nth layer image.

5. The cross-domain semantic segmentation method based on the diffusion model according to claim 4 is characterized in that The method for training a cross-domain semantic segmentation model includes the following steps: the input data needs to be properly preprocessed before being input into the network. The preprocessing methods include data enhancement strategies such as rotation, cropping, adjusting sharpness and brightness, Fourier transform, motion blur, grayscale adjustment, etc. The AdamW optimizer is used for training for 40,000 epochs; the initial learning rate is set to 0.00006, and the poly learning rate scheduler with a default factor of 1.0 is adopted.

6. The cross-domain semantic segmentation method based on the diffusion model according to claim 4 is characterized in that The encoding perturbation module adds Gaussian noise to the target image in T steps to generate an initial noise feature map of the target image.

7. The cross-domain semantic segmentation method based on the diffusion model according to claim 4 is characterized in that The FPN network outputs the l-th layer image feature map to the i-th style projector. The calculation formula of the i-th style projector is as follows: F style =Linear2(ReLU(Linear1(Avg(F l )))); Where: F style is the style feature map of the l-th layer image output by the i-th style projector; F l It is the lth layer image feature data output by the FPN network; Avg() represents the mean calculation function; Linear1() represents the first linear transformation function; ReLU() represents the activation function; Linear2() represents the second linear transformation function.

8. The cross-domain semantic segmentation method based on the diffusion model according to claim 7 is characterized in that: The calculation formula for the mean adjustment module to calculate the mean of style features is as follows: μ adjustment =Linear μ (ReLU(F style )); The formula for calculating the style feature variance in the variance adjustment module is as follows: σ adjustment =Linear σ (ReLU(F style )); Where: F style It is a stylistic feature; Linear μ () represents the mean calculation function of feature map data; Linear σ () represents the variance calculation function of feature map data; ReLU() represents the activation function; μ adjustment is the mean of the image style features; σ adjustment is the variance of the image style feature.

9. The cross-domain semantic segmentation method based on the diffusion model according to claim 4 is characterized in that When the feature fusion module combines the mean and variance of the image style features of each layer with the initial noise feature map of the image, the noise image calculation formula of the feature fusion module is as follows: F adjusted =μ adjustment +F noise ×s adjustment ; Where: μ adjustment is the mean of the image style features; σ adjustment is the variance of the image style feature; F noise is the initial noise characteristic; F adjusted is the adjusted noise characteristic.

Images