Cross-domain semantic segmentation system and method based on diffusion model

Abstract

The invention discloses a cross-domain semantic segmentation system based on a diffusion model. The cross-domain semantic segmentation system comprises a backbone network, an FPN network, a coding disturbance module, a style projector, a mean value adjustment module, a variance adjustment module, a feature fusion module and a mask decoder. The backbone network and coding disturbance module inputs a target image; the coding disturbance module outputs an initial noise characteristic pattern; the backbone network is connected with the FPN network; the FPN network outputs multi-scale features of the target image; the trunk and / or FPN network outputs the lth layer of image feature map to the ith style projector; the ith style projector outputs a style feature map of the lth layer of image to a mean value and variance adjustment module; the mean value and variance adjusting module correspondingly outputs mean values and variances of style features; and the feature fusion module inputs signals from the coding disturbance module, the FPN network and the mean value and variance adjustment module respectively, generates a mixed feature pattern and sends the mixed feature pattern to the mask decoder to obtain a prediction mask. The problem of low target semantic segmentation precision caused by domain migration is solved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to a cross-domain semantic segmentation system and method based on a diffusion model. Background Art

[0002] Currently, the task of semantic segmentation is to classify each pixel of an image in order to achieve accurate segmentation of different objects. The accuracy of this task relies on the model to classify each pixel in order to generate accurate semantic labels for each part in the image. Compared with classification or detection tasks, semantic segmentation requires a more fine-grained analysis because it not only identifies objects but also determines their precise boundaries. Due to the complexity of images in real scenes, such as lighting changes, different perspectives, occlusions, etc., the generalization ability of semantic segmentation models faces higher requirements. The model must be able to accurately identify and segment target objects in different scenarios and conditions. This places higher requirements on the model's training data, data enhancement methods, and network architecture.

[0003] Existing methods usually need to rely on domain adaptation technology to solve the domain shift problem. Although this approach can alleviate the distribution difference between the source domain and the target domain to a certain extent, it often relies on a large amount of labeled data and has limited performance in unlabeled scenarios. By adopting the strategy of domain generalization, the generalization ability of the model can be improved without the need for target domain data, and the distribution difference between different domains can be effectively dealt with, especially in unsupervised or weakly supervised scenarios. The advantages of this method are particularly obvious. However, most existing methods focus on global feature alignment or adversarial training, and fail to fully consider the differences in fine-grained features, which results in insufficient performance when dealing with complex scenes or significant changes in local structures, resulting in poor ability of the model to capture details in practical applications. Most semantic segmentation models assume that the test data and training data have the same distribution, but in practical applications, this assumption is often not true. Especially in open environments, models often face domain shift problems. This shift may be caused by semantic differences (such as unseen categories) or environmental differences (such as samples from different scenes or styles). This makes semantic segmentation more complicated, and the model must have good generalization ability to adapt to changing scenarios and challenges.

[0004] In summary, in order to achieve a wider range of application scenarios, the model must have stronger adaptability. Not only does it need to perform well on standard segmentation tasks, but it must also be able to flexibly respond to different visual styles and environmental conditions. Therefore, for cross-domain semantic segmentation tasks, in addition to considering the generalization ability of the model between domains, it is also necessary to focus on the processing of inter-domain differences, such as changes in visual style and diversity of the environment. Only in this way can we ensure the robustness and efficiency of the model in practical applications and meet the needs of complex and changing scenarios. Summary of the invention

[0005] The present invention provides a cross-domain semantic segmentation system and method based on a diffusion model to solve the technical problems existing in the known technology.

[0006] The technical solution adopted by the present invention to solve the technical problems existing in the known technology is:

[0007] A cross-domain semantic segmentation system based on a diffusion model, including a backbone network, an FPN network, a coding perturbation module, a style projector, a mean adjustment module, a variance adjustment module, a feature fusion module and a mask decoder;

[0008] The backbone network is used to extract the feature map of the target image from bottom to top;

[0009] The FPN network is used to fuse the low-level feature maps and high-level feature maps in the feature maps extracted by the backbone network to generate multi-scale feature maps;

[0010] The encoding perturbation module is used to add Gaussian noise to the target image to generate an initial noise feature map;

[0011] The style projector is used to extract the style features of each layer of images from the multi-layer image features to obtain the multi-layer style features of the image;

[0012] The mean adjustment module is used to extract the mean information of each channel in each layer from the multi-layer style features, and expand the size to make it correspond to the size of the initial noise feature;

[0013] The variance adjustment module is used to extract the variance information of each layer and each channel from the multi-layer style features, and expand the size to make it correspond to the size of the initial noise feature;

[0014] The feature fusion module is used to combine the mean and variance of the image style features of each layer of images with the initial noise feature map of the image; and connect it with the multi-scale feature map generated by the FPN network to obtain a mixed feature map;

[0015] The backbone network and the coding perturbation module input the target image respectively; the coding perturbation module outputs the initial noise feature map; the backbone network outputs the N-layer image feature map to the FPN network; the FPN network outputs the multi-scale features of the target image; the backbone network and / or the FPN network outputs the l-th layer image feature map to the i-th style projector, i=l≤N; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the mean adjustment module and the variance adjustment module; the mean adjustment module outputs the mean of the style features of the first to N-th layer images; the variance adjustment module outputs the variance of the style features of the first to N-th layer images; the feature fusion module inputs the output signals from the coding perturbation module, the FPN network, the mean adjustment module and the variance adjustment module respectively to generate a mixed feature map; the mask decoder inputs the mixed feature map generated by the feature fusion module and outputs a predicted mask.

[0016] Furthermore, the FPN network outputs the l-th layer image feature map to the i-th style projector, and the i-th style projector includes a mean calculation module, a first linear layer, a first activation layer, and a second linear layer connected in sequence: the mean calculation module calculates the average value of the input data; the first and second linear layers are used to map the input data to the output through a linear transformation, and the first activation layer is used to process the input data through a nonlinear activation function and output it; the FPN network outputs the l-th layer image feature map to the mean calculation module of the i-th style projector, and the second linear layer of the i-th style projector outputs the style feature map of the l-th layer image.

[0017] Furthermore, the mean adjustment module includes a second activation layer and a feature map mean module connected in sequence; the second activation layer is used to process and output the input data through a nonlinear activation function, and the feature map mean module is used to calculate the mean of the feature map; the variance adjustment module includes a third activation layer and a feature map variance module connected in sequence; the third activation layer is used to process and output the input data through a nonlinear activation function, and the feature map variance module is used to calculate the variance of the feature map; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the second activation layer of the mean adjustment module and the third activation layer of the variance adjustment module; the feature map mean module of the mean adjustment module outputs the mean of the style features of the first to N-th layers of images; the feature map variance module of the variance adjustment module outputs the variance of the style features of the first to N-th layers of images.

[0018] Furthermore, the feature fusion module includes a first matrix multiplier and a first matrix adder connected in sequence, the first matrix multiplier is used to multiply two matrices; the first matrix adder is used to add two matrices; the first matrix multiplier inputs the variance matrix of the image style features of each layer of images and the initial noise feature map, multiplies the two and outputs them to the first input end of the first matrix adder, the second input end of the first matrix adder inputs the mean matrix of the image style features of each layer of images, and the first matrix adder outputs the combined feature map.

[0019] The present invention also provides a cross-domain semantic segmentation method based on a diffusion model, which constructs a cross-domain semantic segmentation model; a backbone network, an FPN network, a coding perturbation module, a style projector, a mean adjustment module, a variance adjustment module, a feature fusion module and a mask decoder are set in the cross-domain semantic segmentation model;

[0020] The backbone network is used to extract the feature map of the target image from bottom to top;

[0021] The FPN network is used to fuse the low-level feature maps and high-level feature maps in the feature maps extracted by the backbone network to generate multi-scale feature maps;

[0022] The encoding perturbation module is used to add Gaussian noise to the target image to generate an initial noise feature map;

[0023] The style projector is used to extract the style features of each layer of images from the multi-layer image features to obtain the multi-layer style features of the image;

[0024] The mean adjustment module is used to extract the mean information of each channel in each layer from the multi-layer style features, and expand the size to make it correspond to the size of the initial noise feature;

[0025] The variance adjustment module is used to extract the variance information of each layer and each channel from the multi-layer style features, and expand the size to make it correspond to the size of the initial noise feature;

[0026] The feature fusion module is used to combine the mean and variance of the image style features of each layer of images with the initial noise feature map of the image; and connect it with the multi-scale feature map generated by the FPN network to obtain a mixed feature map;

[0027] The backbone network and the coding perturbation module are respectively input with the target image; the coding perturbation module outputs the initial noise feature map; the backbone network outputs the N-layer image feature map to the FPN network; the FPN network outputs the multi-scale features of the target image; the backbone network and / or the FPN network outputs the l-th layer image feature map to the i-th style projector, i=l≤N; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the mean adjustment module and the variance adjustment module; the mean adjustment module outputs the style feature mean of the first to N-th layer images; the variance adjustment module outputs the style feature variance of the first to N-th layer images; the feature fusion module is respectively input with the output signals from the coding perturbation module, the FPN network, the mean adjustment module and the variance adjustment module to generate a mixed feature map; the mask decoder is input with the mixed feature map generated by the feature fusion module to output a predicted mask.

[0028] Furthermore, the method for training a cross-domain semantic segmentation model includes the following steps: the input data needs to be properly preprocessed before being input into the network, and the preprocessing methods include data enhancement strategies such as rotation, cropping, adjusting sharpness and brightness, Fourier transform, motion blur, and grayscale adjustment; the AdamW optimizer is used for training for 40,000 rounds; the initial learning rate is set to 0.00006, and the poly learning rate scheduler with a default factor of 1.0 is used.

[0029] Furthermore, the coding perturbation module is configured to add Gaussian noise to the target image in T steps to generate an initial noise feature map of the target image.

[0030] Furthermore, the FPN network outputs the l-th layer image feature map to the i-th style projector, and the calculation formula of the i-th style projector is as follows:

[0031] F style =Linear2(ReLU(Linear1(Avg(F l ))));

[0032] Where:

[0033] F style It is the style feature map of the l-th layer image output by the i-th style projector;

[0034] F l It is the lth layer image feature data output by the FPN network;

[0035] Avg() represents the mean calculation function;

[0036] Linear1() represents the first linear transformation function;

[0037] ReLU() represents the activation function;

[0038] Linear2() represents the second linear transformation function.

[0039] Furthermore, the calculation formula for calculating the mean value of the style feature by the mean adjustment module is as follows:

[0040] μ adjustment =Linear μ (ReLU(F style ));

[0041] The formula for calculating the style feature variance by the variance adjustment module is as follows:

[0042] σ adjustment =Linear σ (ReLU(F style ));

[0043] Where:

[0044] F style It is a style feature;

[0045] Linear μ () represents the mean calculation function of feature map data;

[0046] Linear σ () represents the variance calculation function of feature map data;

[0047] ReLU() represents the activation function;

[0048] μ adjustment is the mean of the image style features;

[0049] σ adjustment is the variance of the image style feature.

[0050] Furthermore, when the feature fusion module combines the mean and variance of the image style features of each layer of the image with the initial noise feature map of the image, the noise image calculation formula of the feature fusion module is as follows:

[0051] F adjusted =μ adjustment +F noise ×σ adjustment ;

[0052] Where:

[0053] μ adjustment is the mean of the image style features;

[0054] σ adjustment is the variance of the image style feature;

[0055] F noise is the initial noise characteristic;

[0056] F adjusted is the adjusted noise characteristic.

[0057] The advantages and positive effects of the present invention are:

[0058] 1. This paper proposes a semantic segmentation method based on a diffusion model, which fully utilizes the advantages of the conditional diffusion model in controllable image generation and achieves more efficient semantic segmentation by introducing its characteristics. The trained model has strong versatility and robustness on different domain data and can adapt to the needs of various complex scenarios.

[0059] 2. This paper proposes a cross-domain feature enhancement method, which effectively solves the problem of domain distribution difference between training data and target images. Under the premise of keeping the internal structure of the image unchanged, by perturbing the channel statistics of the source domain image features, various potential styles can be flexibly captured and expressed, thereby improving the adaptability and performance of the model in semantic segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of the workflow of a cross-domain semantic segmentation system based on a diffusion model of the present invention.

[0061] In the figure: FPN represents FPN network; T-steps represents T steps; P represents style projector. DETAILED DESCRIPTION

[0062] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0063] The Chinese meanings of the following English words, English phrases and English abbreviations are as follows:

[0064] FPN network: (Feature Pyramid Network) is a deep learning network structure used for computer vision tasks such as target detection and semantic segmentation. It solves multi-scale problems by constructing a feature pyramid and improves the model's ability to detect targets of different scales.

[0065] AdamW: It is a variant of the Adam optimizer, mainly used for deep learning model training. The full name of AdamW is "Decoupled Weight Decay Regularization". AdamW adds a weight decay function based on Adam to prevent overfitting.

[0066] See also Figure 1, a cross-domain semantic segmentation system based on a diffusion model, including a backbone network, an FPN network, a coding perturbation module, a style projector, a mean adjustment module, a variance adjustment module, a feature fusion module and a mask decoder;

[0067] The backbone network is used to extract the feature map of the target image from bottom to top;

[0068] The FPN network is used to fuse the low-level feature maps and high-level feature maps in the feature maps extracted by the backbone network to generate multi-scale feature maps;

[0069] The encoding perturbation module is used to add Gaussian noise to the target image to generate an initial noise feature map;

[0070] The style projector is used to extract the style features of each layer of images from the multi-layer image features to obtain the multi-layer style features of the image;

[0071] The mean adjustment module is used to extract the mean information of each channel in each layer from the multi-layer style features, and expand the size to make it correspond to the size of the initial noise feature;

[0072] The variance adjustment module is used to extract the variance information of each layer and each channel from the multi-layer style features, and expand the size to make it correspond to the size of the initial noise feature;

[0073] The feature fusion module is used to combine the mean and variance of the image style features of each layer of images with the initial noise feature map of the image; and connect it with the multi-scale feature map generated by the FPN network to obtain a mixed feature map;

[0074] The backbone network and the coding perturbation module input the target image respectively; the coding perturbation module outputs the initial noise feature map; the backbone network outputs the N-layer image feature map to the FPN network; the FPN network outputs the multi-scale features of the target image; the backbone network and / or the FPN network outputs the l-th layer image feature map to the i-th style projector, i=l≤N; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the mean adjustment module and the variance adjustment module; the mean adjustment module outputs the mean of the style features of the first to N-th layer images; the variance adjustment module outputs the variance of the style features of the first to N-th layer images; the feature fusion module inputs the output signals from the coding perturbation module, the FPN network, the mean adjustment module and the variance adjustment module respectively to generate a mixed feature map; the mask decoder inputs the mixed feature map generated by the feature fusion module and outputs a predicted mask.

[0075] Preferably, the FPN network outputs the l-th layer image feature map to the i-th style projector, and the i-th style projector may include a mean calculation module, a first linear layer, a first activation layer, and a second linear layer connected in sequence: the mean calculation module calculates the average value of the input data; the first and second linear layers are used to map the input data to the output through a linear transformation, and the first activation layer is used to process the input data through a nonlinear activation function and output it; the FPN network outputs the l-th layer image feature map to the mean calculation module of the i-th style projector, and the second linear layer of the i-th style projector outputs the style feature map of the l-th layer image.

[0076] Preferably, the mean adjustment module may include a second activation layer and a feature map mean module connected in sequence; the second activation layer is used to process and output the input data through a nonlinear activation function, and the feature map mean module is used to calculate the mean of the feature map; the variance adjustment module may include a third activation layer and a feature map variance module connected in sequence; the third activation layer is used to process and output the input data through a nonlinear activation function, and the feature map variance module is used to calculate the variance of the feature map; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the second activation layer of the mean adjustment module and the third activation layer of the variance adjustment module; the feature map mean module of the mean adjustment module outputs the mean of the style features of the first to N-th layers of images; the feature map variance module of the variance adjustment module outputs the variance of the style features of the first to N-th layers of images.

[0077] Preferably, the feature fusion module may include a first matrix multiplier and a first matrix adder connected in sequence, the first matrix multiplier is used to multiply two matrices; the first matrix adder is used to add two matrices; the first matrix multiplier inputs the variance matrix of the image style features of each layer of images and the initial noise feature map, multiplies the two and outputs them to the first input end of the first matrix adder, the second input end of the first matrix adder inputs the mean matrix of the image style features of each layer of images, and the first matrix adder outputs the combined feature map.

[0078] The present invention also provides a cross-domain semantic segmentation method based on a diffusion model, which constructs a cross-domain semantic segmentation model; a backbone network, an FPN network, a coding perturbation module, a style projector, a mean adjustment module, a variance adjustment module, a feature fusion module and a mask decoder are set in the cross-domain semantic segmentation model;

[0079] The backbone network is used to extract the feature map of the target image from bottom to top;

[0080] The FPN network is used to fuse the low-level feature maps and high-level feature maps in the feature maps extracted by the backbone network to generate multi-scale feature maps;

[0081] The encoding perturbation module is used to add Gaussian noise to the target image to generate an initial noise feature map;

[0082] The style projector is used to extract the style features of each layer of images from the multi-layer image features to obtain the multi-layer style features of the image;

[0083] The mean adjustment module is used to extract the mean information of each channel in each layer from the multi-layer style features, and expand the size to make it correspond to the size of the initial noise feature;

[0084] The variance adjustment module is used to extract the variance information of each layer and each channel from the multi-layer style features, and expand the size to make it correspond to the size of the initial noise feature;

[0085] The feature fusion module is used to combine the mean and variance of the image style features of each layer of images with the initial noise feature map of the image; and connect it with the multi-scale feature map generated by the FPN network to obtain a mixed feature map;

[0086] The backbone network and the coding perturbation module are respectively input with the target image; the coding perturbation module outputs the initial noise feature map; the backbone network outputs the N-layer image feature map to the FPN network; the FPN network outputs the multi-scale features of the target image; the backbone network and / or the FPN network outputs the l-th layer image feature map to the i-th style projector, i=l≤N; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the mean adjustment module and the variance adjustment module; the mean adjustment module outputs the style feature mean of the first to N-th layer images; the variance adjustment module outputs the style feature variance of the first to N-th layer images; the feature fusion module is respectively input with the output signals from the coding perturbation module, the FPN network, the mean adjustment module and the variance adjustment module to generate a mixed feature map; the mask decoder is input with the mixed feature map generated by the feature fusion module to output a predicted mask.

[0087] The trained cross-domain semantic segmentation model can be used to input the target image set, and the mask can be predicted by retrieving the target on the target image set and classifying it pixel by pixel to obtain the final semantic segmentation result.

[0088] Preferably, the method for training a cross-domain semantic segmentation model may include the following method steps: the input data needs to be properly preprocessed before being input into the network, and the preprocessing method may include data enhancement strategies such as rotation, cropping, adjusting sharpness and brightness, Fourier transform, dynamic blur, grayscale adjustment, etc.; the AdamW optimizer can be used for training for 40,000 rounds; the initial learning rate can be set to 0.00006, and a poly learning rate scheduler with a default factor of 1.0 is used.

[0089] Preferably, the coding perturbation module may add Gaussian noise to the target image in T steps to generate an initial noise feature map of the target image.

[0090] Preferably, the FPN network outputs the l-th layer image feature map to the i-th style projector, and the calculation formula of the i-th style projector can be as follows:

[0091] F style =Linear2(ReLU(Linear1(Avg(F l ))));

[0092] Where:

[0093] F style It is the style feature map of the l-th layer image output by the i-th style projector;

[0094] F l It is the lth layer image feature data output by the FPN network;

[0095] Avg() represents the mean calculation function;

[0096] Linear1() represents the first linear transformation function;

[0097] ReLU() represents the activation function;

[0098] Linear2() represents the second linear transformation function.

[0099] Preferably, the calculation formula for the mean value adjustment module to calculate the mean value of the style feature may be as follows:

[0100] μ adjustment =Linea r μ(ReLU(F style ));

[0101] The calculation formula for calculating the style feature variance by the variance adjustment module can be as follows:

[0102] σ adjustment =Linear σ (ReLU(F style ));

[0103] Where:

[0104] F style It is a style feature;

[0105] Linear μ () represents the mean calculation function of feature map data;

[0106] Linear σ () represents the variance calculation function of feature map data;

[0107] ReLU() represents the activation function;

[0108] μ adjustment is the mean of the image style features;

[0109] σ adjustment is the variance of the image style feature.

[0110] Preferably, when the input target image passes through the FPN network, the vector is defined Capture the output feature map of the l-th layer feature extraction backbone network The channel-by-channel mean and variance of , calculate the feature map The calculation formulas for the mean and variance of each channel data can be as follows:

[0111]

[0112] Where:

[0113] l is the FPN network feature layer number;

[0114] c is the channel number of the feature map;

[0115] is the mean of the feature map data of the cth channel in the lth layer;

[0116] is the variance of the feature map data of the cth channel in the lth layer;

[0117] W is the feature map of the lth layer Width;

[0118] H is the feature map of the lth layer Height;

[0119] is the feature map data of the cth channel of the lth layer;

[0120] x is the input target image;

[0121] w is the feature map The width unit;

[0122] h is the feature map The height unit.

[0123] Preferably, when the feature fusion module combines the mean and variance of the image style features of each layer of images with the initial noise feature map of the image, the noise image calculation formula of the feature fusion module can be as follows:

[0124] F adjusted =μ adjustment +F noise ×σ adjustment ;

[0125] Where:

[0126] μ adjustment is the mean of the image style features;

[0127] σ adjustment is the variance of the image style feature;

[0128] F noise is the initial noise characteristic;

[0129] F adjusted is the adjusted noise characteristic.

[0130] The noise characteristic F after the above adjustment adjusted The signal output by the FPN network is input to the mask decoder. The mask decoder generates the mask structure based on noise + condition. The signal output by the FPN network is used as the conditional signal in the process of restoring the mask, assisting the mask decoder to obtain the adjusted noise feature F adjusted Generates the output prediction mask.

[0131] The following is a preferred embodiment of the present invention to further illustrate the working process and working principle of the present invention:

[0132] The semantic segmentation model is trained using the following steps:

[0133] (i) The backbone network is used to extract features from the target image, and the encoded perturbation module generates initial noise features from the target image mask.

[0134] (1) The input data needs to be properly preprocessed before being input into the network. The preprocessing methods include data enhancement strategies such as rotation, cropping, adjusting sharpness and brightness, Fourier transform, dynamic blur, grayscale adjustment, etc.

[0135] (2) The encoding perturbation module adds Gaussian noise to the input image in T steps until it becomes a noisy image.

[0136] (3) The backbone network processes the input target image and outputs a feature map that contains important visual and spatial information.

[0137] (ii) Use FPN network to generate multi-scale features.

[0138] The FPN network inputs N layers of image features and generates a feature map of the target image The mean and variance of each channel data are calculated as follows:

[0139]

[0140] Where:

[0141] l is the FPN network feature layer number;

[0142] c is the channel number of the feature map;

[0143] is the mean of the feature map data of the cth channel in the lth layer;

[0144] is the variance of the feature map data of the cth channel in the lth layer;

[0145] W is the feature map of the lth layer Width;

[0146] H is the feature map of the lth layer Height;

[0147] is the feature map data of the cth channel of the lth layer;

[0148] x is the input target image;

[0149] w is the feature map The width unit;

[0150] h is the feature map The height unit.

[0151] (iii) Use a style projector to extract style features from multi-scale features.

[0152] The style projector is used to extract the style features of the input image from the multi-level image features. The style feature calculation formula is as follows:

[0153] F style =Linear2(ReLU(Linear(Avg(F l ))));

[0154] Where:

[0155] F style It is the style feature map of the l-th layer image output by the i-th style projector;

[0156] F l It is the lth layer image feature data output by the FPN network;

[0157] Avg() represents the mean calculation function;

[0158] Linear1() represents the first linear layer calculation function;

[0159] ReLU() represents the activation layer calculation function;

[0160] Linear2() represents the second linear layer calculation function.

[0161] (iv) Use the mean adjustment layer and variance adjustment layer to calculate the mean and variance of the image style features.

[0162] In step 4, the mean adjustment layer and the variance adjustment layer calculate the mean and variance from the style features. The calculation formula is as follows:

[0163] μ adjustment =Linear μ (ReLU(F style ));

[0164] σ adjustment =Linear σ (ReLU(F style ));

[0165] Where:

[0166] F style It is a style feature;

[0167] Linear μ () represents the mean calculation function of feature map data;

[0168] Linear σ () represents the variance calculation function of feature map data;

[0169] ReLU() represents the activation layer calculation function;

[0170] μ adjustment is the mean of the image style features;

[0171] σ adjustment is the variance of the image style feature.

[0172] The mean calculation function of the feature map data can be calculated using the following formula:

[0173]

[0174] The variance calculation function of the feature map data can be calculated using the following formula:

[0175]

[0176] Where:

[0177] m is the layer number of the feature map;

[0178] e is the channel number of the feature map;

[0179] is the mean value of the cth channel data of the mth layer feature map;

[0180] is the variance of the cth channel data of the mth layer feature map;

[0181] A is the feature map of the mth layer Width;

[0182] B is the feature map of the mth layer Height;

[0183] is the e-th channel data of the m-th layer feature map;

[0184] z is the input feature map;

[0185] a is the feature map The width unit;

[0186] b is the feature map The height unit.

[0187] is the feature graph.

[0188] (V) The initial noise features output by the coding perturbation module are adjusted using mean and variance, and are input into the mask decoder together with the target image features. The mask decoder outputs the predicted mask.

[0189] In order to narrow the style or domain differences between the target image and the source domain, and between the training and test datasets, and to improve the network's ability to perform semantic segmentation in images of various styles, the present invention adopts a mean and variance adjustment method to narrow the difference.

[0190] Since the statistics of the image feature channels (mean and standard deviation) are closely related to the image style, changing these channel statistics can be seen as an implicit way to change the style of the input image. The adjusted mean and variance are used to adjust the noisy image output by the encoding perturbation module.

[0191] When the feature fusion module combines the mean and variance of the image style features of each layer with the initial noise feature map of the image, the noise image calculation formula of the feature fusion module is as follows:

[0192] F adjusted =μ adjustment +F noise ×σ adjustment ;

[0193] Where:

[0194] μ adjustment is the mean of the image style features;

[0195] σ adjustment is the variance of the image style feature.

[0196] F noise is the initial noise characteristic;

[0197] F adjusted is the adjusted noise characteristic.

[0198] (vi) Obtain the optimal model and use it for semantic segmentation.

[0199] Specifically, data from the visible domain is used for training. After training, the network can be generalized to achieve good semantic segmentation results directly on unseen style features. This greatly increases the difficulty and is closer to real-world scenes.

[0200] (1) The present invention uses the AdamW optimizer for training for 40,000 rounds.

[0201] (2) The initial learning rate is set to 0.00006, and the poly learning rate scheduler with a default factor of 1.0 is used.

[0202] (3) Repeat the above steps until the training is completed. Use the trained model to perform semantic segmentation, that is, input the target image set, the model retrieves the target on the target image set and classifies it pixel by pixel to obtain the final semantic segmentation result.

[0203] The above-mentioned neural networks or functional modules such as the backbone network, FPN network, coding perturbation module, style projector, mean adjustment layer, variance adjustment layer, feature fusion module and mask decoder can all adopt applicable neural networks or functional modules in the prior art, or adopt neural networks or functional modules in the prior art and construct them by conventional technical means.

[0204] The embodiments described above are only used to illustrate the technical ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The patent scope of the present invention cannot be limited only by this embodiment, that is, any equivalent changes or modifications made to the spirit disclosed by the present invention still fall within the patent scope of the present invention.

Claims

1. A cross-domain semantic segmentation system based on a diffusion model, characterized in that: It includes backbone network, FPN network, coding perturbation module, style projector, mean adjustment module, variance adjustment module, feature fusion module and mask decoder; The backbone network is used to extract the feature map of the target image from bottom to top; The FPN network is used to fuse the low-level feature maps and high-level feature maps in the feature maps extracted by the backbone network to generate multi-scale feature maps; The encoding perturbation module is used to add Gaussian noise to the target image to generate an initial noise feature map; The style projector is used to extract the style features of each layer of images from the multi-layer image features to obtain the multi-layer style features of the image; The mean adjustment module is used to extract the mean information of each channel in each layer from the multi-layer style features, and expand the size to make it correspond to the size of the initial noise feature; The variance adjustment module is used to extract the variance information of each layer and each channel from the multi-layer style features, and expand the size to make it correspond to the size of the initial noise feature; The feature fusion module is used to combine the mean and variance of the image style features of each layer of images with the initial noise feature map of the image; and connect it with the multi-scale feature map generated by the FPN network to obtain a mixed feature map; The backbone network and the encoding perturbation module input the target image respectively; the encoding perturbation module outputs the initial noise feature map; the backbone network outputs the N-layer image feature map to the FPN network; the FPN network outputs the multi-scale features of the target image; The backbone network and / or the FPN network outputs the l-th layer image feature map to the i-th style projector, i=l≤N; the i-th style projector outputs the style feature map of the l-th layer image to the mean adjustment module and the variance adjustment module respectively; the mean adjustment module outputs the mean of the style features of the first layer to the N-th layer images; The variance adjustment module outputs the style feature variance of the first to Nth layer images; the feature fusion module inputs the output signals from the encoding perturbation module, FPN network, mean adjustment module and variance adjustment module respectively to generate a mixed feature map; the mask decoder inputs the mixed feature map generated by the feature fusion module and outputs a predicted mask.

2. The cross-domain semantic segmentation system based on the diffusion model according to claim 1, characterized in that: The FPN network outputs the l-th layer image feature map to the i-th style projector, and the i-th style projector includes a mean calculation module, a first linear layer, a first activation layer, and a second linear layer connected in sequence: the mean calculation module calculates the average value of the input data; the first and second linear layers are used to map the input data to the output through a linear transformation, and the first activation layer is used to process the input data through a nonlinear activation function and output it; the FPN network outputs the l-th layer image feature map to the mean calculation module of the i-th style projector, and the second linear layer of the i-th style projector outputs the style feature map of the l-th layer image.

3. The cross-domain semantic segmentation system based on the diffusion model according to claim 1, characterized in that: The mean adjustment module includes a second activation layer and a feature map mean module connected in sequence; the second activation layer is used to process and output the input data through a nonlinear activation function, and the feature map mean module is used to calculate the mean of the feature map; the variance adjustment module includes a third activation layer and a feature map variance module connected in sequence; the third activation layer is used to process and output the input data through a nonlinear activation function, and the feature map variance module is used to calculate the variance of the feature map; the i-th style projector outputs the style feature map of the l-th layer image, respectively to the second activation layer of the mean adjustment module and the third activation layer of the variance adjustment module; the feature map mean module of the mean adjustment module outputs the mean of the style features of the first to N-th layers of images; The feature map variance module of the variance adjustment module outputs the style feature variance of the first layer to the Nth layer of images.

4. The cross-domain semantic segmentation system based on the diffusion model according to claim 1, characterized in that: The feature fusion module includes a first matrix multiplier and a first matrix adder connected in sequence, the first matrix multiplier is used to multiply two matrices; the first matrix adder is used to add two matrices; the first matrix multiplier inputs the variance matrix of the image style features of each layer of images and the initial noise feature map, multiplies the two and outputs them to the first input end of the first matrix adder, the second input end of the first matrix adder inputs the mean matrix of the image style features of each layer of images, and the first matrix adder outputs the combined feature map.

5. A cross-domain semantic segmentation method based on a diffusion model, characterized in that: The method constructs a cross-domain semantic segmentation model; a backbone network, an FPN network, a coding perturbation module, a style projector, a mean adjustment module, a variance adjustment module, a feature fusion module and a mask decoder are set in the cross-domain semantic segmentation model; The backbone network is used to extract the feature map of the target image from bottom to top; The FPN network is used to fuse the low-level feature maps and high-level feature maps in the feature maps extracted by the backbone network to generate multi-scale feature maps; The encoding perturbation module is used to add Gaussian noise to the target image to generate an initial noise feature map; The style projector is used to extract the style features of each layer of images from the multi-layer image features to obtain the multi-layer style features of the image; The mean adjustment module is used to extract the mean information of each channel in each layer from the multi-layer style features, and expand the size to make it correspond to the size of the initial noise feature; The variance adjustment module is used to extract the variance information of each layer and each channel from the multi-layer style features, and expand the size to make it correspond to the size of the initial noise feature; The feature fusion module is used to combine the mean and variance of the image style features of each layer of images with the initial noise feature map of the image; and connect it with the multi-scale feature map generated by the FPN network to obtain a mixed feature map; The backbone network and the coding perturbation module are respectively input with the target image; the coding perturbation module is output with the initial noise feature map; the backbone network is output with the N-layer image feature map to the FPN network; the FPN network is output with the multi-scale features of the target image; The backbone network and / or the FPN network outputs the l-th layer image feature map to the i-th style projector, i=l≤N; the i-th style projector outputs the style feature map of the l-th layer image to the mean adjustment module and the variance adjustment module respectively; The mean adjustment module is made to output the mean of the style features of the first layer to the Nth layer of images; The variance adjustment module is configured to output the variance of style features of the first to Nth layers of images; The feature fusion module is respectively input with output signals from the encoding perturbation module, the FPN network, the mean adjustment module and the variance adjustment module to generate a mixed feature map; the mask decoder is input with the mixed feature map generated by the feature fusion module to output a predicted mask.

6. The cross-domain semantic segmentation method based on the diffusion model according to claim 5 is characterized in that: The method for training a cross-domain semantic segmentation model includes the following steps: the input data needs to be properly preprocessed before being input into the network. The preprocessing method includes data enhancement strategies such as rotation, cropping, adjusting sharpness and brightness, Fourier transform, dynamic blur, grayscale adjustment, etc.; The AdamW optimizer was used for training for 40,000 epochs; the initial learning rate was set to 0.00006, and a poly learning rate scheduler with a default factor of 1.0 was used.

7. The cross-domain semantic segmentation method based on the diffusion model according to claim 5 is characterized in that: The encoding perturbation module is made to add Gaussian noise to the target image in T steps to generate an initial noise feature map of the target image.

8. The cross-domain semantic segmentation method based on the diffusion model according to claim 5, characterized in that: The FPN network outputs the l-th layer image feature map to the i-th style projector. The calculation formula of the i-th style projector is as follows: F style =Linear2(ReLU(Linear1(Avg(F l )))); Where: F style It is the style feature map of the l-th layer image output by the i-th style projector; F l It is the lth layer image feature data output by the FPN network; Avg() represents the mean calculation function; Linear1( ) represents the first linear transformation function; ReLU( ) represents the activation function; Linear2( ) represents the second linear transformation function.

9. The cross-domain semantic segmentation method based on the diffusion model according to claim 8, characterized in that: The calculation formula for calculating the mean value of style features by the mean adjustment module is as follows: μ adjustment =Linear μ (ReLU(F style )); The formula for calculating the style feature variance by the variance adjustment module is as follows: σ adjustment =Linear σ (ReLU(F style )); Where: F style It is a style feature; Linear μ ( ) represents the mean calculation function of feature map data; Linear σ ( ) represents the variance calculation function of feature map data; ReLU( ) represents the activation function; μ adjustment is the mean of the image style features; σ adjustment is the variance of the image style feature.

10. The cross-domain semantic segmentation method based on diffusion model according to claim 5, characterized in that: When the feature fusion module combines the mean and variance of the image style features of each layer with the initial noise feature map of the image, the noise image calculation formula of the feature fusion module is as follows: m adjusted =μ adjustment +F noise ×s adjustment ; Where: μ adjustment is the mean of the image style features; σ adjustment is the variance of the image style feature; F noise is the initial noise characteristic; F adjusted is the adjusted noise characteristic.

Images