An image segmentation method and system fusing self-supervised semantic guidance and conditional modulation mechanism

Abstract

The application discloses an image segmentation method and system fusing self-supervised semantic guidance and conditional modulation mechanism. The application realizes the segmentation of an image by inputting a two-dimensional image into a trained diffusion segmentation network. A double-channel conditional modulation mechanism is introduced in the main path to map the time embedding into the coefficient regulation feature evolution, map the semantic prior information extracted by the conditional path into the affine parameter, scale / shift the features by the channel, perform the layer-by-layer affine transformation in the channel dimension, realize the deep fusion of the semantic condition, improve the semantic consistency maintenance ability of the model under different spatial scales, and help to recover the fuzzy boundary and fine structure. A structure perception composite loss function is introduced to improve the segmentation integrity and boundary clarity of the model on the target region. The application effectively solves the technical defects of the existing image segmentation method in structure consistency, boundary accuracy and label dependence by fusing the self-supervised semantic guidance and the conditional modulation mechanism.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image processing and deep learning, and particularly relates to an image segmentation method and system fusing self-supervised semantic guidance and conditional modulation mechanism. BACKGROUND

[0002] Image segmentation, as a core task of computer vision, aims to divide input images into regions with consistent semantics, and has wide application value in automatic driving, industrial quality inspection, remote sensing interpretation, medical diagnosis, etc. Traditional methods mainly rely on fully convolutional neural networks (FCN), such as UNet, DeepLabV3+, etc. architecture, to realize pixel-level classification through end-to-end learning. However, such methods have two inherent limitations:

[0003] High dependence on labeling: requires a large amount of accurate pixel-level labeled data to train the model, which is costly and difficult to cover all scenario changes;

[0004] Insufficient generalization ability: in scenarios with fuzzy target edges, variable shapes, or complex textures (such as transparent object segmentation, natural scene fine-grained object segmentation), segmentation accuracy significantly decreases.

[0005] In recent years, generative segmentation methods (such as diffusion model-based segmentation networks) have shown stronger detail reconstruction capabilities by learning data distribution through a gradual denoising process. Diffusion models simulate the Markov process of gradually reconstructing data from noise state, and have achieved breakthroughs in natural image generation tasks. However, they have obvious defects in segmentation tasks:

[0006] Lack of structure perception: the generation process lacks explicit modeling of image semantic structure, resulting in inconsistency between output mask and target object topology;

[0007] Insufficient boundary guidance: the denoising process is easily disturbed by local noise, and the boundary area is prone to blur or artifacts, making it difficult to maintain clear contours.

[0008] At the same time, the development of self-supervised learning technology provides a new way to solve the labeling dependence problem. Self-supervised encoder (SSE) based on VisionTransformer (ViT) learns structured representation of images through unlabelled data, and has verified its strong feature extraction ability in classification and detection tasks. However, existing technologies have not effectively integrated the global semantic prior of SSE and the detail generation ability of diffusion model, making it difficult to balance semantic consistency and local accuracy in segmentation tasks.

[0009] In addition, existing segmentation models generally rely on a single pixel-level loss function (such as Dice loss, cross-entropy loss), which only measures pixel overlap and ignores high-order semantic constraints: there is a lack of explicit supervision on the structural integrity and boundary continuity of the target; and in the case of edge blur, the prediction may be broken or have a region hole problem.

[0010] In summary, the current image segmentation technology needs to solve the following key problems:

[0011] (1) How to reduce the dependence on labeled data and improve the generalization of complex scenes;

[0012] (2) How to integrate global structure prior and local detail generation capability;

[0013] (3) How to design a loss function to optimize region overlap, structural consistency and boundary clarity at the same time. SUMMARY

[0014] The purpose of the present application is to overcome the shortcomings of the prior art and provide an image segmentation method and system that integrates self-supervised semantic guidance and conditional modulation mechanism.

[0015] In a first aspect, the present application provides an image segmentation method integrating self-supervised semantic guidance and conditional modulation mechanism, comprising:

[0016] Obtaining a two-dimensional image and its corresponding mask image, and constructing a data set;

[0017] Constructing a diffusion segmentation network integrating self-supervised semantic guidance and conditional modulation mechanism, and training it using the data set;

[0018] Inputting the two-dimensional image to be processed into the trained diffusion segmentation network to realize segmentation of the image;

[0019] The diffusion segmentation network integrating self-supervised semantic guidance and conditional modulation mechanism comprises a conditional path and a backbone path; the conditional path obtains semantic prior information of the two-dimensional image, and the backbone path performs step-by-step denoising under the semantic conditional modulation mechanism of the semantic prior information and predicts the image segmentation result.

[0020] Preferably, the conditional path comprises an image block embedding layer, a classification label embedding layer, a position encoding layer and a Transformer encoding module in sequence, and the input is a two-dimensional image and the output is semantic prior information of the two-dimensional image.

[0021] Preferably, the backbone path adopts a diffusion model, and each time step in the reverse Markov chain process of the diffusion model adopts a conditional U-Net architecture based on residual blocks to perform step-by-step denoising.

[0022] Preferably, the conditional U-Net architecture based on the residual module includes an encoding module, a bottleneck module, and a decoding module;

[0023] Each coding layer in the coding module includes a plurality of residual blocks and a convolution layer arranged between adjacent residual blocks, and a downsampling unit is arranged between adjacent coding layers;

[0024] The structure of the decoding module is symmetrical to that of the encoding module, and an upsampling unit is provided between adjacent decoding layers;

[0025] The bottleneck module is used to connect the encoding and decoding paths, and includes multiple residual blocks and convolution layers arranged between adjacent residual blocks.

[0026] More preferably, in the encoding module, bottleneck module, and decoding module, each residual block integrates time step conditional modulation and semantic conditional modulation, wherein the time step conditional modulation embeds the time step into the vector The time step conditional modulation layer is mapped to the time step modulation coefficient and then added to the intermediate activation channel for time step modulation to obtain the intermediate activation feature, which is used to characterize the time change in the diffusion process; the semantic conditional modulation is to extract the semantic prior information of the conditional path After being mapped into semantic conditional modulation coefficients through the semantic conditional modulation layer, the intermediate activation features after time step conditional modulation are semantically modulated as the final output features of the residual block;

[0027] The time step embedding vector The current diffusion model time step t is generated by the sine and cosine functions to generate the position code , and then mapped to a fixed dimension by a multi-layer perceptron

[0028] Preferably, in the training process of the diffusion segmentation network that integrates self-supervised semantic guidance and conditional modulation mechanism, the composite loss function Measuring the difference between the backbone network output mask and the true annotation, the composite loss function Combined with the diffuse noise prediction error , structural perception error , edge preservation loss and Dice matching loss ; The network continuously iteratively updates parameters through the back-propagation algorithm to minimize the composite loss function.

[0029] In a second aspect, the present invention provides an image segmentation system, comprising:

[0030] Data acquisition module, responsible for acquiring the two-dimensional image to be processed;

[0031] An image segmentation module is responsible for inputting a two-dimensional image into the trained diffusion segmentation network to realize segmentation of the image.

[0032] In a third aspect, the present application provides a computer readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method.

[0033] In a fourth aspect, the present application provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method.

[0034] The beneficial effects of the present application are as follows:

[0035] 1. The present application introduces a conditional path of visual Transformer structure to extract global semantic features of the input image and inject the conditional information into the diffusion model as a structure prior guide to realize fine mask recovery, thereby improving the segmentation accuracy and robustness of the model for complex morphological targets with blurred boundaries and rich structural details.

[0036] 2. The present application introduces a double-channel conditional modulation mechanism in the backbone path to map the time embedding into coefficient regulation feature evolution. The semantic prior information extracted by the conditional path is mapped into affine parameters, the features are scaled / shifted according to the channel, and the affine transformation is performed layer by layer in the channel dimension to realize deep fusion of semantic conditions and improve the semantic consistency maintenance ability of the model at different spatial scales, which is helpful for restoring blurred boundaries and fine structures.

[0037] 3. The present application adopts a linear noise scheduling strategy and an importance sampling time step scheduling mechanism in the diffusion modeling process. On the one hand, the linear scheduling design ensures uniform modeling ability from strong disturbance to weak disturbance; on the other hand, the importance sampling strategy of time steps constructs a sampling density function according to the learning difficulty distribution at different time points, and preferentially learns the middle time steps in training, which significantly improves the middle recovery performance of the model in the denoising generation path, accelerates the convergence and improves the generation quality.

[0038] 4. The present application introduces a structure-aware composite loss function, which covers diffusion noise prediction error, structure-aware error, edge preservation loss and Dice matching loss. The Dice loss constrains the region overlap, the structure loss measures the semantic consistency through the intermediate layer features of the encoder, the boundary loss calculates the edge accuracy based on the gradient map, and the diffusion noise prediction error enhances the recovery ability of the model at key time steps, which jointly constructs a strong supervision target to improve the segmentation integrity of the target region and the boundary clarity of the model.

[0039] 5. During the inference phase, this method requires only a 2D image as input. Conditional features are generated in one go via a conditional path. The diffusion model then performs progressive reverse sampling and denoising in a fixed number of steps, outputting a binary or multi-class mask image with the same size as the input. The overall model operates stably and exhibits good generalization performance, making it suitable for image segmentation tasks involving objects with blurred edges, variable morphology, and complex textures, possessing significant practical application value.

[0040] In summary, the present invention effectively solves the technical deficiencies of existing image segmentation methods in terms of structural consistency, boundary accuracy and annotation dependence by integrating self-supervised semantic guidance with conditional modulation mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is the overall flow chart of the diffusion segmentation network of the present invention, showing the complete process from image input, conditional feature extraction, diffusion modeling to segmentation mask output; represents the image after T steps of noise addition, represents the image at step t, represents the image at step t, represents the original image, represents the denoised distribution predicted by the model during the sampling phase, represents the forward noise distribution in the training phase, and b represents the semantic prior information.

[0042] Figure 2 This is a structural diagram of the conditional path in the diffusion segmentation network of the present invention, showing the process of image block embedding, classification label embedding, position encoding, Transformer encoding and final feature map construction.

[0043] Figure 3 The first 50 samples in the ISIC2018 validation set are compared with the conditional features of the validation set extracted by the conditional path of the present invention and their lesion maps.

[0044] Figure 4 The last 50 samples in the ISIC2018 validation set are compared with the conditional features of the validation set extracted by the conditional path of the present invention and their lesion maps.

[0045] Figure 5 Schematic diagram of the structure of the residual block-based conditional U-Net architecture and its residual block used in the present invention, where (a) is a schematic diagram of the structure of the residual block-based conditional U-Net architecture, and (b) is a schematic diagram of the residual block structure.

[0046] Figure 6 It is a schematic diagram of the conditional modulation in the present invention, which illustrates how the conditional guidance image output by SSE modulates the features in the trunk residual module through scaling and offset coefficients.

[0047] Figure 7 This is the result of the linear noise scheduling strategy and the importance sampling time step scheduling mechanism adopted in the diffusion modeling process of the present invention, where (a) is the linear noise scheduling curve of the linear noise intensity at different time steps, and (b) is the sampling density distribution of importance sampling at different time steps.

[0048] Figure 8 This is a control experiment diagram before and after the use of the composite loss proposed in the present invention, which fully demonstrates the advanced performance of the proposed loss function.

[0049] Figure 9 This is a visual comparison chart of the method of the present invention on the skin disease image segmentation task, showing the difference in prediction effect between the present invention and the existing method under different lesion types. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0051] The terms "including," "having," and any variations thereof, as used in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to the process, method, product, or apparatus.

[0052] like Figure 1 As shown, the present invention provides an image segmentation method that integrates self-supervised semantic guidance and conditional modulation mechanism, the method comprising:

[0053] Obtain a two-dimensional image and its corresponding mask image to construct a data set;

[0054] Construct a diffusion segmentation network that integrates self-supervised semantic guidance and conditional modulation mechanism, and train it using the dataset;

[0055] The two-dimensional image to be processed is input into the trained diffusion segmentation network to achieve image segmentation.

[0056] Among them, the diffusion segmentation network that integrates self-supervised semantic guidance and conditional modulation mechanism includes a conditional path and a trunk path, and adopts a conditional diffusion structure as a whole; the conditional path obtains the semantic prior information of the two-dimensional image, and the trunk path performs step-by-step denoising under the semantic conditional modulation mechanism of the semantic prior information to predict the image segmentation result.

[0057] This diffusion segmentation network achieves fine modeling of the target area and high-quality mask prediction by jointly training the conditional path and the backbone path, and introducing semantic conditions and time step condition information at each time step of diffusion. It is particularly suitable for target images with blurred boundaries and complex morphology.

[0058] Specifically, if Figure 2 The conditional path shown is an image encoding network based on the Visual Transformer (ViT) architecture, namely a self-supervised encoder (SSE), which includes an image block embedding layer, a classification label embedding layer, a position encoding layer, and a Transformer encoding module; the input of the conditional path is a two-dimensional image, and the output is the semantic prior information of the two-dimensional image.

[0059] The image block embedding layer is implemented using a two-dimensional convolution unit with a convolution kernel size equal to the stride, which is used to divide the input image into equal-sized image blocks and map them into high-dimensional conditional vectors called patch tokens. The convolution operation does not include normalization and activation functions, and the output tensor is flattened to form a two-dimensional sequence structure.

[0060] The classification token embedding layer is a set of learnable fixed vectors that are shared across all samples and concatenated with the conditional vector patch token in the sequence dimension.

[0061] The position encoding layer is a set of learnable vectors that are added element by element to the conditional vector token sequence to encode the spatial structure order of the image;

[0062] The Transformer encoding module consists of multiple stacked encoding layers, each of which includes a multi-head self-attention submodule and a feedforward neural network, with internal integration layer normalization and residual connections. The multi-head self-attention submodule contains multiple parallel attention heads and outputs an attention map of uniform dimension. The feedforward neural network consists of two layers of linear transformations, with a constant embedding dimension. All encoding layers model the conditional vector token sequence layer by layer, and the output sequence remains consistent in dimension. The final output is normalized through the standard layer, removing the class segmenter CLS token and retaining only the conditional vector patch token representation. The output conditional vector token sequence can be reconstructed into a two-dimensional spatially arranged embedding map, which serves as an external conditional feature and participates in subsequent modulation.

[0063] Specifically, the input image size of the conditional path is 3x224x224, first image patch embedding layer patch embedding is performed for image block division, each image block patch size is 16x16, and is mapped into a fixed dimension conditional vector token sequence; then a learnable class tokenizer is spliced through a classification label embedding layer, and position encoding is added through the position encoding layer to form a unified conditional vector token sequence input into the Transformer encoding module. The conditional vector patch token without the class tokenizer is output after the processing of the Transformer encoding module, and is reshaped and transposed in the channel to form a two-dimensional semantic feature map with a shape of CxHxW, i.e., semantic prior information, as the modulation condition of the diffusion UNet.

[0064] The main path adopts a diffusion model, for example, the diffusion denoising probability model can be adopted in the embodiment. In the reverse Markov chain process of the diffusion model, each time step adopts a conditional U-Net architecture based on residual blocks to perform step-by-step denoising.

[0065] The input of the conditional U-Net architecture based on residual blocks is a noisy segmentation mask image, and the output is a predicted mask image after step-by-step denoising; it includes an encoding module, a bottleneck module, and a decoding module, and has a symmetric down-sampling and up-sampling path.

[0066] Each encoding layer in the encoding module includes a plurality of residual blocks (ResBlock) and a convolution layer arranged between adjacent residual blocks, and a down-sampling unit is arranged between adjacent encoding layers for resolution compression and feature extraction in deep layers.

[0067] The bottleneck module is used to connect the encoding and decoding paths, and includes a plurality of residual blocks and a convolution layer arranged between adjacent residual blocks, for aggregating context information in the deepest layer.

[0068] The decoding module is symmetrical to the encoding module, and each decoding layer includes the same residual blocks (ResBlock) and convolution layers arranged between adjacent residual blocks as the same layer encoding layer, and an up-sampling unit is arranged between adjacent decoding layers; in each layer, the decoding features and the skip connection features from the encoding module are spliced in the channel dimension and input to the residual block of the current layer decoding layer for processing; the up-sampling unit can be in the form of deconvolution or linear interpolation, and each layer up-sampling unit recovers the resolution layer by layer.

[0069] The decoded feature map output by the final decoding module is mapped into a single-channel or multi-channel prediction mask by a single 2D convolutional unit. The output tensor has the same spatial dimensions as the input image, and the number of channels depends on the segmentation task objectives (e.g., 1 for a binary mask or N for multi-class output).

[0070] Conditional modulation is integrated within the ResBlock structure of each residual block in the backbone network, which specifically includes multiple serial convolutional layers, group normalization, GELU activation function, and conditional modulation.

[0071] See attached Figure 6 The conditional modulation includes time step conditional modulation and semantic conditional modulation, wherein the time step conditional modulation embeds the time step into the vector The time step conditional modulation layer is mapped to the time step modulation coefficient and then added to the intermediate activation channel for time step modulation, which is used to characterize the time changes in the diffusion process; the semantic conditional modulation is to extract the semantic prior information of the conditional path After being mapped into semantic conditional modulation coefficients through the semantic conditional modulation layer, semantic conditional modulation is performed on the features after time step conditional modulation, which is used as the final output feature of the residual block, ensuring that the generation process of each time step is jointly guided by the temporal evolution dynamics and the segmentation target structure prior. The specific implementation process is as follows:

[0072]

[0073]

[0074]

[0075] in, It is the intermediate activation feature of the residual block, which is obtained after the input is processed by the convolution layer, group normalization, and GELU activation function; is the final output feature of the residual block, Represents the semantic prior information output by the two-dimensional image z after being processed by the conditional path; represents the time step conditional modulation layer, which can be implemented using a 1×1 convolutional layer. Represents the semantic conditional modulation layer, which can be implemented using a 1×1 convolutional layer. represents the conditional modulation coefficient, represents the time step modulation coefficient. and It is an affine transformation parameter used to scale and translate the intermediate feature channels of the residual block in the backbone network to achieve semantic modulation across paths.

[0076] The time step embedding vector The current diffusion model time step t is generated by the sine and cosine functions to generate the position code , and mapped to a fixed dimension by a multi-layer perceptron (MLP):

[0077]

[0078] wherein represents a positive sine encoder;

[0079] Conditional modulation maps the conditional features token output by the encoder to affine parameters to scale and shift the backbone features channel by channel, thereby realizing conditional guidance; the standard layer normalization LayerNorm and GELU activation unit are added in the encoder to keep the feature distribution stable and improve the convergence speed of the Transformer; the channel and spatial attention mechanism is integrated in the decoder structure to strengthen the expression ability of the salient region of the features in the segmentation process and improve the boundary preservation effect.

[0080] The linear noise scheduling strategy and the importance sampling time step scheduling mechanism are used in the diffusion modeling process. Figure 7 Fig. (a) is a linear noise scheduling curve of the linear noise intensity used in the present application at different time steps. Figure 7 Fig. (b) is a sampling density distribution of the importance sampling used in the present application at different time steps.

[0081] As an example, the noise scheduling function used by the backbone path in the training process adopts a linear strategy, that is, at each time step of the diffusion process The noise variance coefficient is linearly increased As follows:

[0082]

[0083] wherein represents the maximum noise, represents the minimum noise, represents the total sampling time step.

[0084] The cumulative attenuation coefficient is used for the reconstruction disturbance in the noise adding process:

[0085]

[0086] wherein represents the image at the t step, represents the original image, represents the real Gaussian noise.

[0087] In the training process, in order to improve the segmentation quality and boundary restoration effect of the backbone path model, a composite loss function composed of three types of loss functions is introduced The composite loss function measures the difference between the mask output by the backbone network and the real label Combined with the diffuse noise prediction error , structural perception error , edge preservation loss and Dice matching loss ; The network continuously iteratively updates parameters through the back-propagation algorithm to minimize the total loss and improve the segmentation accuracy.

[0088] As an example, the composite loss function for:

[0089]

[0090] in 、 、 is the weight.

[0091] In order to improve the training efficiency and enhance the recovery ability of the model at key time steps, the time step scheduling adopts the importance sampling strategy, that is, the time step in the training process It is not uniformly sampled, but obeys an error-driven non-uniform distribution This method improves the training weight of the intermediate time step and introduces a sampling weight compensation term , to ensure that the training contribution of the intermediate time steps is amplified, the sampling density is often set to concentrate on Interval. Final diffuse noise prediction error for:

[0092]

[0093] in is the prediction result of the diffusion network for noise; is the input image after adding noise, Represents the global semantic features of the conditional path output; represents true Gaussian noise, is the probability density distribution driven by error, is the sampling weight compensation term.

[0094] The Dice loss is defined as:

[0095]

[0096] in 、 The predicted mask and the real mask are respectively The value of is a numerical stability term, and its value is generally , prevent division by zero;

[0097] Structure-aware loss Use the Euclidean distance between perceptual features:

[0098]

[0099] in Represents the structure-aware encoder Feature maps extracted by layers; represents the prediction mask, represents the true mask;

[0100] Constructing boundary preservation loss based on Sobel operator :

[0101]

[0102] in Represents the edge extraction operation, using the standard 3×3 Sobel operator to extract image gradients in the x and y directions respectively.

[0103] The inference phase only requires a single 2D image as input. The conditional path generates a conditional feature map (i.e., the semantic prior information of the 2D image) and then passes it to the backbone path. The backbone path starts from a randomly initialized Gaussian noise map, iterates through linear scheduling denoising sampling, and gradually restores the target mask within a fixed time step range. The final output size is the same as the input. Figure 1 A consistent single-channel prediction map is generated to achieve automatic and fine segmentation of image targets.

[0104] In one implementation, before inputting into the diffusion segmentation network that integrates self-supervised semantic guidance and conditional modulation mechanism, the image data needs to be standardized and preprocessed to meet the model input requirements and unify the image size and pixel range.

[0105] The present invention will be further described below with reference to specific embodiments. The above image segmentation method is applied to medical image segmentation, targeting the task of segmenting skin lesions with fuzzy boundaries and rich structural details in medical images, achieving precise automatic segmentation of skin lesions (moles, melanomas, pigmented patches, etc.), specifically comprising the following steps:

[0106] Step 1: Data Acquisition

[0107] The dataset used in this example is the ISIC 2018 Dermatology Image Dataset, which contains color images of various common skin lesion types (such as moles, melanomas, and keratotic lesions), along with their pixel-level segmentation mask labels. The images are sourced reliably and have clear medical diagnostic annotations, making them suitable for model training and evaluation for the automatic segmentation of skin lesions. The dataset is divided into training, validation, and test sets. The training set is used for model parameter learning, the validation set is used for tuning and setting the early stopping strategy, and the test set is used for final performance evaluation.

[0108] Step two: image preprocessing

[0109] To ensure the consistency of model input and the stability of training, the original image and mask data are standardized as follows:

[0110] Size normalization: all input images and their corresponding masks are uniformly scaled to the size of 256x256 or 224x224 to match the input size requirements of the conditional path and the backbone path;

[0111] Pixel normalization: linear normalization is performed on the image pixels, and the RGB channel values are scaled to the interval [0, 1];

[0112] Format conversion: convert the image and mask into floating-point tensor format respectively, and save them as binary format that can be directly read by deep learning frameworks such as PyTorch;

[0113] Channel standardization (optional): if a pre-trained model is used to initialize the conditional path, the image input is standardized according to the channel mean and standard deviation of the ImageNet dataset;

[0114] Data augmentation (training phase): during training, random horizontal flipping, brightness perturbation, affine transformation, and other data augmentation operations are performed on the image to improve the robustness of the model to skin image morphological changes.

[0115] Step three: neural network building and training

[0116] 3-1. Build network

[0117] The neural network structure used in this embodiment is a diffusion segmentation network that integrates self-supervised semantic guidance and conditional modulation mechanism. The network mainly consists of two functional paths:

[0118] Condition path: The self-supervised encoder SSE is adopted, and the structure is based on the visual Transformer (ViT) architecture, including image block embedding layer, classification label embedding layer, position encoding layer, and Transformer encoding module. This path is responsible for extracting global semantic features, i.e., semantic prior information, from the input image, which is used as a modulating condition for the diffusion model of the backbone path. Specifically, the image block embedding layer patch embedding layer divides the input lesion image into image blocks of size 16x16, forming 196 image blocks patch (for a 224x224 size image), and mapping them into a 1024-dimensional token sequence through linear transformation; then the positional encoding (positional encoding) is added to maintain the spatial information. The sequence is input into the main encoder containing 24 layers of standard Transformer Block, each Block including a multi-head self-attention module (Multi-Head Self-Attention, MHSA), a feedforward neural network module (Feedforward Network, FFN), and a layer normalization LayerNorm and residual connection before each layer. The class tokenizer [CLS] token in the final extraction output is used as a structural semantic embedding vector to guide the subsequent conditional modulation of the main branch network.

[0119] Main path: A diffusion model is adopted, and a residual block-based conditional U-Net architecture is used to perform step-by-step denoising in the reverse Markov chain process of the diffusion model. As shown in Figure 5 , the residual block-based diffusion U-Net architecture includes an encoding module, a bottleneck module, and a decoding module, and has a typical symmetric upsampling and downsampling structure.

[0120] The encoding module includes 4 layers, each encoding layer including 2 residual blocks (ResBlock) and a 3x3 convolution between the 2 residual blocks, and a convolution with a step of 2 is set between adjacent encoding layers as a downsampling unit to complete downsampling. The bottleneck module is used to connect the encoding and decoding paths, and includes 2 residual blocks and a 3x3 convolution between the 2 residual blocks. A downsampling unit is provided between the bottleneck module and the last encoding layer, and an upsampling unit is provided between the bottleneck module and the adjacent decoding layer. The decoding module structure is symmetrical to the encoding module, each decoding layer includes 2 residual blocks (ResBlock) and a 3x3 convolution between the 2 residual blocks, and a convolution with a step of 2 is set between adjacent decoding layers as an upsampling unit to complete upsampling. A skip connection is provided between the same layer encoding layer and decoding layer for cross-scale feature fusion.

[0121] All residual blocks ResBlock in the encoding module, the bottleneck module and the decoding module are introduced into the double-channel modulation condition modulation composed of the time step condition modulation combined with the semantic condition modulation, and the global semantic features output by the condition path are semantically modulated with the intermediate activation features after the time step condition modulation. Specifically, each ResBlock includes two layers of 3x3 convolution, group normalization, GELU activation function and conditional modulation.

[0122] The diffusion modeling process adopts a standard forward noise adding and reverse noise removing mechanism, and gradually recovers the image from Gaussian noise to a clear segmentation mask within time steps t [0, T].

[0123] 3-2. Model training

[0124] The image data required for segmentation is input into the diffusion segmentation network constructed in 3-1 for training. During the training process, a composite loss function is used The difference between the network output mask and the real label is measured. The loss function is composed of diffusion residual loss, structure perception loss, boundary preservation loss and Dice loss, and the specific composition is as follows:

[0125] The diffusion residual loss is used to measure the difference between the predicted noise and the real noise;

[0126] The structure perception loss measures the output semantic consistency by the Euclidean distance of the perception features in the intermediate layer of the encoder;

[0127] The boundary preservation loss enhances the edge structure through the gradient operator;

[0128] The Dice loss is used to measure the overlap degree of the predicted region and the real mask.

[0129] The training adopts the back propagation algorithm, and the network parameters are continuously iteratively optimized according to the above loss function. The overall design of the loss function is used to optimize the structure restoration ability and the boundary clarity at the same time, and improve the modeling ability of the model to the skin lesion area.

[0130] The noise scheduling function adopts a linear strategy during the training process, that is, the noise variance of each time step in the diffusion process increases in a linear manner. At the same time, the time step scheduling mechanism of importance sampling is used to sample the time step from the error-driven probability density distribution, improve the training proportion of the middle time step, and introduce a sampling weight compensation term to further improve the generation quality of the middle section.

[0131] The diffusion segmentation network described in the application is trained on an A800 (80GB) GPU, and the training data set selects the public medical image data set ISIC2018, so as to ensure the generalization ability and clinical adaptability of the network in the skin lesion segmentation task.

[0132] Step four, input the skin disease image into the diffusion segmentation network built and trained in step three, finally realize the automatic recognition and segmentation of the lesion area in the skin disease image, and generate the mask result meeting the medical diagnosis requirements.

[0133] The skin disease image segmentation network workflow is as follows: the lesion image with an input size of 3x224x224 is input into the conditional branch SSE in the network, a token sequence of 196x1024 is generated after patch embedding and position encoding, and is sent into a 24-layer Transformer encoder for structural semantic feature extraction, and finally a [CLS] token representing the global structure is output as the conditional embedding. At the same time, the backbone network inputs the diffusion noise image Figure 1 and the corresponding time step t, which is mapped into a time embedding vector with a length of D after sine position encoding, and is used together with the semantic prior information output by the conditional branch to control the conditional modulation in each ResBlock. The backbone path adopts a diffusion model, and each time step in the reverse Markov chain process of the diffusion model adopts a conditional U-Net architecture based on residual blocks to perform step-by-step denoising. The encoding module part of the conditional U-Net architecture based on residual blocks contains 4 layers, and the channel dimension after down-sampling of each layer is 128, 256, 512, and 1024 in turn; the bottleneck module retains all the skip connections and introduces double-channel conditional modulation; the decoder part contains 4 layers, and the image spatial structure is recovered layer by layer after up-sampling of each layer, and finally a segmentation mask with the same size as the original is output. The network receives time evolution information and structural condition input at each layer to realize step-by-step refinement of the lesion area and generate clear and structure-aligned segmentation results. The entire inference process remains consistent with the training stage, supports multi-step denoising reconstruction, and has strong robustness and generalization performance.

[0134] The lesion image with an input size of 3x224x224 is input into the SSE in the network , a token sequence of 196x1024 is generated after patch embedding, classification label embedding and position encoding, and is sent into a 24-layer Transformer encoder for structural semantic feature extraction, and finally a [CLS] token representing the global structure is output as the conditional embedding vector c, as shown in the following formula:

[0135]

[0136] The diffusion noise image is input into the backbone network, and the corresponding time step is input into the backbone network. The backbone network is represented as a conditional diffusion mapping function :

[0137]

[0138] Sin-Cos position encoding The time step Mapped to a vector of dimension D, its i-th dimension is defined as follows:

[0139]

[0140] in Represents sine and cosine encoding.

[0141] The diffusion time step t is mapped to a time embedding vector through the sine and cosine functions , used to describe the time position in the diffusion process, see the following formula:

[0142]

[0143] During the training process, the Figure 8 The composite structure-aware loss function, composed of three sub-losses, is used to simultaneously constrain the segmentation mask's regional overlap, structural consistency, and edge clarity, improving the network's expressiveness in fine-grained lesion modeling. The total loss function used in model training includes the diffusion prediction residual loss, the Dice overlap loss, the structure-aware loss, and the edge-preserving loss. The four are weighted to form the final optimization objective function, as shown in the following formula:

[0144]

[0145] Among them, the weight coefficients of each loss item are optimized through experiments and are typically set as: to ensure comprehensive constraints in terms of structure, boundaries and regional overlap.

[0146] By jointly optimizing the above four losses, the network can effectively guide the model to converge synchronously in four dimensions: temporal dynamics, structural consistency, boundary clarity, and regional segmentation accuracy during training, thereby improving the segmentation performance and boundary perception capabilities in complex skin lesion images.

[0147] To validate the effectiveness of our method, we used several common segmentation performance metrics, including the Dice coefficient and Intersection-over-Union (IoU). These two metrics are widely used in medical image segmentation tasks, measuring the overlap and intersection accuracy between the predicted and true masks, respectively.

[0148] The closer the Dice coefficient is to 1, the more fully the predicted area overlaps with the true area. It is defined as follows:

[0149]

[0150] In the formula, A represents a mask area predicted by a model, B represents a real labeled mask area, represents the number of non-zero pixels in the area, is the intersection area of the two.

[0151] At the same time, in order to further evaluate the positioning accuracy of the prediction result, the IoU (intersection over union) index is calculated, and the value is closer to 1, which means that the predicted mask is more consistent with the real mask, and is defined as shown in the following formula:

[0152]

[0153] In the formula, represents the number of pixels in the union of the predicted area and the real area.

[0154] The experimental results are shown in Figure 3 , Figure 4 , Figure 8 , Figure 9 and Tables 1, 2 and 3. In the experiments of the present application, the Dice and IoU values are counted on the validation set and test set of the ISIC2018 dataset respectively, and are compared and analyzed with traditional segmentation methods (such as nnUNet, TransUNet) and generative methods (such as EnsemDiff, SegDiff), to verify the advantages of the present method in structure restoration, boundary preservation and overall segmentation accuracy.

[0155] Table 1 Comparison of Dice values and IoU values between results obtained by the method of the present application and existing methods and content images

[0156]

[0157] Table 2 Comparison of parameter quantities (M) of the network model method of the present application and existing methods

[0158]

[0159] Table 3 Comparison of single inference time of each method when inputting a 256x256 image

[0160] (diffusion 100 steps, A800 GPU, unit: seconds)

[0161]

[0162] The embodiment also provides an image segmentation system for implementing the above method, and the system comprises:

[0163] A data acquisition module is responsible for acquiring a two-dimensional image to be processed;

[0164] An image segmentation module is responsible for inputting the two-dimensional image to the trained diffusion segmentation network to implement segmentation of the image.

[0165] The embodiment also provides an electronic device, in particular, the electronic device includes a memory and a processor, the memory stores executable code, and the processor executes the executable code to implement the method of any one of the embodiments.

[0166] The memory can include a high-speed random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0167] The bus can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0168] The memory is used to store a program, and the processor executes the program after receiving an execution instruction. The method executed by the device defined by the flow process disclosed in any one of the preceding embodiments of the application can be applied to the processor or implemented by the processor.

[0169] The processor can be an integrated circuit chip with a signal processing capability. In implementation, the steps of the above method can be completed by an integrated logic circuit or an instruction in the form of software in the processor. The processor mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. It can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed by the processor. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or other mature storage media in the art. The storage medium is located in the storage memory, and the processor reads the information in the storage memory and combines the hardware to complete the steps of the above method.

[0170] The computer program product of the readable storage medium provided by the embodiments of the present application includes a computer readable storage medium storing program codes, and the program codes include instructions for executing the method described in the foregoing method embodiments. The specific implementation can be referred to the foregoing method embodiments, and will not be described here.

[0171] When the function is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0172] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. An image segmentation method integrating self-supervised semantic guidance and conditional modulation mechanism, the method comprising: Obtain a two-dimensional image and its corresponding mask image to construct a data set; Construct a diffusion segmentation network that integrates self-supervised semantic guidance and conditional modulation mechanism, and train it using the dataset; Input the two-dimensional image to be processed into the trained diffusion segmentation network to achieve image segmentation; The diffusion segmentation network integrating self-supervised semantic guidance and conditional modulation mechanism includes a conditional path and a backbone path; the conditional path obtains semantic prior information of the two-dimensional image, and the backbone path performs step-by-step denoising under the semantic conditional modulation mechanism of the semantic prior information to predict the image segmentation result; The backbone path adopts a diffusion model, and in the reverse Markov chain process of the diffusion model, a conditional U-Net architecture based on residual blocks is used to perform step-by-step denoising at each time step; The conditional U-Net architecture based on the residual module includes an encoding module, a bottleneck module, and a decoding module; Each coding layer in the coding module includes a plurality of residual blocks and a convolution layer arranged between adjacent residual blocks, and a downsampling unit is arranged between adjacent coding layers; The structure of the decoding module is symmetrical to that of the encoding module, and an upsampling unit is provided between adjacent decoding layers; The bottleneck module is used to connect the encoding and decoding paths, and includes multiple residual blocks and convolutional layers arranged between adjacent residual blocks; In the encoding module, bottleneck module, and decoding module, each residual block internally integrates time step conditional modulation and semantic conditional modulation, wherein the time step conditional modulation maps the time step embedding vector to a time step modulation coefficient through a time step conditional modulation layer, and then adds it to the intermediate activation channel for time step modulation, thereby obtaining an intermediate activation feature for characterizing the temporal changes in the diffusion process; the semantic conditional modulation maps the semantic prior information extracted by the conditional path to a semantic conditional modulation coefficient through a semantic conditional modulation layer, and then performs semantic conditional modulation on the intermediate activation feature after time step conditional modulation, which serves as the final output feature of the residual block; The time step embedding vector generates a position code by using the sine and cosine functions for the current diffusion model time step t, and then maps it into a fixed dimension through a multi-layer perceptron.

2. The method according to claim 1, characterized in that The conditional path includes an image block embedding layer, a classification tag embedding layer, a position encoding layer, and a Transformer encoding module in sequence. Its input is a two-dimensional image, and its output is the semantic prior information of the two-dimensional image.

3. The method according to claim 1, characterized in that During the training process of the diffusion segmentation network that integrates self-supervised semantic guidance and conditional modulation mechanism, a composite loss function is used to measure the difference between the backbone network output mask and the true annotation. This composite loss function combines the diffusion noise prediction error, structure perception error, edge preservation loss and Dice matching loss; the network continuously iteratively updates parameters through the backpropagation algorithm to minimize the composite loss function.

4. An image segmentation system implementing the method according to any one of claims 1 to 3, characterized in that The system comprises: Data acquisition module, responsible for acquiring the two-dimensional image to be processed; The image segmentation module is responsible for inputting the two-dimensional image into the trained diffusion segmentation network to achieve image segmentation.

5. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 3.

6. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 3 is implemented.

Images