Image segmentation method, system, medium and device based on double-branch network architecture

Abstract

The application relates to the technical field of image processing, and provides an image segmentation method, system, medium and equipment based on a double-branch network architecture, which comprises the following steps: for an image to be segmented, the complexity of the image to be segmented is measured through edge detection and texture analysis, a denoising step length of a diffusion model is allocated, and semantic features are extracted; after random occlusion and blurring processing is performed on the image to be segmented, local features and global features are extracted through a mask auto-encoding model, fusion features are obtained through multi-scale feature fusion; the semantic features and the fusion features are paired, the similarity between the two paired features is calculated, attention weights are generated, the paired features are fused based on the attention weights, and the image to be segmented is segmented. The segmentation accuracy, the detail extraction of a lesion boundary and the multi-scale lesion capturing capability are significantly improved, and the challenge of different scales and complex backgrounds in medical images is adapted.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to an image segmentation method, system, medium and device based on a dual-branch network architecture. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, with the progress of deep learning in the field of image segmentation, various convolutional neural networks (CNNs) and models based on self-attention mechanisms have gradually been applied to medical image segmentation tasks.

[0004] However, medical images usually have complex background interference, subtle lesion boundaries and multi-scale structural features. Traditional medical image segmentation usually relies on feature extraction and segmentation prediction of a single model. A single model cannot simultaneously take into account semantic information of different scales and levels, resulting in limited segmentation effect; in addition, noise and blur problems in medical images further increase the difficulty of accurate segmentation. Summary of the Invention

[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides an image segmentation method, system, medium and equipment based on a dual-branch network architecture, which uses a dual-branch architecture for image segmentation. One branch uses a diffusion model to process the image, and the other branch uses a masked autoencoder model to process the image. Through the cross-modal attention mechanism, the features extracted by the two models are aligned, the semantic representation is enhanced, and there are significant improvements in segmentation accuracy, detail extraction of lesion boundaries, and the ability to capture multi-scale lesions, adapting to the challenges of different scales and complex backgrounds in medical images.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A first aspect of the present invention provides an image segmentation method based on a dual-branch network architecture, comprising:

[0008] Obtain the image to be segmented;

[0009] For the image to be segmented, the complexity of the image to be segmented is measured through edge detection and texture analysis. After allocating the denoising step size of the diffusion model according to the complexity, the semantic features are extracted through the diffusion model.

[0010] After random occlusion and blurring of the segmented image, local and global features are extracted through the mask autoencoder model, and fused features are obtained through multi-scale feature fusion.

[0011] The semantic features and fusion features are paired, the similarity between the two paired features is calculated, and the attention weight is generated. After the paired features are fused based on the attention weight, the segmentation representation is obtained through the activation function, and the image to be segmented is segmented based on the segmentation representation.

[0012] Furthermore, the denoising step size is inversely proportional to the complexity of the image to be segmented.

[0013] Furthermore, the diffusion model and the mask autoencoder model are trained using pixel-level loss, feature alignment loss, and hierarchical reconstruction loss.

[0014] Furthermore, the feature alignment loss is: ,in, L is the total number of layers, The diffusion model and the masked autoencoder model are respectively l Layer characteristics, is a hyperparameter.

[0015] Furthermore, the masked autoencoder model simulates the missing image blocks by performing random occlusion and blurring on the image to be segmented, and then reconstructs the missing parts through the encoder. The encoder extracts local features through the multi-scale convolutional layer and captures global features through the Transformer layer; the local features and global features are fused at a multi-scale level to obtain a fused feature map.

[0016] Furthermore, the complexity is: , texture complexity , average edge strength complexity ,in, is the edge strength, and are the gradients in the horizontal and vertical directions respectively, N represents the total number of edge pixels involved in the calculation, α and β is the weight coefficient.

[0017] A second aspect of the present invention provides an image segmentation system based on a dual-branch network architecture, comprising:

[0018] An image acquisition module is configured to: acquire an image to be segmented;

[0019] A first feature extraction module is configured to: measure the complexity of the image to be segmented through edge detection and texture analysis, assign a denoising step size of a diffusion model according to the complexity, and then extract semantic features through the diffusion model;

[0020] The second feature extraction module is configured to: after random occlusion and blur processing on the image to be segmented, extract local features and global features through a mask auto-encoding model, and obtain fusion features through multi-scale feature fusion.

[0021] The image segmentation module is configured to: pair the semantic features and the fusion features, calculate the similarity between the paired two features, generate attention weights, fuse the paired features based on the attention weights, and obtain segmentation representation through an activation function, and segment the image to be segmented based on the segmentation representation.

[0022] Further, the denoising step is inversely proportional to the complexity of the image to be segmented.

[0023] A third aspect of the present application provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the image segmentation method based on the dual-branch network architecture as described above.

[0024] A fourth aspect of the present application provides a computer device comprising a computer readable storage medium, a processor and a computer program stored on the computer readable storage medium and executable on the processor, the processor executing the program to implement the steps of the image segmentation method based on the dual-branch network architecture as described above.

[0025] Compared with the prior art, the present application has the following advantages:

[0026] The present application uses a dual-branch architecture for image segmentation, one branch uses a diffusion model to process the image, and the other branch uses a mask auto-encoding model to process the image, and through a cross-modal attention mechanism, the features extracted by the two models are aligned, which enhances the semantic representation, and significantly improves the segmentation accuracy, the extraction of lesion boundary details and the ability to capture multi-scale lesions, and adapts to the challenges of different scales and complex backgrounds in medical images.

[0027] The present application aligns the features of the diffusion model and the MIM model through a cross-modal attention mechanism, and fuses semantic features and fusion features at each layer, which is more coordinated in capturing details and overall structure, and this mechanism enables the image segmentation model to simultaneously focus on subtle boundary features and overall semantic consistency at the pixel level, thereby improving the segmentation effect.

[0028] The application introduces an adaptive back propagation mechanism, so that the diffusion model can dynamically adjust the denoising step according to the image feature distribution and the current denoising state, to more finely preserve and extract semantic features. The denoising step of the diffusion model is allocated according to the complexity, if the edge and texture are complex and the detail information is rich, the step is reduced to more finely denoise and preserve the detail features, to ensure that the details of the complex area can be gradually recovered, and information loss is avoided; if the overall complexity is low, the step is increased, and the fineness of denoising is reduced, to more quickly denoise in the simple area.

[0029] The training of the diffusion model and the mask auto-encoding model of the application adopts pixel-level loss, feature alignment loss and hierarchical reconstruction loss, wherein the feature alignment loss can promote the diffusion model and the mask auto-encoding model to be more similar in different layers, to ensure the consistency of the features extracted by the two models in structure and semantics, thereby improving the overall performance of the model; the use of hierarchical reconstruction loss and other loss functions also helps the model to accurately learn and optimize at different levels, further improving the segmentation ability of the model.

[0030] The mask auto-encoding model of the application simulates the image block missing condition by performing random occlusion and blur processing on the image to be segmented, and then learns to reconstruct the missing part, which enhances the model's processing ability for incomplete information in the image, so that it can still maintain good segmentation performance when facing the image information missing or unclear situation that may occur in the actual scene, and improves the robustness and generalization ability of the model.

[0031] The encoder in the mask auto-encoding model extracts local features through multi-scale convolution layers and captures global features through Transformer layers, and the multi-scale feature fusion can fully combine local detail information and global structure information, so that the model can better understand the image, which is helpful for more accurately identifying and segmenting different objects and regions in the image. BRIEF DESCRIPTION OF DRAWINGS

[0032] The drawings accompanying the specification of the application form part of the application and serve to provide further understanding of the application, the illustrative embodiments of the application and their description serve to explain the application without constituting an improper limitation thereof.

[0033] Figure 1 is a flowchart of the image segmentation method based on the dual-branch network architecture of embodiment one of the application;

[0034] Figure 2 is a structure diagram of the diffusion model of embodiment one of the application;

[0035] Figure 3 is an experimental result diagram of embodiment one of the application;

[0036] Figure 4Fig. 4 is a structural schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0038] It should be pointed out that the following detailed description is exemplary and is intended to provide further illustration of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those of ordinary skill in the art to which the present application belongs.

[0039] Embodiment one

[0040] The embodiment provides an image segmentation method based on a double-branch network architecture.

[0041] The image segmentation method based on the double-branch network architecture provided by the embodiment proposes a double-branch network architecture that fuses multiple features to enhance the feature expression capability of the image segmentation model and improve the segmentation accuracy. One branch extracts different levels of semantic features using a diffusion model, and the other branch uses a pre-trained network of MIM to extract features. Through a cross-modal attention mechanism, the image segmentation model can align the two features and enhance the semantic representation, so as to make a prediction through a pixel classifier.

[0042] The diffusion model performs well in generative tasks, can gradually remove noise and extract multi-level semantic features, and is suitable for recovering detailed information of an image from noisy data; and the Masked Image Modeling (MIM) network learns to capture the global structural features of an image through the reconstruction of a masked image. The two models have complementarity in feature extraction at different levels, and combined use can obtain a comprehensive semantic representation of the target region.

[0043] The image segmentation method based on the double-branch network architecture provided by the embodiment is suitable for medical image segmentation.

[0044] The image segmentation method based on the double-branch network architecture provided by the embodiment, as shown in Figure 1 includes the following steps:

[0045] Step 1, obtaining an image to be segmented, for the image to be segmented, measuring the complexity of the image to be segmented through edge detection and texture analysis, assigning a denoising step length of a diffusion model according to the current complexity, and then extracting semantic features of different scales through the diffusion model.

[0046] The principle of the diffusion model is to convert a clean image into a noisy image by gradually adding noise, and then to restore the original image by denoising through the reverse process. The diffusion model generates image features at multiple time steps T, using a noise predictor (i.e., a U-Net network model) to generate the activation states at different stages. The U-Net (U-shaped network) model is the core module in the diffusion model, which mainly iteratively denoises the Gaussian noise matrix in the diffusion cycle. Using the reverse diffusion process of the diffusion model, the intermediate activation states of the labeled image are used to extract semantic features, and at different levels of the U-Net architecture in the reverse diffusion process, different levels of semantic features can be extracted for subsequent analysis and processing.

[0047] In particular, the present embodiment proposes an adaptive reverse diffusion mechanism, which introduces dynamic step adjustment based on image complexity in the reverse denoising process. Specifically, the complexity of the current state of the image is calculated before each reverse diffusion step, and a more detailed step size is assigned according to the complexity.

[0048] Step 101, initial setting of reverse diffusion: first, a diffusion model is pre-trained based on a large-scale image dataset (ImageNet); then, the pre-trained model is fine-tuned using a specific domain ultrasound dataset to adapt to the feature distribution of medical images; after fine-tuning is completed, the diffusion model starts the reverse diffusion process: gradually denoising the noisy image using the noise predictor of the U-Net architecture, this process restores the image from a random noise state to a clear image by gradually removing noise.

[0049] Before entering each reverse diffusion step, the diffusion model will dynamically calculate the complexity of the current image to determine the step size of the current denoising.

[0050] Step 102, calculate the complexity of the current image: before each reverse diffusion step, the current image state is extracted, and the complexity of the image is measured through edge detection and texture analysis.

[0051] Based on the complexity determination of edge detection, the edge can represent the outline of the lesion area, and the edge intensity and edge density of the image can be used to judge the image complexity, and the Sobel operator or Canny edge detection operator is used to obtain the edge gradient of the image.

[0052] Specifically, for a given image , the Sobel operators in the horizontal and vertical directions are used to obtain the gradients and respectively, where and are the horizontal and vertical Sobel kernels respectively.

[0053] Edge intensity which can be expressed as: .

[0054] The average edge intensity complexity formula is as follows: , N represents the total number of edge pixels participating in the calculation, which can be expressed as: , where W and H are the width and height of the image, respectively; 1(·) is an indicator function that takes the value 1 when the condition is true, and 0 otherwise; , indicating that the current pixel is considered an edge.

[0055] Based on the texture feature complexity judgment, on the basis of the Sobel operator, the average value of the local gradient intensity is calculated to represent the texture complexity: .

[0056] The complexity of the two can be combined into a comprehensive complexity measure by linear combination: , where α and β are weight coefficients.

[0057] Step 103, adjust the de-noising step size according to the comprehensive complexity: in the reverse diffusion process, the diffusion model assigns a dynamic step size according to the current complexity: if the comprehensive complexity is high (e.g. edge and texture complexity, rich in detail information), then reduce the step size to more carefully de-noise and preserve the detail features; if the comprehensive complexity is low, then increase the step size to reduce the de-noising precision to improve efficiency.

[0058] The adaptive de-noising of the dynamic step size enhances the preservation of local details in the image, making the extracted multi-scale semantic features more rich and accurate.

[0059] The reverse de-noising process of the traditional diffusion model is to gradually restore the image through a fixed noise step size. This strategy may not perform well on complex or subtle feature medical images. By introducing an adaptive reverse diffusion mechanism, the diffusion model can dynamically adjust the de-noising step size and intensity according to the image feature distribution and the current de-noising state, to more finely preserve and extract semantic features. For example, if the diffusion model detects that the current image features are very important, it can reduce the step size for more careful de-noising; on the contrary, if the noise has little effect on the image semantics, it can increase the step size to reduce the number of de-noising steps to improve efficiency.

[0060] Step 104, perform reverse diffusion de-noising with dynamic step size: use the adjusted step size to perform de-noising in each reverse diffusion step. Small step size ensures that the details of complex areas can be gradually restored, avoiding information loss; large step size allows faster de-noising in simple areas.

[0061] In each de-noising step, different levels of U-Net generate multi-scale activation states, and different levels of semantic features are extracted through these activation states.

[0062] Step 105, multi-scale feature fusion and semantic feature extraction.

[0063] In the middle layer of each back-propagation step, semantic features are extracted from the activation states of different scales of the U-Net, and a multi-scale feature pyramid structure is used to ensure the comprehensiveness of the feature information.

[0064] As shown in Figure 2 To extract higher quality and multi-scale fine-grained semantic features, the embodiment introduces a multi-scale dynamic weight feature pyramid mechanism to improve the expression ability of different scale features in medical images. A multi-scale U-Net architecture is used to capture rich information in images from global structure to local details by simultaneously performing feature extraction and noise prediction at different scales during the diffusion process. Specifically, in the U-Net encoder stage, a multi-scale feature pyramid is constructed, each scale feature is dynamically assigned a weight through a scale evaluation module, and the weight ratio is adjusted through Softmax normalization for further weighted fusion. To ensure that each scale feature can effectively represent information of different granularities, different sizes of convolution kernels are used in the generation process of the feature pyramid, so as to promote the diffusion model to consider both macrostructure and fine-grained features of small lesions when restoring features. In addition, to further improve feature selectivity, a hybrid weighted attention mechanism is introduced in the U-Net encoder stage, which is a dual-branch structure of channel attention and spatial attention. It can dynamically focus on key areas and channels with more diagnostic significance during multi-scale feature fusion, improving the model's response ability to key information.

[0065] Step 2, for the image to be segmented, the fusion feature map is obtained through the MIM model and multi-scale feature fusion.

[0066] The MIM model simulates the situation of missing parts of the image by randomly occluding and blurring the image. The MIM model learns the ability to reconstruct these missing parts through the encoder, thereby extracting the detailed representation of the image. This process uses convolution operations for local feature extraction and Vision Transformers to capture global information. Transformer is a deep learning model architecture that introduces a self-attention mechanism, and Vision Transformer applies Transformer to visual tasks, with strong ability to model global information.

[0067] The MIM model combines multi-scale convolution and Vision Transformer structure. Local convolution operations help capture details and local information, while global Transformer operations effectively capture long-range dependencies and global information. This approach ensures that the model can handle details while maintaining the coherence of global semantics.

[0068] Step 201, input module: random occlusion and blur processing.

[0069] The input image is preprocessed, and several image blocks are randomly occluded to make the information in these areas missing. At the same time, the occluded areas are blurred to simulate the situation of image detail blur or loss. This allows the MIM model to learn how to complete and restore the missing areas.

[0070] Step 202, MIM encoder structure.

[0071] Multi-scale convolution layer: First, apply multi-scale convolution kernels in the encoder to obtain different scale details of the image. These convolution operations capture local features at different scales, allowing the model to have a deeper understanding of the details of the occluded areas and their adjacent parts.

[0072] Use convolution kernels of different sizes (such as 3x3, 5x5, etc.) to extract multi-scale features to ensure that the details of the image are captured.

[0073] Transformer layer: After the multi-scale convolution layer, introduce the Transformer module to extract global semantic information. The Transformer structure uses self-attention mechanisms to model the long-range dependencies between regions in the image, thereby obtaining global information.

[0074] In the attention mechanism, consider the features output by the multi-scale convolution, so that the information of the occluded area can better integrate with its surrounding areas.

[0075] Through the self-attention mechanism, the MIM model can complete and restore the semantic content of the occluded area at the global level.

[0076] Step 203, multi-scale feature fusion module:

[0077] First, the multi-scale feature fusion module receives the local features extracted by the multi-scale convolution layer (such as 3x3, 5x5, 7x7 convolution output) and the global features of the Transformer layer. Each feature may be different in dimension, so it needs to be aligned and dimensionally processed.

[0078] Among them, the feature alignment and dimension processing includes:

[0079] (1) Channel lifting dimension: Since the features from different scale convolutions and Transformers may differ in the number of channels, first align the channel numbers of these features by 1x1 convolution, ensuring that all features have consistent channel dimensions.

[0080] (2) Spatial alignment: If the spatial resolutions of multi-scale feature maps are different, bilinear interpolation or transpose convolution can be used to upsample or downsample the feature maps, aligning all feature maps to the same spatial size for subsequent fusion.

[0081] Next, the multi-scale feature weight generator includes:

[0082] (1) Feature importance weight: To dynamically adjust the importance of different scale features, a weight generator is introduced, usually using a lightweight fully connected layer or attention mechanism to assign weights to each scale feature. By learning these weights, the MIM model can adaptively adjust the contribution of each scale feature according to the input image content.

[0083] (2) Calculation method: Each scale feature is input to the weight generator to generate corresponding weight values, and normalized by softmax (normalized exponential function) to make the sum of all scale feature weights equal to 1.

[0084] Next, feature weighted fusion includes:

[0085] (1) Scale-wise weighting: Multiply different scale features with their corresponding weights to ensure that the contribution ratio of each scale feature in the fusion process conforms to the weight distribution.

[0086] (2) Feature addition: Element-wise add all scale weighted features to get the fused feature map. At this time, the detailed information and global semantic information are integrated to form a unified multi-scale feature representation.

[0087] Step 204, output layer (fused feature enhancement): The fused feature map is further processed by a 3x3 convolution kernel to enhance local feature details and reduce information loss in the fusion process.

[0088] Step 205, through an activation function (such as ReLU or Leaky ReLU) for nonlinear transformation, to ensure that the feature map representation ability is further enhanced, and the segmentation accuracy of the MIM model is improved. ReLU (Rectified Linear Unit) is an activation function commonly used in neural networks, which refers to a ramp function in mathematics, the formula is as follows: The Leaky Rectified Linear Unit (LeakyReLU) is a variant of the ReLU activation function.

[0089] Step 3: Pair the semantic features with the fused feature map, calculate the similarity between the two paired features, generate attention weights, fuse the paired features based on the attention weights, obtain the segmentation representation through the activation function, and segment the image to be segmented based on the segmentation representation.

[0090] A cross-attention module is proposed to align features extracted from the diffusion model and the MIM model. Features extracted from different diffusion decoder layers are aligned with corresponding MIM features. The diffusion model and the MIM model each process different aspects of the image. The diffusion model focuses on capturing semantic features through progressive denoising, while the MIM model learns global and local information of the image by reconstructing occluded areas. The cross-attention module aligns the features extracted from different diffusion decoder layers with the corresponding MIM model features.

[0091] The local features extracted by each diffusion decoder layer are aligned with the corresponding hierarchical features of the MIM model to ensure that the fusion of local and global information can be fully utilized. This alignment operation not only enhances the mutual understanding of features from different models, but also improves segmentation accuracy.

[0092] To address the problem of insufficient context caused by attention calculation limited to the current level, the cross-modal attention module introduces a cross-level attention transfer mechanism. Through this mechanism, low-level detail features can effectively flow to high-level feature layers, helping to generate a more complete semantic representation.

[0093] The cross-level attention transfer mechanism specifically includes:

[0094] Information transfer between layers: Setting up cross-layer connections to combine low-level features with high-level features so that low-level details can provide support for high-level features;

[0095] By introducing hierarchical information into cross-modal data through layer-by-layer fusion, the alignment accuracy is improved.

[0096] In this embodiment, the specific processing steps of the cross-modal attention module are as follows:

[0097] Step 301: Obtain features of the diffusion model decoder layer and the MIM model encoder layer, and pair them at the same level to form feature pairs.

[0098] Step 302, cross-model attention calculation: for each pair of features, first generate attention weights by calculating the similarity between features, which measure the similarity of the current diffusion feature (semantic feature) and the MIM feature (fusion feature). The specific implementation is as follows:

[0099] Calculate the feature similarity matrix by dot product: ; where: and represent the feature vectors of the i-th and j-th positions in the semantic feature and the fusion feature ; respectively. represents the dot product, represents the norm of the feature vector; S(i,j) represents the similarity between position i and position j, ranging from [-1, 1];

[0100] Based on the similarity matrix S, calculate the attention weight by the softmax function:

[0101] ; where: S(i,j) is the similarity of and , k represents the feature index. measure the influence of on , normalized by softmax, ensuring that the sum of weights for each position is 1.

[0102] , measure the influence of on , normalized by softmax, ensuring that the sum of weights for each position is 1.

[0103] Step 303, weighted fusion feature output.

[0104] According to the attention weight, weight and add each pair of features to obtain the final fused feature. Process the fusion feature through the activation function to form the final segmentation representation, ensuring that global and local information is integrated and optimized.

[0105] According to the attention weight, weight and add two kinds of features to obtain the fused feature representation . The bidirectional attention mechanism ensures that the fusion process considers the interaction of the two features, avoiding bias towards a single source.

[0106] ; where, α and β are hyperparameters, used to adjust the weights of diffusion features and MIM features.

[0107] Step 304, output result. The fusion features output by the cross-modal attention module enter the downstream pixel classifier to further improve the segmentation accuracy.

[0108] In this embodiment, the diffusion model, the mask auto-encoding model, the cross-modal attention module, and the pixel classifier constitute an image segmentation model.

[0109] In this embodiment, two loss functions are used to train the image segmentation model:

[0110] Pixel-level loss: The quality of the segmentation result is measured by the Dice coefficient to optimize the features extracted by the diffusion model and the MIM model. The Dice coefficient is a set similarity measure function, commonly used to calculate the similarity of two samples, and is a common evaluation index for semantic segmentation, with a value range of [0, 1];

[0111] Cross-modal alignment loss: used to ensure that the features extracted by the diffusion model and the MIM model are well aligned and fused in the cross-modal attention mechanism.

[0112] Specifically, the cross-modal alignment loss is used to optimize the feature fusion of the diffusion model and the MIM model. This loss function measures the similarity between the two modal features to ensure that they are well aligned in high-dimensional space. The cross-modal alignment loss can be divided into two main parts: feature alignment loss and hierarchical reconstruction loss, each of which helps to effectively align between cross-modal features.

[0113] Feature alignment loss: measures the similarity between features from different modalities: where L is the total number of layers, and are the features of the diffusion model and the MIM model at the l layer, respectively, is a hyperparameter, cos and represents the cosine similarity.

[0114] Hierarchical reconstruction loss can be used to ensure that the cross-modal attention module maintains the integrity of information when fusing features. Mean squared error (MSE) can be used to measure the difference between the reconstructed features and the true labels: where, is the output of the l layer reconstruction, is the true label, is a hyperparameter that controls the importance of different levels of reconstruction loss, and N is the number of samples.

[0115] The image segmentation model training is performed by joint optimization, gradually adjusts the feature extraction processes of the diffusion model and the MIM model, and ensures the alignment of the features of the two through the cross-modal attention mechanism. The whole training process includes forward propagation, loss calculation, backward propagation and gradient update, and the segmentation performance is gradually optimized through multiple iterations.

[0116] The experimental results of the image segmentation method based on the dual-branch network architecture provided in the embodiment are as shown in the following table. Figure 3

[0117] The image segmentation method based on the dual-branch network architecture provided in the embodiment uses a dual-branch architecture for image segmentation, one branch uses a diffusion model to process images, and the other branch uses a pre-trained network of MIM to capture details in medical images. Through the cross-modal attention mechanism, the two kinds of features are aligned, the semantic representation is enhanced, and prediction is performed through a pixel classifier. The following technical effects are achieved:

[0118] Rich multi-level semantic feature extraction: the diffusion model branch gradually restores the image, and the diffusion model can generate multi-level semantic features, including both the detailed features of the lesion area and the boundary features between the background and the lesion, providing strong support for fine segmentation;

[0119] Enhanced global structure representation: through the MIM network branch, the MIM model can learn large-scale structure information in the image from a global perspective, has stronger semantic understanding ability in the discrimination of lesion areas and background areas, and helps to improve the robustness of the image segmentation model in complex backgrounds;

[0120] Cross-modal alignment enhances feature fusion: through the cross-modal attention mechanism, the features of the diffusion model and the MIM model are aligned, local and global features are fused at each layer, and the capture of detailed and overall structure is more coordinated. This mechanism enables the image segmentation model to simultaneously focus on subtle boundary features and overall semantic consistency at the pixel level, thereby improving segmentation performance;

[0121] Precise segmentation prediction: after combining the features of the dual-branch, a pixel classifier is used to classify the enhanced semantic features at the pixel level. Through this fusion strategy, the image segmentation model has significant improvement in segmentation accuracy, detail extraction of lesion boundaries, and capture ability of multi-scale lesions, adapting to the challenges of different scales and complex backgrounds in medical images.

[0122] Embodiment Two

[0123] The embodiment provides an image segmentation system based on a dual-branch network architecture, which comprises:

[0124] An image acquisition module configured to acquire an image to be segmented; ​

[0125] A first feature extraction module is configured to: measure the complexity of the image to be segmented through edge detection and texture analysis, assign a denoising step size of a diffusion model according to the complexity, and then extract semantic features through the diffusion model;

[0126] The second feature extraction module is configured to: after randomly occluding and blurring the image to be segmented, extract local features and global features through a mask autoencoder model, and obtain fused features through multi-scale feature fusion;

[0127] The image segmentation module is configured to: pair semantic features and fusion features, calculate the similarity between the two paired features, generate attention weights, fuse the paired features based on the attention weights, obtain segmentation representations through activation functions, and segment the image to be segmented based on the segmentation representations.

[0128] Furthermore, the denoising step size is inversely proportional to the complexity of the image to be segmented.

[0129] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation processes are the same, which will not be repeated here.

[0130] Example 3

[0131] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the image segmentation method based on the dual-branch network architecture as described in the first embodiment above are implemented.

[0132] Example 4

[0133] This embodiment provides a computer device, such as Figure 4 As shown, the present invention includes a computer-readable storage medium 1003, a processor 1001, a communication interface 1002, and a computer program stored on the computer-readable storage medium 1003 and executable on the processor 1001. The processor 1001, the communication interface 1002, and the computer-readable storage medium 1003 may be connected via a bus or other means. The communication interface 1002 is used to receive and send data, and when the processor 1001 executes the program, the steps of the image segmentation method based on the dual-branch network architecture described in the first embodiment are implemented.

[0134] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. An image segmentation method based on a dual-branch network architecture, characterized in that, The method comprises the following steps: acquiring an image to be segmented; measuring the complexity of the image to be segmented by edge detection and texture analysis, assigning a denoising step of a diffusion model according to the complexity, and extracting semantic features by the diffusion model; after random occlusion and blurring processing of the image to be segmented, extracting local features and global features by a mask auto-encoding model, and obtaining fused features by multi-scale feature fusion; pairing the semantic features and the fused features, calculating the similarity between the paired two features to generate attention weights, fusing the paired features based on the attention weights, and obtaining segmentation representation by an activation function, and segmenting the image to be segmented based on the segmentation representation; the denoising step is inversely proportional to the complexity of the image to be segmented; the mask auto-encoding model simulates the image block missing condition by random occlusion and blurring processing on the image to be segmented, learns to reconstruct the missing part by an encoder, extracts local features by a multi-scale convolution layer of the encoder, and captures global features by a Transformer layer; and the local features and the global features are fused by multi-scale feature fusion to obtain a fused feature map. 2.The method of claim 1, wherein, The training of the diffusion model and the mask auto-encoding model adopts pixel-level loss, feature alignment loss and hierarchical reconstruction loss. 3.The method of claim 2, wherein, The feature alignment loss is: wherein, L is the total number of layers, are the features of the diffusion model and the mask autoencoding model at the l layer, respectively, is a hyperparameter. 4.The method of claim 1, wherein, The complexity is: , texture complexity , average edge intensity complexity wherein, is the edge intensity, and are the horizontal and vertical gradients, respectively, and N represents the total number of edge pixels participating in the calculation, α and β are weight coefficients.

5. The image segmentation system based on a dual-branch network architecture, characterized in that, The method comprises the following steps: an image acquisition module configured to acquire an image to be segmented; a first feature extraction module configured to measure the complexity of the image to be segmented by edge detection and texture analysis, assign a denoising step of a diffusion model according to the complexity, and extract semantic features by the diffusion model; a second feature extraction module configured to, after random occlusion and blurring processing of the image to be segmented, extract local features and global features by a mask auto-encoding model, and obtain fused features by multi-scale feature fusion; an image segmentation module configured to pair the semantic features and the fused features, calculate the similarity between the paired two features to generate attention weights, fuse the paired features based on the attention weights, obtain segmentation representation by an activation function, and segment the image to be segmented based on the segmentation representation; the denoising step is inversely proportional to the complexity of the image to be segmented; the mask auto-encoding model simulates the image block missing condition by random occlusion and blurring processing on the image to be segmented, learns to reconstruct the missing part by an encoder, extracts local features by a multi-scale convolution layer of the encoder, and captures global features by a Transformer layer; and the local features and the global features are fused by multi-scale feature fusion to obtain a fused feature map.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the steps in the image segmentation method based on the dual-branch network architecture in any one of claims 1-4.

7. A computer device, comprising a computer readable storage medium, a processor, and a computer program stored on the computer readable storage medium and executable on the processor, wherein, The processor executes the program to realize the steps in the image segmentation method based on the dual-branch network architecture in any one of claims 1-4.

Images