Abdomen multi-organ segmentation method based on dynamic fusion

Abstract

An abdomen multi-organ segmentation method based on dynamic fusion constructs an abdomen multi-organ segmentation network dynamically fusing CNN and Transform features, the network adopts a U-shaped structure with jump connection, local and global features are extracted through the CNN and Transform modules connected in parallel, weights are updated in a frequency domain by using a dynamic fusion module, and the dynamic fusion module is used for extracting the local and global features. Effective fusion of local and global information is realized; according to the method, the accuracy of abdomen multi-organ segmentation is improved, and powerful support is provided for disease diagnosis, treatment planning and curative effect evaluation; experiments show that the method achieves a good segmentation effect on a Synapse data set, and compared with a traditional method, the method has the advantages that the segmentation accuracy is improved by about 10%, and meanwhile, higher robustness is shown when a complex abdomen image is processed; in addition, due to the design of the network, the network can be easily expanded to other medical image segmentation tasks, and wide application potential and value are shown.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to an abdominal multi-organ segmentation method based on dynamic fusion. Background Art

[0002] Abdominal multi-organ segmentation plays an important role in medical image analysis and is an indispensable key step in disease diagnosis, treatment planning and efficacy evaluation. The abdomen contains many complex and interrelated organs, such as the liver, kidneys, spleen, pancreas, etc. These organs have different morphologies, overlap with each other, and are affected by multiple factors such as individual differences, pathological changes, and imaging conditions, making abdominal multi-organ segmentation a very challenging task.

[0003] At present, abdominal multi-organ segmentation technology is mainly divided into two categories: traditional methods and deep learning-based methods.

[0004] Traditional methods usually rely on manually extracted image features for image segmentation. These methods include but are not limited to threshold methods, graph cut methods, region growing methods, multi-atlas marker fusion methods, and statistical shape model methods. Although these methods can achieve certain segmentation effects in certain specific cases, they generally have problems such as being time-consuming and labor-intensive, complex to operate, and difficult to reproduce segmentation results. In addition, the segmentation accuracy of traditional methods is often limited by the operator's experience and knowledge, making it difficult to achieve high-precision segmentation, especially for the segmentation task of complex abdominal organs, where its limitations are particularly prominent.

[0005] With the rapid development of deep learning technology, deep learning-based methods have gradually become the mainstream technology for abdominal multi-organ segmentation. Deep learning models, especially convolutional neural networks (CNN) and Transformer models, have shown great potential in the field of image segmentation. CNN has local perception and can capture edge information well, while the Transformer model focuses more on modeling global information and can better capture long-distance dependencies. Therefore, combining CNN with Transformer can consider both local and global information, avoid the shortcomings of using a single architecture, and complement the advantages of the two, thereby significantly improving the segmentation effect.

[0006] However, existing deep learning-based abdominal multi-organ segmentation methods still have some shortcomings. For example, some methods use CNN and Transformer in series, but the input to the Transformer model is usually highly abstract and has extremely low spatial resolution feature maps. The spatial information of these feature maps has been greatly compressed and cannot fully utilize the global modeling capabilities of the Transformer. Although other methods try to use CNN and Transformer in parallel, the lack of effective feature interaction during the feature fusion process may lead to information loss, inability to adaptively adjust the importance of features, and loss of feature independence, resulting in decreased organ segmentation accuracy.

[0007] Therefore, how to overcome the shortcomings of the existing technology and propose a method for abdominal multi-organ segmentation that can fully utilize the respective advantages of CNN and Transformer and effectively integrate the characteristics of both has become a technical problem that needs to be solved urgently. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a method for abdominal multi-organ segmentation based on dynamic fusion to solve the technical difficulties faced by abdominal multi-organ segmentation in medical image analysis. Specifically, it is to address the problems of insufficient accuracy, low efficiency and poor segmentation effect of complex organ structures and small target lesion areas in the existing abdominal multi-organ segmentation technology.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for abdominal multi-organ segmentation based on dynamic fusion, comprising the following steps: Step 1: Establish a training dataset containing abdominal CT images and their corresponding multi-organ segmentation results; Step 2: Build an abdominal multi-organ segmentation network that dynamically integrates CNN and Transformer features; Step 3: Use the test set to test the trained multi-organ segmentation network and obtain the segmentation results.

[0010] In a preferred embodiment, the implementation steps of Step 2 of constructing an abdominal multi-organ segmentation network that dynamically integrates CNN and Transformer features include: Step 2.1: A U-shaped structure with skip connections is used. The encoding path and decoding path are composed of at least one encoding block and decoding block cascaded, respectively. Each encoding block contains a CNN feature extraction module, a Transformer feature extraction module, a downsampling layer, and a dynamic fusion module. Step 2.2: Local and global feature extraction, select CNN and Transformer in parallel to extract features; Step 2.3: Dynamically fuse features. Use the dynamic fusion module in Step 2.1 to convert the feature image from the spatial domain to the frequency domain, update the weights in the frequency domain, and finally convert the frequency domain features with updated weights back to the spatial domain. Enhance the features through the pooling activation function layer and then assign weights.

[0011] In the preferred solution, the CNN feature extraction module in Step 2.1 uses deep convolution, which is composed of deep convolution and Activation functions are composed of cascades.

[0012] In the preferred solution, the Transformer feature extraction module in Step 2.1 adopts the transposed attention mechanism to achieve feature interaction by calculating attention across channels.

[0013] In a preferred solution, the dynamic fusion module in Step 2.1 includes a fast Fourier transform, a convolution activation function layer, an inverse fast Fourier transform layer and a pooling activation function layer, which are used to dynamically fuse local features and global feature information.

[0014] In a preferred solution, the encoding block and the decoding block in Step 2.1 add a skip connection between the encoding and decoding blocks at the same level, pass the feature map of the encoder to the decoder through the skip connection, and merge it with the features output by the upsampling layer in the decoder.

[0015] In a preferred solution, the implementation steps of Step 2.2 local and global feature extraction include: Step 2.2.1: For input features , generated using linear projection , and matrix: (1) In the formula, is the projection matrix, for The projection matrix is ​​used to project the input vector into the query space. is the generated query matrix; for The projection matrix is ​​used to project the input vector into the key space, is the generated key matrix; for The projection matrix is ​​used to project the input vector into the value space, is the generated value matrix; Step 2.2.2: Get Perform a Reshape operation to get the correct dimension; Step2.2.3: and Perform a dot multiplication operation and then normalize it through the Softmax function; Step 2.2.4: Get the C×C similarity matrix attn, and finally attn and Multiply to get the final output, the specific formula is: (2) In the formula, is the transpose of the key matrix, is the dimension of the key vector; The CNN module uses deep convolution, which is composed of deep convolution and The activation function cascades together, and the operation of the depth convolution is expressed as follows: (3) In the formula, represents the convolution operation, It is feature maps of output channels, It is An image with input channels, It is convolution kernels; The operation of the CNN module can be expressed by the following formula: (4) In the formula, is the output feature of the CNN module.

[0016] In a preferred solution, the implementation steps of Step 2.3 dynamic fusion features include: Step 2.3.1: Given non-local features and local features ,connect and Get Features ; Step 2.3.2: Features enter In the Two-Dimensional Fast Fourier Transform (FFT), the image is converted from the spatial domain to the frequency domain, and the amplitude Amp and phase Pha information are separated: (5) In the formula, is the representation in the frequency domain, that is, the result after two-dimensional fast Fourier transform. is the amplitude information, is the phase information.

[0017] Then, the amplitude information and phase information are passed through the convolution layer and Activation function to enhance image contrast and boundary information: (6) In the formula, is the enhanced amplitude information, is the enhanced phase information.

[0018] Step 2.3.3: The enhanced frequency domain features are passed through (Two-Dimensional Inverse Fast Fourier Transform) Convert back to the spatial domain to get the features ; Step 2.3.4: Features The channels are split and the weight map is generated by pooling and Sigmoid activation function : (7) In the formula, is the Sigmoid activation function; Step 2.3.5: Local features and non-local features Perform weight distribution to achieve dynamic fusion: (8) The scale transformation process of the features extracted in Step 2.3.2 in the above steps is as follows: (9) In the formula, represents the image features of the original input, represents the transformed image features, , , Respectively represent the length, width and dimension of the feature image; It means to concatenate two features, and the number of channels is added; represents fast Fourier transform; , represents the activation function; represents inverse fast Fourier transform; Indicates the feature channel splitting operation.

[0019] In a preferred solution, the specific implementation steps of Step 3 of using a test set to test the trained multi-organ segmentation network to obtain a segmentation result include: Step 3.1: Set the resolution and patch size of the input image. During training, all models are trained using the SGD optimizer. Step 3.2: Use the Synapse dataset, including CT or MRI images, covering at least eight categories of abdominal organs; Step 3.3: Use DSC (Dice Similarity Cofficient) and HD (Hausdorff , Hausdorff distance) as an evaluation index to evaluate whether the segmentation results meet the purpose of authenticity: DSC is a metric function used to evaluate the similarity between two samples. Its value range is between 0 and 1. The larger the value, the more similar it is. Assuming the ground truth is The prediction result is , the calculation formula is as follows: (10) In the formula, Represent two sets, and Respectively and The number of elements of represents true positives, that is, the number of pixels marked as positive samples by both the segmentation result and the true segmentation; It represents false negatives, that is, the number of pixels that the segmentation result marks as negative samples but the true segmentation marks as positive samples; It represents false positive, that is, the number of pixels that the segmentation result marks as a positive sample but the actual segmentation marks as a negative sample. The numerator is multiplied by 2 to ensure that the denominator is in the range of [0,1] after repeated calculation. Equivalent to , represents the harmonic mean of precision and recall, represents the performance score of the classifier; HD represents the maximum distance between the predicted segmentation region boundary and the true region boundary. The smaller the value, the smaller the predicted boundary segmentation error and the better the quality. The calculation formula is as follows: (11) In the formula, Represent two sets, Represents a point set Every point in Calculate its point set Every point in The distance of , retain the shortest distance, and then find the maximum distance among the retained shortest distances, Represents a point set Every point in Calculate its point set Every point in distances, retain the shortest distance, and then find the maximum distance among the retained shortest distances.

[0020] The present invention provides a method for abdominal multi-organ segmentation based on dynamic fusion, which has the following beneficial effects: 1. The present invention solves the technical difficulties faced by abdominal multi-organ segmentation in medical image analysis. Specifically, it solves the problems of insufficient accuracy, low efficiency and poor segmentation effect of complex organ structures and small target lesion areas in the existing abdominal multi-organ segmentation technology. 2. The parallel structure of CNN and Transformer adopted in the present invention is different from the CNN and Transformer series connection in the prior art. This paper adopts the parallel structure of CNN and Transformer, which can simultaneously obtain local details and global structures in the image, understand the data more comprehensively, and reduce computing resources; 3. The present invention designs a dynamic fusion module, which transforms the image from the spatial domain to the frequency domain through Fourier transform, enhances the amplitude and phase information respectively, and then returns to the spatial domain through inverse Fourier transform, dynamically generates two weight maps to adaptively re-weight the non-local features and local features. This dynamic fusion method enables the model to weight the features of both to different degrees at different layers, thereby achieving more effective feature fusion; 4. The present invention overcomes the challenges of segmenting the abdomen, which contains multiple complex and interrelated organs, such as the liver, kidneys, spleen, and pancreas. It solves the problems of abdominal organ diversity, shape variation, overlap between adjacent organs, and noise, artifacts, and lack of contrast in the image. 5. The present invention solves the problem that most existing methods combining CNN and Transformer are insufficient in feature expression, resulting in poor segmentation of the positional relationship between multiple organs and blurred tissue edges. By designing a dynamic fusion module, the feature expression ability of the model is improved; 6. The present invention dynamically integrates the features of CNN and Transformer, giving full play to the advantages of both and improving the segmentation accuracy of multiple abdominal organs; 7. Compared with the prior art method of connecting CNN and Transformer in series, the present invention adopts a parallel method and combines a dynamic fusion module to reduce the demand for computing resources; 8. The present invention has broad application prospects in medical image analysis and can assist doctors in disease diagnosis, treatment planning and efficacy evaluation; 9. The present invention also provides visualized segmentation results, enabling doctors to more intuitively understand and analyze the anatomical structure of multiple abdominal organs, further improving the accuracy and efficiency of clinical decision-making, and is expected to become an important technological breakthrough in the field of medical image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will be further described below in conjunction with the accompanying drawings and implementation examples: Figure 1 It is a technical flow chart of the present invention; Figure 2 is a structural diagram of the transposed attention module in the present invention; Figure 3 It is a structural diagram of the CNN module in the present invention; Figure 4 It is a structural diagram of the DF dynamic fusion module in the present invention; Figure 5 This is a schematic diagram of the abdominal multi-organ segmentation network structure of Example 3 of the present invention. DETAILED DESCRIPTION

[0022] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments: Example 1 like Figures 1 to 5 As shown, a method for abdominal multi-organ segmentation based on dynamic fusion includes the following steps: Step 1: Create a training dataset containing abdominal CT (Computed Tomography) images and their corresponding multi-organ segmentation results; Step 2: Build an abdominal multi-organ segmentation network that dynamically integrates CNN (Convolutional Neural Networks) and Transformer features; Step 3: Use the test set to test the trained multi-organ segmentation network and obtain the segmentation results.

[0023] In this embodiment, the implementation steps of Step 2 of constructing an abdominal multi-organ segmentation network that dynamically integrates CNN and Transformer features include: Step 2.1: A U-shaped structure with skip connections is used. The encoding path and decoding path are composed of at least one encoding block and decoding block cascaded, respectively. Each encoding block contains a CNN feature extraction module, a Transformer feature extraction module, a downsampling layer, and a dynamic fusion module. Step 2.2: Local and global feature extraction, select CNN and Transformer in parallel to extract features; Step 2.3: Dynamically fuse features. Use the dynamic fusion module in Step 2.1 to convert the feature image from the spatial domain to the frequency domain, update the weights in the frequency domain, and finally convert the frequency domain features with updated weights back to the spatial domain. Enhance the features through the pooling activation function layer and then assign weights.

[0024] Furthermore, the CNN feature extraction module in Step 2.1 uses deep convolution, which is composed of deep convolution and Activation functions are composed of cascades.

[0025] Furthermore, the Transformer feature extraction module in Step 2.1 adopts the transposed attention mechanism to achieve feature interaction by calculating attention across channels.

[0026] Furthermore, the dynamic fusion module in Step 2.1 includes a fast Fourier transform, a convolution activation function layer, an inverse fast Fourier transform layer and a pooling activation function layer, which are used to dynamically fuse local features and global feature information.

[0027] Furthermore, in the encoding block and the decoding block in Step 2.1, a skip connection is added between the encoding and decoding blocks at the same level, and the feature map of the encoder is passed to the decoder through the skip connection to be merged with the features output by the upsampling layer in the decoder.

[0028] Furthermore, the implementation steps of Step 2.2 local and global feature extraction include: Step 2.2.1: For input features , generated using linear projection , and matrix: (1) In the formula, is the projection matrix, for The projection matrix is ​​used to project the input vector into the query space. is the generated query matrix; for The projection matrix is ​​used to project the input vector into the key space, is the generated key matrix; for The projection matrix is ​​used to project the input vector into the value space, is the generated value matrix; Step 2.2.2: Get Perform a Reshape operation to get the correct dimensions; Step2.2.3: and Perform a dot multiplication operation, and then normalize it through the Softmax function (normalized exponential function); Step 2.2.4: Get the C×C similarity matrix attn, and finally attn and Multiply to get the final output, the specific formula is: (2) In the formula, is the transpose of the key matrix, is the dimension of the key vector; The CNN module uses deep convolution, which is composed of deep convolution and The activation function cascades together, and the operation of the depth convolution is expressed as follows: (3) In the formula, represents the convolution operation, It is The feature map of the output channels, It is An image with input channels, It is convolution kernels; The operation of the CNN module can be expressed by the following formula: (4) In the formula, is the output feature of the CNN module.

[0029] Furthermore, the implementation steps of the Step 2.3 dynamic fusion feature include: Step 2.3.1: Given non-local features and local features ,connect and Get Features ; Step 2.3.2: Features enter In the Two-Dimensional Fast Fourier Transform (FFT), the image is converted from the spatial domain to the frequency domain, and the amplitude Amp and phase Pha information are separated: (5) In the formula, is the representation in the frequency domain, that is, the result after two-dimensional fast Fourier transform. is the amplitude information, is the phase information.

[0030] Then, the amplitude information and phase information are passed through the convolution layer and Activation function to enhance image contrast and boundary information: (6) In the formula, is the enhanced amplitude information, is the enhanced phase information.

[0031] Step 2.3.3: The enhanced frequency domain features are passed through (Two-Dimensional Inverse Fast Fourier Transform) Convert back to the spatial domain to get the features ; Step 2.3.4: Features The channels are split and the weight map is generated by pooling and Sigmoid activation function : (7) In the formula, is the Sigmoid activation function; Step 2.3.5: Local features and non-local features Perform weight distribution to achieve dynamic fusion: (8) The scale transformation process of the features extracted in Step 2.3.2 in the above steps is as follows: (9) In the formula, represents the image features of the original input, represents the transformed image features, , , Respectively represent the length, width and dimension of the feature image; It means to concatenate two features, and the number of channels is added; represents fast Fourier transform; , represents the activation function; represents inverse fast Fourier transform; Indicates the feature channel splitting operation.

[0032] Furthermore, the specific implementation steps of Step 3 of using the test set to test the trained multi-organ segmentation network to obtain the segmentation result include: Step 3.1: Set the resolution and patch size of the input image. During training, all models are trained using the SGD (Stochastic Gradient Descent) optimizer. Step 3.2: Use the Synapse dataset (multi-organ segmentation dataset), including CT or MRI (Magnetic Resonance Imaging) images, covering at least eight categories of abdominal organs; Step 3.3: Use DSC (Dice Similarity Cofficient) and HD (Hausdorff , Hausdorff distance) as an evaluation index to evaluate whether the segmentation results meet the purpose of authenticity: DSC is a metric function used to evaluate the similarity of two samples. Its value range is between 0 and 1. The larger the value, the more similar it is. Assume that the ground truth (real data) is The prediction result is , the calculation formula is as follows: (10) In the formula, Represent two sets, and Respectively and The number of elements of (True Positive) means true positive, that is, the number of pixels marked as positive samples by both the segmentation result and the true segmentation; (False Negative) means false negative, that is, the number of pixels where the segmentation result marks a pixel as a negative sample but the actual segmentation marks it as a positive sample; (False Positive) indicates false positive, that is, the number of pixels that are marked as positive samples by the segmentation result but are marked as negative samples by the actual segmentation. The numerator is multiplied by 2 to ensure that the denominator is within the range of [0,1] after repeated calculation. Equivalent to , Represents the harmonic mean of precision and recall. represents the performance score of the classifier; HD represents the maximum distance between the predicted segmentation region boundary and the true region boundary. The smaller the value, the smaller the predicted boundary segmentation error and the better the quality. The calculation formula is as follows: (11) In the formula, Represent two sets, Represents a point set Every point in Calculate its point set Every point in The distance of , retain the shortest distance, and then find the maximum distance among the retained shortest distances, Represents a point set Every point in Calculate its point set Every point in distances, retain the shortest distance, and then find the maximum distance among the retained shortest distances.

[0033] Example 2 In another preferred embodiment, based on the above embodiment 1, Figures 1 to 5 As shown, this embodiment describes in detail the implementation steps of a method for segmenting multiple abdominal organs based on dynamic fusion, which are as follows: Step 1: Create a training dataset First, a training dataset A containing abdominal CT images and their corresponding multi-organ segmentation results is established. This dataset is used to train the subsequent abdominal multi-organ segmentation network.

[0034] Step 2: Build an abdominal multi-organ segmentation network Next, a dynamic fusion of CNN and Transformer features is constructed for abdominal multi-organ segmentation. The network adopts a U-shaped structure with skip connections as the baseline framework, and the encoding path and decoding path are composed of 3 encoding blocks and 3 decoding blocks cascaded respectively.

[0035] Each encoding block contains a CNN feature extraction module, a Transformer feature extraction module, a downsampling layer, and a dynamic fusion module (DF). The CNN module is used to extract local features, and the Transformer module is used to extract global features. The dynamic fusion module is used to dynamically fuse local features with global features to improve segmentation accuracy.

[0036] In the encoding path, the input abdominal CT image passes through a 3×3 convolution, the first encoder, the second encoder, and the third encoder in sequence. Each encoder contains a CNN feature extraction module, a Transformer feature extraction module, a downsampling layer, and a dynamic fusion module. The first encoder is used to extract low-level features, such as edges and textures in the image; the second encoder further extracts mid-level features, such as the shape of the object, based on the first encoder; the third encoder is used to extract higher-level abstract features, including object categories, scene semantics, etc.

[0037] On the decoding path, the feature map output by the third encoder is used as the input of the decoder, and passes through the first decoder, the second decoder, and the third decoder in sequence. In order to avoid redundant transmission of information and enhance the recognition ability of small objects and edges, a jump connection is added between the encoding and decoding blocks at the same level. The feature maps output by the first encoder, the second encoder, and the third encoder are passed to the third decoder, the second decoder, and the first decoder respectively through the jump connection, and the feature map from the encoder is fused with the features output by the upsampling layer in the decoder.

[0038] 1. Local and global feature extraction In order to avoid the features of the input transformer being too abstract in the serial mode, the present invention selects the parallel mode of CNN and Transformer to extract features.

[0039] In the Transformer module, the transposed attention mechanism, i.e., cross-channel self-attention, is used. Transposed attention significantly reduces the computational burden by calculating attention across channels and promotes more effective and efficient feature interactions.

[0040] The CNN module uses deep convolution, which is composed of deep convolution and The deep convolution is composed of a cascade of activation functions. The deep convolution performs an independent convolution operation on each input channel, which enables it to effectively extract the features of the input image on each channel and retain the independence between channels.

[0041] 2. Dynamic Fusion Features In order to give full play to the advantages of CNN and Transformer, the present invention designs a dynamic fusion module (DF) to dynamically fuse local information and global information. The module includes fast Fourier transform (2DFFT), convolution activation function layer, inverse fast Fourier transform layer (2DIFFT) and pooling activation function layer in sequence.

[0042] In the dynamic fusion module, the feature image is first converted from the spatial domain to the frequency domain, the weights are updated in the frequency domain, and finally the frequency domain features with updated weights are converted back to the spatial domain, and the features are enhanced through the pooling activation function layer before the weights are assigned.

[0043] Given non-local features (extracted by Transformer) and local features (extracted by CNN), the two are concatenated and input into the dynamic fusion module. The module first transforms the features from the spatial domain to the frequency domain and separates the amplitude and phase information. Then, the amplitude information and phase information are respectively passed through the convolutional layer and The activation function is used to enhance the image contrast and boundary information. The enhanced frequency domain features are converted back to the spatial domain through 2DIFFT to obtain new features. Then, the channels of the new features are split, and the weight map W is generated through pooling and Sigmoid activation function. Finally, the weights of local features and non-local features are assigned according to the weight map W to achieve dynamic fusion.

[0044] Step 3: Test the segmentation network The trained multi-organ segmentation network was tested with the test set to obtain the segmentation results. Specifically, the resolution and patch size of the input image were set to 224×224 and 16, respectively. During training, the SGD optimizer with a learning rate of 0.01, momentum of 0.9, and weight decay of 1e-4 was used for training. The default batch size is 24. All experiments were performed on an Nvidia RTX3090 GPU (Graphic Processing Unit).

[0045] The experiment uses the Synapse dataset, which contains 1,250 CT or MRI images (1,000 for training and 250 for testing), covering at least 8 categories of abdominal organs. DSC and HD are used as evaluation indicators to evaluate the quality of the segmentation results.

[0046] Through the above steps, the present invention realizes a method for abdominal multi-organ segmentation based on dynamic fusion. The method makes full use of the local feature extraction capability of CNN and the global feature extraction capability of Transformer, and realizes the complementary advantages of the two through the dynamic fusion module, thereby improving the accuracy and efficiency of abdominal multi-organ segmentation.

[0047] Example 3 In another preferred embodiment, based on the above embodiments 1 and 2, as Figures 1 to 5 As shown, this embodiment further describes the implementation steps of a method for abdominal multi-organ segmentation based on dynamic fusion, which are as follows: Step 1: Establish a training dataset A containing abdominal CT images and their corresponding multi-organ segmentation results; Step 2: Build an abdominal multi-organ segmentation network that dynamically integrates CNN and Transformer features, giving full play to the local feature extraction capability of CNN and the global feature extraction capability of Transformer encoder; Step 3: Use the test set to test the trained multi-organ segmentation network and obtain the segmentation results; Through the above steps, a dynamic fusion abdominal multi-organ segmentation network based on U-shaped network is constructed.

[0048] 1. Step 2 includes the following sub-steps: 1. A U-shaped structure with jump connections is used as the network benchmark framework, in which the encoding path and the decoding path are respectively composed of 3 encoding blocks and 3 decoding blocks in cascade: each encoding block contains a CNN feature extraction module, a Transformer feature extraction module, a downsampling layer and a dynamic fusion module; each encoding block contains a CNN feature extraction module, a Transformer feature extraction module, a downsampling layer and a dynamic fusion module. The process of segmenting the input CT image using the abdominal CT image multi-organ segmentation model includes: the original CT image is input into the abdominal CT image multi-organ segmentation model, and passes through a 3×3 convolution, a first encoder, a second encoder, and a third encoder in sequence. The first encoder is used to extract low-level features, which are usually related to the original information of the input data, such as edges and textures in the image. The second encoder further extracts mid-level features based on the first encoder, such as more complex structures such as the shape of the object. The third encoder is used to extract higher-level abstract features, including high-level concepts such as object categories and scene semantics. Multiple encoders can make the multi-organ segmentation model of abdominal CT images gradually learn and integrate the features of different levels of the original CT image, and deepen the model's understanding of the image; the feature map output by the third encoder is used as the input of the decoder, and passes through the first decoder, the second decoder, and the third decoder in sequence; in order to avoid redundant information transmission and enhance the recognition ability of small targets and edges, a jump connection is added between the encoding and decoding blocks of the same level. The feature maps output by the first encoder, the second encoder, and the third encoder are passed to the third decoder, the second decoder, and the first decoder through jump connections, respectively, and the feature maps from the encoder are fused with the features output by the upsampling layer in the decoder; finally, a 3×3 convolution is used to output an organ segmentation image of the same size as the original CT image. In the decoding process, the features from the encoder are fused with the feature information in the decoder upsampling, which can make full use of shallow and deep features to retain more detailed information and improve the understanding ability of the multi-organ segmentation model of abdominal CT images to the image content.

[0049] 2. Local and global feature extraction: Traditional Unet networks (U-shaped networks) usually use continuous layer-by-layer convolution to obtain features. However, the convolution kernel can only focus on its own and boundary feature information, lacking interactive fusion between different features. Therefore, an attention module is introduced in the encoding path. In order to avoid the input transformer features being too abstract in the serial mode, we choose to extract features by connecting CNN and Transformer in parallel. The Transformer module uses the transposed attention mechanism, i.e., cross-channel self-attention. Its structure is based on Figure 3As shown in the figure, unlike the conventional self-attention mechanism that calculates dot products in the spatial dimension, transposed attention significantly reduces the computational burden by calculating attention across channels. Moreover, it is spatially free, allowing the calculation of attention features across scales, promoting more efficient and effective feature interactions. The specific steps are: (1) For input features , generated using linear projection , and matrix: (1) (2) For the obtained Perform a Reshape operation to get the correct dimension; (3) and Perform a dot multiplication operation and then normalize it through the Softmax function; (4) Get the C×C similarity matrix attn. Finally, attn and Multiply to get the final output, the specific formula is: (2) The CNN module uses deep convolution, and its structure is based on Figure 4 As shown, the depth convolution and The activation function cascades, and the deep convolution performs independent convolution operations on each input channel, which enables it to effectively extract the features of the input image on each channel and retain the independence between channels. The operation of the deep convolution of the deep convolution is expressed as follows: (3) CNN module (deep convolution + The operation of activation function can be expressed by the following formula: (4) 3. Dynamic fusion features: In addition, in order to give full play to the advantages of CNN and Transformer, a frequency interaction module is designed to dynamically fuse local information and global information, namely the dynamic fusion module (DF for short), with the following structure: Figure 5 As shown in the figure, the module includes fast Fourier transform, convolution activation function layer, inverse fast Fourier transform layer and pooling activation function layer in sequence. In the dynamic fusion module, the feature image is first converted from the spatial domain to the frequency domain, the weight is updated in the frequency domain, and finally the frequency domain features with updated weights are converted back to the spatial domain. The features are enhanced through the pooling activation function layer and then the weights are distributed. The specific steps are as follows: (1) Given non-local features and local features ,connect and Get Features ; (2) Features enter In , the image is converted from the spatial domain to the frequency domain, and the amplitude Amp and phase Pha information are separated: (5) Then, the amplitude information and phase information are passed through the convolution layer and Activation function to enhance image contrast and boundary information: (6) (3) The enhanced frequency domain features are obtained by Convert back to the spatial domain to get the features ; (4) Features The channels are split and the weight map is generated by pooling and Sigmoid activation function : (7) (5) Local features and non-local features Perform weight distribution to achieve dynamic fusion: (8) The features extracted in step (2) are scaled as follows: (9) represents fast Fourier transform; , represents the activation function; represents inverse fast Fourier transform; Indicates the feature channel splitting operation.

[0050] In step 2, the following improvements are made to the prior art: (1) Due to the characteristics of CNN model construction, the final output feature map often condenses a large amount of semantic information (i.e., high-level features). However, due to the continuous downsampling, the resolution of this part of the feature map is low and the local details are less preserved. If CNN and Transformer are connected in series, the input to the Transformer model is a highly abstract feature map with extremely low spatial resolution. The spatial information of these feature maps has been greatly compressed, and the global modeling ability of Transformer cannot be fully utilized. Therefore, the segmentation effect of the lesion area and boundary of small targets in medical images is not satisfactory. Combining CNN and Transformer in parallel allows CNN and Transformer to process input data simultaneously, so that local details and global structures in the image can be obtained at the same time, and the data can be understood more comprehensively. The existing technology uses a dual encoder to achieve the parallelization of the two, which increases computing resources and memory compared with a single encoder. Therefore, a single encoder architecture is selected to keep the parallel structure of CNN and Transformer unchanged, and improve the dual-branch structure to combine the two into one module, which reduces computing resources while maintaining good experimental performance.

[0051] (2) Most existing methods combining CNN and Transformer have insufficient feature expression capabilities, resulting in poor segmentation of the positional relationship between multiple organs and areas with blurred tissue edges. Therefore, a dynamic fusion module is designed to enable the model to dynamically fuse global and local information. The features of the two are weighted to different degrees.

[0052] Since CNN uses convolution sliders to extract edge texture features, and edge texture belongs to high-frequency information, Transformer tends to extract the overall structure, shape and semantic information of the object, and the overall structure and shape belong to low-frequency information. Simply fusing these two features (such as adding or splicing) may cause feature conflicts and lose important information. Therefore, we use Fourier transform to convert the image from the spatial domain to the frequency domain, separate the amplitude and phase, and enhance them separately. Amplitude can help analyze the global contrast and image blur in medical images. It is more suitable for describing the overall brightness change or the general structure of the organ; phase information mainly reflects the geometric structure and spatial position of the image, and determines the shape and details of the object in the image. Finally, through the inverse Fourier transform back to the spatial domain, two weight maps are dynamically generated to adaptively re-weight the non-local features and local features. This dynamic fusion allows the model to weight the two features to different degrees at different layers, thereby achieving more effective feature fusion and giving full play to the powerful advantages of both.

[0053] 2. Step 3 includes the following sub-steps: We set the resolution and patch size of the input image to 224×224 and 16, respectively; during training, all models are trained using the SGD optimizer with a learning rate of 0.01, momentum of 0.9, and weight decay of 1e-4. The default batch size is 24. All experiments are performed on an Nvidia RTX3090 GPU.

[0054] The experiment used the Synapse dataset, which contains 1,250 CT or MRI images (1,000 for training and 250 for testing), covering at least 8 categories of abdominal organs.

[0055] 3. Use DSC and HD as evaluation indicators to evaluate the authenticity of this experiment: DSC (similarity coefficient) is a metric function used to evaluate the similarity of two samples. Its value range is between 0 and 1. The larger the value, the more similar it is. Assume that the ground truth is The prediction result is , the calculation formula is as follows: (10) HD represents the maximum distance between the predicted segmentation region boundary and the true region boundary. The smaller the value, the smaller the predicted boundary segmentation error and the better the quality. The calculation formula is as follows: (11) In the preferred solution, the CNN feature extraction module in Step 2.1 uses deep convolution, which is composed of deep convolution and The structure is composed of a cascade of activation functions; this structure is not a simple stacking, but a clever cascade combination of deep convolution and activation functions. The deep convolution is responsible for capturing the deep features in the image, while the activation function introduces nonlinear factors to enhance the expressiveness of the model; the two complement each other and together constitute an efficient and powerful feature extraction network. This setting not only improves the accuracy of feature extraction, but also optimizes the generalization ability of the model, laying a solid foundation for subsequent processing steps.

[0056] In the preferred solution, the Transformer feature extraction module in Step 2.1 uses the transposed attention mechanism to achieve feature interaction by calculating attention across channels. The above settings effectively improve the model's ability to capture complex feature relationships when processing data, allowing the model to more accurately understand and learn deep connections in the data. At the same time, by optimizing algorithms and structural design, the redundancy in the calculation process is significantly reduced, and the overall computing efficiency is improved. In addition, the module also innovatively introduces a local perception mechanism, which further enhances the model's sensitivity to these detail features by paying attention to local detail features in the data, thereby showing higher accuracy and robustness when processing complex tasks.

[0057] In the preferred solution, the dynamic fusion module in Step 2.1 includes a fast Fourier transform, a convolution activation function layer, an inverse fast Fourier transform layer, and a pooling activation function layer, which are used to dynamically fuse local features and global feature information; the above settings can significantly improve the robustness and accuracy of feature representation, which is specifically manifested in that the model can still accurately capture key information in the case of noise interference or data loss. In addition, this improvement can also greatly enhance the model's ability to understand complex scenes, including the recognition and analysis of different backgrounds, lighting conditions, and object occlusion; through these optimizations, the entire system can not only adapt to various data distributions more flexibly when processing diversified data, but also significantly improve processing speed and accuracy, thereby showing stronger adaptability and stability in various application scenarios.

[0058] In the preferred solution, the encoding block and the decoding block in Step 2.1 add a skip connection between the encoding and decoding blocks at the same level, pass the feature map of the encoder to the decoder through the skip connection, and merge it with the features output by the upsampling layer in the decoder; the above setting enables the decoder to obtain richer contextual information, thereby improving the model's ability to understand complex image content; in addition, the skip connection also enhances the back propagation of the gradient, which helps to improve the training stability and performance of the model.

[0059] In summary, the present invention proposes a method for abdominal multi-organ segmentation based on dynamic fusion, which solves the problems of insufficient accuracy, low efficiency and poor segmentation effect of complex organ structures and small target lesion areas in the existing technology of abdominal multi-organ segmentation in medical image analysis; the parallel structure of CNN and Transformer adopted by the present invention is different from the serial use of CNN and Transformer in the traditional method. The scheme innovatively adopts a parallel structure, which allows CNN and Transformer to process input data simultaneously, so that local details and global structures in the image can be obtained simultaneously, providing more comprehensive feature information for abdominal multi-organ segmentation; the present invention designs a dynamic fusion module (DF), which transforms the image from the spatial domain to the frequency domain through Fourier transform, and then enhances the amplitude and phase information respectively, and finally returns to the spatial domain through inverse Fourier transform. This module can dynamically generate two weight maps to adaptively re-weight non-local features and local features. This dynamic fusion method enables the model to weight the features of CNN and Transformer to different degrees at different layers, thereby achieving more effective feature fusion; although the combination of CNN and Transformer has applications in many fields, the present invention The invention applies it to the specific medical problem of abdominal multi-organ segmentation and achieves remarkable results. This application not only expands the scope of application of the combination of CNN and Transformer, but also provides a new solution for medical image segmentation. The invention realizes a major improvement in the technology of abdominal multi-organ segmentation by connecting CNN and Transformer in parallel and combining them with a dynamic fusion module. This innovation of technical ideas not only improves the segmentation accuracy, but also reduces the demand for computing resources. The invention proposes a new solution to the challenges encountered in abdominal multi-organ segmentation, such as organ diversity, shape changes, overlap between adjacent organs, and noise, artifacts and insufficient contrast in images. By dynamically fusing the features of CNN and Transformer, the invention successfully solves these problems and improves the segmentation effect. The invention has broad application prospects in medical image analysis and can assist doctors in disease diagnosis, treatment planning and efficacy evaluation. This innovation in practical application not only improves the level of medical services, but also brings better treatment effects to patients. At the same time, the popularization and application of this technology will also promote the rapid development of the field of medical image analysis, inject new vitality into medical research and clinical practice, and help the medical industry move towards a more intelligent and precise future.

Claims

1. A method for abdominal multi-organ segmentation based on dynamic fusion, characterized in that: The segmentation method comprises the following steps: Step 1: Establish a training dataset containing abdominal CT images and their corresponding multi-organ segmentation results; Step 2: Build an abdominal multi-organ segmentation network that dynamically integrates CNN and Transformer features; Step 3: Use the test set to test the trained multi-organ segmentation network and obtain the segmentation results.

2. The abdominal multi-organ segmentation method based on dynamic fusion according to claim 1, characterized in that: The implementation steps of Step 2 to construct an abdominal multi-organ segmentation network that dynamically integrates CNN and Transformer features include: Step 2.1: A U-shaped structure with skip connections is used. The encoding path and decoding path are composed of at least one encoding block and decoding block cascaded, respectively. Each encoding block contains a CNN feature extraction module, a Transformer feature extraction module, a downsampling layer, and a dynamic fusion module. Step 2.2: Local and global feature extraction, select CNN and Transformer in parallel to extract features; Step 2.3: Dynamically fuse features. Use the dynamic fusion module in Step 2.1 to convert the feature image from the spatial domain to the frequency domain, update the weights in the frequency domain, and finally convert the frequency domain features with updated weights back to the spatial domain. Enhance the features through the pooling activation function layer and then assign weights.

3. The abdominal multi-organ segmentation method based on dynamic fusion according to claim 2, characterized in that: The CNN feature extraction module in Step 2.1 uses deep convolution, which is composed of deep convolution and Activation functions are composed of cascades.

4. The abdominal multi-organ segmentation method based on dynamic fusion according to claim 3 is characterized in that: The Transformer feature extraction module in Step 2.1 adopts the transposed attention mechanism to achieve feature interaction by calculating attention across channels.

5. The abdominal multi-organ segmentation method based on dynamic fusion according to claim 4, characterized in that: The dynamic fusion module in Step 2.1 includes a fast Fourier transform, a convolution activation function layer, an inverse fast Fourier transform layer and a pooling activation function layer, which are used to dynamically fuse local features and global feature information.

6. The abdominal multi-organ segmentation method based on dynamic fusion according to claim 5, characterized in that: In the encoding block and the decoding block in Step 2.1, a skip connection is added between the encoding block and the decoding block at the same level, and the feature map of the encoder is passed to the decoder through the skip connection to be merged with the feature output of the upsampling layer in the decoder.

7. The method for abdominal multi-organ segmentation based on dynamic fusion according to claim 6, characterized in that: The implementation steps of Step 2.2 local and global feature extraction include: Step 2.2.1: For input features , generated using linear projection , and matrix: （1）； In the formula, is the projection matrix, for The projection matrix is ​​used to project the input vector into the query space. is the generated query matrix; for The projection matrix is ​​used to project the input vector into the key space, is the generated key matrix; for The projection matrix is ​​used to project the input vector into the value space, is the generated value matrix; Step 2.2.2: Get Perform a Reshape operation to get the correct dimension; Step2.2.3: and Perform a dot multiplication operation and then normalize it through the Softmax function; Step 2.2.4: Get the C×C similarity matrix attn, and finally attn and Multiply to get the final output, the specific formula is: （2）； In the formula, is the transpose of the key matrix, is the dimension of the key vector; The CNN module uses deep convolution, which is composed of deep convolution and The activation function cascades together, and the operation of the depth convolution is expressed as follows: （3）； In the formula, represents the convolution operation, It is The feature map of the output channels, It is An image with input channels, It is convolution kernels; The operation of the CNN module can be expressed by the following formula: （4）； In the formula, is the output feature of the CNN module.

8. The abdominal multi-organ segmentation method based on dynamic fusion according to claim 7, characterized in that: The implementation steps of Step 2.3 dynamic fusion features include: Step 2.3.1: Given non-local features and local features ,connect and Get Features ; Step 2.3.2: Features enter In , the image is converted from the spatial domain to the frequency domain, and the amplitude Amp and phase Pha information are separated: （5）； In the formula, is the representation in the frequency domain, that is, the result after two-dimensional fast Fourier transform. is the amplitude information, is the phase information. Then, the amplitude information and phase information are passed through the convolutional layer and Activation function to enhance image contrast and boundary information: （6）； In the formula, is the enhanced amplitude information, is the enhanced phase information. Step 2.3.3: The enhanced frequency domain features are passed through Convert back to the spatial domain to get the features ; Step 2.3.4: Features The channels are split and the weight map is generated by pooling and Sigmoid activation function : （7）； In the formula, is the Sigmoid activation function; Step 2.3.5: Local features and non-local features Perform weight distribution to achieve dynamic fusion: （8）； The scale transformation process of the features extracted in Step 2.3.2 in the above steps is as follows: （9）； In the formula, represents the image features of the original input, represents the transformed image features, , , Respectively represent the length, width and dimension of the feature image; It means to concatenate two features, and the number of channels is added; represents fast Fourier transform; , represents the activation function; represents inverse fast Fourier transform; Indicates the feature channel splitting operation.

9. The abdominal multi-organ segmentation method based on dynamic fusion according to claim 1, characterized in that: The specific implementation steps of Step 3 using the test set to test the trained multi-organ segmentation network to obtain the segmentation result include: Step 3.1: Set the resolution and patch size of the input image. During training, all models are trained using the SGD optimizer. Step 3.2: Use the Synapse dataset, including CT or MRI images, covering at least eight categories of abdominal organs; Step 3.3: Use DSC and HD as evaluation indicators to evaluate the segmentation results for authenticity purposes: DSC is a metric function used to evaluate the similarity between two samples. Its value range is between 0 and 1. The larger the value, the more similar it is. Assuming the ground truth is The prediction result is , the calculation formula is as follows: （10）； In the formula, Represent two sets, and Respectively and The number of elements of represents true positives, that is, the number of pixels marked as positive samples by both the segmentation result and the true segmentation; It represents false negatives, that is, the number of pixels that the segmentation result marks as negative samples but the true segmentation marks as positive samples; It represents false positive, that is, the number of pixels that the segmentation result marks as a positive sample but the actual segmentation marks as a negative sample. The numerator is multiplied by 2 to ensure that the denominator is in the range of [0,1] after repeated calculation. Equivalent to , Represents the harmonic mean of precision and recall. represents the performance score of the classifier; HD represents the maximum distance between the predicted segmentation region boundary and the true region boundary. The smaller the value, the smaller the predicted boundary segmentation error and the better the quality. The calculation formula is as follows: （11）； In the formula, Represent two sets, Represents a point set Every point in Calculate its point set Every point in The distance of , retain the shortest distance, and then find the maximum distance among the retained shortest distances, Represents a point set Every point in Calculate its point set Every point in distances, retain the shortest distance, and then find the maximum distance among the retained shortest distances.

Images