Transform 3D medical image segmentation method based on feature focusing
By introducing focus mapping and sparse matrix into Transformer, combining spatial feature fusion connection module and dynamic characterization position coding module, the problems of large amount of data and high computational complexity in 3D medical image segmentation are solved, and a more efficient image segmentation effect is achieved.
Patent Information
- Application Number
- CN202410976438.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-20
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-20
AI Technical Summary
Due to the huge data volume and feature space complexity, existing convolutional neural networks are difficult to fully capture the contextual connections of images, resulting in poor segmentation effect. The high computational complexity of the Transformer network leads to large computing resource overhead.
Using the Transformer method based on feature focus, the calculation overhead is reduced through focus mapping and matrix sparseness, and a spatial feature fusion connection module and dynamic characterization position coding module are proposed to integrate the encoding codec features and extract global information.
It effectively reduces the computational complexity and storage requirements, improves the accuracy and efficiency of 3D medical image segmentation, and enhances the understanding of image context.
Smart Images

Figure CN118941572A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of image segmentation and computer vision technology, and in particular to a Transformer 3D medical image segmentation method based on feature focusing. Background Art
[0002] In the field of medical image segmentation, 3D medical image segmentation is an important and challenging topic. Due to the stereoscopic nature of 3D medical images, the amount of data is usually N times that of 2D images, with complex spatial features and enhanced sensitivity to position information. However, as a result, the position, size, and shape of the segmentation target are more variable, which increases the complexity of the segmentation task. When using convolutional neural networks (CNNs) to process such large-scale 3D data, due to the huge amount of data and the complexity of the feature space, the network often cannot fully capture the contextual connections of the image, resulting in poor segmentation results. To meet this challenge, some current methods consider using Transformer networks that can perform global modeling. Transformer networks can effectively capture the relationship between distant pixels through the self-attention mechanism, thereby better understanding the global structure of the image. However, since the computational complexity of the self-attention mechanism is O(N 2 ), and the data volume of each sample of 3D medical images is huge, which may lead to huge overhead of computing resources. Summary of the invention
[0003] In view of this, the purpose of the present invention is to provide a Transformer 3D medical image segmentation method based on feature focusing, which reduces the computational overhead by focusing mapping and matrix sparsification. Secondly, a spatial feature fusion connection module is proposed to alleviate the problem of spatial position information ambiguity by fusing the features of the encoder and decoder. Finally, a dynamic representation position encoding module is proposed to extract and fuse global information at different scales to increase context understanding.
[0004] To achieve the above object, the present invention adopts the following technical solution: a Transformer 3D medical image segmentation method based on feature focusing, comprising the following steps:
[0005] Step S1: By training the 3D medical image dataset and using data enhancement methods on the data, the data is sent to the V-Net backbone network for a first-stage preliminary training and feature extraction;
[0006] Step S2: Use the trained feature extraction network as the first-stage feature extraction module, and then propose the Transformer based on focus mapping and sparse matrix as the basic module of the second-stage extraction network. Through focus mapping, the intrinsic distribution change is stimulated. At the same time, the amount of calculation and storage space requirements are reduced by using sparse matrix, so that the model can capture key information when processing large-size data;
[0007] Step S3: A spatial feature fusion link module SFFC is proposed to fuse the feature information in the encoder and decoder by retaining and extracting the spatial position information;
[0008] Step S4: In view of the limitations of static position coding and conventional dynamic coding, a dynamic representation position coding module DPRE is proposed. It extracts local features at different scales, fuses global information to increase contextual understanding, implements dynamic position coding according to content, and finally realizes the segmentation of 3D medical images.
[0009] In a preferred embodiment, step S1 specifically includes the following steps:
[0010] Step S11: Obtain a public medical image segmentation training set from the Internet and obtain relevant annotations of the training data; mainly obtain three commonly used 3D medical image datasets ADCD, Synapse, and LA, which respectively include data of multiple parts of the heart and the whole body trunk;
[0011] Step S12: After the data is loaded, different random data enhancements are performed on the data, including random color jitter, grayscale, Gaussian blur and random shearing to simulate the complex changes in real medical scenes and force the model to learn more robust feature representations;
[0012] Step S13: Initialize network weights and parameters of the image to be segmented in the medical image segmentation dataset using a pre-trained V-Net feature extraction network.
[0013] In a preferred embodiment, step S2 specifically includes the following steps:
[0014] Step S21: First, perform linear transformation on the query matrix Q and the key matrix K, and use the similarity calculation method to calculate the similarity between them, so as to obtain the self-attention matrix A. In the self-attention matrix A, each element represents the similarity score between different positions in the sequence; then, perform sparse processing on the self-attention matrix A to obtain a sparse self-attention matrix A'; most of the elements in the sparse self-attention matrix are zero, and only some important similarity scores are retained. The specific calculation process is as follows:
[0015] Q = xW Q , K = xW K , V = xWV
[0016]
[0017] Among them, x represents the input feature, K T represents the transpose of the key matrix, d represents the dimension of the value matrix, Sim uses the common Sotfmax function, and after sparsely processing the self-attention matrix A, we get A′ ij , where k is a hyperparameter of the sparsification matrix, which is also a hyperparameter of the focus map; A′ ij Represents the sparse self-attention matrix; A ij represents the self-attention matrix;
[0018] Step S22: In the V processing of the value matrix, a focus mapping function is proposed, which is carefully designed and applied to the value matrix; the specific calculation formula is as follows:
[0019]
[0020] Where V′ represents the value matrix after stimulating the intrinsic characteristics, and the RELU function ensures the non-negativity of the input and the validity of the denominator;
[0021] Step S23: A local rank restoration module is proposed to restore the order of the attention matrix by applying additional depthwise separable convolution (DWC) to supplement the details lost in the model; the final complete and specific calculation process is as follows:
[0022]
[0023] DWC is a separable convolution, where DWC(V) represents the operation of the separable convolution on the V matrix.
[0024] In a preferred embodiment, step S3 specifically includes the following steps:
[0025] Step S31: improving the skip connection module;
[0026] First, obtain the low-level feature A from the encoder, and adopt the maximum pooling and average pooling strategies; through these two pooling operations, two feature weight matrices with different characteristics are obtained, representing the prominent features and global features of A respectively; add these two pooling matrices to fuse the prominent features and global features of feature A, which can better capture the correlation and importance between features, and process them through the sigmoid activation function to limit the weighted features between 0 and 1; then, multiply the weighted weight matrix with the original feature matrix A to obtain a new feature matrix A′; the purpose of this step is to make A′ pay more attention to the important spatial position information in the encoder, as well as the prominent and global features through the weighted multiplication operation. The specific calculation is shown in the formula:
[0027] A′=(MLP(Max(A))+MLP(Mean(A)))×A
[0028] Among them, MLP is a multi-layer perceptron, Max is a 3D maximum pooling layer, and Mean is a 3D average pooling layer;
[0029] Step S32: Upsample the high-level feature B from the decoder to make it have the same size as the feature matrix A′; after upsampling, use a separable convolution operation to further extract the feature representation of B and maintain the consistency of the spatial position information; after the separable convolution process, a decoder feature B′ with twice the spatial position information is obtained. The specific calculation process is as follows:
[0030] B′=DWC(Upsample(B))
[0031] Step S33: Finally, we concatenate the feature matrix A′ from the encoder and the feature matrix B′ from the decoder to obtain a comprehensive feature representation, which contains important features from the encoder and decoder as well as spatial position information; the complete calculation process is shown in the formula:
[0032]
[0033] MLP is a multi-layer perceptron, a basic structure of a neural network; Upsample is to upsample the feature;
[0034] Step S34: Through a series of detailed operations, including pooling, weighted multiplication, upsampling, separable convolution and feature splicing, the feature information in the encoder and decoder is effectively integrated and the spatial position information is retained.
[0035] In a preferred embodiment, step S4 specifically includes the following steps:
[0036] Step S41: The dynamic representation position encoding module DPRE extracts local features from different scales through separable convolution, and then performs global attention operation to obtain global information to increase context understanding, thereby realizing dynamic position encoding according to the content;
[0037] Step S42: First, the input features are put into two 3D separable convolutional layers with different kernel sizes for feature extraction; by extracting local features at different scales, two features F3 and F5 with different sizes are obtained. The specific calculation process of F3 and F5 is as follows:
[0038] F i =DWC i×i (F input )
[0039] Where DWC stands for depthwise separable convolution, F i It is represented as the feature after depth-wise separable convolution, where i takes the values of 3 and 5, and F input Represented as input features;
[0040] Step S43: Next, the self-attention mechanism is used to establish links between patch features to capture the dependencies between positions of different sizes in the input sequence;
[0041] Step S44: A fully connected layer is used to further adjust the dynamic position encoding; by learning the weight and bias parameters of the position encoding, the position information is effectively integrated with the image features, and finally the feature F with dynamic position encoding is obtained. out , the specific calculation process is shown in the formula:
[0042] F pos =LN(F5)×Softmax(Reshape(LN(F3))×LN(F5))
[0043] F out =F input +Reshape(F pos )
[0044] F pos is the feature after learning the position feature; LN is the linear layer, F5 is the feature after the separable convolution of size 5 mentioned above, F3 is the feature after the separable convolution of size 3 mentioned above, Softmax is the normalization function, and Reshape is the shape transformation, so that the previous and next features can be multiplied according to the correct rank;
[0045] Step S45: Testing the trained medical image segmentation model on the validation set to obtain the final detection accuracy.
[0046] Compared with the prior art, the present invention has the following beneficial effects:
[0047] 1. Different from the common 3D medical image segmentation, considering the problem of excessive amount of 3D medical image data, sparse matrix and focusing function are introduced on the basis of Transformer to make it more suitable for actual application scenarios.
[0048] 2. A spatial feature fusion connection module is proposed to alleviate the problem of spatial position information ambiguity by fusing the features of the encoder and decoder.
[0049] 3. A dynamic representation position encoding module is proposed to extract and fuse global information at different scales to increase contextual understanding. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 The present invention is a flowchart of a method implementation of a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0052] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0053] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.
[0054] like Figure 1 As shown, the trial example of the present invention provides a Transformer3D medical image segmentation method based on feature focusing, comprising the following steps:
[0055] Step S1: training with 3D medical image datasets, using data enhancement methods on the data, and sending the data into the V-Net backbone network for a first-stage preliminary training and feature extraction;
[0056] Step S2: The trained feature extraction network is used as the first-stage feature extraction module, and the Transformer (F-Transformer) based on focus mapping and sparse matrix is proposed as the basic module of the second-stage extraction network. The focus mapping is used to stimulate the change of intrinsic distribution. At the same time, the use of sparse matrix reduces the amount of calculation and storage space requirements, so that the model can capture key information more efficiently when processing large-size data;
[0057] Step S3: In view of the limitations of the traditional skip connection method in terms of information utilization and spatial position information retention, a spatial feature fusion connection module (SFFC) is proposed, which can effectively fuse the feature information in the encoder and decoder by retaining and extracting the spatial position information;
[0058] Step S4: In view of the limitations of static position coding and conventional dynamic coding, a dynamic representation position coding module (DPRE) is proposed. By extracting local features at different scales and fusing global information to increase contextual understanding, dynamic position coding can be better implemented based on content, and finally the segmentation of 3D medical images can be achieved.
[0059] Step S1 specifically includes the following steps:
[0060] Step S11: Obtain a public medical image segmentation training set from the Internet and obtain relevant annotations of the training data; mainly obtain three commonly used 3D medical image datasets (ADCD, Synapse, LA), which respectively include data of the heart, multiple parts of the whole body, etc.;
[0061] Step S12: After the data is loaded, different random data enhancements are performed on the data, including random color jitter, grayscale, Gaussian blur, and random shearing, to simulate the complex changes in real medical scenes and force the model to learn more robust feature representations;
[0062] Step S13: Initialize network weights and parameters of the image to be segmented in the medical image segmentation dataset using a pre-trained V-Net feature extraction network.
[0063] Step S2 specifically includes the following steps:
[0064] Step S21: First, we perform linear transformation on the query matrix Q and the key matrix K, and use the similarity calculation method to calculate the similarity between them, so as to obtain the self-attention matrix A, in which each element represents the similarity score between different positions in the sequence. Then, in order to reduce the computational complexity, we perform sparse processing on the self-attention matrix A to obtain a sparse self-attention matrix A'. Most of the elements in the sparse self-attention matrix are zero, and only some important similarity scores are retained, which significantly reduces the amount of calculation and storage space requirements. This sparse processing method can effectively improve the computational efficiency of the self-attention mechanism. The specific calculation process is as follows:
[0065] Q = xW Q , K = xW K , V = xW V
[0066]
[0067] Among them, x represents the input feature, K T represents the transpose of the key matrix, d represents the dimension of the value matrix, Sim uses the common Sotfmax function, and after sparsely processing the self-attention matrix A, we get A′ ij , where k is a hyperparameter of the sparsification matrix, which is also a hyperparameter of the focus map. ij Represents the sparse self-attention matrix; A ij Represents the self-attention matrix.
[0068] Step S22: In the V processing of the value matrix, an innovative focus mapping function is proposed. This function is carefully designed and applied to the value matrix. Its purpose is to stimulate the distribution changes within the value matrix through a specific mapping method, so that the features corresponding to different positions are more prominent and significant. The design of this mapping function takes into account the intrinsic characteristics of the value matrix and the application scenarios in the self-attention mechanism, in order to achieve more accurate and effective feature extraction and representation. At the same time, enhancing the corresponding features can also alleviate the feature smoothing caused by the sparse matrix. The specific calculation formula is as follows:
[0069]
[0070] Where V′ represents the value matrix after stimulating intrinsic characteristics, the RELU function ensures the non-negativity of the input and the validity of the denominator, and k is a hyperparameter of the sparsification matrix, which is also a hyperparameter of the focus map.
[0071] Step S23: A local rank restoration module is proposed to restore the order of the attention matrix by applying additional depthwise separable convolution (DWC) to supplement the lost details in the model. The final complete and specific calculation process is as follows:
[0072]
[0073] Where Sparse() is the sparse function, O is the final output of self-attention; DWC is the separable convolution, where DWC(V) represents the operation of the separable convolution on the V matrix.
[0074] The proposed Transformer based on focused mapping and sparse matrix reduces the subsequent computational complexity through the sparse self-attention matrix A, and performs focused mapping on the value matrix V to stimulate intrinsic distribution changes and enhance feature contrast. Finally, deep separable convolution is used to maintain the matrix rank lost due to the sparse matrix, so that the model can capture key information more efficiently when processing large-size data and maintain the diversity of output features.
[0075] Step S3 specifically includes the following steps:
[0076] Step S31: In order to enhance the features generated by the model, better utilize the information in the feature map, and retain and utilize the spatial location information to further improve the performance and generalization ability of the model, we improve the skip connection module;
[0077] First, we obtain the low-level feature A from the encoder. In order to retain its spatial position information, the maximum pooling and average pooling strategies are adopted. Through these two pooling operations, we obtain two feature weight matrices with different characteristics, representing the salient features and global features of A respectively. Adding these two pooling matrices can fuse the salient features and global features of feature A, which can better capture the correlation and importance between features. The weighted features are limited to between 0 and 1 through the sigmoid activation function. Next, we multiply the weighted weight matrix with the original feature matrix A to obtain a new feature matrix A′. The purpose of this step is to make A′ pay more attention to the important spatial position information in the encoder, as well as the salient and global features through weighted multiplication operations. The specific calculation is shown in the formula:
[0078] A′=(MLP(Max(A))+MLP(Mean(A)))×A
[0079] Among them, MLP is a multi-layer perceptron, Max is a 3D maximum pooling layer, and Mean is a 3D average pooling layer.
[0080] Step S32: Next, we upsample the high-level features B from the decoder to make them the same size as the feature matrix A′. After upsampling, we use a separable convolution operation to further extract the feature representation of B and maintain the consistency of the spatial position information. Separable convolution is an effective convolution operation that can reduce the number of parameters while maintaining the effectiveness of the model. After the separable convolution process, we get a decoder feature B′ that is magnified twice and has spatial position information. The specific calculation process is as follows:
[0081] B′=DWC(Upsample(B))
[0082] Step S33: Finally, we concatenate the feature matrix A′ from the encoder and the feature matrix B′ from the decoder to obtain a comprehensive feature representation, which contains important features from the encoder and decoder as well as spatial position information. This operation process aims to make full use of the low-level and high-level features in the encoder and decoder, and to retain and extract spatial position information through weighted fusion and convolution operations, thereby providing a more accurate and effective feature representation for medical image segmentation tasks. The complete calculation process is shown in the formula:
[0083]
[0084] MLP is a multi-layer perceptron, a basic structure of a neural network; Upsample is to upsample the feature;
[0085] Step S34: Through a series of detailed operations, including pooling, weighted multiplication, upsampling, separable convolution and feature concatenation, the feature information in the encoder and decoder is effectively fused and the spatial position information is retained.
[0086] Step S4 specifically includes the following steps:
[0087] Step S41: There are many methods for dynamic position coding, among which the current mainstream methods include dynamic position coding based on convolution and dynamic position coding based on global information. The convolution-based method utilizes the local receptive field characteristics of the convolutional neural network, which can better capture the local features of the image. However, due to its strong locality, it is always unable to effectively consider the situation involving global information, resulting in the inability to fully utilize the overall semantic information of the image, limiting the performance of the model. Another method based on global information adopts a strategy of global analysis of the entire image, and generates position coding by comprehensively considering the information of all pixels in the image. This method can better capture the global features and semantic information of the image, increase contextual understanding, and thus improve the model's ability to understand the overall structure of the image. A novel dynamic representation position encoding module (DPRE) is designed, which extracts local features from different scales through separable convolution, and then performs global attention operations to obtain global information to increase contextual understanding, so that dynamic position coding can be better implemented according to the content;
[0088] Step S42: First, the input features are put into two 3D separable convolution layers with different kernel sizes for feature extraction. The advantage of this design is that it can quickly extract local features in each patch image. By extracting local features at different scales, two features F3 and F5 with different sizes are obtained. The model can better capture the details in the image, thereby enhancing the ability to understand the local structure. At the same time, the use of separable convolution makes the model more efficient, which can reduce the number of parameters and reduce the computational complexity. The specific calculation process of F3 and F5 is as follows:
[0089] F i =DWC i×i (F input )
[0090] Where DWC stands for depthwise separable convolution, F i It is represented as the feature after depth-wise separable convolution, where i takes the values of 3 and 5, and F input Represented as input features.
[0091] Step S43: Next, the self-attention mechanism is used to establish links between patch features to capture the dependencies between positions of different sizes in the input sequence. By paying global attention to the entire sequence, the model can better understand the overall structure and semantic information of the input data, thereby improving the ability to capture global features and enhancing the understanding of the overall semantics of the image. ;
[0092] Step S44: At the same time, it only needs to perform self-attention on the features between patches, which can greatly reduce the computational complexity of training and reasoning, and has a significant effect on large-size images. Finally, a fully connected layer is used to further adjust the dynamic position encoding. By learning the weights and bias parameters of the position encoding, the position information is effectively integrated with the image features, and finally the feature F with dynamic position encoding is obtained. out , the specific calculation process is shown in the formula:
[0093] F pos =LN(F5)×Softmax(Reshape(LN(F3))×LN(F5))
[0094] F out =F input +Reshape(F pos )
[0095] F pos is the feature after learning the position feature; LN is the linear layer, F5 is the feature after the separable convolution of size 5 mentioned above, F3 is the feature after the separable convolution of size 3 mentioned above, Softmax is the normalization function, and Reshape is the shape transformation, so that the previous and next features can be multiplied according to the correct rank;
[0096] Step S45: Testing the trained medical image segmentation model on the validation set to obtain the final detection accuracy.
[0097] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0098] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.
[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0101] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.
[0102] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other forms of Transformer3D medical image segmentation methods based on feature focusing under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.
Claims
1. A Transformer 3D medical image segmentation method based on feature focusing, It is characterized in that The following steps are involved: Step S1: By training the 3D medical image dataset and using data enhancement methods on the data, the data is sent to the V-Net backbone network for a first-stage preliminary training and feature extraction; Step S2: Use the trained feature extraction network as the first-stage feature extraction module, and then propose the Transformer based on focus mapping and sparse matrix as the basic module of the second-stage extraction network. Through focus mapping, the intrinsic distribution change is stimulated. At the same time, the amount of calculation and storage space requirements are reduced by using sparse matrix, so that the model can capture key information when processing large-size data; Step S3: A spatial feature fusion link module SFFC is proposed to fuse the feature information in the encoder and decoder by retaining and extracting the spatial position information; Step S4: In view of the limitations of static position coding and conventional dynamic coding, a dynamic representation position coding module DPRE is proposed. It extracts local features at different scales, fuses global information to increase contextual understanding, implements dynamic position coding according to content, and finally realizes the segmentation of 3D medical images.
2. The Transformer 3D medical image segmentation method based on feature focusing according to claim 1, It is characterized in that Step S1 specifically includes the following steps: Step S11: Obtain a public medical image segmentation training set from the Internet and obtain relevant annotations of the training data; mainly obtain three commonly used 3D medical image datasets ADCD, Synapse, and LA, which respectively include data of multiple parts of the heart and the whole body trunk; Step S12: After the data is loaded, different random data enhancements are performed on the data, including random color jitter, grayscale, Gaussian blur and random shearing to simulate the complex changes in real medical scenes and force the model to learn more robust feature representations; Step S13: Initialize network weights and parameters of the image to be segmented in the medical image segmentation dataset using a pre-trained V-Net feature extraction network.
3. The Transformer 3D medical image segmentation method based on feature focusing according to claim 1, It is characterized in that Step S2 specifically includes the following steps: Step S21: First, perform linear transformation on the query matrix Q and the key matrix K, and use the similarity calculation method to calculate the similarity between them, so as to obtain the self-attention matrix A. In the self-attention matrix A, each element represents the similarity score between different positions in the sequence; then, perform sparse processing on the self-attention matrix A to obtain a sparse self-attention matrix A'; most of the elements in the sparse self-attention matrix are zero, and only some important similarity scores are retained. The specific calculation process is as follows: Q=xW Q ,K=xW K ,V=xW V Among them, x represents the input feature, K T represents the transpose of the key matrix, d represents the dimension of the value matrix, Sim uses the common Sotfmax function, and after sparsely processing the self-attention matrix A, we get A′ ij , where k is a hyperparameter of the sparsification matrix, which is also a hyperparameter of the focus map; A′ ij Represents the sparse self-attention matrix; A ij represents the self-attention matrix; Step S22: In the V processing of the value matrix, a focus mapping function is proposed, which is carefully designed and applied to the value matrix; the specific calculation formula is as follows: Where V′ represents the value matrix after stimulating the intrinsic characteristics, and the RELU function ensures the non-negativity of the input and the validity of the denominator; Step S23: A local rank restoration module is proposed to restore the order of the attention matrix by applying additional depthwise separable convolution (DWC) to supplement the details lost in the model; the final complete and specific calculation process is as follows: Where Sparse() is the sparse function, O is the final output of self-attention; DWC is the separable convolution, where DWC(V) represents the operation of the separable convolution on the V matrix.
4. The Transformer 3D medical image segmentation method based on feature focusing according to claim 1, It is characterized in that Step S3 specifically includes the following steps: Step S31: improving the skip connection module; First, obtain the low-level feature A from the encoder, and adopt the maximum pooling and average pooling strategies; through these two pooling operations, two feature weight matrices with different characteristics are obtained, representing the prominent features and global features of A respectively; add these two pooling matrices to fuse the prominent features and global features of feature A, which can better capture the correlation and importance between features, and process them through the sigmoid activation function to limit the weighted features between 0 and 1; then, multiply the weighted weight matrix with the original feature matrix A to obtain a new feature matrix A′; the purpose of this step is to make A′ pay more attention to the important spatial position information in the encoder, as well as the prominent and global features through the weighted multiplication operation. The specific calculation is shown in the formula: A′=(MLP(Max(A))+MLP(Mean(A)))×A Among them, MLP is a multi-layer perceptron, Max is a 3D maximum pooling layer, and Mean is a 3D average pooling layer; Step S32: Upsample the high-level feature B from the decoder to make it have the same size as the feature matrix A′; after upsampling, use a separable convolution operation to further extract the feature representation of B and maintain the consistency of the spatial position information; after the separable convolution process, a decoder feature B′ with twice the spatial position information is obtained. The specific calculation process is as follows: B′=DWC(Upsample(B)) Step S33: Finally, we concatenate the feature matrix A′ from the encoder and the feature matrix B′ from the decoder to obtain a comprehensive feature representation, which contains important features from the encoder and decoder as well as spatial position information; the complete calculation process is shown in the formula: MLP is a multi-layer perceptron, a basic structure of a neural network; Upsample is to upsample the feature; Step S34: Through a series of detailed operations, including pooling, weighted multiplication, upsampling, separable convolution and feature splicing, the feature information in the encoder and decoder is effectively integrated and the spatial position information is retained.
5. The Transformer 3D medical image segmentation method based on feature focusing according to claim 1, It is characterized in that Step S4 specifically includes the following steps: Step S41: The dynamic representation position encoding module DPRE extracts local features from different scales through separable convolution, and then performs global attention operation to obtain global information to increase context understanding, thereby realizing dynamic position encoding according to the content; Step S42: First, the input features are put into two 3D separable convolutional layers with different kernel sizes for feature extraction; by extracting local features at different scales, two features F with different sizes are obtained. 3 and F 5 , specific F 3 and F 5 The calculation process is as follows: F i =DWC i×i (F input ) Where DWC stands for depthwise separable convolution, F i It is represented as the feature after depth-wise separable convolution, where the values of i are 3 and 5, and F input Represented as input features; Step S43: Next, the self-attention mechanism is used to establish links between patch features to capture the dependencies between positions of different sizes in the input sequence; Step S44: A fully connected layer is used to further adjust the dynamic position encoding; by learning the weight and bias parameters of the position encoding, the position information is effectively integrated with the image features, and finally the feature F with dynamic position encoding is obtained. out , the specific calculation process is shown in the formula: F pos =LN(F 5 )×Softmax(Reshape(LN(F 3 ))×LN(F 5 )) F out =F input +Reshape(F pos ) F pos is the feature after learning the position feature; LN is the linear layer, F 5 is the feature after the separable convolution of size 5 mentioned above, F 3 The above mentioned features after the separable convolution of size 3, Softmax is a normalization function, and Reshape is a shape transformation, so that the previous and next features can be multiplied in the correct rank; Step S45: Testing the trained medical image segmentation model on the validation set to obtain the final detection accuracy.
Citation Information
Patent Citations
Medical image segmentation method based on multi-stage aggregation
CN116385454A
Medical image segmentation method based on joint encoder network GPA-TUNet
CN117197151A
Multi-context brain tumor segmentation system based on scale fusion guidance
CN117237320A
Image semantic segmentation method and system based on differentiated contexts
CN117253034A
Global and local feature reconstruction network-based medical image segmentation method
US20230274531A1
