A whole brain segmentation method, system, device, and medium
By combining a whole-brain segmentation model with CNN and Transformer encoders, the problem of low soft tissue contrast in CT images was solved, and accurate segmentation of brain tissues such as gray matter, white matter, cerebrospinal fluid and brainstem was achieved, thus enhancing the application value of CT images in the diagnosis and treatment of neurological diseases.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG CANCER HOSPITAL
- Filing Date
- 2026-01-29
- Publication Date
- 2026-05-19
AI Technical Summary
Existing CT image segmentation methods struggle to accurately distinguish between brain gray matter, white matter, cerebrospinal fluid, and brainstem tissues when soft tissue contrast is low, resulting in low segmentation accuracy and poor robustness, which fails to meet clinical needs.
A whole-brain segmentation model based on CNN encoder and Transformer encoder is adopted, combined with cross-domain fusion module and feature enhancement module, to extract local detail features and global semantic features of CT images. Through cross-domain fusion and feature enhancement, dynamic interaction and fusion are performed, and finally upsampling decoding is performed through decoder to achieve accurate segmentation of multiple brain tissues.
It significantly improves the ability to distinguish the boundaries of soft tissues such as gray matter, white matter, cerebrospinal fluid, and brainstem on CT images. The segmentation boundaries are clearer and conform to the anatomical structure, providing high-precision whole-brain segmentation results to support disease assessment and surgical planning.
Smart Images
Figure CN121616610B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of medical image processing and artificial intelligence technology, specifically to a whole-brain segmentation method, system, device, and medium. Background Technology
[0002] Whole-brain segmentation is a key task in medical image analysis, aiming to accurately separate different tissue structures (such as gray matter, white matter, cerebrospinal fluid, and brainstem) in brain images, providing important information for the diagnosis, surgical planning, and efficacy evaluation of brain diseases. Currently, whole-brain segmentation mainly relies on two imaging methods: computed tomography (CT) and magnetic resonance imaging (MRI).
[0003] MRI images offer high soft tissue contrast, enabling clear differentiation of different brain structures. Therefore, most existing high-precision brain segmentation methods are developed based on MRI data. However, the high cost of MRI equipment, long scan times, high patient compliance requirements, and susceptibility to motion artifacts limit its application in emergency situations, large-scale screenings, and resource-constrained scenarios.
[0004] In contrast, CT imaging has advantages such as fast scanning speed, low cost, and strong adaptability, and is widely used in the diagnosis of acute conditions such as stroke and traumatic brain injury. However, CT images have significantly lower soft tissue contrast than MRI, with severe gray-level overlap and blurred boundaries between different brain tissues. This poses a huge challenge to traditional image segmentation methods (such as thresholding, region growing, and edge detection), resulting in low segmentation accuracy and poor robustness, making it difficult to meet clinical needs.
[0005] In recent years, with the development of deep learning technology, automatic segmentation methods based on convolutional neural networks (CNNs) have made significant progress in medical image analysis. However, most existing CT-based brain segmentation methods employ pure CNN structures. While these models excel in local feature extraction, they have limited ability to capture long-range dependencies and global semantic information. In low-contrast CT images, relying solely on local features makes it difficult to accurately distinguish different brain tissues, especially structures with similar gray levels such as gray matter, white matter, and cerebrospinal fluid, which can easily lead to missegmentation and blurred boundaries. Summary of the Invention
[0006] The technical problem to be solved by this invention is: a whole-brain segmentation method, system, device, and medium that addresses the problem of low soft tissue contrast in CT image data, making it difficult to distinguish between brain gray matter, brain white matter, cerebrospinal fluid, and brainstem tissue. This method extracts local detail features using a CNN encoder and global semantic features using a Transformer encoder, and dynamically interacts and fuses the local and global semantic features to accurately segment the whole brain gray matter, brain white matter, cerebrospinal fluid, and brainstem tissue.
[0007] To address the aforementioned technical problems, the present invention provides a whole-brain segmentation method, which includes at least the following steps:
[0008] Acquire CT image data and preprocess the CT image data to obtain standard image data;
[0009] The standard image data is input into the whole-brain segmentation model to obtain the whole-brain segmentation result;
[0010] The whole-brain segmentation model is trained based on labeled sample data, and the whole-brain segmentation model includes at least:
[0011] A CNN encoder includes at least a plurality of sequentially connected three-dimensional convolutional residual blocks for extracting local features at different scales from the standard image data;
[0012] The Transformer encoder includes at least multiple Transformer modules based on a self-attention mechanism, used to extract global semantic features at different scales from the standard image data;
[0013] And multiple cross-domain fusion modules are set up one-to-one with the three-dimensional convolutional residual block and the Transformer module, which are used to fuse the local features output by the corresponding three-dimensional convolutional residual block and the global semantic features output by the corresponding Transformer module to obtain fused features;
[0014] A feature enhancement module, which is set up one-to-one with the cross-domain fusion module, is used to splice the fused feature with the corresponding local feature and enhance the feature through the channel attention mechanism to obtain the enhanced feature;
[0015] The decoder is used to perform upsampling decoding by sequentially fusing different scale enhancement features based on the global semantic features output by the Transformer module located at the end, to obtain a whole-brain segmentation result, which includes a segmentation map of at least one brain tissue, such as gray matter, white matter, cerebrospinal fluid, or brainstem.
[0016] In a preferred embodiment, any of the three-dimensional convolutional residual blocks includes at least a first convolutional layer for extracting local features from the input features, a first normalization layer connected to the output of the first convolutional layer for batch normalization of the output features of the first convolutional layer, a first activation function layer connected to the output of the first normalization layer for nonlinear activation of the output features of the first normalization layer, and a residual connection establishing the output of the first activation function layer and the output of the first convolutional layer. Specifically, this is achieved using the following calculation method:
[0017] ;
[0018] in, Let be the weight parameters of the i-th 3D convolutional residual block. For the local features output by the i-th 3D convolutional residual block, For batch normalization, This is the Sigmoid activation function.
[0019] In a preferred embodiment, the Transformer encoder includes at least:
[0020] The Patch Embedding layer is used to segment the input standard image data into multiple patches and embed each patch into a feature vector through linear projection.
[0021] Multiple Transformer modules, including two sequentially connected SwinTransformer modules, capture long-distance dependency features through window attention mechanism and shift window attention mechanism;
[0022] A Patch Merging layer is interspersed between the Transformer modules. The Patch Merging layer is used to downsample the output of the Transformer module located at its input and transmit the sampled result to the Transformer module located at its output.
[0023] In a preferred embodiment, the cross-domain fusion module includes at least:
[0024] A cross-domain basic fusion module is used to receive the local features output by the 3D convolutional residual block and the global semantic features output by the corresponding Transformer module, and perform preliminary fusion through the SwinTransformer module with self-attention mechanism to obtain the first feature data.
[0025] A correlation enhancement module is used to obtain second feature data by multiplying the global semantic features and local features element-wise.
[0026] The channel attention module includes at least a pooling layer for performing global average pooling on the global semantic features, a fully connected layer connected to the pooling layer for mapping the pooling result to obtain a first channel weight parameter, and a fusion layer connected to the fully connected layer, wherein the fusion layer is used to fuse the first channel weight parameter and the global semantic features by element-wise multiplication to obtain third feature data.
[0027] The feature fusion module is used to process the concatenation result of the first feature data, the second feature data, and the third feature data through three-dimensional convolution processing, batch normalization processing, and activation function to introduce nonlinear processing, and establish a residual connection between the processed result and the concatenation result to obtain the fused feature.
[0028] In a preferred embodiment, the initial fusion of the first feature data using the Swin Transformer module with a self-attention mechanism specifically includes the following steps:
[0029] The local features output by the three-dimensional convolutional residual block are subjected to global average pooling to obtain the fourth feature data;
[0030] The global semantic features output by the Transformer module are adjusted in dimension to obtain the fifth feature data with the same dimension as the fourth feature data, and global average pooling is used to obtain the sixth feature data.
[0031] The fourth feature data is concatenated with the fifth feature data, and the concatenation result is used to extract features through the SwinTransformer module to obtain the sixth feature data.
[0032] The fourth feature data is concatenated with the sixth feature data, and the concatenation result is used to extract features through the SwinTransformer module to obtain the seventh feature data.
[0033] The sixth feature data and the seventh feature data are concatenated, and the concatenation result is channel-fused through a 1×1 convolutional layer to obtain the first feature data.
[0034] In a preferred embodiment, the step of concatenating the fused feature with the corresponding local feature and then performing feature enhancement through a channel attention mechanism to obtain the enhanced feature specifically includes the following steps:
[0035] The fused feature is concatenated with the corresponding local feature to obtain the eighth feature data, and the spatial dimension of the eighth feature data is compressed by global average pooling.
[0036] The eighth feature data after compression of spatial dimension is mapped through a fully connected layer, and the mapping result is activated by the Sigmoid activation function to obtain the second channel weight parameters.
[0037] The enhanced feature is obtained by fusing the second channel weight parameter with the eighth feature data through element-wise multiplication.
[0038] In a preferred embodiment, the decoder includes at least a DecoderBlock module that is sequentially connected and corresponds one-to-one with the three-dimensional convolutional residual blocks;
[0039] The DecoderBlock module establishes a skip connection with the feature enhancement module at the corresponding level, and the DecoderBlock module at the end establishes a skip connection with the three-dimensional convolutional residual block at the beginning.
[0040] The input terminal of the DecoderBlock module located at the beginning is connected to the output terminal of the Transformer module located at the end;
[0041] Any of the DecoderBlock modules includes at least an upsampling layer for restoring resolution, a second convolutional layer connected to the output of the upsampling layer for feature extraction of the upsampled features, a second normalization layer connected to the output of the second convolutional layer for batch normalization, a second activation function layer connected to the output of the second normalization layer for non-linear activation, and a concatenation layer connected to the output of the second activation function layer, wherein the concatenation layer is used to concatenate the data output by the second activation function layer with the data input by the skip connection.
[0042] The present invention also provides a system employing any of the above-described whole-brain segmentation methods, comprising at least:
[0043] A data preprocessing module is used to acquire the CT image data and preprocess the CT image data to obtain standard image data.
[0044] A whole-brain segmentation model construction module is used to construct the whole-brain segmentation model, train the whole-brain segmentation model based on sample labeled data to obtain the optimal weight parameters, and load the optimal weight parameters into the whole-brain segmentation model.
[0045] The segmentation module, through the trained whole-brain segmentation model, segments the standard image data to obtain a whole-brain segmentation result including a segmentation map of at least one brain tissue, such as gray matter, white matter, cerebrospinal fluid, or brainstem.
[0046] The present invention also provides an apparatus including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the whole-brain segmentation method as described in any of the preceding claims.
[0047] The present invention also provides a medium having a computer program stored thereon, which, when executed by a processor, implements the whole-brain segmentation method as described in any of the preceding claims.
[0048] The whole-brain segmentation method of the present invention has the following advantages compared with the prior art:
[0049] (1) A whole-brain segmentation method of the present invention includes at least the following steps: acquiring CT image data and preprocessing the CT image data to obtain standard image data. Through standardization preprocessing, the heterogeneity of CT images is eliminated, the consistency of input data is improved, the foundation for the subsequent model to extract stable features is laid, the segmentation error caused by data differences is reduced, and the robustness of the model is enhanced; the standard image data is input into the whole-brain segmentation model to obtain the whole-brain segmentation result.
[0050] The whole-brain segmentation model is trained based on labeled sample data and includes at least a CNN encoder, a Transformer encoder, a cross-domain fusion module, a feature enhancement module, and a decoder. The CNN encoder includes at least multiple sequentially connected 3D convolutional residual blocks to extract local detail features at different scales from standard image data, improving localization accuracy at blurred boundaries. The Transformer encoder includes at least multiple Transformer modules based on self-attention mechanisms. These mechanisms establish long-range dependencies between any two pixels in the image, extracting global semantic features at different scales from standard image data to understand the image content holistically. This provides a global understanding of the overall anatomical structure of the whole brain, overcoming ambiguity caused by local gray-level overlap and compensating for the limited receptive field of traditional CNNs. The dual-path parallel architecture of the CNN and Transformer encoders fully leverages the complementary advantages of local details and global contextual information, significantly improving the model's ability to distinguish soft tissue boundaries such as gray matter, white matter, cerebrospinal fluid, and brainstem in low-contrast CT images, resulting in clearer segmentation boundaries that better conform to anatomical structures.
[0051] Multiple cross-domain fusion modules are configured one-to-one with 3D convolutional residual blocks and Transformer modules. These modules fuse the local features output by the corresponding 3D convolutional residual block with the global semantic features output by the corresponding Transformer module to obtain fused features. This ensures that information interacts at the same level of abstraction, avoiding fusion noise caused by scale mismatch. The cross-domain fusion modules do not simply stitch together features; they engage in deep interaction. This allows global semantics to guide the focusing and correction of local features in real time, while local details enrich the accuracy of global understanding. Consequently, more discriminative fused representations are generated at each feature scale, improving the feature representation ability of complex and blurred CT images.
[0052] The feature enhancement module, which corresponds one-to-one with the cross-domain fusion module, is used to concatenate the fused features with the corresponding local features and enhance the features through the channel attention mechanism to obtain enhanced features. This achieves secondary refinement and focusing of features, on the one hand, retaining the original local information as a reference; on the other hand, it strengthens the feature channels that are crucial to the current segmentation task, while suppressing redundant or noisy channels, ensuring that the enhanced features are highly optimized and purified.
[0053] The decoder, based on the global semantic features output by the Transformer module at the end, sequentially fuses enhanced features at different scales for upsampling decoding to obtain whole-brain segmentation results. It recovers precise spatial details from high-level semantic features, continuously acquiring fused and enhanced detail information from different levels during the gradual restoration of spatial resolution, thus ensuring that the final segmentation result is both semantically correct and has precise boundaries. The whole-brain segmentation result includes segmentation maps of at least one brain tissue from the cerebral gray matter, white matter, cerebrospinal fluid, or brainstem. It can accurately segment multiple key brain tissues in a single inference, providing structured information that greatly facilitates subsequent quantitative analysis, disease assessment, and surgical planning, enhancing the practical value of CT imaging in the diagnosis and treatment of neurological diseases.
[0054] (2) The present invention provides a whole-brain segmentation method, wherein the cross-domain fusion module includes at least a cross-domain basic fusion module, a correlation enhancement module, a channel attention module, and a feature fusion module. The cross-domain basic fusion module dynamically evaluates and integrates the correlation between two features at different spatial locations through a self-attention mechanism, allowing the features to interact and discover which local details are most relevant to which global contexts, thereby generating first feature data with richer information and stronger correlation, laying the foundation for deep fusion.
[0055] The correlation enhancement module is used to obtain the second feature data by multiplying the global semantic features and local features element by element. It performs pixel-level recalibration on local detail features, amplifies the local region response consistent with the global semantics, suppresses inconsistent or irrelevant noise, and enhances the expression of fused features on key anatomical structures, which is beneficial for making correct judgments in areas with blurred boundaries.
[0056] The channel attention module includes at least a pooling layer for global average pooling of global semantic features, a fully connected layer connected to the pooling layer for mapping the pooling result to the first channel weight parameters, and a fusion layer connected to the fully connected layer. The fusion layer is used to fuse the first channel weight parameters and global semantic features in an element-wise multiplication manner to obtain the third feature data, realizing feature recalibration in the channel dimension. By multiplying the weights with the original features, it can adaptively emphasize the feature channels with rich information and suppress the channels with less information. This allows the global semantic features themselves to be optimized before fusion, highlighting their most discriminative parts and improving the quality of subsequent fusion.
[0057] The feature fusion module takes the concatenated first feature data, second feature data, and third feature data and introduces nonlinear processing through three-dimensional convolution, batch normalization, and activation functions. It then establishes a residual connection between the processed result and the concatenated result to obtain the fused feature. This ensures that features from different paths and enhanced in different ways can be smoothly and efficiently integrated into a unified fused feature with stronger expressive power, while avoiding possible information degradation during the fusion process.
[0058] (3) The decoder includes at least a DecoderBlock module that is sequentially connected and corresponds one-to-one with the 3D convolutional residual blocks. This ensures that each scale feature extracted during the encoding process has a corresponding module to receive and process during the decoding and resolution restoration. This symmetry ensures that the path of multi-scale information transmission is the shortest and most direct, avoiding information attenuation or distortion in complex paths, and laying the structural foundation for the final output of a high spatial precision segmentation map. The DecoderBlock module establishes a skip connection with the feature enhancement module at the corresponding level to achieve high-precision boundary reconstruction. The output of the feature enhancement module is the most discriminative feature that has undergone cross-domain fusion and attention optimization. By skipping the connection, it is directly injected into the decoding stage of the same scale, providing a "high-quality reference template from the same level as the encoder" for the decoding process that is restoring the resolution. This supplements the details that may be lost during upsampling, ensures that the fine structure and local texture of the tissue boundary are accurately restored, and effectively overcomes the boundary blurring problem caused by the low contrast of CT images. The DecoderBlock module at the end establishes a jump connection with the 3D convolutional residual block at the beginning, realizing a direct connection between the "deepest semantics" and the "shallowest details". The deepest features contain rich category semantics, while the shallowest features retain the most original high-resolution spatial details (such as subtle edges and noise). Introducing the shallowest details in the last step of decoding can perform pixel-level final fine-tuning and sharpening on the segmentation map to be output, so that the segmentation boundary has the correct semantic attribution while achieving the highest spatial fit.
[0059] The input of the DecoderBlock module at the beginning is connected to the output of the Transformer module at the end. The output of the Transformer module at the end contains the richest global context information and is the "brain" of the segmentation decision. Taking this as the starting point for decoding ensures that the entire upsampling and feature fusion process is always carried out under the strongest global semantic guidance and control, preventing the decoding process from deviating from the correct anatomical structure and ensuring the overall rationality and consistency of the segmentation results. Attached Figure Description
[0060] Figure 1 This is a flowchart illustrating an embodiment of a whole-brain segmentation method, system, device, and medium according to the present invention.
[0061] Figure 2 This is a schematic diagram of a whole-brain segmentation model, representing an embodiment of a whole-brain segmentation method, system, device, and medium according to the present invention.
[0062] Figure 3 This is a schematic diagram of a cross-domain fusion module, representing an embodiment of a whole-brain segmentation method, system, device, and medium of the present invention.
[0063] Figure 4 This is a schematic diagram illustrating the gold standard annotation results of an embodiment of the whole-brain segmentation method, system, device, and medium of the present invention;
[0064] Figure 5 This is a schematic diagram of the whole-brain segmentation model and the whole-brain segmentation result, representing an embodiment of the whole-brain segmentation method, system, device, and medium of the present invention.
[0065] Figure 6 This is a schematic diagram of the whole-brain segmentation results of the existing UNet model. Detailed Implementation
[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0067] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0068] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, an integral connection, or a detachable connection; they can refer to the internal connection of two components; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0069] Example
[0070] This embodiment describes a whole-brain segmentation method, such as... Figure 1 As shown, it includes at least the following steps:
[0071] Step S1: Acquire CT image data and preprocess it to obtain standard image data. Standardization preprocessing eliminates the heterogeneity of CT images, improves the consistency of input data, lays the foundation for subsequent model extraction of stable features, reduces segmentation errors caused by data differences, and enhances model robustness.
[0072] In this embodiment, the preprocessing of CT image data includes interpolation completion, normalization, and spatial alignment, specifically including the following steps:
[0073] If there are inconsistencies in the dimensions of CT image data or if slices need to be resampled, methods such as cubic spline interpolation or linear interpolation are used to interpolate or downsample in three spatial dimensions to unify the size of all data to the preset standard size.
[0074] CT image data is normalized to scale the data values to a relatively stable range, eliminating differences in grayscale distribution caused by different CT scanning equipment and parameters. This accelerates model training convergence and improves generalization ability. The specific calculation method is as follows:
[0075] ;
[0076] in, and The mean and standard deviation of the pixel values in the CT images in the sample labeled data. Original CT image data, These are normalized CT image data.
[0077] Aligning CT image data to a common, standardized anatomical space eliminates individual differences in head pose, position, and angle, ensuring the model learns features of the anatomical structures themselves, rather than positional variations. Specifically, this includes:
[0078] (1) Select a reference frame. Select a CT image with standard orientation and position as reference frame I.r .
[0079] (2) Determine the spatial transformation operator. The spatial transformation operator is determined by minimizing the difference between the normalized image data and the reference frame. The specific calculation method is as follows:
[0080] ;
[0081] Among them, I r For reference frame, For spatial transformation operators, These are normalized CT image data.
[0082] (3) Spatial alignment: Based on the spatial transformation operator, the CT image is spatially transformed. The specific calculation method is as follows:
[0083] ;
[0084] in, For spatial transformation operators, The CT image data after spatial transformation These are normalized CT image data.
[0085] Preprocessing, through interpolation completion (uniform size) → normalization (uniform grayscale distribution) → spatial alignment (uniform anatomical space), solves the inherent heterogeneity problem of clinical CT data and improves the training stability, generalization performance and final segmentation accuracy of deep learning models.
[0086] Step S2 involves inputting standard image data into the whole-brain segmentation model to obtain the whole-brain segmentation result. The whole-brain segmentation model is trained based on labeled sample data, which uses the gold standard for annotation.
[0087] Whole brain segmentation models, such as Figure 2 As shown, it includes at least a CNN encoder, a Transformer encoder, a cross-domain fusion module, a feature enhancement module, and a decoder.
[0088] A CNN encoder comprises at least multiple sequentially connected 3D convolutional residual blocks for extracting local features at different scales from standard image data, improving localization accuracy at blurred boundaries. In this embodiment, the CNN encoder includes four sequentially connected 3D convolutional residual blocks: 3D convolutional residual block one, 3D convolutional residual block two connected to the output of 3D convolutional residual block one, 3D convolutional residual block three connected to the output of 3D convolutional residual block two, and 3D convolutional output block four connected to the output of 3D convolutional residual block three. It should be noted that 3D convolutional residual block one is located at the beginning of the CNN encoder, and 3D convolutional output block four is located at the end of the CNN encoder.
[0089] Any three-dimensional convolutional residual block includes at least a first convolutional layer for extracting local features from the input features, a first normalization layer connected to the output of the first convolutional layer for batch normalization of the output features of the first convolutional layer, a first activation function layer connected to the output of the first normalization layer for nonlinear activation of the output features of the first normalization layer, and a residual connection establishing the output of the first activation function layer and the output of the first convolutional layer. Specifically, it is implemented using the following calculation method:
[0090] ;
[0091] in, Let be the weight parameters of the i-th 3D convolutional residual block. Let be the local features output by the i-th 3D convolutional residual block, and BN be the batch normalization. This is the Sigmoid activation function. It addresses the issue of blurred boundaries in CT images. The 3D convolutional residual blocks fully utilize the spatial volume information of CT images. The residual connection structure effectively alleviates gradient vanishing, allowing the network to be built deeper. This enables multi-level and efficient capture of detailed texture features from subtle edges to local structures. The rich local information can accurately depict tissue boundaries and distinguish areas with similar gray levels.
[0092] The Transformer encoder includes at least several Transformer modules based on a self-attention mechanism to extract global semantic features at different scales from standard image data. The self-attention mechanism establishes long-range dependencies between any two pixels in an image, enabling the extraction of global semantic features at different scales from standard image data. This allows for a holistic understanding of image content, establishing a global understanding of the overall anatomical structure of the brain, overcoming ambiguity caused by local gray-level overlap, and compensating for the limited receptive field of traditional CNNs. Employing a dual-path parallel architecture of CNN and Transformer encoders, it fully leverages the complementary advantages of local details and global contextual information, significantly improving the model's ability to distinguish soft tissue boundaries such as gray matter, white matter, cerebrospinal fluid, and brainstem in low-contrast CT images, resulting in clearer segmentation boundaries that better conform to anatomical structures.
[0093] A Transformer encoder includes at least:
[0094] The Patch Embedding layer is used to segment the input standard image data into multiple patches. Each patch is embedded into a feature vector through linear projection, transforming the structured information of the image into a modelable semantic unit, and providing an input format for subsequent self-attention mechanisms to perform global relation calculations.
[0095] Multiple Transformer modules, including two sequentially connected Swing Transformer modules, capture long-range dependency features through window attention and shifted window attention mechanisms. On one hand, window attention computes self-attention within a local window, significantly reducing computational cost; on the other hand, shifted window attention enables information exchange between adjacent windows. The alternating use of these two mechanisms maintains the ability to model long-range dependencies (overcoming the limitations of pure CNNs) while significantly improving the model's computational efficiency.
[0096] Patch Merging layers are interspersed between Transformer modules. These layers downsample the output of the Transformer module at its input and transmit the sampled result to the Transformer module at its output, constructing a hierarchical feature pyramid. As the network deepens, the Patch Merging layers gradually reduce the spatial resolution and increase the feature dimension. This allows shallow features to capture details, while deeper features fuse broader contextual information. Consequently, the global semantic features output by the Transformer encoder itself possess multi-scale characteristics, facilitating subsequent alignment and fusion with the multi-scale local features of the CNN.
[0097] In this embodiment, the Transformer encoder includes a Patch Embedding layer, a first Transformer module connected to the output of the Patch Embedding layer, a first Patch Merging layer connected to the output of the first Transformer module, a second Transformer module connected to the output of the first Patch Merging layer, a second Patch Merging layer connected to the output of the second Transformer module, a third Transformer module connected to the output of the second Patch Merging layer, a third Patch Merging layer connected to the output of the third Transformer module, and a fourth Transformer module connected to the output of the third Patch Merging layer. It should be noted that the fourth Transformer module is the Transformer module located at the end of the Transformer encoder.
[0098] Multiple cross-domain fusion modules are configured one-to-one with 3D convolutional residual blocks and Transformer modules. Each cross-domain fusion module is used to fuse the local features output by the corresponding 3D convolutional residual block and the global semantic features output by the corresponding Transformer module to obtain fused features. This ensures that information interacts at the same level of abstraction and avoids fusion noise caused by scale mismatch. The cross-domain fusion modules do not simply stitch together features but engage in deep interaction, enabling global semantics to guide the focusing and correction of local features in real time. At the same time, local details can enrich the accuracy of global understanding, thereby generating more discriminative fused representations at each feature scale and improving the feature's ability to represent complex and blurred CT images.
[0099] Cross-domain integration modules, such as Figure 3 As shown, it includes at least a cross-domain basic fusion module, a correlation enhancement module, a channel attention module, and a feature fusion module.
[0100] The cross-domain basic fusion module receives local features from the output of the 3D convolutional residual block and global semantic features from the output of the corresponding Transformer module. It performs initial fusion through the Swin Transformer module with self-attention mechanism to obtain the first feature data. The cross-domain basic fusion module dynamically evaluates and integrates the correlation between the two features at different spatial locations through self-attention mechanism, allowing the features to interact and discover which local details are most relevant to which global contexts, thereby generating first feature data with richer information and stronger correlation, laying the foundation for deep fusion.
[0101] The correlation enhancement module is used to obtain the second feature data by multiplying the global semantic features and local features element by element. It performs pixel-level recalibration on local detail features, amplifies the local region response consistent with the global semantics, suppresses inconsistent or irrelevant noise, and enhances the expression of fused features on key anatomical structures, which is beneficial for making correct judgments in areas with blurred boundaries.
[0102] The channel attention module includes at least a pooling layer for global average pooling of global semantic features, a fully connected layer connected to the pooling layer for mapping the pooling result to the first channel weight parameters, and a fusion layer connected to the fully connected layer. The fusion layer is used to fuse the first channel weight parameters and global semantic features in an element-wise multiplication manner to obtain the third feature data, realizing feature recalibration in the channel dimension. By multiplying the weights with the original features, it can adaptively emphasize the feature channels with rich information and suppress the channels with less information. This allows the global semantic features themselves to be optimized before fusion, highlighting their most discriminative parts and improving the quality of subsequent fusion.
[0103] The feature fusion module is used to combine the first, second, and third feature data and introduce nonlinear processing through three-dimensional convolution, batch normalization, and activation functions. The processed result is then connected with the combined result to establish a residual connection to obtain the fused feature. This ensures that features from different paths and enhanced in different ways can be smoothly and efficiently integrated into a unified and more expressive fused feature, while avoiding possible information degradation during the fusion process.
[0104] The first feature data is obtained through preliminary fusion using the Swin Transformer module with a self-attention mechanism, specifically including the following steps:
[0105] The local features output by the 3D convolutional residual block are subjected to global average pooling to obtain the fourth feature data;
[0106] The global semantic features output by the corresponding Transformer module are adjusted in dimension to obtain the fifth feature data with the same dimension as the fourth feature data, and global average pooling is used to obtain the sixth feature data.
[0107] The fourth feature data is concatenated with the fifth feature data, and the concatenation result is used to extract features through the Swin Transformer module to obtain the sixth feature data.
[0108] The fourth feature data is concatenated with the sixth feature data, and the concatenation result is used to extract features through the Swin Transformer module to obtain the seventh feature data.
[0109] The sixth feature data and the seventh feature data are concatenated, and the concatenation result is channel-fused through a 1×1 convolutional layer to obtain the first feature data.
[0110] Multiple feature enhancement modules are set up one-to-one with the cross-domain fusion module. The feature enhancement module is used to splice the fused features with the corresponding local features and enhance the features through the channel attention mechanism to obtain enhanced features. This realizes the secondary refinement and focusing of features. On the one hand, the original local information is retained as a reference; on the other hand, the feature channels that are crucial to the current segmentation task are strengthened, while redundant or noisy channels are suppressed to ensure that the enhanced features are highly optimized and purified.
[0111] The fused features are concatenated with the corresponding local features, and the enhanced features are obtained by using a channel attention mechanism. The specific steps include the following:
[0112] The fused features are concatenated with the corresponding local features to obtain the eighth feature data, and the spatial dimension of the eighth feature data is compressed by global average pooling.
[0113] The eighth feature data after compression of spatial dimensions is mapped through a fully connected layer, and the mapping result is activated by the Sigmoid activation function to obtain the second channel weight parameters.
[0114] The enhanced features are obtained by fusing the weight parameters of the second channel with the eighth feature data through element-wise multiplication.
[0115] The decoder is used to upsample and decode based on the global semantic features output by the Transformer module at the end, sequentially fusing enhanced features at different scales to obtain the whole-brain segmentation result. Accurate spatial details are recovered from high-level semantic features. In the process of gradually restoring spatial resolution, it continuously acquires fused and enhanced detail information from different levels, thus ensuring that the final segmentation result is both semantically correct and has precise boundaries.
[0116] The decoder includes at least a DecoderBlock module that is sequentially connected and corresponds one-to-one with the 3D convolutional residual block. This ensures that each scale feature extracted during the encoding process has a corresponding module to receive and process during the decoding and resolution restoration. This symmetry ensures that the path for multi-scale information transmission is the shortest and most direct, avoiding information attenuation or distortion in complex paths, and laying the structural foundation for the final output of a high spatial precision segmentation map.
[0117] The DecoderBlock module establishes skip connections with the corresponding level of feature enhancement modules to achieve high-precision boundary reconstruction. The output of the feature enhancement module is the most discriminative feature that has undergone cross-domain fusion and attention optimization. Through skip connections, it is directly injected into the decoding stage of the same scale, providing a "high-quality reference template from the same level as the encoder" for the decoding process that is restoring resolution. This supplements the details that may be lost during upsampling, ensuring that the fine structure and local texture of the tissue boundary are accurately restored, and effectively overcoming the boundary blurring problem caused by the low contrast of CT images.
[0118] The DecoderBlock module at the end establishes a jump connection with the 3D convolutional residual block at the beginning, realizing a direct connection between the "deepest semantics" and the "shallowest details". The deepest features contain rich category semantics, while the shallowest features retain the most original high-resolution spatial details (such as subtle edges and noise). Introducing the shallowest details in the last step of decoding can perform pixel-level final fine-tuning and sharpening on the segmentation map to be output, so that the segmentation boundary has the correct semantic attribution while achieving the highest spatial fit.
[0119] The input of the DecoderBlock module at the beginning is connected to the output of the Transformer module at the end. The output of the Transformer module at the end contains the richest global context information and is the "brain" of the segmentation decision. Using this as the starting point for decoding ensures that the entire upsampling and feature fusion process is always carried out under the strongest global semantic guidance and control, preventing the decoding process from deviating from the correct anatomical structure and ensuring the overall rationality and consistency of the segmentation results.
[0120] In this embodiment, the decoder includes four DecoderBlock modules connected in sequence: a first DecoderBlock module, a second DecoderBlock module connected to the output of the first DecoderBlock module, a third DecoderBlock module connected to the output of the second DecoderBlock module, and a fourth DecoderBlock module connected to the output of the third DecoderBlock module. It should be noted that the first DecoderBlock module is located at the beginning of the decoder, and the fourth DecoderBlock module is located at the end of the decoder.
[0121] Any DecoderBlock module includes at least an upsampling layer for restoring resolution, a second convolutional layer connected to the output of the upsampling layer for feature extraction of the upsampled features, a second normalization layer connected to the output of the second convolutional layer for batch normalization, a second activation function layer connected to the output of the second normalization layer for non-linear activation, and a concatenation layer connected to the output of the second activation function layer, the concatenation layer being used to concatenate the data output by the second activation function layer with the data input by the skip connections.
[0122] In this embodiment, three cross-domain fusion modules are employed, specifically a first cross-domain fusion module, a second cross-domain fusion module, and a third cross-domain fusion module. Correspondingly, three feature enhancement modules are also provided, specifically a first feature enhancement module, a second feature enhancement module, and a third feature enhancement module.
[0123] The input of the first cross-domain feature fusion module is connected to the output of the second three-dimensional convolutional residual block and the output of the first Transformer module, respectively. The output of the first cross-domain feature fusion module and the output of the second three-dimensional convolutional residual block are connected to the input of the first feature enhancement module, respectively. The output of the first feature enhancement module is connected to the input of the third DecoderBlock module.
[0124] The input of the second cross-domain feature fusion module is connected to the three outputs of the three-dimensional convolutional residual block and the output of the second Transformer module, respectively. The outputs of the second cross-domain feature fusion module and the three outputs of the three-dimensional convolutional residual block are connected to the input of the second feature enhancement module, respectively. The output of the second feature enhancement module is connected to the input of the second DecoderBlock module.
[0125] The input of the third cross-domain feature fusion module is connected to the output of the four-stage 3D convolutional residual block and the output of the third Transformer module, respectively. The output of the third cross-domain feature fusion module and the four-stage 3D convolutional residual block are connected to the input of the third feature enhancement module, respectively. The output of the third feature enhancement module is connected to the input of the first DecoderBlock module.
[0126] The input of the first DecoderBlock module is connected to the output of the fourth Transformer module, and the input of the fourth DecoderBlock module is connected to the output of the three-dimensional convolutional residual block.
[0127] Whole-brain segmentation results include segmentation maps of at least one brain tissue from the cerebral gray matter, white matter, cerebrospinal fluid, or brainstem. It enables precise segmentation of multiple key brain tissues in a single inference operation, providing structured information that greatly facilitates subsequent quantitative analysis, disease assessment, and surgical planning, thus enhancing the practical value of CT imaging in the diagnosis and treatment of neurological diseases. For example... Figure 4 , Figure 5 and Figure 6 The images shown are the gold standard annotation image, the segmentation result image of this embodiment, and the segmentation result image of the traditional UNet, respectively. The segmentation method of this embodiment has the advantages of clearer boundaries and higher segmentation accuracy. In whole-brain segmentation on low-contrast CT images, the Dice coefficient, an overlap evaluation index, is improved to 0.79, which is superior to the traditional methods UNet (0.72), Swin-UNet (0.76), Full-UNet (0.74), and Trans-UNet (0.75).
[0128] This embodiment also provides a system employing the whole-brain segmentation method described above, comprising at least:
[0129] The data preprocessing module is used to acquire CT image data and preprocess the CT image data to obtain standard image data.
[0130] The whole brain segmentation model construction module is used to construct a whole brain segmentation model, train the whole brain segmentation model based on sample labeled data to obtain the optimal weight parameters, and load the optimal weight parameters into the whole brain segmentation model.
[0131] The segmentation module, through a trained whole-brain segmentation model, segments standard image data to obtain whole-brain segmentation results including segmentation maps of at least one brain tissue, such as gray matter, white matter, cerebrospinal fluid, or brainstem.
[0132] This embodiment also provides a device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the whole-brain segmentation method as described above.
[0133] This embodiment also provides a medium on which a computer program is stored, which, when executed by a processor, implements the whole-brain segmentation method as described above.
[0134] In summary, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A whole-brain segmentation method, characterized in that, It should include at least the following steps: Acquire CT image data and preprocess the CT image data to obtain standard image data; The standard image data is input into the whole-brain segmentation model to obtain the whole-brain segmentation result; The whole-brain segmentation model is trained based on labeled sample data, and the whole-brain segmentation model includes at least: A CNN encoder includes at least a plurality of sequentially connected three-dimensional convolutional residual blocks for extracting local features at different scales from the standard image data; The Transformer encoder includes at least multiple Transformer modules based on a self-attention mechanism, used to extract global semantic features at different scales from the standard image data; And multiple cross-domain fusion modules are set up one-to-one with the three-dimensional convolutional residual block and the Transformer module, which are used to fuse the local features output by the corresponding three-dimensional convolutional residual block and the global semantic features output by the corresponding Transformer module to obtain fused features; A feature enhancement module, which is set up one-to-one with the cross-domain fusion module, is used to concatenate the fused feature with the corresponding local feature and enhance the feature through the channel attention mechanism to obtain the enhanced feature. Specifically, it includes the following steps: concatenating the fused feature with the corresponding local feature to obtain the eighth feature data, and compressing the spatial dimension of the eighth feature data through global average pooling. The eighth feature data after compression of spatial dimension is mapped through a fully connected layer, and the mapping result is activated by the Sigmoid activation function to obtain the second channel weight parameters. The enhanced feature is obtained by fusing the second channel weight parameters with the eighth feature data through element-wise multiplication. The decoder is used to perform upsampling decoding based on the global semantic features output by the Transformer module located at the end, sequentially fusing enhancement features at different scales, to obtain a whole-brain segmentation result. The whole-brain segmentation result includes a segmentation map of at least one brain tissue, such as gray matter, white matter, cerebrospinal fluid, or brainstem. The decoder includes at least a DecoderBlock module that is sequentially connected and corresponds one-to-one with the three-dimensional convolutional residual block, wherein the DecoderBlock module located at the end establishes a skip connection with the three-dimensional convolutional residual block located at the beginning.
2. The whole-brain segmentation method according to claim 1, characterized in that: Any of the three-dimensional convolutional residual blocks includes at least a first convolutional layer for extracting local features from the input features, a first normalization layer connected to the output of the first convolutional layer for batch normalization of the output features of the first convolutional layer, a first activation function layer connected to the output of the first normalization layer for nonlinear activation of the output features of the first normalization layer, and a residual connection establishing the output of the first activation function layer and the output of the first convolutional layer. Specifically, this is achieved using the following calculation method: ； in, Let be the weight parameters of the i-th 3D convolutional residual block. For the local features output by the i-th 3D convolutional residual block, For batch normalization, This is the Sigmoid activation function.
3. A whole-brain segmentation method according to claim 1 or 2, characterized in that, The Transformer encoder includes at least: The Patch Embedding layer is used to segment the input standard image data into multiple patches and embed each patch into a feature vector through linear projection. Multiple Transformer modules, including two sequentially connected SwinTransformer modules, capture long-distance dependency features through window attention mechanism and shift window attention mechanism; A Patch Merging layer is interspersed between the Transformer modules. The Patch Merging layer is used to downsample the output of the Transformer module located at its input and transmit the sampled result to the Transformer module located at its output.
4. A whole-brain segmentation method according to claim 1 or 2, characterized in that, The cross-domain fusion module includes at least: A cross-domain basic fusion module is used to receive the local features output by the 3D convolutional residual block and the global semantic features output by the corresponding Transformer module, and perform preliminary fusion through the SwinTransformer module with self-attention mechanism to obtain the first feature data. A correlation enhancement module is used to obtain second feature data by multiplying the global semantic features and local features element-wise. The channel attention module includes at least a pooling layer for performing global average pooling on the global semantic features, a fully connected layer connected to the pooling layer for mapping the pooling result to obtain a first channel weight parameter, and a fusion layer connected to the fully connected layer, wherein the fusion layer is used to fuse the first channel weight parameter and the global semantic features by element-wise multiplication to obtain third feature data. The feature fusion module is used to process the concatenation result of the first feature data, the second feature data, and the third feature data through three-dimensional convolution processing, batch normalization processing, and activation function to introduce nonlinear processing, and establish a residual connection between the processed result and the concatenation result to obtain the fused feature.
5. A whole-brain segmentation method according to claim 4, characterized in that, The initial fusion of the first feature data using the Swin Transformer module with a self-attention mechanism specifically includes the following steps: The local features output by the three-dimensional convolutional residual block are subjected to global average pooling to obtain the fourth feature data; The global semantic features output by the Transformer module are adjusted in dimension to obtain the fifth feature data with the same dimension as the fourth feature data, and global average pooling is used to obtain the sixth feature data. The fourth feature data is concatenated with the fifth feature data, and the concatenation result is used to extract features through the Swin Transformer module to obtain the sixth feature data. The fourth feature data is concatenated with the sixth feature data, and the concatenation result is used to extract features through the Swin Transformer module to obtain the seventh feature data. The sixth feature data and the seventh feature data are concatenated, and the concatenation result is channel-fused through a 1×1 convolutional layer to obtain the first feature data.
6. A whole-brain segmentation method according to any one of claims 1-2 or 5, characterized in that, The decoder includes at least a DecoderBlock module that is sequentially connected and corresponds one-to-one with the three-dimensional convolutional residual block; The DecoderBlock module establishes a skip connection with the feature enhancement module at the corresponding level, and the input end of the DecoderBlock module at the beginning is connected to the output end of the Transformer module at the end. Any of the DecoderBlock modules includes at least an upsampling layer for restoring resolution, a second convolutional layer connected to the output of the upsampling layer for feature extraction of the upsampled features, a second normalization layer connected to the output of the second convolutional layer for batch normalization, a second activation function layer connected to the output of the second normalization layer for non-linear activation, and a concatenation layer connected to the output of the second activation function layer, wherein the concatenation layer is used to concatenate the data output by the second activation function layer with the data input by the skip connection.
7. A system employing a whole-brain segmentation method as described in any one of claims 1-6, characterized in that, At least including: A data preprocessing module is used to acquire the CT image data and preprocess the CT image data to obtain standard image data. A whole-brain segmentation model construction module is used to construct the whole-brain segmentation model, train the whole-brain segmentation model based on sample labeled data to obtain the optimal weight parameters, and load the optimal weight parameters into the whole-brain segmentation model. The segmentation module, through the trained whole-brain segmentation model, segments the standard image data to obtain a whole-brain segmentation result including a segmentation map of at least one brain tissue, such as gray matter, white matter, cerebrospinal fluid, or brainstem.
8. An apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the whole-brain segmentation method as described in any one of claims 1-6.
9. A medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the whole-brain segmentation method as described in any one of claims 1-6.