Brain colloid nodule region segmentation method and electronic equipment
By using a segmentation model with encoder and decoder branches, combined with multi-scale convolutional fusion of self-attention and pseudo-label generation, the problems of high annotation cost and low accuracy in glioma segmentation are solved, and efficient glioma sub-region segmentation is achieved.
Patent Information
- Application Number
- CN202411129256.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-19
AI Technical Summary
Existing automated segmentation methods for gliomas suffer from high annotation costs and low segmentation accuracy. Traditional methods rely on manual feature representation, which is insufficient, while fully supervised methods based on pixel-level labels are too costly.
A segmentation model with encoder and decoder branches is adopted, combined with multi-scale convolutional fusion self-attention module and pseudo-label generation. Through multi-stage feature fusion and loss function optimization, accurate segmentation of glioma sub-regions is achieved.
It reduces annotation costs, improves the accuracy and efficiency of glioma segmentation, and can accurately determine the location and size of tumors in different MRI image sequences.
Smart Images

Figure CN118982533A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the fields of artificial intelligence, medical image processing technology, and medical imaging technology, and specifically to a brain glioma sub-region segmentation method and electronic equipment. Background Art
[0002] Gliomas have the characteristics of malignant growth and invasion, and are prone to develop into highly malignant tumors. They are relatively difficult to treat and have a relatively poor prognosis. This is affected by factors such as unclear early symptoms, complex imaging manifestations, difficulty in obtaining tissue samples, and obvious tumor heterogeneity.
[0003] Traditional automated glioma segmentation methods based on magnetic resonance imaging (MRI) are usually based on manual features such as color and texture for segmentation. Most of the currently popular automated glioma segmentation methods based on neural networks rely on high-precision pixel-level labels to train the model. However, manual features are not expressive enough, and the fully supervised methods that rely on pixel-level labels require too high annotating costs. In summary, the automated glioma segmentation methods of the above two methods have technical problems of high annotating costs and low segmentation accuracy. Summary of the invention
[0004] In view of the above problems, the present disclosure provides a method for segmenting brain glioma sub-regions and an electronic device.
[0005] According to a first aspect of the present disclosure, a method for sub-region segmentation of a brain glioma is provided, comprising: preprocessing original nuclear magnetic resonance image data of a brain region to obtain preprocessed nuclear magnetic resonance image data; inputting the nuclear magnetic resonance image data into an encoder branch of a segmentation model, and outputting encoding features of multiple scales; inputting the encoding features of multiple scales into a decoder branch of the segmentation model, determining the segmentation results of each of the multiple scales, and outputting regional segmentation results for the nuclear magnetic resonance image data based on the segmentation results of each of the multiple scales; wherein the regional segmentation result includes multiple channels, and the regional segmentation result of each channel represents the distribution of a sub-region of the brain glioma.
[0006] According to an embodiment of the present disclosure, a decoder branch includes N feature decoding modules and an N-layer pyramid structure, the N-layer pyramid structure corresponds to N scales, and N is a positive integer greater than or equal to 3; determining the segmentation results for each of the multiple scales includes: inputting the encoding features of each scale into the feature decoding module corresponding to each scale, and outputting the first decoding features of each scale; using the N-layer pyramid structure, fusing the first decoding features of two adjacent scales to obtain (N-1) second decoding features; and obtaining the segmentation results for each of the N scales based on the (N-1) second decoding features and the first decoding features of the highest scale.
[0007] According to an embodiment of the present disclosure, the first decoding features of two adjacent scales are fused to obtain (N-1) second decoding features, including: splicing the first decoding features of two adjacent scales in the channel dimension to obtain a first spliced feature; and performing a linear transformation on the first spliced feature using a single linear layer to obtain a second decoding feature.
[0008] According to an embodiment of the present disclosure, the regional segmentation result for the nuclear magnetic resonance image data is output according to the segmentation results of each of the multiple scales, including: adding the segmentation results of each of the multiple scales according to the calculation weights of each of the multiple scales to obtain the regional segmentation result.
[0009] According to an embodiment of the present disclosure, the segmentation model also includes a classifier branch, which includes a multi-stage feature fusion module, a multi-layer perceptron and a pseudo-label generation module; the method also includes: inputting the encoding features of multiple scales into the multi-stage feature fusion module, and outputting the classification features fused at multiple scales; inputting the classification features into the multi-layer perceptron, enhancing and predicting the classification features, and outputting the classification results; inputting the classification features and the classification results into the pseudo-label generation module to generate pseudo-label information for the nuclear magnetic resonance image data, wherein the data pair consisting of the pseudo-label information, the nuclear magnetic resonance image data and the regional segmentation result is used to optimize the decoder branch.
[0010] According to an embodiment of the present disclosure, the method also includes: the loss function of the segmentation model includes cross entropy loss, Dice loss and Focal loss, wherein the label information in the cross entropy loss is pseudo label information generated by the classifier branch according to the training data.
[0011] According to an embodiment of the present disclosure, an encoder branch includes N identical stacked encoding stages and a linear layer between two adjacent encoding stages, each encoding stage includes multiple identical multi-scale convolution fusion self-attention modules; the nuclear magnetic resonance image data is input into the encoder branch of the segmentation model, and encoding features of multiple scales are output, including: repeating the following operations until the highest scale encoding feature is output in the Nth encoding stage: the encoding feature output from the n-1th encoding stage is input into the linear layer to obtain the encoding feature after the scale is increased; n is an integer greater than or equal to 3 and less than N; the encoding feature after the scale is increased is input into the nth encoding stage, and the encoding feature after the scale is increased is obtained through multiple convolution operations of multiple stacked multi-scale convolution fusion self-attention modules; wherein, the first encoding stage is used to perform multiple convolutions on the nuclear magnetic resonance image data through multiple stacked multi-scale convolution fusion self-attention modules to obtain the lowest scale encoding feature.
[0012] According to an embodiment of the present disclosure, a multi-scale convolution fusion self-attention module includes a convolution module, a first normalization layer, a self-attention module, a second normalization layer and a multi-layer perceptron connected in sequence; the convolution operation of the multi-scale convolution fusion self-attention module includes: using multiple convolution branches of the convolution module to convolve the input features respectively, to obtain convolution features output by multiple convolution branches respectively; and fusing multiple convolution features to output convolution features, wherein the multiple convolution branches have convolution kernels of different sizes; inputting the convolution features processed by the first normalization layer into the self-attention module, and outputting the attention features; inputting the attention features processed by the second normalization layer into the multi-layer perceptron, and outputting the convolution features, wherein the convolution features are the output features of the multi-scale convolution fusion self-attention module.
[0013] According to an embodiment of the present disclosure, the original nuclear magnetic resonance image data of the brain region are preprocessed to obtain preprocessed nuclear magnetic resonance image data, including: aligning the four groups of nuclear magnetic resonance image data sequence data of T1, T2, T1CE, and FLAIR obtained by scanning the brain region; sampling the four groups of aligned nuclear magnetic resonance image data sequence data at preset intervals to obtain slice image sequence data; using a segmentation algorithm to segment the slice image sequence data of the brain tissue region from the slice image sequence data; and performing outlier elimination and standardization processing on the slice image sequence data of the brain tissue region to obtain nuclear magnetic resonance image data.
[0014] A second aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, and when the one or more processors execute the one or more computer programs, the steps of the above-mentioned brain glioma sub-region segmentation method are implemented.
[0015] The third aspect of the present disclosure provides a brain glioma sub-region segmentation device, including: a preprocessing module, used to preprocess the original nuclear magnetic resonance image data of the brain region to obtain preprocessed nuclear magnetic resonance image data; an encoding module, used to input the nuclear magnetic resonance image data into the encoder branch of the segmentation model, and output encoding features of multiple scales; a decoding module, used to input the encoding features of multiple scales into the decoder branch of the segmentation model, determine the segmentation results of each of the multiple scales, and output the regional segmentation results for the nuclear magnetic resonance image data according to the segmentation results of each of the multiple scales; wherein the regional segmentation result includes multiple channels, and the regional segmentation result of each channel represents a sub-region distribution of the brain glioma.
[0016] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instruction stored thereon, which, when executed by a processor, implements the steps of the above-mentioned brain glioma sub-region segmentation method.
[0017] The fifth aspect of the present disclosure also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the above-mentioned brain glioma sub-region segmentation method.
[0018] The embodiment of the present disclosure achieves accurate regional segmentation of brain glioma by inputting the preprocessed nuclear magnetic image data into the encoder branch of the segmentation model to output the coding features of multiple scales; inputting the coding features of multiple scales into the decoder branch of the segmentation model to determine the segmentation results of each of the multiple scales, and outputting the regional segmentation results for the nuclear magnetic image data based on the segmentation results of each of the multiple scales. In the embodiment of the present disclosure, since the coding features of each scale are outputted separately in the decoder branch, the decoder branch can decode each scale to obtain the segmentation result of each scale. Then, the regional segmentation result obtained by fusing the segmentation results of multiple scales can fuse a wider feature dimension and the feature expression within each feature dimension, thereby achieving the technical effect of improving the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0020] Figure 1 The application scenario of the brain glioma sub-region segmentation method according to the embodiment of the present disclosure is schematically shown;
[0021] Figure 2 The flowchart of the method for segmenting brain glioma sub-regions according to an embodiment of the present disclosure is schematically shown;
[0022] Figure 3 The schematic diagram of the structure of the segmentation model according to the first embodiment of the present disclosure is shown;
[0023] Figure 4 The structural diagram of the segmentation model according to the second embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0024] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.
[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.
[0026] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.
[0027] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).
[0028] An embodiment of the present disclosure provides a method for sub-region segmentation of a brain glioma, including: preprocessing the original nuclear magnetic resonance image data of a brain region to obtain preprocessed nuclear magnetic resonance image data; inputting the nuclear magnetic resonance image data into an encoder branch of a segmentation model, and outputting encoding features of multiple scales; inputting the encoding features of multiple scales into a decoder branch of the segmentation model, determining the segmentation results of each of the multiple scales, and outputting a regional segmentation result for the nuclear magnetic resonance image data based on the segmentation results of each of the multiple scales; wherein the regional segmentation result includes multiple channels, and the regional segmentation result of each channel represents the distribution of a sub-region of the brain glioma.
[0029] It should be noted that the brain glioma sub-region segmentation method, device, and electronic device involved in the embodiments of the present disclosure are not intended for the diagnosis or treatment of cancer, but only for image analysis and image segmentation of nuclear magnetic resonance image data. The present disclosure is only used for non-medical treatment purposes, including but not limited to medical research and medical auxiliary diagnosis, and is strictly prohibited from being used for other medical treatment-related purposes. The use of the present disclosure is strictly restricted, and any attempt to violate this restriction will be deemed an infringement.
[0030] Figure 1 The application scenario of the brain glioma sub-region segmentation method according to the embodiment of the present disclosure is schematically shown.
[0031] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.
[0032] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).
[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.
[0034] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.
[0035] It should be noted that the brain glioma sub-region segmentation method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the brain glioma sub-region segmentation device provided in the embodiment of the present disclosure can generally be set in the server 105. The brain glioma sub-region segmentation method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the brain glioma sub-region segmentation device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.
[0036] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0037] The following will be based on Figure 1 The scene described by Figure 2~Figure 4 The brain glioma sub-region segmentation method of the disclosed embodiment is described in detail.
[0038] Figure 2 The flowchart of the method for segmenting brain glioma sub-regions according to an embodiment of the present disclosure is schematically shown.
[0039] like Figure 2 As shown, the method 200 includes operations S210 to S230.
[0040] In operation S210, the original nuclear magnetic resonance image data of the brain region is preprocessed to obtain preprocessed nuclear magnetic resonance image data.
[0041] In operation S220, the nuclear magnetic resonance image data is input into an encoder branch of a segmentation model, and encoding features of multiple scales are output.
[0042] In operation S230, the encoded features of multiple scales are input into the decoder branch of the segmentation model, the segmentation results of each of the multiple scales are determined, and the regional segmentation results for the nuclear magnetic resonance image data are output based on the segmentation results of each of the multiple scales; wherein the regional segmentation results include multiple channels, and the regional segmentation results of each channel represent a sub-region distribution of the brain glioma.
[0043] According to an embodiment of the present disclosure, the raw magnetic image data may be a series of magnetic resonance imaging (MRI) images obtained by scanning a brain region. The preprocessing of the raw magnetic image data includes: data alignment, stripping of skull features in the brain region, outlier removal, standard processing, etc.
[0044] The preprocessed MRI image data only includes the features of the brain region so as to perform regional segmentation of gliomas in the brain region.
[0045] Brain glioma includes three sub-regions: edema region, enhancement region, and necrosis region. In the segmentation model, the above three sub-regions can be used as three categories to achieve segmentation of the three sub-regions.
[0046] According to an embodiment of the present disclosure, the segmentation model adopts an encoding-decoding structure, that is, the segmentation model includes an encoder branch and a decoder branch, which are used to encode image features and decode image features respectively. For example, the segmentation model can be an improved weakly supervised brain tumor subregion segmentation (IWS-BTSS) model.
[0047] After preprocessing, the encoder branch in the preprocessed nuclear magnetic image data output segmentation model can encode the nuclear magnetic image data at multiple scales to obtain encoding features of multiple scales. In an embodiment of the present disclosure, different scales can be understood as different feature dimensions. In a specific embodiment, the scales of the encoding features output by the encoder branch are all different.
[0048] According to an embodiment of the present disclosure, the decoder branch can perform decoding operations on the coded features of different scales to obtain the segmentation results of the corresponding scales. In addition, in the decoder branch, after determining the segmentation results of each of the multiple scales, the segmentation results of the multiple scales can also be fused to obtain the final region segmentation result.
[0049] For example, the regional segmentation result has multiple channels, each channel has a value between 0 and 1, and each channel indicates the distribution of edema, enhancement, and necrosis areas of brain glioma. In a specific embodiment, the regional segmentation result can be superimposed with the original MRI data and visualized to the doctor, thereby assisting the doctor in locating the location of the brain glioma and judging the development of the tumor.
[0050] The embodiment of the present disclosure achieves accurate regional segmentation of brain glioma by inputting the preprocessed nuclear magnetic image data into the encoder branch of the segmentation model to output the coding features of multiple scales; inputting the coding features of multiple scales into the decoder branch of the segmentation model to determine the segmentation results of each of the multiple scales, and outputting the regional segmentation results for the nuclear magnetic image data based on the segmentation results of each of the multiple scales. In the embodiment of the present disclosure, since the coding features of each scale are outputted separately in the decoder branch, the decoder branch can decode each scale to obtain the segmentation result of each scale. Then, the regional segmentation result obtained by fusing the segmentation results of multiple scales can fuse a wider feature dimension and the feature expression within each feature dimension, thereby achieving the technical effect of improving the segmentation accuracy.
[0051] According to an embodiment of the present disclosure, for the above-mentioned operation S210, the original nuclear magnetic resonance image data of the brain area are preprocessed to obtain preprocessed nuclear magnetic resonance image data, including: aligning the four groups of nuclear magnetic resonance image data sequence data of T1, T2, T1CE, and FLAIR obtained by scanning the brain area; sampling the four groups of aligned nuclear magnetic resonance image data sequence data at preset intervals to obtain slice image sequence data; using a segmentation algorithm to segment the slice image sequence data of the brain tissue area from the slice image sequence data; and performing outlier removal and standardization processing on the slice image sequence data of the brain tissue area to obtain nuclear magnetic resonance image data.
[0052] Since the tumor site behaves differently under different sequences, only one sequence cannot accurately determine the location, size, and other information of the tumor. Therefore, the embodiment of the present disclosure uses four sets of nuclear magnetic resonance image data sequences to accurately determine the site information of brain glioma.
[0053] The four sets of MRI image data sequences, T1, T2, T1CE and FLAIR, are all in DICOM format.
[0054] Specifically, the corresponding positions in the four groups of nuclear magnetic resonance image data sequences of T1, T2, T1CE, and FLAIR can be aligned to obtain a cross-sectional image. After that, the cross-sectional image is sampled at a certain time interval to obtain a slice image sequence data. Use segmentation algorithms, such as the BET (Brain Extraction Tool) algorithm, to perform skull stripping and segment the image sequence data that only retains the brain tissue area. .
[0055] Slice image sequence data of brain tissue area The percentile method is used to eliminate outliers, and the 1% with the largest intensity and the 1% with the smallest intensity are eliminated to obtain the slice image. . For slice data Use the Z-score method for standardization, that is, intensity standardization. Specifically, the standardization process can calculate the slice image The mean and standard deviation of the slice image The value range is scaled to between -1 and 1 to obtain the final MRI image data used. , for example, see the following formulas (1) to (3):
[0056] (1)
[0057] (2)
[0058] (3)
[0059] Among them, μ and σ are the mean and standard deviation respectively, and N is the number of pixels in the slice data.
[0060] According to an embodiment of the present disclosure, for the above operation S220, the encoder module includes stacked N identical encoding stages, and a linear layer between two adjacent encoding stages, each encoding stage includes multiple identical multi-scale convolutional fusion self-attention modules; the nuclear magnetic image data is input into the encoder branch of the segmentation model, and encoding features of multiple scales are output, including:
[0061] Repeat the following operations until the highest-scale coding features are output in the Nth coding stage: input the coding features output in the n-1th coding stage into the linear layer to obtain the scaled-up coding features, where n is an integer greater than or equal to 3 and less than N; input the scaled-up coding features into the nth coding stage, and obtain the scaled-up coding features through multiple convolution operations of stacked multiple multi-scale convolutions and fusion self-attention modules; wherein the first coding stage is used to perform multiple convolutions on the nuclear magnetic resonance image data through multiple stacked multi-scale convolutions and fusion self-attention modules to obtain the lowest-scale coding features.
[0062] According to an embodiment of the present disclosure, the encoder branch includes N stacked identical encoding stages and a linear layer between two adjacent encoding stages, each encoding stage is used to encode features of a scale, and the linear layer between two adjacent encoding stages is used for feature dimension transformation.
[0063] In a specific embodiment, the linear layer is used to increase the feature dimension, so that from the first encoding stage to the Nth encoding stage, the feature dimension processed in each encoding stage gradually increases. For example, when N is 4, the feature dimension of the preprocessed nuclear magnetic resonance image data is 128, and the feature dimensions used in the first encoding stage to the fourth encoding stage are 128, 256, 512, and 1024, respectively.
[0064] For each encoding stage, each encoding stage includes multiple identical multi-scale convolution fusion self-attention modules, and multiple multi-scale convolution fusion self-attention module pairs are stacked to form an encoding stage. In each encoding stage, each multi-scale convolution fusion self-attention module performs a convolution operation.
[0065] In a specific embodiment, 12 multi-scale convolution fusion self-attention modules with the same structure can be stacked to obtain an encoder branch. Every three multi-scale convolution fusion self-attention modules are combined into one stage, that is, the encoder branch has four stages in total. The encoding features of the four scales output by the encoder branch are respectively .
[0066] In the embodiments of the present disclosure, feature extraction is performed on MRI data of multiple modalities through encoder branches to obtain semantic representations of multiple scales from coarse to fine. During the encoding process, the feature extraction capability of the encoder can be guaranteed while reducing the model complexity as much as possible by gradually increasing the feature dimension.
[0067] According to an embodiment of the present disclosure, a multi-scale convolution fusion self-attention module includes a convolution module, a first normalization layer, a self-attention module, a second normalization layer and a multi-layer perceptron connected in sequence; the convolution operation of the multi-scale convolution fusion self-attention module includes: using multiple convolution branches of the convolution module to convolve the input features respectively, to obtain convolution features output by multiple convolution branches respectively; and fusing multiple convolution features to output convolution features, wherein the multiple convolution branches have convolution kernels of different sizes; inputting the convolution features processed by the first normalization layer into the self-attention module, and outputting the attention features; inputting the attention features processed by the second normalization layer into the multi-layer perceptron, and outputting the convolution features, wherein the convolution features are the output features of the multi-scale convolution fusion self-attention module.
[0068] The convolution module contains three convolution branches with convolution kernel sizes of 3, 5, and 7 respectively. Each branch is composed of a convolution layer, a batch normalization layer, and a ReLU activation layer connected in sequence. Convolution branches with different convolution kernel sizes can extract features of different scales through different receptive fields. The three sets of features of different scales are spliced and fused in the channel dimension, and then fused through the linear layer to obtain the output of the convolution module, that is, the convolution feature.
[0069] The convolution features output by the convolution module can be used as the input of the self-attention module after being processed by the first normalization layer.
[0070] The self-attention module maps the input features into three sets of features: Q (Query), K (Key), and V (Value) through three sets of linear layers. Features Q and K are multiplied and processed by softmax to obtain the attention weights. After the weights are scaled by the feature dimension, the feature V is weighted to obtain the output features of the self-attention module, that is, the attention features.
[0071] The attention features output by the attention module are processed by the second normalization layer and used as the input of the multi-layer perceptron, so as to enhance the attention features by using the multi-layer perceptron to obtain the output features of the multi-scale convolution fusion attention module. The first normalization layer and the second normalization layer can be the same.
[0072] In summary, the processing flow of the multi-scale convolution fusion attention module can be described as:
[0073] (4)
[0074] (5)
[0075] (6)
[0076] (7)
[0077] in, is the input feature of the current multi-scale convolution fusion attention module, is the convolution feature output by the convolution module (CNN), is the attention feature output by the attention module, is the output feature of the current multi-scale convolution fusion attention module, It includes convolutional layer, batch normalization layer, ReLU activation layer, MLP is a multi-layer perceptron. The first and second normalization layers are the same.
[0078] According to an embodiment of the present disclosure, the decoder branch includes N feature decoding modules and an N-layer pyramid structure, the N-layer pyramid structure corresponds to N scales, and N is a positive integer greater than or equal to 3.
[0079] With respect to determining the segmentation results for each of the multiple scales in operation S230, the operation includes: inputting the coded features of each scale into a feature decoding module corresponding to each scale, and outputting a first decoding feature of each scale; using an N-layer pyramid structure, fusing the first decoding features of two adjacent scales to obtain (N-1) second decoding features; and obtaining the segmentation results for each of the N scales based on the (N-1) second decoding features and the first decoding feature of the highest scale.
[0080] According to an embodiment of the present disclosure, the N feature decoding modules in the decoder branch correspond one-to-one to the N stages in the encoder branch, and the N-layer pyramid structure corresponds one-to-one. That is, the encoding features of a certain scale output by each encoding stage in the encoder branch are input to the feature decoding module of the corresponding scale in the decoder branch. It can be understood that the decoder branch includes N feature decoding modules.
[0081] The (N-1)-layer pyramid structure in the N-layer pyramid structure can fuse decoding features of at least two scales to improve segmentation accuracy, and the last layer of the pyramid structure does not process the first decoding feature of the highest scale.
[0082] For example, the first decoding features of two adjacent scales are fused, and the fused second decoding features are used as the region segmentation benchmark of the current scale. For an N-layer pyramid structure, the first decoding features of two adjacent scales can be fused in a predetermined order to obtain N-1 second decoding features. For example, the first decoding features of the current scale are fused with the first decoding features of the adjacent higher scale, thereby improving the region segmentation accuracy of the current scale through more feature dimensions.
[0083] According to the embodiments of the present disclosure, the (N-1) second decoding features and the first decoding features of the highest scale can be respectively input into the softmax activation layer to obtain the segmentation results of each of the N scales, that is, the probability that each pixel belongs to each sub-region of the brain glioma. It can be understood that the output of the N-layer pyramid structure is the (N-1) second decoding features and the first decoding features of the highest scale.
[0084] According to the embodiments of the present disclosure, the feature decoding module is mainly composed of a convolution layer, an upsampling layer, and a linear layer. Specifically, for the feature decoding module processing the same scale, the convolution layer is used to further aggregate the output features of the encoder branch at the current stage, and the output features of the encoder branch are globally pooled to obtain global representative features. After the aggregated features are added to the representative features, they enter the upsampling layer to improve the spatial resolution, and the linear layer unifies the channel dimensions into one scale (feature dimension) to facilitate the subsequent fusion of the first decoding features of different scales.
[0085] For example, the feature decoding modules of the four scales all generate the first decoding features with a feature dimension of 128 through a linear layer.
[0086] The processing process of the feature decoding module can be expressed as:
[0087] (8)
[0088] in, is the encoding feature output by the i-th stage of the encoder branch, It is the first decoded feature output by the feature decoding module before the feature pyramid structure is processed. The convolution operation Convs(⋅) performed by the convolution layer includes convolution, batch normalization, and ReLU activation. GAP is global pooling, UpSampling is the upsampling operation of the upsampling layer, and Linear is the operation of the linear layer.
[0089] According to an embodiment of the present disclosure, the first decoding features of two adjacent scales are fused to obtain (N-1) second decoding features, including: splicing the first decoding features of two adjacent scales in the channel dimension to obtain a first spliced feature; and performing a linear transformation on the first spliced feature using a single linear layer to obtain a second decoding feature.
[0090] For each first decoded feature output by each feature decoding module, copy a set of first decoded features output by the next feature decoding module, concatenate them with the first decoded features output by the current feature decoding module in the channel dimension, and use a single linear layer to fuse them in the channel dimension, so that the deep features (first decoded features of higher scales) are fused with the shallow features (first decoded features of the current scale).
[0091] Figure 3 The schematic diagram shows the structure of the segmentation model according to the first embodiment of the present disclosure.
[0092] like Figure 3 As shown in the figure, 12 multi-scale convolutional fusion self-attention modules with exactly the same structure can be stacked to obtain the encoder branch. Every three multi-scale convolutional fusion self-attention modules are combined into one encoding stage, and there are three linear layers between the four encoding stages.
[0093] After the encoded features of each scale are input into the feature decoding module corresponding to each scale, the output feature dimension is the first decoding feature of 128 The N-layer pyramid structure copies a set of decoding features output by the next feature decoding module for concatenation with the decoding features output by the previous feature decoding module in the channel dimension, and uses a single linear layer to fuse them in the channel dimension, thereby fusing the deep features with the line layer features to obtain four sets of output decoding features, namely The four sets of decoded features are all passed through the softmax activation layer to obtain the segmentation results of the four sets of brain glioma sub-regions. According to the segmentation results of each of the four scales, the regional segmentation results for the nuclear magnetic resonance image data are output .
[0094] According to an embodiment of the present disclosure, the regional segmentation result for the nuclear magnetic resonance image data is output according to the segmentation results of each of the multiple scales, including: adding the segmentation results of each of the multiple scales according to the calculation weights of each of the multiple scales to obtain the regional segmentation result.
[0095] For example, the segmentation results of the four groups of glioma sub-regions are weighted by equal weights. Add together to get the final region segmentation result map .
[0096] In addition, the annotation cost of the MRI brain glioma segmentation method based on neural network in the related art that only relies on image-level category labels is too high, and the segmentation accuracy depends on the quality of the labels. Therefore, the embodiment of the present disclosure further improves the segmentation model and adds a classifier branch to the segmentation model to automatically construct pseudo-label information for optimizing the model.
[0097] According to an embodiment of the present disclosure, the segmentation model also includes a classifier branch, and the classifier branch includes a multi-stage feature fusion module, a multi-layer perceptron, and a pseudo-label generation module. The method also includes: inputting the encoding features of multiple scales into the multi-stage feature fusion module, and outputting the classification features of the fusion of multiple scales; inputting the classification features into the multi-layer perceptron, enhancing and predicting the classification features, and outputting the classification results; inputting the classification features and the classification results into the pseudo-label generation module to generate pseudo-label information of the nuclear magnetic image data, wherein the data pair consisting of the pseudo-label information, the nuclear magnetic image data, and the regional segmentation result is used to optimize the decoder branch.
[0098] The multi-stage feature fusion module receives the coded features of multiple scales output by the encoder branch, for example, receives the coded features of 128, 256, 512, and 1024 output by the encoder branch of the four encoding stages .
[0099] The multi-stage feature fusion module is used to fuse the coding features of multiple scales and process them to obtain classification features. The classification features are further enhanced by the multi-layer perceptron to form enhanced classification features, and finally processed by the sigmoid activation layer to obtain the classification results, that is, the classification probability of each sub-region of the brain glioma. .
[0100] In addition, the pseudo-label generation module included in the classifier branch can generate pixel-level pseudo-label information to guide the training process of the decoder branch. The pseudo-label information is a rough segmentation result of the brain glioma sub-region.
[0101] In an embodiment of the present disclosure, the multi-stage feature fusion module is composed of a deep separation convolution layer, a global pooling layer, and a linear layer.
[0102] Figure 4 The structural diagram of the segmentation model according to the second embodiment of the present disclosure is schematically shown. Figure 4 The encoder branch and decoder branch refer to Figure 3 , I will not go into details here.
[0103] like Figure 4 As shown in the figure, the multi-stage feature fusion module receives the four encoded features output from the four encoding stages of the encoder branch, unifies the feature dimensions through the linear layer, and then splices them in the channel dimension and inputs them into the deep separation convolution layer. The deep separation convolution performs convolution enhancement on the features from the four encoding stages. After processing, the features are aggregated by the global pooling layer, and the features are fused in the channel dimension through the linear layer, and finally the output features of the multi-stage feature fusion module are obtained. The processing process of the multi-stage feature fusion module can be expressed as:
[0104] (9)
[0105] in, is the encoding feature generated by the i-th encoding stage of the encoder branch, That is, the output features of the multi-stage feature fusion module. represents the depth separation convolutional layer, Represents the global pooling layer.
[0106] The pseudo-label generation module does not contain learnable parameters. The pseudo-label generation module mainly involves the classification features entering the global pooling layer in the multi-stage feature fusion module. , and the weight parameters of the last linear layer of the classifier branch According to the classification result currently predicted by the classifier branch, that is, the category, the weight parameter of the corresponding position is taken
[0107] For classification features The corresponding channels are weighted and accumulated to obtain the class activation map corresponding to each classification category of a single sample.
[0108] For example, when the predicted classification result is an enhanced area, the classification feature is weighted according to the weight parameters of each pixel point at the enhanced area position. The corresponding enhanced area channels are weighted and accumulated, thereby obtaining the class activation map corresponding to the enhanced area category.
[0109] In the embodiment of the present disclosure, a threshold of 0.2 can be used to divide the class activation map into background and tumor areas (edema area, enhancement area, necrosis area), thereby obtaining pseudo-label information corresponding to each predicted category of a single sample. This pixel-level pseudo-label will be used to supervise the training of the decoder branch. This process can be described as:
[0110] (10)
[0111] (11)
[0112] in, is the class activation map of the c-th classification category, is the pseudo label information of the c-th classification category.
[0113] According to an embodiment of the present disclosure, the loss function of the segmentation model includes cross entropy loss, Dice loss and Focal loss, wherein the label information in the cross entropy loss is pseudo label information generated by the classifier branch according to the training data.
[0114] In the embodiments of the present disclosure, cross entropy, Dice loss and Focal loss are used, and the pseudo-label information is utilized to train the segmentation model of the present disclosure, such as the IWS-BTSS model, in a weakly supervised manner.
[0115] Cross entropy as a basic constraint and main constraint for classification problems, including the cross entropy loss that constrains the classifier branches and the cross entropy loss for the constrained decoder branch , the two cross entropy losses can ensure the convergence of the segmentation model. Dice loss is a commonly used loss function for segmentation tasks and is used to constrain the decoder branch. Focal loss is suitable for the scene of imbalanced categories such as tumor sub-region segmentation and can be used as an auxiliary loss to improve the effect of the decoder branch. Therefore, the loss function of the segmentation model is a combination of four losses, namely:
[0116] (12)
[0117] in, is the weight coefficient of each loss item, which is used to adjust the relative importance of each loss item. and Set to 2, and Set to 1.
[0118] Due to the insufficient expression ability of manual features, the labeling cost required by the fully supervised method that relies on pixel-level labels is too high. In order to reduce the labeling cost, the weakly supervised segmentation method of glioma based on point labeling and stroke labeling can save a lot of labeling time. However, in comparison, the above-mentioned glioma segmentation method that only relies on image-level category labels does not require pixel-level labeling at all and cannot be applied to the training of glioma segmentation models based on pixel-level labels. The embodiment of the present disclosure automatically generates pseudo-label information through the classifier branch of the segmentation model, which reduces the labeling cost and labeling time, and improves the efficiency of glioma model training.
[0119] According to an embodiment of the present disclosure, a brain glioma sub-region segmentation device includes: a preprocessing module, which is used to preprocess the original nuclear magnetic resonance image data of the brain region to obtain preprocessed nuclear magnetic resonance image data; an encoding module, which is used to input the nuclear magnetic resonance image data into the encoder branch of the segmentation model and output encoding features of multiple scales; a decoding module, which is used to input the encoding features of multiple scales into the decoder branch of the segmentation model, determine the segmentation results of each of the multiple scales, and output the regional segmentation results for the nuclear magnetic resonance image data based on the segmentation results of each of the multiple scales; wherein the regional segmentation result includes multiple channels, and the regional segmentation result of each channel represents a sub-region distribution of the brain glioma.
[0120] It should be noted that the device part in the embodiments of the present disclosure corresponds to the method part in the embodiments of the present disclosure. The description of the device part specifically refers to the method part and will not be repeated here.
[0121] According to an embodiment of the present disclosure, the electronic device includes one or more processors; a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the above-mentioned brain glioma sub-region segmentation.
[0122] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.
[0123] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.
[0124] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the above method provided by the embodiment of the present disclosure.
[0125] When the computer program is executed by the processor, the above functions defined in the system / device of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.
[0126] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).
[0127] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0128] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.
[0129] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present disclosure. It should be understood that the above description is only a specific embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.
Claims
1. A method for segmenting brain glioma sub-regions, characterized in that: The method comprises: Preprocessing the original nuclear magnetic resonance image data of the brain area to obtain preprocessed nuclear magnetic resonance image data; Inputting the nuclear magnetic resonance image data into an encoder branch of a segmentation model, and outputting encoding features at multiple scales; Inputting the encoded features of the plurality of scales into a decoder branch of the segmentation model, determining the segmentation results of the plurality of scales, and outputting the regional segmentation results for the nuclear magnetic resonance image data according to the segmentation results of the plurality of scales; The region segmentation result includes multiple channels, and the region segmentation result of each channel represents a sub-region distribution of the brain glioma.
2. The method according to claim 1, characterized in that The decoder branch includes N feature decoding modules and N-layer pyramid structures, the N-layer pyramid structures correspond to N scales, and N is a positive integer greater than or equal to 3; The determining of the segmentation results for each of the multiple scales includes: Inputting the encoded features of each scale into a feature decoding module corresponding to each scale, and outputting a first decoded feature of each scale; Using the N-layer pyramid structure, the first decoding features of two adjacent scales are fused to obtain (N-1) second decoding features; According to (N-1) second decoding features and the first decoding features corresponding to the highest scale, segmentation results of the N scales are obtained.
3. The method according to claim 2, characterized in that The fusing of the first decoding features of two adjacent scales to obtain (N-1) second decoding features includes: Concatenate the first decoding features of two adjacent scales in the channel dimension to obtain a first concatenated feature; The first concatenated features are linearly transformed using a single linear layer to obtain the second decoding features.
4. The method according to claim 1 or 2, characterized in that: Outputting the regional segmentation result for the nuclear magnetic resonance image data according to the segmentation results of each of the multiple scales includes: According to the calculation weights of the multiple scales, the segmentation results of the multiple scales are added together to obtain the region segmentation result.
5. The method according to claim 1, characterized in that The segmentation model also includes a classifier branch, and the classifier branch includes a multi-stage feature fusion module, a multi-layer perceptron and a pseudo-label generation module; the method also includes: Inputting the encoding features of multiple scales into the multi-stage feature fusion module, and outputting the classification features fused with the multiple scales; Inputting the classification features into the multilayer perceptron, enhancing and predicting the classification features, and outputting classification results; The classification feature and the classification result are input into the pseudo-label generation module to generate pseudo-label information of the nuclear magnetic resonance image data, wherein a data pair consisting of the pseudo-label information, the nuclear magnetic resonance image data and the region segmentation result is used to optimize the decoder branch.
6. The method according to claim 5, characterized in that The method further comprises: The loss function of the segmentation model includes cross entropy loss, Dice loss and Focal loss, wherein the label information in the cross entropy loss is pseudo label information generated by the classifier branch according to the training data.
7. The method according to claim 1, characterized in that The encoder branch includes stacked N identical encoding stages and a linear layer between two adjacent encoding stages, each encoding stage includes multiple identical multi-scale convolutional fusion self-attention modules; The step of inputting the nuclear magnetic resonance image data into the encoder branch of the segmentation model and outputting encoding features of multiple scales includes: Repeat the following operations until the highest-scale encoding feature is output at the Nth encoding stage: Inputting the encoding features outputted by the n-1th encoding stage into the linear layer to obtain the encoding features after the scale is increased; n is an integer greater than or equal to 3 and less than N; Inputting the scaled-up encoding features into the nth encoding stage, and obtaining the scaled-up encoding features by stacking multiple multi-scale convolutions and fusing multiple convolution operations of the self-attention module; Among them, the first encoding stage is used to perform multiple convolutions on the nuclear magnetic resonance image data through stacking multiple multi-scale convolution fusion self-attention modules to obtain the lowest scale encoding features.
8. The method according to claim 7, characterized in that The multi-scale convolution fusion self-attention module includes a convolution module, a first normalization layer, a self-attention module, a second normalization layer and a multi-layer perceptron connected in sequence; The convolution operation of the multi-scale convolution fusion self-attention module includes: Using the multiple convolution branches of the convolution module to convolve the input features respectively, to obtain the convolution features respectively output by the multiple convolution branches; and fusing the multiple convolution features to output the convolution features, wherein the multiple convolution branches have convolution kernels of different sizes; Inputting the convolutional features processed by the first normalization layer into the self-attention module, and outputting attention features; The attention feature processed by the second normalization layer is input into the multi-layer perceptron, and a convolution feature is output, wherein the convolution feature is the output feature of the multi-scale convolution fusion self-attention module.
9. The method according to claim 1, characterized in that: The preprocessing of the original nuclear magnetic resonance image data of the brain region to obtain the preprocessed nuclear magnetic resonance image data includes: Aligning four sets of magnetic resonance imaging data sequences, namely, T1, T2, T1CE, and FLAIR, obtained by scanning the brain region; Sampling the aligned four groups of nuclear magnetic resonance image data sequences at preset intervals to obtain slice image sequence data; Segmenting the slice image sequence data of the brain tissue area from the slice image sequence data using a segmentation algorithm; The slice image sequence data of the brain tissue area are respectively subjected to outlier elimination and standardization processing to obtain the nuclear magnetic resonance image data.
10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.
Citation Information
Cited By
Unsupervised lesion segmentation method based on large model and pseudo-label learning
CN119762774A
Nuclear power equipment defect detection method based on Mixed-FFM model
CN120259218A
A nuclear power plant defect detection method based on a mixed-FFM model
CN120259218B
