Self-supervised pre-training method and device for brain tumor based on attention symmetric autoencoding

Abstract

The present invention discloses a method, device, computer equipment and readable storage medium for self-supervised pre-training of brain tumors based on attention symmetric autoencoding, wherein the method includes: obtaining a first brain MRI data sample set and a second brain MRI data sample set, and performing pre-processing, wherein the first brain MRI data sample set includes unlabeled brain MRI data, and the second brain MRI data sample set includes labeled brain MRI data; dividing the unlabeled brain MRI data into multiple image blocks, and randomly using masks to cover a preset proportion of image blocks; reconstructing the masked image blocks through an attention symmetric autoencoder to generate a pre-trained model; fine-tuning the pre-trained model through labeled brain MRI data to generate a segmentation model; and segmenting the brain MRI data of the patient to be segmented through the segmentation model to determine the tumor area. The information of the importance and symmetry of the brain area can be effectively utilized to improve the accuracy of segmentation.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation, and in particular to a brain tumor self-supervised pre-training method, device, computer equipment and storage medium based on attention symmetric autoencoding. Background Art

[0002] In clinical practice, two-dimensional MRI slices are usually used to observe the brain for tumor assessment and diagnosis. The tumor area is also manually outlined on the two-dimensional MRI slices as a segmentation label. Since a three-dimensional MRI image contains hundreds of two-dimensional MRI slices, it is very time-consuming to outline the segmentation label, which results in a relatively small number of samples with voxel-level segmentation labels in existing MRI datasets.

[0003] At present, although self-supervised pre-training methods are used to pre-train the parameter weights of the deep neural network model, the pre-trained deep neural network model is then trained on the existing MRI image data and its tumor segmentation labels to achieve tumor image segmentation. Since the self-supervised pre-training method used provides better initialization parameter weights, the model can complete training even with a small number of samples.

[0004] However, the current self-supervised pre-training methods and deep neural network models do not fully consider the symmetry of the brain's anatomical structure, and ignore the fact that images of different brain regions contain different amounts of information and have different importance, resulting in low segmentation accuracy. Summary of the invention

[0005] Based on this, it is necessary to provide a brain tumor self-supervised pre-training method, device, computer equipment and storage medium based on attention symmetric autoencoding to address the above technical problems, so as to solve the problem of low segmentation accuracy of existing self-supervised segmentation models.

[0006] In the first aspect, a self-supervised pre-training method for brain tumors based on attention symmetric autoencoding is provided, comprising:

[0007] Acquire a first brain MRI data sample set and a second brain MRI data sample set, and perform preprocessing, wherein the first brain MRI data sample set includes unlabeled brain MRI data, and the second brain MRI data sample set includes labeled brain MRI data;

[0008] Dividing the unlabeled brain MRI data into a plurality of image blocks, and randomly using masks to cover a preset proportion of the image blocks;

[0009] Perform image reconstruction on the masked image patches through an attention symmetric autoencoder to generate a pre-trained model;

[0010] Initializing the original segmentation model using the parameters of the pre-trained model, and fine-tuning the original segmentation model using labeled brain MRI data to generate an actual segmentation model;

[0011] The brain MRI data of the patient to be segmented is segmented by using the segmentation model to determine the tumor area.

[0012] In one embodiment, the performing image reconstruction on the mask image block by using an attention symmetric autoencoder to generate a pre-trained model includes:

[0013] Arbitrarily selecting a piece of unlabeled MRI data after mask processing and inputting it into the first image block embedding layer for processing to obtain a first feature vector, wherein the unlabeled MRI data after mask processing includes a non-masked image block and a masked image block;

[0014] Performing symmetric position encoding on the non-masked image block to generate a first position encoding vector;

[0015] Adding the first position coding vector and the first feature vector, inputting the sum into the encoder and decoder modules for processing, and outputting the image reconstruction result;

[0016] A loss function between the image reconstruction result and the original image region is calculated, and iterative training is performed according to the loss function until the pre-training model is generated.

[0017] In one embodiment, the adding of the first position coding vector and the first feature vector, inputting the sum into the encoder and decoder modules for processing, and outputting the image reconstruction result includes:

[0018] After adding the first position encoding vector to the first feature vector, the vector is input into a multi-layer linear moving window visual self-attention change network block of an encoder for feature extraction to output an image feature vector of a non-mask position;

[0019] Performing symmetric position encoding on the mask image block to generate a second position encoding vector;

[0020] After adding the second position encoding vector to the mask token, it is spliced ​​with the image feature vector of the non-mask position, and input into the multi-layer linear moving window visual self-attention change network block of the decoder for processing, and then the image reconstruction result is output to generate the pre-trained model.

[0021] In one embodiment, the multi-layer linear moving window visual sub-attention change network block is composed of a multi-head self-attention mechanism based on a linear window, a multi-head self-attention mechanism based on a moving linear window, a normalization layer, and a multi-layer perceptron.

[0022] In one embodiment, the image reconstruction result includes a restored image area, and the calculating of the loss function between the image reconstruction result and the original image area includes:

[0023] Calculate the gradient vector of each voxel in each image block through the filter mask;

[0024] For each image block, a corresponding 2D gradient histogram is constructed;

[0025] Determining the importance of any image block by using the gradient vector and the gradient histogram;

[0026] Based on the importance, a mean square error is used to calculate the voxel-level difference between the restored image region and the original image region to determine the total loss.

[0027] In one embodiment, the initializing the original segmentation model by using the parameters of the pre-trained model, and fine-tuning the original segmentation model by using labeled brain MRI data to generate an actual segmentation model, comprises:

[0028] Initialize the original segmentation model using the parameter weights obtained in the pre-training phase;

[0029] Inputting the labeled brain MRI data into the original segmentation model and generating a segmentation result;

[0030] According to the difference between the segmentation result and the original segmentation label on the labeled brain MRI data, the original segmentation model is iteratively trained to generate the actual segmentation model.

[0031] In one embodiment, the step of inputting the labeled brain MRI data into the original segmentation model and generating a segmentation result comprises:

[0032] Dividing the labeled brain MRI data into a plurality of image blocks, randomly using a mask to cover a preset proportion of the image blocks, and inputting the image blocks into a second image embedding layer for processing to obtain a second feature vector;

[0033] Performing symmetric position encoding on image blocks at non-mask positions in the labeled brain MRI data to generate a third position encoding vector;

[0034] The third position encoding vector is added to the second feature vector and input into a segmentation module for segmentation processing to generate a segmentation result.

[0035] In the second aspect, a brain tumor self-supervised pre-training device based on attention symmetric autoencoding is provided, comprising:

[0036] a data acquisition unit, configured to acquire a first brain MRI data sample set and a second brain MRI data sample set, and perform preprocessing, wherein the first brain MRI data sample set includes unlabeled brain MRI data, and the second brain MRI data sample set includes labeled brain MRI data;

[0037] A mask unit, used for dividing the unlabeled brain MRI data into a plurality of image blocks, and randomly using masks to cover a preset proportion of the image blocks;

[0038] A pre-trained model generation unit, used for performing image reconstruction on the mask image block through an attention symmetric autoencoder to generate a pre-trained model;

[0039] A segmentation model generating unit, configured to initialize the original segmentation model using the parameters of the pre-trained model, and fine-tune the original segmentation model using labeled brain MRI data to generate an actual segmentation model;

[0040] The segmentation unit is used to segment the brain MRI data of the patient to be segmented by using the segmentation model to determine the tumor area.

[0041] In a third aspect, a computer device is provided, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein when the processor executes the computer-readable instructions, the steps of the brain tumor self-supervised pre-training method based on attention symmetric autoencoding as described above are implemented.

[0042] In a fourth aspect, one or more readable storage media are provided, wherein the readable storage media stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the brain tumor self-supervised pre-training method based on attention symmetric autoencoding as described above are implemented.

[0043] In an embodiment of the present application, a method, device, computer equipment and readable storage medium for self-supervised pre-training of brain tumors based on attention symmetric autoencoding are provided, and the method implementation includes: obtaining a first brain MRI data sample set and a second brain MRI data sample set, and performing pre-processing, wherein the first brain MRI data sample set includes unlabeled brain MRI data, and the second brain MRI data sample set includes labeled brain MRI data; dividing the unlabeled brain MRI data into multiple image blocks, and randomly using masks to cover a preset proportion of image blocks; reconstructing the masked image blocks through an attention symmetric autoencoder to generate a pre-training model; fine-tuning the pre-training model through the labeled brain MRI data to generate a segmentation model; segmenting the brain MRI data of the patient to be segmented through the segmentation model to determine the tumor area. In the present application, a large amount of unlabeled brain MRI data is used as data in the pre-training stage, a small amount of brain MRI data is used as data in the fine-tuning training stage, and an attention symmetric autoencoder is used for image reconstruction, which can effectively utilize the information of the importance and symmetry of brain regions and improve segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0045] Figure 1 It is a flow chart of a method for self-supervised pre-training of brain tumors based on attention symmetric autoencoding in one embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of an application environment of a brain tumor self-supervised pre-training model based on attention symmetric autoencoding in one embodiment of the present invention;

[0047] Figure 3 2 is a schematic diagram of a structure of a multi-layer linear moving window visual sub-attention change network block in one embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of an application environment of a segmentation model in one embodiment of the present invention;

[0049] Figure 5 is a structural schematic diagram of a brain tumor self-supervised pre-training device based on attention symmetric autoencoding in one embodiment of the present invention;

[0050] Figure 6 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] In one embodiment, if Figure 1 As shown, a brain tumor self-supervised pre-training method based on attention symmetric autoencoding is provided, comprising the following steps:

[0053] In step S110, a first brain MRI data sample set and a second brain MRI data sample set are acquired and preprocessed, wherein the first brain MRI data sample set includes unlabeled brain MRI data, and the second brain MRI data sample set includes labeled brain MRI data;

[0054] In an embodiment of the present application, the first brain MRI data sample set and the second brain MRI data sample set may be brain MRI data collected from clinical patients, the first brain MRI data sample set includes a large amount of unlabeled brain MRI data, and the second brain MRI data sample set includes a small amount of labeled brain MRI data, and the labeled brain MRI data is a tumor segmentation label manually drawn by a professional doctor on the brain MRI data.

[0055] In the embodiment of the present application, the labeled brain MRI data and the unlabeled brain MRI data are three-dimensional MRI images.

[0056] In the embodiment of the present application, the preprocessing specifically includes: aligning the data, affine transforming the data to the same space, cropping the central area, etc.

[0057] In step S120, the unlabeled brain MRI data is divided into a plurality of image blocks, and a mask is randomly used to cover a preset proportion of the image blocks;

[0058] In an embodiment of the present application, the unlabeled brain MRI is divided into multiple image blocks according to a preset size, for example, a size of H×W, and a preset proportion of the image blocks can be masked by a mask, for example, 75% of the area where the image blocks are located is masked.

[0059] In step S130, the mask image block is reconstructed by an attention symmetric autoencoder to generate a pre-trained model;

[0060] In an embodiment of the present application, the attention symmetric autoencoder may include a symmetric position encoding module, an encoder and decoder module, and a reconstruction loss module based on an attention mechanism.

[0061] In an embodiment of the present application, the masked unlabeled MRI image is input into an attention symmetric autoencoder, and after dimension conversion through the image block embedding layer of the attention symmetric autoencoder, it is added to the encoding vector of the symmetric position encoding and input into the encoder and decoder modules for processing, and then the image reconstruction result is output. Then, the gradient vector of the image block voxel is constructed through a reconstruction loss module based on the attention mechanism, and then the voxel-level difference between the restored image area and the original image area is calculated through the loss function to obtain the total loss, and then the parameters of the attention symmetric autoencoder are adjusted through the total loss, and iterative training is performed to generate the pre-trained model.

[0062] In step S140, the original segmentation model is initialized by the parameters of the pre-trained model, and the original segmentation model is fine-tuned and trained by the labeled brain MRI data to generate an actual segmentation model;

[0063] In an embodiment of the present application, the original segmentation model in the brain tumor image segmentation task is initialized by the model parameters of the pre-trained model, and then the original segmentation model is fine-tuned and trained by the brain MRI data with segmentation labels. Specifically, the brain MRI data with segmentation labels is input into the original segmentation model, firstly the dimension is converted through the image block embedding layer, and then it is added with the encoding vector of the symmetric position encoding module, and then input into the encoder to extract image features, and the image features extracted by each layer of the encoder are connected to the front and back through convolution by jump connection to reduce the accuracy loss, until the segmentation result is output through the last layer of the decoder.

[0064] Furthermore, the segmentation results are compared with the original labels on the brain MRI data to determine the loss, and the pre-trained model is fine-tuned, and then iterative training is performed until the model output reaches the expected value to generate a segmentation model.

[0065] In step S150, the brain MRI data of the patient to be segmented is segmented by using the segmentation model to determine the tumor area.

[0066] In an embodiment of the present application, after the segmentation model is generated, the brain MRI data of clinical patients can be segmented by the segmentation model to help doctors make a diagnosis. Specifically, when the patient takes a brain MRI image, the segmentation model can be used to segment the tumor area in the MRI image to help doctors quickly locate and diagnose.

[0067] In an embodiment of the present application, a self-supervised pre-training method for brain tumors based on attention symmetric autoencoding is provided, including: obtaining a first brain MRI data sample set and a second brain MRI data sample set, and performing pre-processing, wherein the first brain MRI data sample set includes unlabeled brain MRI data, and the second brain MRI data sample set includes labeled brain MRI data; dividing the unlabeled brain MRI data into multiple image blocks, and randomly using masks to cover a preset proportion of image blocks; reconstructing the masked image blocks through an attention symmetric autoencoder to generate a pre-training model; fine-tuning the pre-training model through the labeled brain MRI data to generate a segmentation model; segmenting the brain MRI data of the patient to be segmented through the segmentation model to determine the tumor area. In the present application, a large amount of unlabeled brain MRI data is used as data in the pre-training stage, a small amount of brain MRI data is used as data in the fine-tuning training stage, and an attention symmetric autoencoder is used for image reconstruction, which can effectively utilize the information of the importance and symmetry of brain regions and improve segmentation accuracy.

[0068] In one embodiment of the present application, a brain tumor self-supervised pre-training method based on attention symmetric autoencoding is provided, and its implementation process includes:

[0069] In step S110, a first brain MRI data sample set and a second brain MRI data sample set are acquired and preprocessed, wherein the first brain MRI data sample set includes unlabeled brain MRI data, and the second brain MRI data sample set includes labeled brain MRI data;

[0070] In an embodiment of the present application, the first brain MRI data sample set and the second brain MRI data sample set may be brain MRI data collected from clinical patients, the first brain MRI data sample set includes a large amount of unlabeled brain MRI data, and the second brain MRI data sample set includes a small amount of labeled brain MRI data, and the labeled brain MRI data is a tumor segmentation label manually drawn by a professional doctor on the brain MRI data.

[0071] In the embodiment of the present application, the labeled brain MRI data and the unlabeled brain MRI data are three-dimensional MRI images.

[0072] In the embodiment of the present application, the preprocessing specifically includes: aligning the data, affine transforming the data to the same space, cropping the central area, etc.

[0073] In step S120, the unlabeled brain MRI data is divided into a plurality of image blocks, and a mask is randomly used to cover a preset proportion of the image blocks;

[0074] In an embodiment of the present application, the unlabeled brain MRI is divided into multiple image blocks according to a preset size, for example, a size of H×W, and a preset proportion of the image blocks can be masked by a mask, for example, 75% of the area where the image blocks are located is masked.

[0075] In step S130, the mask image block is reconstructed by an attention symmetric autoencoder to generate a pre-trained model;

[0076] In an embodiment of the present application, the attention symmetric autoencoder may include a symmetric position encoding module, an encoder and decoder module, and a reconstruction loss module based on an attention mechanism.

[0077] In one embodiment of the present application, the mask image block is reconstructed by an attention symmetric autoencoder to generate a pre-trained model, including:

[0078] Inputting the masked unlabeled MRI data into a first image block embedding layer for processing to obtain a first feature vector, wherein the masked unlabeled MRI data includes a non-masked image block and a masked image block;

[0079] Performing symmetric position encoding on the non-masked image block to generate a first position encoding vector;

[0080] Adding the first position coding vector and the first feature vector, inputting the sum into the encoder and decoder modules for processing, and outputting the image reconstruction result;

[0081] A loss function between the image reconstruction result and the original image region is calculated, and iterative training is performed according to the loss function until the pre-training model is generated.

[0082] In an embodiment of the present application, an unlabeled MRI data is randomly selected from the first MRI data sample set, and after being divided into multiple image blocks, the positions of some image blocks are masked, and then input into the first image block embedding layer of the attention symmetric autoencoder for dimensional conversion to obtain a first feature vector, and the image block at the MRI mask position is position-encoded by the symmetric position encoding module to obtain a first position encoding vector, and after adding the first feature vector and the first position encoding vector, they are input into the encoder and decoder modules for feature extraction until the image reconstruction result is output through the last layer of the decoder, and the image reconstruction result includes the restored image area. A loss function between the restored image area and the original image area is constructed, and the model parameters are adjusted according to the loss function, and other unlabeled MRI data are selected from the first MRI data sample set to iteratively train the model until the loss function reaches the expected value, and the pre-trained model is generated at this time.

[0083] In an embodiment of the present application, the position encoding module can be used to encode an image block at a non-mask position into a vector containing, for example, 128 elements. For example, the 2nd row, 3rd column, and 5th position can be represented as (2, 3, 5).

[0084] In the embodiment of the present application, the symmetric position coding allows the symmetrical image blocks in the same row to share the same position coding. Specifically, for example, the position coding of the image block is T×H×W, (t, h, w) represents the number of the image block and the coordinates of the image block, then the symmetric position coding can be calculated by the following formula:

[0085]

[0086] PE(t,h,w,2i)=sin(Pos);

[0087]

[0088] Where D is the dimension of the symmetric position encoding vector, which is the same as the number of cut-in channels of the image block. PE(·) is represented by the image block at (t, h, w) returning the first dimension of the symmetric position encoding vector. elements.

[0089] In an embodiment of the present application, based on the prior information of the left-right symmetry of the brain structure, the brain MRI data is symmetrically position encoded through the symmetrical position encoding, which can reduce the encoding difference between the two symmetrical image positions and encourage the model to obtain better features from the two related areas.

[0090] See also Figure 2 In an embodiment of the present application, the encoder and the decoder both include a multi-layer linear moving window visual self-attention change network block. The unlabeled MRI image after mask processing is H×W×D×C. After being processed by the image block embedding layer and the feature vector added to the symmetric position encoding module is input into the multi-layer linear moving window visual self-attention transformation network block of the encoder for feature extraction layer by layer, H / 8×w / 8×D / 8×384 image features can be obtained, and then input into the decoder, and feature extraction is performed layer by layer in the multi-layer linear moving window visual self-attention transformation network block of the decoder in turn, until the image reconstruction result is output through the last layer of linear moving window visual self-attention transformation network block, and a reconstructed image block H×W×D×C is obtained.

[0091] In the embodiment of the present application, the first position coding vector and the first feature vector are added and input into the encoder and decoder modules for processing, and then the image reconstruction result is output, including:

[0092] After adding the first position encoding vector to the first feature vector, the vector is input into a multi-layer linear moving window visual self-attention change network block of an encoder for feature extraction to output an image feature vector of a non-mask position;

[0093] Performing symmetric position encoding on the mask image block to generate a second position encoding vector;

[0094] After adding the second position encoding vector to the mask token, it is spliced ​​with the image feature vector of the non-mask position, and input into the multi-layer linear moving window visual self-attention change network block of the decoder for processing, and then the image reconstruction result is output to generate the pre-trained model.

[0095] In an embodiment of the present application, feature extraction is performed layer by layer in the multi-layer linear moving window visual self-attention change network block of the encoder until the image feature vector of the non-mask position is output through the last layer of the linear moving window visual self-attention change network block, the image block at the mask position is position-encoded by the symmetric position encoding module, and after adding the mask token at the corresponding position, the image feature vector at the non-mask position is spliced, and the spliced ​​feature vector is input into the multi-layer linear moving window visual self-attention change network block of the decoder for processing, and the image reconstruction result is output.

[0096] See also Figure 3 In an embodiment of the present application, the linear moving window visual self-attention change network block is composed of a linear window-based multi-head self-attention mechanism (Linear Window-based Multi-head Self-attention, LW-MSA), a shifted linear window-based multi-head self-attention mechanism (Shifted Linear Window-based Multi-head Self-attention, SLW-MSA), a normalization layer (Norm layer) and a multi-layer perceptron (MLP layer).

[0097] Specifically, the self-attention mechanism assigns three different vectors to each embedded image block X, namely the query vector (Q); the key vector (K) and the value vector (Q), which are obtained by multiplying the embedding vector X by three different weight matrices, and then calculating a score for each vector, score = Q·K T ; Then normalize the score and divide by is the embedded vector; apply the softmax activation function to the score to form a value vector, and get the weighted score v of each input embedding vector. Finally, add the scores to get the final output result z = ∑{v}. The calculation formula is as follows:

[0098]

[0099] In the embodiment of the present application, the multi-head attention mechanism based on the linear window can flatten the image block into a linear sequence and divide it according to the size of the window S. LW-MSA calculates the attention in each 1D linear window by the above formula, while SLW-MSA shifts the sequence before passing through the LW-MSA module. After calculating LW-WSA, shift the sequence in reverse By alternately stacking LW-MSA and SLW-MSA, a linear moving window visual self-attention transformation network capable of extracting features across windows is obtained. The specific calculation process is as follows:

[0100]

[0101]

[0102]

[0103]

[0104] Among them, l represents the network layer index and MLP represents multi-layer perceptron.

[0105] In an embodiment of the present application, the image reconstruction result includes a restored image area, and the calculating of the loss function between the image reconstruction result and the original image area includes:

[0106] Calculate the gradient vector of each voxel in each image block through the filter mask;

[0107] For each image block, a corresponding 2D gradient histogram is constructed;

[0108] Determining the importance of any image block by using the gradient vector and the gradient histogram;

[0109] Based on the importance, a mean square error is used to calculate the voxel-level difference between the restored image region and the original image region to determine the total loss.

[0110] Specifically, since learning to restore smooth areas is less helpful for the model to obtain discriminative representations, by constructing an attention-based reconstruction loss function, we can better emphasize the information-rich areas in brain MRI data. For each voxel of each image block, a filter mask of [-1, 0, 1] can be used to calculate its gradient vector Among them, they correspond to three dimensions respectively. In the three-dimensional spherical coordinate system, two scalars θ and φ are used to represent the direction of the voxel. θ and φ can be calculated by the following formula:

[0111]

[0112] φ=|atan2(g y ,g x )|

[0113] For each image block, a 2D histogram G is constructed, and the histogram unit distance (bin) is set to b×b. Then, each voxel in the image block is traversed, and the direction of the current voxel is represented by θ and φ. Specifically, the bucket index of the voxel is first determined as:

[0114] Then, the gradient magnitude of the current voxel is Accumulate it to the bucket index (r, c) corresponding to the 2D gradient histogram G. After processing all voxels of the image block, perform L2 normalization on the image block histogram and calculate the average value of each image block histogram G Finally, the importance of the i-th image block p is obtained by normalizing the mean of all image blocks i ,Right now:

[0115]

[0116] The mean square error (MSE) is used to measure the voxel-level difference between the restored image area and the original image area, and the total loss can be expressed as:

[0117]

[0118] Where X, Y are the reconstructed image and the original image, N is the number of image patches masked by the mask, and M is the number of voxels in the image patch. ij represents the j-th voxel value of the i-th block in image X.

[0119] In step S140, the original segmentation model is initialized by the parameters of the pre-trained model, and the original segmentation model is fine-tuned and trained by the labeled brain MRI data to generate an actual segmentation model;

[0120] In an embodiment of the present application, the original segmentation model is initialized by the parameters of the pre-trained model, and the original segmentation model is fine-tuned and trained by labeled brain MRI data to generate an actual segmentation model, including:

[0121] Initialize the original segmentation model using the parameter weights obtained in the pre-training phase;

[0122] Inputting the labeled brain MRI data into the original segmentation model and generating a segmentation result;

[0123] According to the difference between the segmentation result and the original segmentation label on the labeled brain MRI data, the original segmentation model is iteratively trained to generate the actual segmentation model.

[0124] See also Figure 4 In an embodiment of the present application, the original segmentation model includes a brain tumor segmentation neural network module. The parameters of the encoder in the segmentation neural network model are the same as the parameters of the encoder module in the pre-trained model, and both include a multi-layer linear moving window visual self-attention transformation network block stacked together.

[0125] In an embodiment of the present application, after the original segmentation model is initialized by the parameter weights of the pre-trained model, a labeled brain MRI data is arbitrarily selected from the second brain MRI sample set and input into the original segmentation model. The segmentation processing is performed by the original segmentation model, and a segmentation result is generated. Then, the segmentation accuracy is determined by comparing the segmentation result with the label marked in advance on the brain MRI data, and the parameters of the original segmentation model are adjusted. Then, the above steps are repeated with other labeled brain MRI data, and the original segmentation model is iteratively trained until the similarity between its segmentation result and the original label reaches a preset threshold, and then an actual segmentation model is generated.

[0126] In the embodiment of the present application, the step of inputting the labeled brain MRI data into the original segmentation model and generating a segmentation result includes:

[0127] Dividing the labeled brain MRI data into a plurality of image blocks, randomly using a mask to cover a preset proportion of the image blocks, and inputting the image blocks into a second image embedding layer for processing to obtain a second feature vector;

[0128] Performing symmetric position encoding on image blocks at non-mask positions in the labeled brain MRI data to generate a third position encoding vector;

[0129] The third position encoding vector is added to the second feature vector and input into a segmentation module for feature extraction to generate a segmentation result.

[0130] In an embodiment of the present application, the brain tumor segmentation neural network module includes a second image block embedding layer, a U-shaped network composed of a multi-layer moving window visual self-attention change network block and a deconvolution upsampling module.

[0131] Specifically, a labeled brain MRI image is randomly selected from the second brain MRI data sample set, and then the image is divided into multiple image blocks, and a mask is randomly used to cover a preset proportion of image blocks, for example, 75% of the image blocks, and then input into the second image embedding layer for dimensional conversion to obtain a second feature vector, and the image blocks at non-mask positions in the labeled brain MRI data are symmetrically encoded through a symmetrical position encoding module to generate a third position encoding vector, and the second feature vector and the third position encoding vector are added and input into a multi-layer moving window visual self-attention change network block for feature extraction layer by layer, and the image features extracted by convolution of each layer of the moving window visual self-attention change network block can be connected to the front and back through a jump connection, and the final segmentation result is output, which can effectively reduce the accuracy loss.

[0132] The convolution may include 3×3×3 and 1×1×1 convolutions.

[0133] The upsampling can be achieved by deconvolution.

[0134] In step S150, the brain MRI data of the patient to be segmented is segmented by using the segmentation model to determine the tumor area.

[0135] In an embodiment of the present application, after the segmentation model is generated, the brain MRI data of clinical patients can be segmented by the segmentation model to help doctors make a diagnosis. Specifically, when the patient takes a brain MRI image, the segmentation model can be used to segment the tumor area in the MRI image to help doctors quickly locate and diagnose.

[0136] In this application, a large amount of unlabeled brain MRI data is used as data in the pre-training stage, a small amount of brain MRI data is used as data in the fine-tuning training stage, and an attention symmetric autoencoder is used for image reconstruction, which can effectively utilize the information of the importance and symmetry of brain regions and improve segmentation accuracy.

[0137] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0138] In one embodiment, a method and apparatus for self-supervised pre-training of brain tumors based on attention symmetric autoencoding is provided, and the method and apparatus for self-supervised pre-training of brain tumors based on attention symmetric autoencoding corresponds one-to-one to the method and apparatus for self-supervised pre-training of brain tumors based on attention symmetric autoencoding in the above embodiment. Figure 5As shown, the brain tumor self-supervised pre-training method and device based on attention symmetric autoencoding includes a data acquisition unit 10, a mask unit 20, a pre-training model generation unit 30, a segmentation model generation unit 40 and a segmentation unit 50. The components are described in detail as follows:

[0139] A data acquisition unit 10 is used to acquire a first brain MRI data sample set and a second brain MRI data sample set, and perform preprocessing, wherein the first brain MRI data sample set includes unlabeled brain MRI data, and the second brain MRI data sample set includes labeled brain MRI data;

[0140] A mask unit 20, for dividing the unlabeled brain MRI data into a plurality of image blocks, and randomly using masks to cover a preset proportion of the image blocks;

[0141] A pre-trained model generating unit 30, configured to perform image reconstruction on the mask image block through an attention symmetric autoencoder to generate a pre-trained model;

[0142] A segmentation model generating unit 40, configured to initialize the original segmentation model using the parameters of the pre-trained model, and fine-tune the original segmentation model using labeled brain MRI data to generate an actual segmentation model;

[0143] The segmentation unit 50 is used to segment the brain MRI data of the patient to be segmented by using the segmentation model to determine the tumor area.

[0144] In one embodiment of the present application, the pre-training model generation unit 30 is further used to:

[0145] Inputting the masked unlabeled MRI data into a first image block embedding layer for processing to obtain a first feature vector, wherein the masked unlabeled MRI data includes a non-masked image block and a masked image block;

[0146] Performing symmetric position encoding on the non-masked image block to generate a first position encoding vector;

[0147] Adding the first position coding vector and the first feature vector, inputting the sum into the encoder and decoder modules for processing, and outputting the image reconstruction result;

[0148] A loss function between the image reconstruction result and the original image region is calculated, and iterative training is performed according to the loss function until the pre-training model is generated.

[0149] In one embodiment of the present application, the pre-training model generation unit 30 is further used to:

[0150] After adding the first position encoding vector to the first feature vector, the vector is input into a multi-layer linear moving window visual self-attention change network block of an encoder for feature extraction to output an image feature vector of a non-mask position;

[0151] Performing symmetric position encoding on the mask image block to generate a second position encoding vector;

[0152] After adding the second position encoding vector to the mask token, it is spliced ​​with the image feature vector of the non-mask position, and input into the multi-layer linear moving window visual self-attention change network block of the decoder for processing, and then the image reconstruction result is output to generate the pre-trained model.

[0153] In one embodiment of the present application, the multi-layer linear moving window visual sub-attention change network block is composed of a multi-head self-attention mechanism based on a linear window, a multi-head self-attention mechanism based on a moving linear window, a normalization layer, and a multi-layer perception mechanism.

[0154] In one embodiment of the present application, the pre-training model generation unit 30 is further used to:

[0155] Calculate the gradient vector of each voxel in each image block through the filter mask;

[0156] For each image block, a corresponding 2D gradient histogram is constructed;

[0157] Determining the importance of any image block by using the gradient vector and the gradient histogram;

[0158] Based on the importance, a mean square error is used to calculate the voxel-level difference between the restored image region and the original image region to determine the total loss.

[0159] In one embodiment of the present application, the segmentation model generating unit 40 is further configured to:

[0160] Initialize the original segmentation model using the parameter weights obtained in the pre-training phase;

[0161] Inputting the labeled brain MRI data into the original segmentation model and generating a segmentation result;

[0162] According to the difference between the segmentation result and the original segmentation label on the labeled brain MRI data, the original segmentation model is iteratively trained to generate the actual segmentation model.

[0163] In one embodiment of the present application, the segmentation model generating unit 40 is further configured to:

[0164] Dividing the labeled brain MRI data into a plurality of image blocks, randomly using a mask to cover a preset proportion of the image blocks, and inputting the image blocks into a second image embedding layer for processing to obtain a second feature vector;

[0165] Performing symmetric position encoding on image blocks at non-mask positions in the labeled brain MRI data to generate a third position encoding vector;

[0166] The third position encoding vector is added to the second feature vector and input into a segmentation module for segmentation processing to generate a segmentation result.

[0167] In this application, a large amount of unlabeled brain MRI data is used as data in the pre-training stage, a small amount of brain MRI data is used as data in the fine-tuning training stage, and an attention symmetric autoencoder is used for image reconstruction, which can effectively utilize the information of the importance and symmetry of brain regions and improve segmentation accuracy.

[0168] For the specific limitations of the brain tumor self-supervised pre-training device based on attention symmetric autoencoding, please refer to the limitations of the brain tumor self-supervised pre-training method based on attention symmetric autoencoding above, which will not be repeated here. Each module in the above-mentioned brain tumor self-supervised pre-training device based on attention symmetric autoencoding can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0169] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium. The readable storage medium stores computer-readable instructions. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instructions are executed by the processor, a brain tumor self-supervised pre-training method based on attention symmetric autoencoding is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0170] A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the steps of the above-mentioned brain tumor self-supervised pre-training method based on attention symmetric autoencoding are implemented.

[0171] One or more readable storage media, wherein the readable storage medium stores computer-readable instructions, wherein when the computer-readable instructions are executed by a processor, the steps of self-supervised pre-training of brain tumors based on attention symmetric autoencoding as described above are implemented.

[0172] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0173] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0174] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A self-supervised pre-training method for brain tumors based on attention symmetric autoencoders, characterized in that: The method comprises: Acquire a first brain MRI data sample set and a second brain MRI data sample set, and perform preprocessing, wherein the first brain MRI data sample set includes unlabeled brain MRI data, and the second brain MRI data sample set includes labeled brain MRI data; Dividing the unlabeled brain MRI data into a plurality of image blocks, and randomly using masks to cover a preset proportion of the image blocks; Inputting the masked unlabeled MRI data into a first image block embedding layer for processing to obtain a first feature vector, wherein the masked unlabeled MRI data includes a non-masked image block and a masked image block; Performing symmetric position encoding on the non-masked image block to generate a first position encoding vector; Adding the first position coding vector and the first feature vector, inputting the sum into the encoder and decoder modules for processing, and outputting the image reconstruction result; Calculating a loss function between the image reconstruction result and the original image region, and performing iterative training according to the loss function until a pre-training model is generated; Initializing the original segmentation model using the parameters of the pre-trained model, and fine-tuning the original segmentation model using labeled brain MRI data to generate an actual segmentation model; The brain MRI data of the patient to be segmented is segmented by using the actual segmentation model to determine the tumor area.

2. The brain tumor self-supervised pre-training method based on attention symmetric autoencoding according to claim 1, characterized in that: The adding of the first position coding vector and the first feature vector, inputting the sum into the encoder and decoder modules for processing, and outputting the image reconstruction result includes: After adding the first position encoding vector to the first feature vector, the vector is input into a multi-layer linear moving window visual self-attention change network block of an encoder for feature extraction to output an image feature vector of a non-mask position; Performing symmetric position encoding on the mask image block to generate a second position encoding vector; After adding the second position encoding vector to the mask token, it is spliced ​​with the image feature vector of the non-mask position, and input into the multi-layer linear moving window visual self-attention change network block of the decoder for processing, and then the image reconstruction result is output to generate the pre-trained model.

3. The brain tumor self-supervised pre-training method based on attention symmetric autoencoding as claimed in claim 2, characterized in that: The multi-layer linear moving window visual self-attention change network block is composed of a multi-head self-attention mechanism based on a linear window, a multi-head self-attention mechanism based on a moving linear window, a normalization layer and a multi-layer perceptron.

4. The brain tumor self-supervised pre-training method based on attention symmetric autoencoding according to claim 1, characterized in that: The image reconstruction result includes a restored image area, and the calculating of a loss function between the image reconstruction result and the original image area includes: Calculate the gradient vector of each voxel in each image block through the filter mask; For each image block, a corresponding 2D gradient histogram is constructed; Determining the importance of any image block by using the gradient vector and the gradient histogram; Based on the importance, a mean square error is used to calculate the voxel-level difference between the restored image region and the original image region to determine the total loss.

5. The brain tumor self-supervised pre-training method based on attention symmetric autoencoding according to claim 1, characterized in that: The initializing the original segmentation model by using the parameters of the pre-trained model, and fine-tuning the original segmentation model by using labeled brain MRI data to generate an actual segmentation model, includes: Initialize the original segmentation model using the parameter weights obtained in the pre-training phase; Inputting the labeled brain MRI data into the original segmentation model and generating a segmentation result; According to the difference between the segmentation result and the original segmentation label on the labeled brain MRI data, the original segmentation model is iteratively trained to generate the actual segmentation model.

6. The brain tumor self-supervised pre-training method based on attention symmetric autoencoding according to claim 5, characterized in that: The step of inputting the labeled brain MRI data into the original segmentation model and generating a segmentation result comprises: Dividing the labeled brain MRI data into a plurality of image blocks, randomly using a mask to cover a preset proportion of the image blocks, and inputting the image blocks into a second image embedding layer for processing to obtain a second feature vector; Performing symmetric position encoding on image blocks at non-mask positions in the labeled brain MRI data to generate a third position encoding vector; The third position encoding vector is added to the second feature vector and input into a segmentation module for segmentation processing to generate a segmentation result.

7. A brain tumor self-supervised pre-training device based on attention symmetric autoencoding, characterized in that: The device comprises: a data acquisition unit, configured to acquire a first brain MRI data sample set and a second brain MRI data sample set, and perform preprocessing, wherein the first brain MRI data sample set includes unlabeled brain MRI data, and the second brain MRI data sample set includes labeled brain MRI data; A mask unit, used for dividing the unlabeled brain MRI data into a plurality of image blocks, and randomly using masks to cover a preset proportion of the image blocks; A pre-trained model generation unit is used to input the masked unlabeled MRI data into a first image block embedding layer for processing to obtain a first feature vector, wherein the masked unlabeled MRI data includes a non-masked image block and a masked image block; symmetrically position-encode the non-masked image block to generate a first position encoding vector; add the first position encoding vector to the first feature vector, input the sum to an encoder and a decoder module for processing, and output an image reconstruction result; calculate a loss function between the image reconstruction result and the original image region, and perform iterative training according to the loss function until the pre-trained model is generated; A segmentation model generating unit, configured to initialize the original segmentation model using the parameters of the pre-trained model, and fine-tune the original segmentation model using labeled brain MRI data to generate an actual segmentation model; The segmentation unit is used to segment the brain MRI data of the patient to be segmented by using the actual segmentation model to determine the tumor area.

8. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer-readable instructions, the steps of the brain tumor self-supervised pre-training method based on attention symmetric autoencoding as described in any one of claims 1 to 6 are implemented.

9. One or more readable storage media storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the brain tumor self-supervised pre-training method based on attention symmetric autoencoding as described in any one of claims 1 to 6 are implemented.

Images