A brain image processing method, device and storage medium

Abstract

The present application relates to the technical field of image processing, in particular to a brain image processing method, device and storage medium. The present application first divides an original brain image into multiple sub-images of multiple scales based on a residual network structure, and extracts initial feature maps of each sub-image. Then, a convolution algorithm is used to further extract local features of each initial feature map. After that, the individual feature maps of multiple scales are fused to obtain a common feature map. Since the initial feature maps of multiple scales obtained by using the residual network structure cover the global features of the original brain image, and the individual feature maps obtained by using the convolution cover the local features, the common feature map after fusion has both the global features and the local features of the original brain image, so the fusion feature map based on the common feature map can more accurately reflect the features of the original brain image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a brain image processing method, device and storage medium. BACKGROUND

[0002] Based on the analysis result of the human brain image, the doctor can effectively assist in carrying out the research work related to brain diseases, such as Alzheimer's disease (AD), which is a progressive neurodegenerative disease with insidious onset and irreversible nature. AD is the main cause of dementia in the elderly, and often occurs in the elderly over 65 years old, which has caused serious obstacles to the physical and mental health of the elderly. Since it is difficult to locate the lesion area in the ordinary brain image of the patient in the early stage of AD, the structural magnetic resonance imaging (sMRI) and positron emission computed tomography (PET) can show the morphological changes caused by brain atrophy, which plays an important role in AD disease identification. The existing technology often uses convolutional neural network (CNN) and Transformer neural network to process the brain image of the patient with AD, but CNN lacks the ability to model global information (i.e., lacks the ability to extract global features of the brain image), and Transformer lacks the ability to model local invariance (i.e., lacks the ability to extract local features of the brain image). From the above analysis, it can be seen that the existing technology is difficult to achieve comprehensive extraction of global features and local features, thereby resulting in low accuracy of brain image analysis based on brain image features.

[0003] In summary, the features extracted by the existing technology for brain images have limitations.

[0004] Therefore, the existing technology needs to be improved and improved. SUMMARY

[0005] To solve the above technical problems, the present application provides a brain image processing method, device and storage medium, which solves the problem that the features extracted by the existing technology for brain images have limitations.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a brain image processing method, which comprises:

[0008] Applying a residual network structure to the original brain image to obtain initial feature maps corresponding to a plurality of scales of the original brain image;

[0009] Applying a convolution algorithm to the initial feature maps corresponding to a plurality of scales to obtain features of the initial feature maps of each scale output by the convolution algorithm, denoted as individual feature maps;

[0010] fuse the individual scale personal feature maps to obtain a common feature map;

[0011] apply a dynamic fusion attention algorithm to the common feature map and the individual scale personal feature maps to obtain a feature map after fusion of individual channel information of the common feature map, denoted as a fusion feature map of each scale;

[0012] obtain a processing result for the original brain image according to the fusion feature map of each scale, the processing result being used to predict a brain lesion degree of a patient corresponding to the original brain image.

[0013] In an implementation manner, the applying of the residual network structure to the original brain image to obtain initial feature maps corresponding to a plurality of scales of the original brain image comprises:

[0014] obtain a magnetic resonance original image and a positron emission computed tomography original image in the original brain image according to the original brain image;

[0015] apply the residual network structure to the magnetic resonance original image and the positron emission computed tomography original image respectively to obtain feature maps of each scale of the magnetic resonance original image and feature maps of each scale of the positron emission computed tomography original image output by each residual layer in the residual network structure;

[0016] splice the feature maps of each scale of the magnetic resonance original image and the feature maps of each scale of the positron emission computed tomography original image to obtain the initial feature maps.

[0017] In an implementation manner, each residual layer of the residual network structure comprises:

[0018] a first efficient convolution module, an input end of which is configured to input an image;

[0019] a training acceleration convergence module, an input end of which is connected with an output end of the efficient convolution module, and configured to accelerate training of the residual network;

[0020] a first piecewise function module, an input end of which is connected with an output end of the training acceleration convergence module;

[0021] a second efficient convolution module, an input end of which is connected with an output end of the first piecewise function module;

[0022] a second piecewise function module, an input end of which is configured to input the same image as the first efficient convolution module, an input end of which is connected with an output end of the second efficient convolution module, and an output end of which is connected with an input end of the convolution algorithm.

[0023] In an implementation manner, the initial feature maps corresponding to the plurality of scales are applied with a convolution algorithm to obtain features of the initial feature maps of each scale output by the convolution algorithm, denoted as individual feature maps, including:

[0024] A first initial feature map, a second initial feature map, and a third initial feature map in the initial feature maps corresponding to the plurality of scales are determined, and the scales of the first initial feature map, the second initial feature map, and the third initial feature map decrease in turn.

[0025] A first convolution branch, a second convolution branch, a third convolution branch, and a fourth convolution branch constituting the convolution algorithm are determined, the first convolution branch, the second convolution branch, the third convolution branch, and the fourth convolution branch respectively include a front convolution layer and a rear convolution layer, the sizes of the front convolution layer of the first convolution branch, the front convolution layer of the second convolution branch, the front convolution layer of the third convolution branch, and the front convolution layer of the fourth convolution branch increase in turn, and the sizes of the rear convolution layer of the first convolution branch, the rear convolution layer of the second convolution branch, the rear convolution layer of the third convolution branch, and the rear convolution layer of the fourth convolution branch are the same.

[0026] The first convolution branch and the second convolution branch are respectively applied to the third initial feature map, and a feature map output by the first convolution branch and a feature map output by the second convolution branch are spliced to obtain an individual feature map of the third initial feature map.

[0027] The second initial feature map and the third initial feature map after upsampling are spliced to obtain a spliced feature map corresponding to the second initial feature map.

[0028] The first convolution branch, the second convolution branch, and the third convolution branch are respectively applied to the spliced feature map corresponding to the second initial feature map, and a feature map output by the first convolution branch, a feature map output by the second convolution branch, and a feature map output by the third convolution branch are spliced to obtain an individual feature map of the second initial feature map.

[0029] The first initial feature map and the second initial feature map after upsampling are spliced to obtain a spliced feature map corresponding to the first initial feature map.

[0030] The first convolution branch, the second convolution branch, the third convolution branch, and the fourth convolution branch are respectively applied to the spliced feature map corresponding to the first initial feature map, and a feature map output by the first convolution branch, a feature map output by the second convolution branch, a feature map output by the third convolution branch, and a feature map output by the fourth convolution branch are spliced to obtain an individual feature map of the first initial feature map.

[0031] In an implementation manner, the dynamic fusion attention algorithm comprises:

[0032] a channel attention branch configured to extract individual channel feature maps of a common concatenation feature map, the common concatenation feature map being a feature map obtained by concatenating the common feature map and the individual feature map;

[0033] a spatial attention branch configured to extract a global feature of the common concatenation feature map;

[0034] a voxel attention branch configured to extract voxel points of individual channels of the common concatenation feature map;

[0035] a fusion module configured to fuse the individual channel feature maps, the global feature and the voxel points to obtain fusion feature maps of individual scales.

[0036] In an implementation manner, the processing result for the original brain image is obtained according to the fusion feature maps of individual scales, and the processing result is used to predict a brain lesion degree of a patient corresponding to the original brain image, which comprises:

[0037] the fusion feature maps of individual scales are applied with a feature fusion algorithm in a scale-by-scale manner to obtain the fusion feature maps after preprocessing;

[0038] a Transformer algorithm with several axial directions is applied to the fusion feature maps after preprocessing to obtain a final feature map;

[0039] the processing result for the original brain image is obtained according to the final feature map.

[0040] In an implementation manner, the processing result for the original brain image is obtained according to the final feature map, which comprises:

[0041] a ROI attention algorithm is applied to the final feature map and the initial feature map of the smallest scale to obtain individual brain region feature maps and classification features output by the ROI attention algorithm;

[0042] the processing result for the original brain image is obtained according to the individual brain region feature maps and the classification features.

[0043] In an implementation manner, the ROI attention algorithm is applied to the final feature map and the initial feature map of the smallest scale to obtain individual brain region feature maps and classification features output by the ROI attention algorithm, which comprises:

[0044] a 1x1 convolution is applied to the initial feature map of the smallest scale to obtain a first convolution result;

[0045] applying 1x1 convolution to the final feature map to obtain a second convolution result;

[0046] obtaining a correlation between the final feature map and the initial feature map of the minimum scale according to the first convolution result and the second convolution result;

[0047] obtaining classification features of each final feature map according to the correlation and the second convolution result and the final feature map;

[0048] performing global average pooling, convolution and activation function ReLu on the classification features to obtain a brain region feature map.

[0049] In a second aspect, an embodiment of the present application further provides a terminal device, wherein the terminal device comprises a memory, a processor and a brain image processing program stored in the memory and executable on the processor, and the processor implements the steps of the brain image processing method when executing the brain image processing program.

[0050] In a third aspect, an embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a brain image processing program, and the processor implements the steps of the brain image processing method when executing the brain image processing program.

[0051] Beneficial effects: The present application firstly divides an original brain image into multiple sub-images of multiple scales based on a residual network structure, and extracts initial feature maps of each sub-image. Then, a convolution algorithm is used to further extract local features (denoted as individual feature maps) of each initial feature map. After that, the individual feature maps of multiple scales are fused to obtain common feature maps. Since the initial feature maps of multiple scales obtained by using the residual network structure cover the global features of the original brain image, and the individual feature maps obtained by using the convolution cover the local features, the common feature maps after fusion have both the global features and the local features of the original brain image, so the fusion feature maps obtained based on the common feature maps can more accurately reflect the features of the original brain image. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a whole flowchart of the present application;

[0053] Figure 2 is an AD classification flowchart in an embodiment of the present application;

[0054] Figure 3 is a residual layer structure diagram in an embodiment of the present application;

[0055] Figure 4 is a structure diagram of the MSCB module in an embodiment of the present application;

[0056] Figure 5 DFA structure diagram in the embodiment of the present application;

[0057] Figure 6 MSFA structure diagram in the embodiment of the present application;

[0058] Figure 7 ROI flow chart in the embodiment of the present application;

[0059] Figure 8 MRI image preprocessing flow chart in the embodiment of the present application;

[0060] Figure 9 PET image preprocessing flow chart in the embodiment of the present application;

[0061] Figure 10 The internal structure principle block diagram of the terminal device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the present application are described below in conjunction with the embodiments and the accompanying drawings. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0063] It is found through research that the analysis result based on the human brain image can effectively assist doctors in carrying out brain disease related research work. Alzheimer's disease (AD) is a progressive neurodegenerative disease with insidious onset and irreversibility. AD is the main cause of senile dementia, which often occurs in people over 65 years old, and causes serious obstacles to the physical and mental health of the elderly. Since it is difficult to locate the lesion area in the ordinary brain image of patients in the early stage of AD, structural magnetic resonance imaging (sMRI) and positron emission computed tomography (PET) can show morphological changes caused by brain atrophy and play an important role in AD disease identification. The existing technology often uses convolutional neural network (CNN) and Transformer neural network to process the brain image of patients with AD. However, CNN lacks the ability to model global information (i.e., lacks the ability to extract global features of brain images), and Transformer lacks the ability to model local invariance (i.e., lacks the ability to extract local features of brain images). From the above analysis, it can be seen that the existing technology is difficult to achieve comprehensive extraction of global features and local features, thereby resulting in low accuracy of brain image analysis based on brain image features.

[0064] To solve the above technical problems, the application provides a brain image processing method, device and storage medium, which solves the problem of the limitation of the features extracted by the prior art for brain images. In specific implementation, firstly, a residual network structure is applied to an original brain image to obtain initial feature maps corresponding to a plurality of scales of the original brain image; then, a convolution algorithm is applied to the initial feature maps corresponding to the plurality of scales to obtain features of the initial feature map of each scale output by the convolution algorithm (denoted as a personal feature map); thereafter, the personal feature maps of each scale are fused to obtain a common feature map; then, a dynamic fusion attention algorithm is applied to the common feature map and the personal feature maps of each scale to obtain a feature map after the channel information of the common feature map is fused (denoted as a fusion feature map of each scale); finally, a processing result for the original brain image is obtained according to the fusion feature map of each scale, and the processing result is used to predict the brain lesion degree of a patient corresponding to the original brain image. The accuracy of the obtained processing result can be improved.

[0065] For example, an original brain image (with a size or scale of 128*128*128) of a patient suffering from an AD brain disease is taken, and the original brain image is input into the residual network structure. The first residual layer in the residual network structure processes the original brain image to obtain an initial feature map a of 64*64*64. The second residual layer of the residual network structure continues to process the initial feature map a to obtain an initial feature map b of 32*32*32. The third residual layer of the residual network structure continues to process the initial feature map b to obtain an initial feature map c of 16*16*16. Since the scales of the initial feature maps a, b and c are different, the initial feature maps a, b and c constitute multi-scale initial feature maps. Then, the convolution algorithm is applied to the initial feature maps a, b and c respectively to obtain a personal feature map a1 corresponding to a (the personal feature map a1 only covers the features possessed by a), a personal feature map b1 corresponding to b and a personal feature map c1 corresponding to c. Then, a1, b1 and c1 are spliced to obtain a common feature map f. Then, the dynamic fusion attention algorithm DFA is applied to a1 and f, the dynamic fusion attention algorithm DFA is applied to b1 and f, and the dynamic fusion attention algorithm DFA is applied to c1 and f to obtain a fusion feature map a2 of 64*64*64, a fusion feature map b2 of 32*32*32 and a fusion feature map c2 of 16*16*16 respectively. Finally, the fusion feature map a2, the fusion feature map b2 and the fusion feature map c2 are further processed to determine the AD lesion degree according to the processing result.

[0066] Exemplary method

[0067] The brain image processing method of the embodiment can be applied to a terminal device, which can be a terminal product with an image acquisition function, such as a computer. Figure 1As shown in the figure, the brain image processing method specifically comprises the following steps S100, S200, S300, S400, S500:

[0068] S100, applying a residual network structure to the original brain image to obtain initial feature maps corresponding to a plurality of scales of the original brain image.

[0069] In one embodiment, step S100 comprises the following steps S101, S102, S103:

[0070] S101, obtaining a magnetic resonance original image and a positron emission tomography original image in the original brain image according to the original brain image.

[0071] The original brain image in this embodiment can simultaneously comprise a magnetic resonance original image MRI and a positron emission tomography original image PET, or can only comprise one of MRI and PET. MRI can reflect the lesion condition of the brain region in the early stage of AD, and PET can reflect the functional lesion condition. Therefore, combining MRI and PET multi-modal images can effectively mine discriminative features related to the early stage of AD.

[0072] S102, applying the residual network structure to the magnetic resonance original image and the positron emission tomography original image respectively to obtain feature maps of each scale of the magnetic resonance original image and feature maps of each scale of the positron emission tomography original image output by each residual layer in the residual network structure.

[0073] S103, splicing the feature maps of each scale of the magnetic resonance original image and the feature maps of each scale of the positron emission tomography original image to obtain the initial feature map.

[0074] For example, a 128*128*128 magnetic resonance original image MRI and a 128*128*128 positron emission tomography original image PET are input to the first residual layer, the first residual layer outputs the feature map corresponding to MRI (scale 64*64*64) and the feature map corresponding to PET (scale 64*64*64), and then the two feature maps are input to the next residual layer. In this way, the last two residual layers output the feature map corresponding to MRI (scale 32*32*32) and the feature map corresponding to PET (scale 32*32*32), the feature map corresponding to MRI (scale 16*16*16) and the feature map corresponding to PET (scale 16*16*16) respectively. Splicing the 64*64*64 feature map of MRI and the 64*64*64 feature map of PET obtains the 64*64*64 initial feature map S 1_in, the 32*32*32 feature map of the MRI and the 32*32*32 feature map of the PET are spliced to obtain an initial feature map S of 32*32*32 2_in , the 16*16*16 feature map of the MRI and the 16*16*16 feature map of the PET are spliced to obtain an initial feature map S of 16*16*16 3_in .

[0075] In this embodiment, the residual network structure ResNet18 is as shown in Figure 2 This embodiment first uses ResNet18 as a basis to extract initial multi-scale features, and in order to avoid the extracted feature dimension being too small, only the first three residual layers (Residual block) of ResNet18 are used, that is, three scale features are extracted.

[0076] As shown in Figure 3 Each residual layer includes a first efficient convolution module DSConv, a training acceleration convergence module BatchNorm, a first piecewise function module ReLu, a second efficient convolution module DSConv, and a second piecewise function module ReLu.

[0077] S200, applying a convolution algorithm to the initial feature maps corresponding to the plurality of scales to obtain the features of the initial feature map of each scale output by the convolution algorithm, denoted as a personal feature map.

[0078] For example, the initial feature map S 1_in , the initial feature map S 2_in , and the initial feature map S 3_in apply a convolution algorithm MSCB, which specifically includes the following steps S201 to S207:

[0079] S201, determining a first initial feature map S 1_in , a second initial feature map S 2_in , and a third initial feature map S 3_in of the initial feature maps corresponding to the plurality of scales, the scale of the first initial feature map, the scale of the second initial feature map, and the scale of the third initial feature map decrease in turn.

[0080] S202, determine a first convolution branch branch1, a second convolution branch branch2, a third convolution branch branch3, and a fourth convolution branch branch4 constituting the convolution algorithm, the first convolution branch, the second convolution branch, the third convolution branch, and the fourth convolution branch respectively comprising a front convolution layer and a rear convolution layer, the size of the front convolution layer of the first convolution branch, the size of the front convolution layer of the second convolution branch, the size of the front convolution layer of the third convolution branch, and the size of the front convolution layer of the fourth convolution branch increasing in turn, the size of the rear convolution layer of the first convolution branch, the size of the rear convolution layer of the second convolution branch, the size of the rear convolution layer of the third convolution branch, and the size of the rear convolution layer of the fourth convolution branch being the same.

[0081] As shown in Figure 4 , the sizes of the front convolution layers of branch1, branch2, branch3, and branch4 are 1x1x1, 3x3x3, 5x5x5, and 7x7x7 in turn, and the rear convolution layers of the four are all 3x3x3.

[0082] S203, applying the first convolution branch and the second convolution branch to the third initial feature map respectively, and splicing the feature map output by the first convolution branch and the feature map output by the second convolution branch to obtain a personal feature map S 3_o .

[0083] That is, two multi-scale convolution branch operations (branch1 and branch2) are constructed for the third initial feature map S 3_in to learn global features

[0084] S204, splicing the second initial feature map S 2_in and the third initial feature map S 3_in after upsampling to obtain a spliced feature map S2' corresponding to the second initial feature map _in .

[0085] The so-called upsampling is to adjust the scale of S 3_in to the same scale as S 2_in .

[0086] S205, applying the first convolution branch, the second convolution branch, and the third convolution branch to the spliced feature map S2' corresponding to the second initial feature map respectively, and splicing the feature map output by the first convolution branch, the feature map output by the second convolution branch, and the feature map output by the third convolution branch to obtain a personal feature map S _in of the second initial feature map 2_o .

[0087] S206, the first initial feature map and the upsampled second initial feature map are concatenated to obtain the concatenated feature map corresponding to the first initial feature map.

[0088] S207, apply the first convolutional branch, the second convolutional branch, the third convolutional branch, and the fourth convolutional branch to the concatenated feature map corresponding to the first initial feature map, and concatenate the feature maps output by the first convolutional branch, the second convolutional branch, the third convolutional branch, and the fourth convolutional branch to obtain the individual feature map S of the first initial feature map. 1_o .

[0089] Steps S201 to S207 are based on the following principles:

[0090] The sliding window mechanism unique to convolutional operations can effectively focus on key local features, allowing for the capture of detailed information about local disease areas. However, global representation is also crucial for disease diagnosis, but convolutional operations cannot learn global information. Multi-scale convolution can use kernels of different sizes to convolve a unified feature map, resulting in new feature maps of different sizes. This allows for learning image features from a global perspective, enriching the image's features. Therefore, multi-scale convolutional blocks were designed to model the global space, constructing... Figure 4 The multi-scale convolutional block shown can be represented as follows:

[0091]

[0092] in This represents the initial multi-scale features. The multi-scale convolution operation is represented by `Cat(·)`, which represents the channel concatenation operation. The multi-scale convolution operation can be described as follows: First, for low-scale feature learning, the multi-scale convolution block first concatenates the learned high-scale features with the low-scale features, i.e. The concatenated features serve as input features for the low-scale features, where, due to The highest-scale feature (referring to the feature obtained after three residual layers, not S) 3_in (the largest size), therefore, S 3_in The input features for multi-scale convolutional block feature learning are themselves. (This is related to...) Four multi-scale convolutional operation branches (branch1, branch2, branch3, branch4) were constructed because this scale contains the least feature information; therefore, four multi-scale convolutions can effectively learn richer features. Similarly, for... Three multi-scale convolutional operation branches (branch1, branch2, branch3) were constructed to learn global features. For S... 3_in Two multi-scale convolutional branch operations (branch1, branch2) were constructed to learn global features. In the multi-scale convolutional operation, the learned multi-scale convolutional features are concatenated together through a concatenation operation, and then the dimensions are restored to the feature dimensions of the input scale.

[0093] S300, the individual feature maps at each scale are fused to obtain a common feature map.

[0094] like Figure 2 As shown, for example, personality trait map S 1_o Personality Trait Map S 2_o Personality Trait Map S 3_o Concatenating the concatenation maps yields the common feature map f. ms That is, f ms =Cat[S 1_o ,S 2_o ,S 3_o ].

[0095] S400, Apply the Dynamic Fusion Attention Algorithm (DFA) to the common feature map and the individual feature maps at each scale to obtain the feature map after fusing the information of each channel of the common feature map, which is denoted as the fused feature map at each scale.

[0096] First, the individual feature maps at each scale are concatenated with the common feature maps to obtain f. ms_i :f ms_i =[Cat[f ms ,S i_o [,i=1,2,3], and then for each scale f ms_i DFA is applied to obtain the fused feature map f corresponding to each size i. DFA_i In this embodiment, f ms The feature map after dimensionality reduction, i.e., f ms The size and each S i_o same.

[0097] In a real-time example, such as Figure 5 As shown, the Dynamic Fusion Attention Algorithm (DFA) includes channel attention branch, channel attention branch, and voxel attention branch. Figure 5 The top to bottom of the middle section consists of channel attention branch, channel attention branch, voxel attention branch, and fusion module Add.

[0098] like Figure 5As shown, the DFA consists of spatial attention branches, channel attention branches, and voxel attention branches. In the channel attention branch, the feature map f is first processed by a global average pooling layer (GAP). ms_i The spatial information is compressed, then the features of each channel are calculated through two 1×1×1 convolutional layers (Conv). Next, a sigmoid activation function is used to obtain the channel-level weights. Finally, the input feature map f is... ms_i Multiplying by the channel weights yields the output features of the channel attention branch. In the spatial attention branch, the feature map f is first processed... ms_i The channel features are averaged to obtain initial spatial features. Then, a 1×1×1 convolutional layer (Conv) is used to learn spatial features, with spatial weights determined by a sigmoid activation function. Finally, the feature map is obtained by multiplying the input features and spatial weights pixel-wise. In the voxel attention branch, two 1×1×1 convolutional layers are used to learn the unique features of each voxel. Then, a sigmoid activation function is used to obtain voxel-level weights. Similarly, the voxel-level feature map is obtained by multiplying the input features and voxel-level weights. Finally, the outputs of the channel, spatial, and voxel attention branches are summed element-wise to form a new feature map f. DFA_i .

[0099] S500, based on the fused feature maps f at each scale DFA_i The processing results of the original brain image are obtained, and the processing results are used to predict the degree of brain lesions of the patient corresponding to the original brain image.

[0100] In one embodiment, S500 includes the following steps S501 to S508:

[0101] S501, the fused feature maps f at various scales DFA_i The feature fusion algorithm is applied on a scale-by-scale basis to obtain the preprocessed fused feature map.

[0102] For example, f DFA_1 f DFA_2 f DFA_3 The scale decreases sequentially, and f... DFA_1 Input to feature fusion algorithm ( Figure 6 The fused feature map is the output of the feature fusion algorithm after preprocessing, containing the dashed bounding boxes of the two Conv and one Sigmoid. Then with f DFA_2 By splicing After that Input to feature fusion algorithm, feature fusion algorithm output For f DFA_3Perform the same operation to obtain the preprocessed fused feature map f. s .

[0103] S502, the preprocessed fused feature map f s Applying the Transformer algorithm with several axes, the final feature map f is obtained. ost .

[0104] The feature fusion algorithm of S501 and the Transformer algorithm of S502 constitute Multi-Scale Fusion Attention (MSFA). The specific process of the feature fusion algorithm is as follows:

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] in, Features learned at each scale through convolutional blocks and the Sigmoid activation function. For scale-wise fusion features, f s These are the features obtained in the first part.

[0111] The specific process of the Transformer algorithm is as follows:

[0112]

[0113]

[0114]

[0115] in, The features obtained from the first part are standardized and used as input features for the second part. Features are defined by three axes. It features triaxial fusion and MHSA is a multi-head self-attention layer. These are the final learned features. MSFA can be simply described as follows: First, by processing multi-scale features f... DFA_1 ,f DFA_2 ,f DFA_3Scale-wise fusion is performed by first passing low-scale features through convolutional layers and a sigmoid activation function for channel transformation and weight learning (the channels can be RGB). Then, these features are concatenated with high-scale features, and the channel transformation and weight learning processes are repeated. Finally, the multi-scale fused feature f is obtained. s Then, spatial features are learned through an axial spatial Transformer block (OST). Specifically, the OST first standardizes the input features using a LayerNorm layer. Next, transformations are performed along the three axes to obtain the triaxial features. (That is, features along the x, y, and z axes). The three axial features are modeled spatial relationships through MHSA layers and multiplied to obtain fused features. The fused features are then combined with... Adding them together yields the intermediate layer features. Finally, intermediate layer features The final feature f is obtained by learning through LayerNorm and MLP layers and adding it to itself. ost .

[0116] S503, apply a 1x1 convolution to the initial feature map at the smallest scale to obtain the first convolution result Q.

[0117] like Figure 2 As shown, the initial feature map at the smallest scale is S3_ in The feature map obtained after three residual layers can also be represented by f. l express.

[0118] Q = Conv 1×1×1 (f l )

[0119] S504, for the final feature map f ost Applying a 1x1 convolution, we obtain the second convolution result K,V.

[0120] K, V = Conv 1×1×1 (f ost )

[0121] S505, based on the first convolution result and the second convolution result, obtain the correlation position between the final feature map and the initial feature map at the smallest scale:

[0122] Posltion=(Q×K)+(Q×(r D +r H +r W ))

[0123] S506, Based on the correlation, the second convolution result, and the final feature map, the classification feature f of each of the final feature maps is obtained. c .

[0124]

[0125] S507, perform global average pooling, convolution, and ReLU activation function on the classification features to obtain brain region feature maps f. roi :

[0126] f roi =Relu(Conv(Relu(Conv(GAP(f)) c )))))

[0127] In the above formula, It goes through three Convs 1×1×1 (·) yields vector values ​​representing different parts of the input features, where Q represents feature f. l The vector values, K, V represent features. The vector value of r. D ,r H ,r W It is a relative positional encoding calculated based on the input feature dimensions.

[0128] S503 to S507 calculate brain region feature maps and classification features based on ROIs. ROI attention (ROIAttention) learns discriminative brain regions related to the disease. The ROI process is as follows:

[0129] like Figure 7 As shown, the feature f is first calculated through a 1×1×1 convolution. l and f ost The vector values ​​are used to obtain Q, K, and V. Next, Q is multiplied by K and the sum of the three relative position codes, and then summed to obtain Position, which represents the correlation between the two vectors. Afterwards, Position is processed through a Softmax activation function to generate corresponding attention weights, which are then multiplied by V and summed with f. ost The summation yields the final classification feature f. c f c The final ROI feature values ​​are obtained by global average pooling (GAP) and two repeated ROI feature learning blocks consisting of convolutional layers and ReLU activation functions.

[0130] S508, based on the feature maps and classification features of each brain region, the processing results for the original brain image are obtained.

[0131] Brain region feature maps are feature maps corresponding to ninety brain regions in the human brain. In this embodiment, these feature maps can be determined using MRI and PET images. These ninety brain region feature maps can assist doctors in assessing the severity of Alzheimer's disease (AD) in patients. Figure 2 As shown, categorical features are used to help doctors determine which of the three types of patients a patient has: AD (Alzheimer's disease), MCI (mild cognitive impairment), or CN (non-cognitive disorder).

[0132] In one embodiment, the ResNet18 residual network structure, the MSCB convolutional algorithm, the DFA dynamic fusion attention algorithm, the MSFA multi-scale fusion attention algorithm, and the ROI in steps S100 to S500 constitute a multi-scale AD recognition framework. The multi-scale AD recognition framework in steps S100 to S500 is a recognition framework trained using training data. Before training, the training data needs to be collected and preprocessed:

[0133] 1447 participants were screened from three databases of the Alzheimer's Disease Neuroimaging Initiative (ADNI) and the Australian Flagship Study of Aging Imaging, Biomarkers and Lifestyles (AIBL), as shown in Table 1. They were divided into three groups: 356 AD participants, 598 NC participants, and 493 MCI participants. AD was the Alzheimer's group, MCI was the mild cognitive impairment group, and NC was the healthy control group. All participants screened in this embodiment had 3T MRI and FDG-PET image data and clinical scores. Statistical Parametric Mapping (SPM) and CAT were used to preprocess the MRI and PET data. CAT was used to preprocess the MRI data, such as... Figure 8 and Figure 9 The steps shown are: 1) noise filtering; 2) resampling; 3) bias field correction; 4) skull stripping; 5) linear registration; 6) tissue segmentation; 7) spatial normalization; 8) clipping boundaries. Compared to MRI, PET data preprocessing involves fewer steps, mainly including: 1) registration; 2) spatial normalization; 3) skull stripping; 4) smoothing.

[0134] Table 1

[0135]

[0136] In summary, this invention first divides an original brain image into multiple sub-images at various scales based on a residual network structure, and simultaneously extracts an initial feature map for each sub-image. Then, a convolutional algorithm is used to further extract local features (denoted as individual feature maps) from each initial feature map. Finally, the individual feature maps at multiple scales are fused to obtain a common feature map. Because the initial feature maps at multiple scales obtained using the residual network structure cover the global features of the original brain image, while the individual feature maps obtained using convolution cover local features, the fused common feature map contains both global and local features of the original brain image. Therefore, the fused feature map based on the common feature map more accurately reflects the features of the original brain image.

[0137] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 10 As shown, the terminal device includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a brain image processing method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The temperature sensor is pre-installed inside the terminal device to detect the operating temperature of the internal components.

[0138] Those skilled in the art will understand that Figure 10 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0139] In one embodiment, a terminal device is provided, comprising a memory, a processor, and a brain image processing program stored in the memory and executable on the processor. When the processor executes the brain image processing program, it implements the following operation instructions:

[0140] A residual network structure is applied to the original brain image to obtain initial feature maps corresponding to several scales of the original brain image;

[0141] Apply a convolution algorithm to the initial feature maps corresponding to several scales to obtain the features of the initial feature map at each scale output by the convolution algorithm, which are denoted as individual feature maps;

[0142] The individual feature maps at each scale are fused to obtain a common feature map;

[0143] A dynamic fusion attention algorithm is applied to the common feature map and the individual feature maps at each scale to obtain the feature map after fusing the information of each channel of the common feature map, which is denoted as the fused feature map at each scale.

[0144] Based on the fused feature maps at various scales, processing results are obtained for the original brain image, and these processing results are used to predict the degree of brain lesions in the patient corresponding to the original brain image.

[0145] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A brain image processing method, comprising a ResNet18 residual network structure, a MSCB convolutional algorithm, a DFA dynamic fusion attention algorithm, a MSFA multi-scale fusion attention algorithm, and ROIs to form a multi-scale AD recognition framework, characterized in that, include: A residual network structure is applied to the original brain image to obtain initial feature maps corresponding to several scales of the original brain image; Apply a convolution algorithm to the initial feature maps corresponding to several scales to obtain the features of the initial feature map at each scale output by the convolution algorithm, which are denoted as individual feature maps; The individual feature maps at each scale are fused to obtain a common feature map; A dynamic fusion attention algorithm is applied to the common feature map and the individual feature maps at each scale to obtain the feature map after fusing the information of each channel of the common feature map, which is denoted as the fused feature map at each scale. Based on the fusion feature maps at various scales, processing results are obtained for the original brain image, and the processing results are used to predict the degree of brain lesions in the patient corresponding to the original brain image; The dynamic fusion attention algorithm includes: The channel attention branch is used to extract the feature maps of each channel from the common spliced ​​feature map, wherein the common spliced ​​feature map is the feature map obtained by splicing the common feature map and the individual feature map; A spatial attention branch is used to extract global features from the common feature maps. A voxel attention branch is used to extract voxel points from each channel of the common spliced ​​feature map; The fusion module is used to fuse the feature maps of each channel, the global features, and the voxels to obtain fused feature maps at each scale. The processing result for the original brain image is obtained based on the fused feature maps at various scales. The processing result is used to predict the degree of brain lesions in the patient corresponding to the original brain image, including: The fused feature maps at each scale are applied to the feature fusion algorithm scale by scale to obtain the preprocessed fused feature maps; The preprocessed fused feature map is then subjected to a Transformer algorithm with several axes to obtain the final feature map. The ROI attention algorithm is applied to the final feature map and the initial feature map at the minimum scale to obtain the brain region feature maps and classification features output by the ROI attention algorithm. Based on the feature maps and classification features of each brain region, the processing results for the original brain image are obtained.

2. The brain image processing method as described in claim 1, characterized in that, The process of applying a residual network structure to the original brain image to obtain initial feature maps corresponding to several scales of the original brain image includes: Based on the original brain images, the original magnetic resonance imaging (MRI) images and the original positron emission tomography (PET) images of the original brain images are obtained; The residual network structure is applied to the original magnetic resonance image and the original positron emission tomography image, respectively, to obtain feature maps of the original magnetic resonance image and the original positron emission tomography image at various scales output by each residual layer in the residual network structure. The initial feature map is obtained by stitching together the feature maps at various scales of the original magnetic resonance image and the feature maps at various scales of the original positron emission tomography image.

3. The brain image processing method as described in claim 1, characterized in that, Each residual layer of the residual network structure includes: The first high-efficiency convolution module has an input image. The training acceleration convergence module has its input end connected to the output end of the efficient convolution module, and is used to accelerate the training of the residual network. The first piecewise function module has its input end connected to the output end of the training acceleration convergence module; The second high-efficiency convolution module has its input connected to the output of the first piecewise function module. The second piecewise function module has an input end that is used to input the same image as the first efficient convolution module, and the input end is connected to the output end of the second efficient convolution module. The output end is connected to the input end of the convolution algorithm.

4. The brain image processing method as described in claim 1, characterized in that, The convolution algorithm is applied to the initial feature maps corresponding to several scales to obtain the features of the initial feature map at each scale output by the convolution algorithm, denoted as the individual feature map, including: A first initial feature map, a second initial feature map, and a third initial feature map are determined from the initial feature maps corresponding to several scales, wherein the scale of the first initial feature map, the scale of the second initial feature map, and the scale of the third initial feature map decrease in sequence. The first convolutional branch, second convolutional branch, third convolutional branch, and fourth convolutional branch constituting the convolutional algorithm are determined. Each of the first, second, third, and fourth convolutional branches includes a pre-convolutional layer and a post-convolutional layer. The size of the pre-convolutional layer of the first, second, third, and fourth convolutional branches increases sequentially. The size of the post-convolutional layer of the first, second, third, and fourth convolutional branches is the same. The first convolutional branch and the second convolutional branch are applied to the third initial feature map respectively, and the feature maps output by the first convolutional branch and the feature maps output by the second convolutional branch are concatenated to obtain the individual feature map of the third initial feature map; By concatenating the second initial feature map and the upsampled third initial feature map, a concatenated feature map corresponding to the second initial feature map is obtained; The first convolutional branch, the second convolutional branch, and the third convolutional branch are applied to the concatenated feature map corresponding to the second initial feature map, and the feature maps output by the first convolutional branch, the second convolutional branch, and the third convolutional branch are concatenated to obtain the personalized feature map of the second initial feature map. By concatenating the first initial feature map and the upsampled second initial feature map, a concatenated feature map corresponding to the first initial feature map is obtained. Apply the first convolutional branch, the second convolutional branch, the third convolutional branch, and the fourth convolutional branch to the spliced ​​feature map corresponding to the first initial feature map, and splice the feature maps output by the first convolutional branch, the second convolutional branch, the third convolutional branch, and the fourth convolutional branch to obtain the personalized feature map of the first initial feature map.

5. The brain image processing method as described in claim 1, characterized in that, The ROI attention algorithm is applied to the final feature map and the initial feature map at the minimum scale to obtain the brain region feature maps and classification features output by the ROI attention algorithm, including: Apply a 1x1 convolution to the initial feature map at the smallest scale to obtain the first convolution result; Applying a 1x1 convolution to the final feature map yields the second convolution result; Based on the first convolution result and the second convolution result, the correlation between the final feature map and the initial feature map at the minimum scale is obtained; Based on the correlation, the second convolution result, and the final feature map, the classification features of each of the final feature maps are obtained; Global average pooling, convolution, and ReLU activation function are applied to the classification features to obtain brain region feature maps.

6. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a brain image processing program stored in the memory and executable on the processor. When the processor executes the brain image processing program, it implements the steps of the brain image processing method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a brain image processing program, which, when executed by a processor, implements the steps of the brain image processing method as described in any one of claims 1-5.

Images