A Brain Image Classification and Brain Cognitive Score Prediction Method Based on Multi-Task Learning

Abstract

This application is applicable to the field of image classification technology, and provides a method for brain image classification and brain cognitive score prediction based on multi-task learning, including obtaining a training brain magnetic resonance imaging (MRI) image set; constructing a brain image classification and brain cognitive score prediction model; inputting the training brain MRI image set into the model to obtain brain image classification results, brain cognitive score prediction values, and interaction features; respectively constructing first, second, third, and fourth loss functions; determining the loss value of the brain image classification and brain cognitive score prediction model according to the above loss functions; using the loss value to perform backpropagation on the model to obtain a trained brain image classification and brain cognitive score prediction model; inputting the brain MRI image to be recognized into the trained brain image classification and brain cognitive score prediction model to obtain brain image classification results and brain cognitive score prediction values. This application can improve the accuracy of brain image classification and brain cognitive score prediction.

Description

Technical Field

[0001] This application belongs to the technical field of image classification, and particularly relates to a brain image classification and brain cognitive score prediction method based on multi-task learning. Background Art

[0002] Brain image classification is an important part of the medical imaging field and has important applications in fields such as brain age assessment, aging progress judgment, and disease identification. Due to its ability to non-invasively and finely image the brain, neuroimaging techniques such as magnetic resonance imaging (MRI) have been widely used in the field of brain image classification and have become important data carriers for brain image classification. Early MRI-based brain image classification mainly relied on handcrafted features. However, handcrafted features sometimes cannot identify subtle changes within the brain region. Therefore, more research has used convolutional neural networks (CNNs) as the main framework for brain image classification. Some 2D slice-based brain image classification methods have borrowed classical CNN structures, such as Deep Residual Network (ResNet) and Visual Geometry Group Network (VGGNet). To further utilize spatial information, some research has selected image patches in specific regions as the input of 3D CNNs through prior knowledge, and some research has designed new network structures to adapt to the entire brain image as the input. For subtle structural changes, the brain structural changes that MRI can reflect are still not obvious enough to accurately classify brain images.

[0003] From existing clinical experience, brain image classification not only depends on MRI but is also closely related to some indicators related to brain cognitive functions, such as the Mini-mental State Examination (MMSE) and Clinical Dementia Rating (CDR) that measure cognitive functions. Among them, key brain structures such as the hippocampus have been proven to be highly correlated with cognitive functions, and the possibility of predicting brain cognitive scores through MRI images has been verified. Similar to brain image classification methods, prediction methods for brain cognitive scores have also successively emerged, including prediction methods based on traditional machine learning, prediction methods based on CNNs, and prediction methods based on other deep learning technologies such as weakly supervised learning. However, the vast majority of existing methods do not consider the correlation between brain image classification and brain cognitive score prediction. Summary of the Invention

[0004] The present application provides a method for brain image classification and brain cognitive score prediction based on multi-task learning, which can solve the problems of low accuracy of current brain image classification and poor prediction accuracy of brain cognitive scores.

[0005] In a first aspect, the present application provides a method for brain image classification and brain cognitive score prediction based on multi-task learning, including:

[0006] Obtain a training set of brain magnetic resonance images; the training set of brain magnetic resonance images includes N brain magnetic resonance image samples;

[0007] Construct a brain image classification and brain cognitive score prediction model; the brain image classification and brain cognitive score prediction model includes a first feature extraction module for extracting brain image classification features, a second feature extraction module for extracting brain cognitive score features, a feature interaction module for interacting the brain image classification features and the brain cognitive score features, an identification module for generating brain image classification results, and a prediction module for generating brain cognitive score prediction results. The input ends of the first feature extraction module and the second feature extraction module receive brain magnetic resonance images. The first output end of the first feature extraction module outputs brain image classification features. The first output end of the second feature extraction module outputs brain cognitive score features. The first input end of the feature interaction module is connected to the second output end of the first feature extraction module. The second input end of the feature interaction module is connected to the second output end of the second feature extraction module. The first output end of the feature interaction module is connected to the first input end of the identification module. The second input end of the identification module is connected to the first output end of the first feature extraction module. The output end of the identification module outputs brain image classification results. The second output end of the feature interaction module is connected to the first input end of the prediction module. The second input end of the prediction module is connected to the first output end of the second feature extraction module. The output end of the prediction module outputs brain cognitive score prediction values;

[0008] Input the brain magnetic resonance images in the training set of brain magnetic resonance images into the brain image classification and brain cognitive score prediction model one by one to obtain N brain image classification results, N brain cognitive score prediction values, and N interaction features;

[0009] Construct a first loss function according to the N brain image classification results;

[0010] Construct a second loss function according to the N brain cognitive score prediction values;

[0011] Construct a third loss function according to the N interaction features;

[0012] Construct a fourth loss function according to the actual brain cognitive score value of each brain magnetic resonance image sample in the N brain magnetic resonance image samples obtained in advance;

[0013] Determine the loss value of the brain image classification and brain cognitive score prediction model according to the first loss function, the second loss function, the third loss function, and the fourth loss function;

[0014] Use the loss value to perform backpropagation on the brain image classification and brain cognitive score prediction model until the brain image classification and brain cognitive score prediction model converges, and obtain the trained brain image classification and brain cognitive score prediction model;

[0015] Input the brain magnetic resonance image to be classified into the trained brain image classification and brain cognitive score prediction model to obtain the brain image classification result and the brain cognitive score prediction value of the brain magnetic resonance image to be classified.

[0016] Optionally, input the brain magnetic resonance images in the training brain magnetic resonance image set into the brain image classification and brain cognitive score prediction model one by one to obtain N brain image classification results, N brain cognitive score prediction values, and N interaction features, including:

[0017] For each brain magnetic resonance image, perform the following operations:

[0018] Through the calculation formula

[0019] z M1，i = z 0，i - z M2，i

[0020] z M2，i =(CS M (z 0，i )⊙SS M (z 0，i ))×z 0，i

[0021] Obtain the brain image classification feature z M1，i ; where, z M1，i represents the brain image classification feature of the i-th brain magnetic resonance image, z M2，i represents the feature data output by the second output end of the first feature module, CS M represents the spatial attention network in the first feature module, SS M represents the channel attention network in the first feature module, z 0，i represents the original feature of the i-th brain magnetic resonance image, and the original feature is obtained by performing convolution and pooling operations on the brain magnetic resonance image, ⊙ represents the Kronecker product, × represents element-wise multiplication, and i = 1, 2,..., N;

[0022] Through the calculation formula

[0023] z A1，i = z 0，i - z A2，i

[0024] z A2，i =(CS A (z 0，i )⊙SS A (z 0，i ))×z 0，i

[0025] Obtain the brain cognitive score feature z A1，i ; where z A1，i represents the brain cognitive score feature of the i-th brain magnetic resonance image, and z A2，i represents the feature data output from the second output end of the second feature extraction module. CS A represents the spatial attention network in the second feature extraction module, and SS A represents the channel attention network in the second feature extraction module;

[0026] Through the calculation formula

[0027] z S，i =Conv 111 (z L，i )+z R，i

[0028] z L，i =(CS S1 (z A2，i )⊙SS S1 (z A2，i ))×z A2，i +(CS S2 (z M2，i )⊙SS S2 (z M2，i ))×z M2，i

[0029] z R，i =E(Concat(z A2，i ,z M2，i ,E C (z S，i，-1 )))

[0030] Obtain the interaction feature z S，i ; where z S，i represents the interaction feature of the i-th brain magnetic resonance image. CS S2 , CS S1 represents two different spatial attention networks in the feature interaction module, and SS S1 , SS S2 represents two different channel attention networks in the feature interaction module. Conv 111 represents a convolution with a convolution kernel of 1, and E(·) represents a feature extraction layer containing two convolutional layers. E C(·) represents a feature extraction layer containing a convolutional layer and a pooling layer, and Concat(·) represents the concatenation of features, z S，i，-1 represents the result output by the previous feature interaction module;

[0031] Based on the interaction feature z S，i and the brain image classification feature z M1，i , a brain image classification feature sequence is obtained, and the brain image classification feature sequence is subjected to a fully connected process to obtain the brain image classification result p i ; where p i represents the brain image classification result of the i-th brain magnetic resonance image, and the brain image classification result includes the probability that the image belongs to the corresponding category;

[0032] Based on the interaction feature z S，i and the brain cognitive score feature z A1，i , a brain cognitive score feature sequence is obtained, and the brain cognitive score feature sequence is subjected to a fully connected process to obtain the brain cognitive score prediction value where, represents the brain cognitive score prediction value of the i-th brain magnetic resonance image.

[0033] Optionally, the expression of the first loss function is as follows:

[0034]

[0035] where L CE represents the first loss function, and y i represents the true label of the i-th brain magnetic resonance image.

[0036] Optionally, the expression of the second loss function is as follows:

[0037]

[0038]

[0039] where L S represents the second loss function, and m i represents the actual brain cognitive score value of the i-th sample obtained in advance.

[0040] Optionally, the expression of the third loss function is as follows:

[0041]

[0042]

[0043] where L CIt represents the third loss function, where j = 1, 2,..., I, and I represents the total number of each module in the brain image classification and brain cognitive score prediction model. The number of modules in the brain image classification and brain cognitive score prediction model is the same. It represents the feature data output from the second output end of the j-th second feature extraction module. It represents the feature data output from the second output end of the j-th first feature extraction module. SA represents average pooling, SM represents max pooling, and ||·||1 represents the 1-norm of the vector.

[0044] Optionally, the expression of the fourth loss function is as follows:

[0045]

[0046]

[0047] Among them, L D represents the fourth loss function, F c represents the distribution function, and AU(F c ) represents the area under the distribution function F c . The gradient calculation expression of the fourth loss function is:

[0048]

[0049] Optionally, according to the first loss function, the second loss function, the third loss function, and the fourth loss function, determine the loss value of the brain image classification and brain cognitive score prediction model, including:

[0050] By using the calculation formula

[0051] L Total = L CE + L S + L C + L D

[0052] obtain the loss value L Total ; among them, L CE represents the first loss function, L S represents the second loss function, L C represents the third loss function, and L D represents the fourth loss function.

[0053] Optionally, use the loss value to perform backpropagation on the brain image classification and brain cognitive score prediction model until the brain image classification and brain cognitive score prediction model converges, and obtain the trained brain image classification and brain cognitive score prediction model, including:

[0054] Step i: Perform backpropagation on the brain image classification and brain cognitive score prediction model according to the loss value to obtain a new brain image classification and brain cognitive score prediction model;

[0055] Step ii: Calculate the new loss value corresponding to the new brain image classification and brain cognitive score prediction model, and determine whether the new loss value is less than or equal to a preset loss threshold;

[0056] Step iii: If the new loss value is less than or equal to the preset loss threshold, it is determined that the new brain image classification and brain cognitive score prediction model has converged, and the new brain image classification and brain cognitive score prediction model is used as the trained brain image classification and brain cognitive score prediction model; otherwise, it is determined that the new brain image classification and brain cognitive score prediction model has not converged, and the new brain image classification and brain cognitive score prediction model is used as the brain image classification and brain cognitive score prediction model in Step i, and return to execute Step i.

[0057] In a second aspect, the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned brain image classification and brain cognitive score prediction method based on multi-task learning is implemented.

[0058] In a third aspect, the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned brain image classification and brain cognitive score prediction method based on multi-task learning is implemented.

[0059] The above solution of the present application has the following beneficial effects:

[0060] The brain image classification and brain cognitive score prediction model provided by the present application effectively combines the correlation between the two tasks of brain cognitive score and brain image classification, greatly improving the accuracy of brain image classification and brain cognitive score prediction; the present application performs backpropagation on the brain image classification and brain cognitive score prediction model by constructing four different loss functions, making the classification and prediction effects of the brain image classification and brain cognitive score prediction model more accurate; the brain image classification and brain cognitive score prediction method provided by the present application can effectively improve the accuracy of brain magnetic resonance image classification and cognitive score prediction by classifying and predicting the brain magnetic resonance image to be recognized through an accurate brain image classification and brain cognitive score prediction method model.

[0061] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation part. Description of the Drawings

[0062] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0063] Figure 1 Flowchart of a brain image classification and brain cognitive score prediction method based on multi-task learning provided by an embodiment of the present application;

[0064] Figure 2 Structural schematic diagram of a brain image classification and brain cognitive score prediction model provided by an embodiment of the present application;

[0065] Figure 3 Distribution schematic diagram of brain cognitive scores provided by an embodiment of the present application;

[0066] Figure 4 Structural schematic diagram of a first feature extraction module provided by an embodiment of the present application;

[0067] Figure 5 Structural schematic diagram of a feature interaction module provided by an embodiment of the present application;

[0068] Figure 6 Structural schematic diagram of a terminal device provided by an embodiment of the present application. Detailed implementation manners

[0069] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0070] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0071] It should also be understood that the term "and / or" as used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0072] As used in the specification of this application and the appended claims, the term "if" may be construed, depending on the context, as "when", "once", "in response to determining", or "in response to detecting". Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be construed, depending on the context, to mean "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]".

[0073] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be construed as indicating or implying relative importance.

[0074] Reference to "one embodiment" or "some embodiments" or the like described in the specification of this application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0075] Aiming at the problems of low accuracy of current brain image classification and poor prediction accuracy of brain cognitive scores, this application provides a method for brain image classification and brain cognitive score prediction based on multi-task learning. The brain image classification and brain cognitive score prediction model provided by this application effectively combines the correlation between brain cognitive scores and brain image classification, greatly improving the accuracy of brain image classification and reducing the error of brain cognitive score prediction; this application performs backpropagation on the brain image classification and brain cognitive score prediction model by constructing four different loss functions, making the classification effect of the brain image classification and brain cognitive score prediction model more accurate; the brain image classification method provided by this application classifies the brain magnetic resonance image to be recognized and predicts the brain cognitive score through an accurate brain image classification method model, and can effectively improve the accuracy of brain magnetic resonance image classification and brain cognitive score prediction.

[0076] As Figure 1 shown, a method for brain image classification and brain cognitive score prediction based on multi-task learning provided by this application includes the following steps:

[0077] Step 111, obtaining a training brain magnetic resonance image set.

[0078] In one embodiment of the present application, the above-mentioned training brain magnetic resonance image set includes N brain magnetic resonance image samples.

[0079] It should be noted that in order to ensure that the brain magnetic resonance images in the training brain magnetic resonance image set can effectively train the brain image classification and brain cognitive score prediction model, it is also necessary to preprocess the obtained brain magnetic resonance image samples. The specific steps are as follows:

[0080] Step 111.1, brain magnetic resonance image registration.

[0081] Use the linear image registration tool (FLIRT, FMRIB’s Linear Image Registration Tool) in FSL (a set of tools for brain imaging data analysis and processing developed by the FMRIB Center at the University of Oxford, UK) to transform the nuclear magnetic resonance images (MRI, NMR Imaging) of N samples from the original space to the standard MNI space. MNI (Montreal Neurological Institute) is a standard coordinate system established by the Montreal Neurological Institute based on the MRI data of the normal human brain.

[0082] Step 111.2, skull stripping.

[0083] Due to the scarcity of medical images and the high-dimensional data brought by three-dimensional (3D) images, it is necessary to remove the task-irrelevant regions in the original data. In the embodiment of the present application, this step can be performed using the brain extraction tool (BET, Brain Extraction Tool) in FSL to strip the skull. This tool can retain the brain part in the brain magnetic resonance image data of the sample and remove the remaining organs, tissues, etc.

[0084] Step 111.3, data normalization.

[0085] Specifically, set the voxel mean and standard deviation of the brain magnetic resonance images of the samples to 0 and 1 respectively to obtain the preprocessed training brain magnetic resonance image set. Among them, the size of each brain magnetic resonance image is 181×217×181.

[0086] Step 112, construct a brain image classification and brain cognitive score prediction model.

[0087] As Figure 2 shown, the brain image classification and brain cognitive score prediction model includes a first feature extraction module for extracting brain image classification features (such as Figure 2as shown in 21), a second feature extraction module for extracting brain cognitive score features (such as Figure 2 as shown in 22), a feature interaction module for interacting between brain image classification features and brain cognitive score features (such as Figure 2 as shown in 23), an identification module for generating brain image classification results (such as Figure 2 as shown in 24), and a prediction module for generating brain cognitive score prediction results (such as Figure 2 as shown in 25). The input ends of the first feature extraction module and the second extraction module receive magnetic resonance images of the brain (magnetic resonance images of the brain obtained after preprocessing). The first output end of the first feature extraction module outputs brain image classification features. The first input end of the feature interaction module is connected to the second output end of the first feature extraction module. The second input end of the feature interaction module is connected to the second output end of the second feature extraction module. The first output end of the feature interaction module is connected to the first input end of the identification module. The second input end of the identification module is connected to the first output end of the first feature extraction module. The output end of the identification module outputs brain image classification results. The second output end of the feature interaction module is connected to the first input end of the prediction module. The second input end of the prediction module is connected to the first output end of the second feature extraction module. The output end of the prediction module outputs brain cognitive score prediction values.

[0088] It should be noted here that brain image classification features refer to features that can be used to classify brain images extracted from MRI through multi-layer convolution and pooling operations. For example: hippocampal features related to age changes and aging, amygdala features related to emotions and gender, and cerebral cortex features related to abnormal changes in structure. Brain cognitive score features refer to features for predicting brain cognitive scores of brain images extracted from MRI through multi-layer convolution and pooling operations. For example: hippocampal features related to memory cognition, parietal lobe features related to spatial cognition and visual cognition, and frontal lobe features related to attention and language cognition.

[0089] It should be understood that for the sake of convenience of description, only the case where the number of each module in the brain image classification and brain cognitive score prediction model is 1 is described here. In another embodiment of the present application, the number of each module in the brain image classification and brain cognitive score prediction model may be multiple. Correspondingly, these modules can be divided into multiple groups. The types, functions, numbers of the modules in each group, and the connection methods between the modules are the same as the above content. The difference between each group lies in the different network structure parameters in each module, such as the different numbers of convolution kernels.

[0090] It should be noted that in the embodiments of the present application, the above recognition module is composed of two fully connected layers. The first fully connected layer includes 32 neurons, and its input is the brain image classification feature and the interaction feature. The second fully connected layer includes 2 neurons, and its input is the output of the upper layer, and its output is the brain image classification result corresponding to the input brain magnetic resonance image.

[0091] The processing process of the recognition module is as follows:

[0092] First, the brain image classification feature and the interaction feature are concatenated, and then the concatenated result is linearly transformed, activated, and dimension-reduced through two fully connected layers, and the output is the brain image classification result corresponding to the input brain magnetic resonance image.

[0093] The above prediction module is composed of two fully connected layers. The first fully connected layer includes 32 neurons, and its input is the brain image classification feature and the interaction feature, and this interaction feature is the same as the interaction feature in the recognition module. The second fully connected layer includes 1 neuron, and its input is the output of the upper layer, and its output is the predicted value of the brain cognitive score corresponding to the input brain magnetic resonance image.

[0094] The processing process of the prediction module is as follows:

[0095] First, the brain cognitive score feature and the interaction feature are concatenated, and then the concatenated result is linearly transformed, activated, and dimension-reduced through two fully connected layers, and the output is the predicted value of the brain cognitive score corresponding to the input brain magnetic resonance image.

[0096] Step 113: Input the brain magnetic resonance images in the training brain magnetic resonance image set into the brain image classification and brain cognitive score prediction model one by one, and obtain N brain image classification results, N predicted values of brain cognitive scores, and N interaction features.

[0097] Step 114: Construct a first loss function according to the N brain image classification results.

[0098] Specifically, the expression of the first loss function is as follows:

[0099]

[0100] Among them, L CE represents the first loss function, y i represents the true label of the i-th brain magnetic resonance image, and the true label is the category to which the brain image belongs in the classification task. Exemplarily, the categories include elderly / young, male / female, disease / health. Exemplarily, the brain cognitive scores include the Mini-Mental State Examination, Clinical Dementia, and Montreal Cognitive Assessment Scale (MoCA).

[0101] It should be noted that the first loss function is constructed based on the classification results of N brain images to enable the brain image classification and brain cognitive score prediction model to better classify brain magnetic resonance images, thereby improving the accuracy of brain image classification.

[0102] Step 115: Construct a second loss function according to the N brain cognitive score prediction values.

[0103] Specifically, the expression of the second loss function is as follows:

[0104]

[0105]

[0106] where L S represents the second loss function, and m i represents the actual brain cognitive score value of the i-th sample obtained in advance.

[0107] It should be noted that the second loss function is constructed based on the N brain cognitive score prediction values to improve the accuracy of the brain image classification and brain cognitive score prediction model in predicting brain cognitive scores.

[0108] Step 116: Construct a third loss function according to the N interaction features.

[0109] Specifically, the expression of the third loss function is as follows:

[0110]

[0111]

[0112] where L C represents the third loss function, j = 1, 2,..., I, where I represents the total number of each module in the brain image classification and brain cognitive score prediction model, and the number of each module in the brain image classification and brain cognitive score prediction model is the same. represents the feature data output from the second output end of the j-th second feature extraction module. represents the feature data output from the second output end of the j-th first feature extraction module, SA represents average pooling, SM represents max pooling, and ||·||1 represents the 1-norm of the vector.

[0113] It should be noted that, in this case, the third loss function is constructed based on N interaction features. This is because when there are multiple feature interaction modules in the brain image classification and brain cognitive score prediction model, feature interaction between the brain image classification features and the brain cognitive score features through multiple feature interaction modules may cause the risk of overfitting in the brain image classification and brain cognitive score prediction model. Since the brain image classification and brain cognitive score prediction model provided in this application is effective for both brain magnetic resonance image classification and brain cognitive score prediction, based on prior knowledge, the interaction features obtained from the brain image classification features and the brain cognitive score features often have a consistent feature distribution. Therefore, this application constructs the third loss function to constrain the distribution difference between the interaction features.

[0114] Step 117: Construct a fourth loss function according to the actual brain cognitive score value of each brain magnetic resonance image sample in the pre-acquired N brain magnetic resonance image samples.

[0115] Specifically, the expression of the fourth loss function is as follows:

[0116]

[0117]

[0118] where L D represents the fourth loss function, F c represents the distribution function, AU(F c ) represents the area under the distribution function F c . The gradient calculation expression of the fourth loss function is:

[0119]

[0120] It should be noted that, from the perspective of the population, one manifestation that the brain cognitive score is related to the category of the brain image is that the category and the distribution correspond to each other, that is, the brain cognitive score follows a specific distribution under a specific category. In order to utilize the distribution information of the brain cognitive score and achieve the fitting of the distribution, this application pre-performs the brain cognitive score on the samples and constructs the MMSE distribution diagram as shown in Figure 3 . According to the MMSE distribution diagram, discrete points with non-zero function values in the original distribution are selected, and the distribution function (describing the correspondence between the brain cognitive score and the number of people belonging to category c) is constructed by connecting these discrete points, so as to construct the fourth loss function.

[0121] It is worth mentioning that, in an embodiment of this application, when the MMSE predicted value is not in the set A c , the set A c represents the interval corresponding to the brain cognitive score, and the gradient of backpropagation is 0, which makes the distribution loss (the fourth loss function) unable to perform backpropagation at this time. Therefore, according to the rules, the present application will be modified to a fixed value λF c (min(A c )) and λF c (max(A c )) to ensure stable training of the distribution loss.

[0122] Step 118, determine the loss value of the brain image classification and brain cognitive score prediction model according to the first loss function, the second loss function, the third loss function, and the fourth loss function.

[0123] Specifically, through the calculation formula

[0124] L Total = L CE + L S + L C + L D

[0125] obtain the loss value; where L CE represents the first loss function, L S represents the second loss function, L C represents the third loss function, L D represents the fourth loss function.

[0126] Step 119, perform backpropagation on the brain image classification and brain cognitive score prediction model using the loss value until the brain image classification and brain cognitive score prediction model converges, and obtain the trained brain image classification and brain cognitive score prediction model.

[0127] Specifically, perform the following steps:

[0128] Step i, perform backpropagation on the brain image classification and brain cognitive score prediction model according to the loss value to obtain a new brain image classification and brain cognitive score prediction model.

[0129] Step ii, calculate the new loss value corresponding to the new brain image classification and brain cognitive score prediction model, and determine whether the new loss value is less than or equal to a preset loss threshold.

[0130] Exemplarily, in an embodiment of the present application, the loss threshold can be preset to 0.05.

[0131] Step iii, if the new loss value is less than or equal to the preset loss threshold, it is determined that the new brain image classification and brain cognitive score prediction model has converged, and the new brain image classification and brain cognitive score prediction model is used as the trained brain image classification and brain cognitive score prediction model; otherwise, it is determined that the new brain image classification and brain cognitive score prediction model has not converged, and the new brain image classification and brain cognitive score prediction model is used as the brain image classification and brain cognitive score prediction model in Step i, and Step i is executed again by returning.

[0132] Step 120, input the brain magnetic resonance image to be classified into the trained brain image classification and brain cognitive score prediction model to obtain the brain image classification result and the brain cognitive score prediction value of the brain magnetic resonance image to be classified.

[0133] The above-mentioned brain magnetic resonance image to be recognized refers to the brain magnetic resonance image that needs to be classified and the brain cognitive score predicted.

[0134] Next, an exemplary description of the process of Step 113 (inputting the brain magnetic resonance images in the training brain magnetic resonance image set into the brain image classification and brain cognitive score prediction model one by one to obtain N brain image classification results, N brain cognitive score prediction values, and N interaction features) will be given.

[0135] For each brain magnetic resonance image, the following operations are performed:

[0136] Step 113.1, through the calculation formula

[0137] z M1，i = z 0，i - z M2，i

[0138] z M2，i = (CS M (z 0，i ) ⊙ SS M (z 0，i )) × z 0，i

[0139] The brain image classification feature z M1，i is obtained.

[0140] Among them, z M1，i represents the brain image classification feature of the i-th brain magnetic resonance image, z M2，i represents the feature data output from the second output end of the first feature module, CS M represents the spatial attention network in the first feature module, SS M represents the channel attention network in the first feature module, z 0，iDenote the original features of the \(i\)-th brain magnetic resonance image, which are obtained by performing convolution and pooling operations on the brain magnetic resonance image. \(\odot\) represents the Kronecker product, and \(\times\) represents element-wise multiplication, where \(i = 1, 2,\cdots, N\).

[0141] As Figure 4 shown, in the embodiment of the present application, the above first feature extraction module includes a spatial attention network (shown as 41 in the figure) and a channel attention network (shown as 42 in the figure), where CA and CM represent global average pooling and global max pooling in the channel dimension, and SA and SM represent global average pooling and global max pooling in the spatial dimension. The present application uses global average pooling and max pooling to obtain information in different spaces and channels. Then, multiple fully connected layers and convolutional layers are respectively used to obtain attention weights in the channel and space. Specifically, the processing process of the spatial attention network is:

[0142] CS M (z 0，i ) = E CS (Concat(CA(z 0，i ), CM(z 0，i )))

[0143] where E CS represents a multi-layer perceptron containing two fully connected layers.

[0144] The processing process of the channel attention network is:

[0145] SS M (z 0，i ) = E SS (Concat(SA(z 0，i ), SM(z 0，i )))

[0146] where E SS represents an encoding layer containing two convolutional layers.

[0147] The first feature extraction module multiplies the attention weights in the channel and spatial dimensions element-wise to obtain an attention matrix of the interaction features. Then, this attention matrix is used to select the interaction features.

[0148] Step 113.2, through the calculation formula

[0149] z A1，i = z 0，i - z A2，i

[0150] z A2，i = (CS A (z 0，i ) \(\odot\) SS A (z0，i )) × z 0，i

[0151] Obtain the brain cognitive score feature z A1，i .

[0152] Among them, z A1，i represents the brain cognitive score feature of the i-th brain magnetic resonance image, and z A2，i represents the feature data output from the second output end of the second feature extraction module, CS A represents the spatial attention network in the second feature extraction module, and SS A represents the channel attention network in the second feature extraction module.

[0153] Since in the embodiments of the present application, the network structure of the second feature extraction module is the same as that of the brain image classification feature extraction module, except for the difference in parameter values, the network structure of the first feature extraction module can be specifically referred to and will not be elaborated here.

[0154] Step 113.3, through the calculation formula

[0155] z S，i = Conv 111( z L，i ) + z R，i

[0156] z L，i = (CS S1 (z A2，i ) ⊙ SS S1 (z A2，i )) × z A2，i + (CS S2 (z M2，i ) ⊙ SS S2 (z M2，i )) × z M2，i

[0157] z R，i = E(Concat(z A2，i , z M2，i , E C (z S，i，-1 )))

[0158] Obtain the interaction feature z S，i .

[0159] Among them, z S，i represents the interaction feature of the i-th brain magnetic resonance image, CS S2 , CS S1 represents two different spatial attention networks in the feature interaction module, and SS S1 , SS S2represent two different channel attention networks in the feature interaction module, Conv 111 represents a convolution with a kernel size of 1, E(·) represents a feature extraction layer containing two convolutional layers, E C (·) represents a feature extraction layer containing one convolutional layer and one pooling layer, Concat(·) represents the concatenation of features, z S，i-1 represents the result output by the previous feature interaction module.

[0160] In the embodiments of the present application, the network structure of the above-mentioned feature interaction module is as Figure 5 shown. Brain image classification and brain cognitive score prediction both have more concerned brain regions. Therefore, it is necessary to generate specific weights for feature interaction for different brain regions. At the same time, due to the possible representational differences between channels, a spatial attention network (as shown in 52 and 53 in Figure 5 ) and a channel attention network (as shown in 51 and 54 in Figure 5 ) are also used in the feature interaction module to generate different weights for different spatial positions and channels. In addition, since one of the feasibilities of using linear transformation for interaction is that there is a spatial correspondence between features and the original image region, because convolution and pooling satisfy spatial position invariance. However, since the convolution kernel aggregates feature information from multiple spatial positions, after passing through multiple convolutional and pooling layers, this correspondence may not be very precise. Therefore, the present application additionally adds a path composed of convolutions (as shown in the dashed box in Figure 5 ) in the feature interaction. This path can not only provide useful information for feature interaction, but also correct the errors caused by possible spatial mismatches.

[0161] Step 113.4, according to the interaction feature z S，i and the brain image classification feature z M1，i , obtain a brain image classification feature sequence, and perform a fully connected process on the brain image classification feature sequence to obtain a brain image classification result p i .

[0162] Exemplarily, in the embodiments of the present application, the fully connected process means using two fully connected layers to process the i-th brain image classification feature sequence to obtain a brain image classification result

[0163] where p i represents the brain image classification result of the i-th brain magnetic resonance image. The brain image classification result includes the probability that the MRI image belongs to or does not belong to a certain specific category.

[0164] Step 113.5, according to the interaction feature z S，i and the brain cognitive score feature z A1，i, a brain cognitive score feature sequence is obtained, and the brain cognitive score feature sequence is subjected to a fully connected process to obtain a brain cognitive score prediction value

[0165] Wherein, represents the brain cognitive score prediction value of the i-th brain magnetic resonance image.

[0166] The process of performing a fully connected process on the brain cognitive score feature sequence is similar to the process of performing a fully connected process on the brain image classification feature sequence. For details, reference can be made to the process of performing a fully connected process on the brain image classification feature sequence.

[0167] As Figure 6 shown, an embodiment of the present application provides a terminal device. As Figure 6 shown, the terminal device D10 in this embodiment includes: at least one processor D100 ( Figure 6 only one processor is shown in ), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100. When the processor D100 executes the computer program D102, the steps in any of the above method embodiments are implemented.

[0168] Specifically, when the processor D100 executes the computer program D102, it first obtains a training set of brain magnetic resonance images, then constructs a brain image classification and brain cognitive score prediction model, and then inputs the brain magnetic resonance images in the training set of brain magnetic resonance images into the brain image classification and brain cognitive score prediction model one by one to obtain N brain image classification results, N brain cognitive score prediction values, and N interaction features. Subsequently, according to the N brain image classification results, a first loss function is constructed, then according to the N brain cognitive score prediction values, a second loss function is constructed, then according to the N interaction features, a third loss function is constructed, and then according to the actual brain cognitive score value of each sample in the pre-obtained N samples, a fourth loss function is constructed. Subsequently, according to the first loss function, the second loss function, the third loss function, and the fourth loss function, the loss value of the brain image classification and brain cognitive score prediction model is determined, and then the loss value is used to perform backpropagation on the brain image classification and brain cognitive score prediction model until the brain image classification and brain cognitive score prediction model converges, and a trained brain image classification and brain cognitive score prediction model is obtained. Finally, the brain magnetic resonance image to be recognized is input into the trained brain image classification and brain cognitive score prediction model to obtain the recognition result of the brain magnetic resonance image to be recognized. The brain image classification and brain cognitive score prediction model provided by this application effectively combines the influence of brain cognitive scores on brain image classification, greatly improving the accuracy of brain image classification; this application performs backpropagation on the brain image classification and brain cognitive score prediction model by constructing four different loss functions, making the recognition effect of the brain image classification and brain cognitive score prediction model more accurate; the brain image classification method provided by this application can effectively improve the accuracy of brain magnetic resonance image recognition by accurately identifying the brain magnetic resonance image to be recognized through the brain image classification method model.

[0169] The so-called processor D100 may be a central processing unit (CPU, Central Processing Unit), and this processor D100 may also be other general-purpose processors, digital signal processors (DSP, Digital Signal Processor), application-specific integrated circuits (ASIC, Application Specific Integrated Circuit), field-programmable gate arrays (FPGA, Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0170] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as the hard disk or memory of the terminal device D10. In some other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk equipped on the terminal device D10, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory D101 may also include both the internal storage unit and the external storage device of the terminal device D10. The memory D101 is used to store an operating system, application programs, a BootLoader, data, and other programs, such as the program code of the computer program. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0171] An embodiment of the present application also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments can be implemented.

[0172] An embodiment of the present application provides a computer program product, and when the computer program product runs on a terminal device, the terminal device is caused to execute the steps in the above method embodiments.

[0173] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of the present application, a computer program may be used to instruct relevant hardware to complete. The computer program may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps in the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the terminal device, a recording medium, a computer memory, a Read-Only Memory (ROM), a Random Access Memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium may not be an electrical carrier signal and a telecommunications signal.

[0174] In the above embodiments, the descriptions of the various embodiments each have their own emphasis. For parts not described in detail or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0175] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0176] In the embodiments provided in this application, it should be understood that the disclosed device / network device and method can be implemented in other ways. For example, the device / network device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0177] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] The method for brain image classification and brain cognitive score prediction based on multi-task learning provided by this application has the following advantages:

[0179] 1) This application proposes a multi-task feature interaction mechanism for joint learning of brain image classification and brain cognitive score prediction, which utilizes the correlation of the representations of different tasks. This mechanism can obtain interactive representations with cross-task generalization ability, and obtain shared representations that are effective for both tasks through the complementarity of the interactive representations from different tasks.

[0180] 2) This application proposes a multi-loss joint learning mechanism, which jointly optimizes the brain image classification and brain cognitive score prediction models by using multiple losses such as the smooth L1 loss for brain cognitive score prediction, the cross-entropy loss for brain image classification, the feature consistency loss for constraining feature interaction, and the distribution loss for fully obtaining the distribution information in the brain cognitive score.

[0181] The above are the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle described in the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A brain image classification and brain cognitive score prediction method based on multi-task learning, characterized in that, Including: Obtain a training brain magnetic resonance image set; the training brain magnetic resonance image set includes N brain magnetic resonance image samples; Construct a brain image classification and brain cognitive score prediction model; the brain image classification and brain cognitive score prediction model includes a first feature extraction module for extracting brain image classification features, a second feature extraction module for extracting brain cognitive score features, a feature interaction module for interacting the brain image classification features and the brain cognitive score features, an identification module for generating a brain image classification result, and a prediction module for generating a brain cognitive score prediction result. The input ends of the first feature extraction module and the second feature extraction module receive brain magnetic resonance images. The first output end of the first feature extraction module outputs brain image classification features. The first output end of the second feature extraction module outputs brain cognitive score features. The first input end of the feature interaction module is connected to the second output end of the first feature extraction module. The second input end of the feature interaction module is connected to the second output end of the second feature extraction module. The first output end of the feature interaction module is connected to the first input end of the identification module. The second input end of the identification module is connected to the first output end of the first feature extraction module. The output end of the identification module outputs a brain image classification result. The second output end of the feature interaction module is connected to the first input end of the prediction module. The second input end of the prediction module is connected to the first output end of the second feature extraction module. The output end of the prediction module outputs a brain cognitive score prediction value; Input the brain magnetic resonance images in the training brain magnetic resonance image set into the brain image classification and brain cognitive score prediction model one by one, and obtain N brain image classification results, N brain cognitive score prediction values, and N interaction features; Construct a first loss function according to the N brain image classification results; Construct a second loss function according to the N brain cognitive score prediction values; Construct a third loss function according to the N interaction features; Construct a fourth loss function according to the actual brain cognitive score value of each brain magnetic resonance image sample in the N brain magnetic resonance image samples obtained in advance; Determine the loss value of the brain image classification and brain cognitive score prediction model according to the first loss function, the second loss function, the third loss function, and the fourth loss function; Perform backpropagation on the brain image classification and brain cognitive score prediction model using the loss value until the brain image classification and brain cognitive score prediction model converges, and obtain a trained brain image classification and brain cognitive score prediction model; Input the brain magnetic resonance image to be classified into the trained brain image classification and brain cognitive score prediction model, and obtain the brain image classification result and the brain cognitive score prediction value of the brain magnetic resonance image to be classified.

2. The brain image classification and brain cognitive score prediction method according to claim 1, wherein The step of inputting the brain magnetic resonance images in the training brain magnetic resonance image set into the brain image classification and brain cognitive score prediction model one by one, and obtaining N brain image classification results, N brain cognitive score prediction values, and N interaction features includes: For each brain magnetic resonance image, perform the following operations: Through the calculation formula z M1，i = z 0，i -z M2，i z M2，i = (CS M (z 0，i ) ⊙ SS M (z 0，i )) × z 0，i Obtain the brain image classification feature z M1，i ; where, z M1，i represents the brain image classification feature of the i-th brain magnetic resonance image, and z M2，i represents the feature data output from the second output end of the first feature extraction module, CS M represents the spatial attention network in the first feature extraction module, SS M represents the channel attention network in the first feature extraction module, and z 0，i represents the original feature of the i-th brain magnetic resonance image, and the original feature is obtained by performing convolution and pooling operations on the brain magnetic resonance image. ⊙ represents the Kronecker product, and × represents the element-wise multiplication, where i = 1, 2,..., N; Through the calculation formula z A1，i = z 0,i -z A2,i z A2,i = (CS A (z 0,i ) ⊙ SS A (z 0,i )) × z 0,i Obtain the brain cognitive scoring feature z A1，i ; where z A1，i represents the brain cognitive scoring feature of the i-th brain magnetic resonance image, and z A2，i represents the feature data output from the second output end of the second feature extraction module, CS A represents the spatial attention network in the second feature extraction module, and SS A represents the channel attention network in the second feature extraction module; Through the calculation formula z S，i = Conv 111 (z L,i ) + z R,i z L，i = (CS S1 (z A2，i ) ⊙ SS S1 (z A2,i )) × z A2，i + (CS S2 (z M2,i ) ⊙ SS S2 (z M2,i )) × z M2,i z R，i = E(Concat(z A2，i , z M2，i , E C (z S，i，-1 ))) Obtain the interaction feature z S，i ; where z S，i represents the interaction feature of the i-th brain magnetic resonance image, CS S2 , CS S1 represents two different spatial attention networks in the feature interaction module, SS S1 , SS S2 represents two different channel attention networks in the feature interaction module, Conv 111 represents a convolution with a convolution kernel of 1, E(·) represents a feature extraction layer containing two convolutional layers, E C (·) represents a feature extraction layer containing one convolutional layer and one pooling layer, Concat(·) represents the concatenation of features, z S，i，-1 represents the result output by the previous feature interaction module; According to the interaction feature z S,i and the brain image classification feature z M1，i , a brain image classification feature sequence is obtained, and the brain image classification feature sequence is subjected to a fully connected process to obtain the brain image classification result p i ; where p i represents the brain image classification result of the i-th brain magnetic resonance image, and the brain image classification result includes the probability that the image belongs to the corresponding category; According to the interaction feature z S，i and the brain cognitive score feature z A1，i , the brain cognitive score feature sequence is obtained, and the brain cognitive score feature sequence is subjected to a fully connected process to obtain the brain cognitive score prediction value wherein represents the brain cognitive score prediction value of the i-th brain magnetic resonance image.

3. The brain image classification and brain cognitive score prediction method according to claim 2, characterized in that The expression of the first loss function is as follows: Among them, L CE represents the first loss function, and y i represents the true label of the i-th brain magnetic resonance image, and the true label is the category to which the brain magnetic resonance image belongs.

4. The brain image classification and brain cognitive score prediction method according to claim 2, characterized in that The expression of the second loss function is as follows: Among them, L S represents the second loss function, and m i represents the actual value of the brain cognitive score of the i-th pre-acquired brain magnetic resonance image sample.

5. The brain image classification and brain cognitive score prediction method according to claim 2, characterized in that The expression of the third loss function is as follows: Among them, L C represents the third loss function, j = 1, 2,..., I, where I represents the total number of each module in the brain image classification and brain cognitive score prediction model, and the number of each module in the brain image classification and brain cognitive score prediction model is the same. represents the feature data output from the second output end of the j-th second feature extraction module. represents the feature data output from the second output end of the j-th first feature extraction module, SA represents average pooling, SM represents max pooling, and ||·||1 represents the 1-norm of the vector.

6. The brain image classification and brain cognitive score prediction method according to claim 2, characterized in that The expression of the fourth loss function is as follows: Among them, L D represents the fourth loss function, and F c represents the distribution function. AU(F c ) represents the area under the distribution function F c . The gradient calculation expression of the fourth loss function is as follows:

7. The brain image classification and brain cognitive score prediction method according to claim 1, characterized in that Determining the loss value of the brain image classification and brain cognitive score prediction model according to the first loss function, the second loss function, the third loss function, and the fourth loss function includes: Through the calculation formula L Total = L CE + L S + L C + L D Obtain the loss value L Total ; where, L CE represents the first loss function, L S represents the second loss function, L C represents the third loss function, L D represents the fourth loss function.

8. The brain image classification and brain cognitive score prediction method according to claim 1, characterized in that Using the loss value to perform backpropagation on the brain image classification and brain cognitive score prediction model until the brain image classification and brain cognitive score prediction model converges, obtaining the trained brain image classification and brain cognitive score prediction model, includes: Step i, perform backpropagation on the brain image classification and brain cognitive score prediction model according to the loss value to obtain a new brain image classification and brain cognitive score prediction model; Step ii, calculate the new loss value corresponding to the new brain image classification and brain cognitive score prediction model, and determine whether the new loss value is less than or equal to a preset loss threshold; Step iii, if the new loss value is less than or equal to the preset loss threshold, determine that the new brain image classification and brain cognitive score prediction model has converged, and use the new brain image classification and brain cognitive score prediction model as the trained brain image classification and brain cognitive score prediction model; otherwise, determine that the new brain image classification and brain cognitive score prediction model has not converged, use the new brain image classification and brain cognitive score prediction model as the brain image classification and brain cognitive score prediction model in step i, and return to execute step i.

Images