Transform-based schizophrenia identification and interpretation system and method, equipment and medium

Abstract

The invention provides a schizophrenia identification and interpretation system and method based on Transform, equipment and a medium, according to the schizophrenia identification and interpretation system and method based on Transform, a region-of-interest image is cut from a functional magnetic resonance image, diagnosis analysis is carried out based on the region-of-interest image, a brain region most likely to be related to schizophrenia is selected for key analysis, and the schizophrenia identification and interpretation accuracy is improved. The calculation amount can be reduced, the attention of the model to key features can be improved, and the processing time of the deep learning model is shortened. According to the schizophrenia diagnosis method and the schizophrenia diagnosis system, the Transform neural network framework is introduced, and the objectivity and the accuracy of schizophrenia diagnosis are effectively improved by utilizing the strong sequence processing capability, the attention mechanism and the accurate capture of a complex mode of the Transform neural network framework. Besides, the feature map generated by the Vision Transform model is extracted and visualized, the interpretability of the model decision-making process is enhanced, a plurality of brain areas most related to the schizophrenia can be visualized, and the specificity table of the schizophrenia in the brain image is further disclosed.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a Transformer-based schizophrenia recognition and interpretation system and method, device, and medium. Background Art

[0002] Schizophrenia, as a chronic mental disorder with complex and yet to be fully understood etiology, is listed as one of the most serious mental health problems. Epidemiological studies have revealed that the global lifetime prevalence of the disease hovers between 3.8‰ and 8.4‰. Patients not only face a higher risk of comorbidities, but also often show violent tendencies such as emotional instability and increased aggression, which increases the economic and emotional burden on the family and poses a potential threat to social public safety. Therefore, achieving early and accurate diagnosis and effective intervention of schizophrenia is of vital importance to improving the cure rate and preventing the disease from worsening.

[0003] However, current clinical diagnosis mainly relies on doctors' experience, cognitive assessment scales, and comprehensive analysis of patients' and family medical histories. This process lacks objective and unified diagnostic criteria, leading to frequent missed and misdiagnoses. In this context, it is particularly urgent to explore and develop an objective and accurate diagnostic method to assist doctors in the precise diagnosis and treatment of schizophrenia. Summary of the invention

[0004] The present disclosure at least provides a Transformer-based schizophrenia recognition and interpretation system and method, device, and medium to provide an objective and unified diagnostic solution, improve the accuracy of schizophrenia diagnosis, and assist doctors in the accurate diagnosis and treatment of schizophrenia.

[0005] According to one aspect of the present disclosure, a Transformer-based schizophrenia recognition and interpretation system is provided, comprising:

[0006] An image acquisition module, used for acquiring a functional magnetic resonance image of the brain of a target subject;

[0007] An image preprocessing module, used for clipping a region of interest image from the functional magnetic resonance image based on preset position information of a brain region corresponding to schizophrenia, and inputting the region of interest image into an image recognition module;

[0008] A model processing module, used to use the trained Vision Transformer model to perform the following operations: crop the image of the region of interest into multiple image blocks of preset sizes, convert each image block into an embedding vector using a linear mapping, set a learnable classification tag, add a position code to each embedding vector, and form an embedding sequence using all the embedding vectors with added position codes and the classification tags; input the embedding sequence into the Transformer encoder of the Vision Transformer model, and use the Transformer encoder to extract global features and local features of the embedding vector; extract the classification tag from the output of the Transformer encoder, and input the extracted classification tag and the features output by the Transformer encoder into a multilayer perceptron; the multilayer perceptron uses the received classification tags and features to classify schizophrenia, and outputs a predicted category and the probability corresponding to each predicted category; wherein, during the training process, the Vision Transformer model is first pre-trained on a public data set, and then the pre-trained Vision Transformer model is trained using a training sample data set corresponding to schizophrenia;

[0009] an image classification decision module, configured to perform binary classification based on the predicted categories and the probabilities corresponding to each predicted category, determine a first probability that the target object suffers from schizophrenia and a second probability that the target object is in a healthy state, and determine that the target object suffers from schizophrenia when the first probability is higher than the second probability;

[0010] A feature visualization module is used to obtain a feature map output by the Transformer encoder, and to generate and display an attention feature map based on the feature map, wherein the attention feature map includes attention distribution information of each layer of the multi-head attention mechanism in the Transformer encoder;

[0011] The key area visualization module is used to obtain the feature map output by the Vision Transformer model, generate a heat map based on the feature map output by the Vision Transformer model, and display the heat map.

[0012] In a possible implementation, the model processing module is further used to generate a training sample data set corresponding to schizophrenia:

[0013] Acquire functional magnetic resonance imaging sample images of the brains of multiple sample subjects, and annotate each functional magnetic resonance imaging sample image respectively;

[0014] Based on the preset location information of the brain region corresponding to schizophrenia, a sample image of the region of interest is obtained by clipping from each functional magnetic resonance imaging sample image; wherein the annotation information of the sample image of the region of interest is the same as the annotation information of the corresponding functional magnetic resonance imaging sample image;

[0015] Flipping each of the sample images of the region of interest in the horizontal direction according to a first preset angle to obtain a horizontally enhanced sample image; wherein the annotation information of the horizontally enhanced sample image is the same as the annotation information of the corresponding functional magnetic resonance sample image;

[0016] Flipping each sample image of the region of interest in the vertical direction according to a second set angle to obtain a vertically enhanced sample image; wherein the annotation information of the vertically enhanced sample image is the same as the annotation information of the corresponding functional magnetic resonance sample image;

[0017] The training sample data set is formed by using the interest region sample image, the annotation information of the interest region sample image, the horizontal enhancement sample image, the annotation information of the horizontal enhancement sample image, the vertical enhancement sample image, and the annotation information of the vertical enhancement sample image.

[0018] In a possible implementation, the public data set includes a MathVista data set.

[0019] In a possible implementation, when the key area visualization module generates a heat map based on the feature map output by the VisionTransformer model, it is specifically used to:

[0020] The Grad-CAM technique is used to generate a heat map based on the feature map output by the Vision Transformer model.

[0021] In one possible implementation, the Transformer encoder includes a multi-head attention mechanism layer, a feedforward neural network layer, and a normalization layer.

[0022] In a possible implementation, the Vision Transformer model includes: an embedding layer, a Transformer encoder, and a multi-layer perceptron.

[0023] In a possible implementation, the size of the interest region image is 120 pixels x 140 pixels x 120 pixels.

[0024] According to another aspect of the present disclosure, a Transformer-based schizophrenia identification and interpretation method is provided, comprising:

[0025] Acquiring functional magnetic resonance images of the target subject's brain;

[0026] A region of interest image is obtained by clipping the image from the functional magnetic resonance image based on the preset location information of the brain region corresponding to schizophrenia;

[0027] The trained Vision Transformer model is used to perform the following operations: the image of the region of interest is cropped into multiple image blocks of a preset size, each image block is converted into an embedding vector using a linear mapping, a learnable classification tag is set, a position code is added to each embedding vector, and an embedding sequence is formed using all the embedding vectors with added position codes and the classification tags; the embedding sequence is input into the Transformer encoder of the Vision Transformer model, and the Transformer encoder is used to extract global features and local features of the embedding vector; the classification tag is extracted from the output of the Transformer encoder, and the extracted classification tag and the features output by the Transformer encoder are input into a multilayer perceptron; the multilayer perceptron uses the received classification tags and features to classify schizophrenia, and outputs the predicted category and the probability corresponding to each predicted category; wherein, during the training process, the Vision Transformer model is first pre-trained on a public dataset, and then the pre-trained Vision Transformer model is trained using a training sample dataset corresponding to schizophrenia;

[0028] Performing binary classification based on the predicted categories and the probabilities corresponding to each predicted category, determining a first probability that the target object suffers from schizophrenia and a second probability that the target object is in a healthy state, and determining that the target object suffers from schizophrenia when the first probability is higher than the second probability;

[0029] Obtain a feature map output by the Transformer encoder, and generate and display an attention feature map based on the feature map, wherein the attention feature map includes attention distribution information of each layer of the multi-head attention mechanism in the Transformer encoder;

[0030] A feature map output by the Vision Transformer model is obtained, and a heat map is generated based on the feature map output by the Vision Transformer model, and the heat map is displayed.

[0031] According to another aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0032] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned methods is implemented.

[0033] The Transformer-based schizophrenia recognition and interpretation system and method, device, and medium disclosed in the present invention utilize functional magnetic resonance images of the brain for processing and analysis. The functional magnetic resonance images are functional activities of the brain in a basic state captured without external stimulation, which can reflect the main part of the brain's energy consumption and provide an important basis for a deep understanding of the abnormal brain function of schizophrenia. The present invention clips the region of interest image from the functional magnetic resonance image and continues to perform diagnostic analysis based on the region of interest image. This step selects the brain area most likely to be related to schizophrenia for key analysis, which is based on features. This not only reduces the amount of calculation, but also improves the model's attention to key features, strikes a balance between image resolution and the computational complexity of the deep learning model (Vision Transformer model), and reduces the processing time of the deep learning model. Based on functional magnetic resonance imaging, this paper introduces the Transformer neural network framework, and uses its powerful sequence processing capabilities, attention mechanism, and accurate capture of complex patterns to bring new breakthroughs to the field of medical image processing. The Vision Transformer model can dig out hidden feature information that is difficult for the human eye to detect, providing strong support for the objective diagnosis of schizophrenia, and effectively improving the objectivity and accuracy of schizophrenia diagnosis. In addition, this paper extracts the feature map generated by the Vision Transformer model and performs visualization processing to enhance the interpretability of the model's decision-making process. At the same time, this paper also visualizes multiple brain regions that are most relevant to schizophrenia, further revealing the specific manifestations of schizophrenia in brain images.

[0034] In summary, the disclosed technical solution extracts key image features and enhances the interpretability of the model through the integration of deep learning and advanced image analysis technology, providing an objective and accurate auxiliary means for the early diagnosis of schizophrenia. This solution not only improves the accuracy and efficiency of diagnosis, but also improves the interpretability of deep learning-assisted diagnosis of schizophrenia, providing a new perspective for the study of the pathogenesis of schizophrenia and a scientific basis for the formulation of precise treatment strategies.

[0035] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0037] Figure 1 is a schematic diagram of the structure of the Transformer-based schizophrenia recognition and interpretation system according to the present disclosure;

[0038] Figure 2A is a schematic diagram of visualizing an attention feature map according to an embodiment of the present disclosure;

[0039] Figure 2B is a visualization diagram of key areas according to an embodiment of the present disclosure;

[0040] Figure 3 is a schematic diagram of interest region image enhancement according to an embodiment of the present disclosure;

[0041] Figure 4 is a schematic diagram of a Vision Transformer model according to an embodiment of the present disclosure;

[0042] Figure 5 is a flowchart of the Transformer-based schizophrenia identification and interpretation method according to the present disclosure;

[0043] Figure 6 is a schematic structural diagram of an electronic device according to the present disclosure. DETAILED DESCRIPTION

[0044] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0045] The present disclosure is based on the problem of lack of objective and unified diagnostic standards in the current diagnosis of schizophrenia, and the frequent occurrence of missed diagnosis and misdiagnosis. It provides a schizophrenia identification and interpretation system and method, equipment, and medium based on Transformer. The present disclosure uses functional magnetic resonance images of the brain for processing and analysis. The functional magnetic resonance images are functional activities of the brain in the basic state captured without external stimulation. They can reflect the main part of brain energy consumption and provide an important basis for in-depth understanding of brain dysfunction in schizophrenia. The present disclosure cuts out the region of interest image from the functional magnetic resonance image, and continues to perform diagnostic analysis based on the region of interest image. This step selects the brain area most likely to be related to schizophrenia for key analysis, which is based on features. This can not only reduce the amount of calculation, but also improve the model's attention to key features, strike a balance between image resolution and the computational complexity of the deep learning model (Vision Transformer model), and reduce the processing time of the deep learning model. Based on functional magnetic resonance imaging, this paper introduces the Transformer neural network framework, and uses its powerful sequence processing capabilities, attention mechanism, and accurate capture of complex patterns to bring new breakthroughs to the field of medical image processing. The Vision Transformer model can dig out hidden feature information that is difficult for the human eye to detect, providing strong support for the objective diagnosis of schizophrenia, and effectively improving the objectivity and accuracy of schizophrenia diagnosis. In addition, this paper extracts the feature map generated by the Vision Transformer model and performs visualization processing to enhance the interpretability of the model's decision-making process. At the same time, this paper also visualizes multiple brain regions that are most relevant to schizophrenia, further revealing the specific manifestations of schizophrenia in brain images.

[0046] The technical solution of the present disclosure is described below through specific embodiments.

[0047] like Figure 1 As shown, it is a structural diagram of the Transformer-based schizophrenia recognition and interpretation system of this embodiment. The executor of this embodiment is a system with data processing capabilities. Specifically, the system of this embodiment may include an image acquisition module 110, an image preprocessing module 120, a model processing module 130, an image classification decision module 140, a feature visualization module 150, and a key area visualization module 160.

[0048] The image acquisition module 110 is used to acquire a functional magnetic resonance image of the target object's brain.

[0049] Functional magnetic resonance images (fMRI images) are 3D images that are taken using a dedicated functional magnetic resonance imaging machine.

[0050] The above functional magnetic resonance images can also be replaced by images collected by magnetic resonance imaging (MRI) technology or images collected by resting-state functional magnetic resonance imaging (rs-fMRI). Among them, resting-state functional magnetic resonance imaging (rs-fMRI) has become an important tool for studying mental illnesses such as schizophrenia due to its non-invasiveness, high spatiotemporal resolution, and high sensitivity to brain connectivity. rs-fMRI can capture the functional activities of the brain in the basal state without external stimulation, reflecting the main part of the brain's energy consumption, and provides an important basis for a deep understanding of the abnormal brain function in schizophrenia.

[0051] The image preprocessing module 120 is used to clip a region of interest image from the functional magnetic resonance image based on preset position information of the brain region corresponding to schizophrenia, and input the region of interest image into the image recognition module.

[0052] Here, the fMRI image is preprocessed using a feature-based method to obtain the region of interest image. This not only reduces the amount of calculation, but also increases the Vision Transformer model's attention to key features.

[0053] The size of the above-mentioned interest region image may be 120 pixels x 140 pixels x 120 pixels.

[0054] The model processing module 130 is used to perform the following operations using the trained Vision Transformer model: crop the image of the region of interest into multiple image blocks of a preset size, convert each image block into an embedding vector using a linear mapping, set a learnable classification tag, add a position code to each embedding vector, and form an embedding sequence using all the embedding vectors with added position codes and the classification tags; input the embedding sequence into the Transformer encoder of the Vision Transformer model, and use the Transformer encoder to extract global features and local features from the embedding vector; extract the classification tag from the output of the Transformer encoder, and input the extracted classification tag and the features output by the Transformer encoder into a multilayer perceptron; the multilayer perceptron uses the received classification tags and features to classify schizophrenia, and outputs a predicted category and the probability corresponding to each predicted category; wherein, during the training process, the Vision Transformer model is first pre-trained on a public data set, and then the pre-trained Vision Transformer model is trained using a training sample data set corresponding to schizophrenia.

[0055] The above prediction categories include having schizophrenia and healthy status.

[0056] The above Transformer encoder may include but is not limited to: a multi-head attention mechanism layer, a feedforward neural network layer, and a normalization layer. The Vision Transformer model includes but is not limited to: an embedding layer, a Transformer encoder, and a multi-layer perceptron.

[0057] The above public dataset may include the MathVista dataset. The Vision Transformer (ViT) model is pre-trained on the public dataset MathVista to obtain a preliminary understanding of brain image features.

[0058] The training sample data set is formed by using functional magnetic resonance imaging sample images of the brain. The ViT model is trained on the training sample data set to learn the brain features unique to schizophrenia. During the training process, the training is iterated continuously until the number of iterations reaches the preset upper limit of the number of iterations to ensure that the ViT model is fully learned and converged.

[0059] The image classification decision module 140 is used to perform binary classification based on the predicted category and the probability corresponding to each predicted category, determine a first probability that the target object suffers from schizophrenia and a second probability that the target object is in a healthy state, and determine that the target object suffers from schizophrenia when the first probability is higher than the second probability.

[0060] When the first probability is lower than the second probability, the target object is determined to be in a healthy state. According to the predicted probability output by the ViT model (i.e., the first probability and the second probability), a binary classification strategy is adopted to select the category with a higher probability as the final judgment, and the image is classified as schizophrenia or healthy.

[0061] The feature visualization module 150 is used to obtain the feature map output by the Transformer encoder, and generate and display the attention feature map based on the feature map, wherein the attention feature map includes the attention distribution information of the multi-head attention mechanism of each layer in the Transformer encoder.

[0062] The feature visualization module 150 extracts the feature map output by the Transformer encoder of the ViT model and performs visualization processing to enhance the interpretability of the model decision process, and helps to observe the response of the ViT model to different inputs and understand how the ViT model extracts image features.

[0063] Figure 2AThe image shows the feature map generated by the Transformer encoder in the ViT model and visualized. The attention expansion method is used to average / maximize the attention features of each "head", where "head" refers to the head in the multi-head attention mechanism in the ViT model. The multiple small boxes arranged at the top of each layer in the figure represent the attention features of each "head" in the layer, showing the final attention distribution of each layer of the Transformer encoder. Figure 2A As shown in the figure, as the number of network layers increases, the attention feature map gradually changes from capturing low-level features such as edges and textures to focusing on high-level semantic features. Based on the visualization results of the attention feature map, a consistent pattern linking the control group Health and the schizophrenia group SZ was determined, and the schizophrenia image features can be captured at different levels, which can better understand the decision-making process of the ViT model and evaluate the model's strong reliability.

[0064] The key area visualization module 160 is used to obtain the feature map output by the Vision Transformer model, generate a heat map based on the feature map, and display the heat map.

[0065] The key area visualization module 160 can generate a heat map based on the feature map output by the Vision Transformer model using the Grad-CAM (Gradient-weighted ClassActivation Mapping) technology. By generating a heat map, it is possible to intuitively see which areas the model focuses on when making decisions. These areas are generally considered to be the parts that the model considers to be critical to the classification task. Here, the key areas that the ViT model focuses on are visualized, and the categories predicted by the ViT model are associated with the fMRI image features, which helps to understand which areas of the fMRI image the ViT model has a higher response to during the classification process, and further reveals the specific manifestations of schizophrenia in brain images.

[0066] Using Grad-CAM to visualize the output of the ViT model, we can get a heat map showing the image regions that the ViT model pays most attention to when making classification decisions. In the schizophrenia classification task, the heat map will highlight the brain regions that contribute most to the model's prediction of the schizophrenia class. Figure 2BAs shown, it is found that the detailed information of the saliency map of the attention feature map of the ViT model, the main areas include gray matter in the right temporal lobe, thalamus and visual area, and the patient shows more posterior temporal lobe and thalamus. For patients, the functional network connection pairs are more significant in the internal connection of the cerebellum and the auditory-motor and auditory-visual connections. This shows that the method disclosed in the present invention will highlight specific brain areas related to the pathophysiological mechanism of schizophrenia, and improve the transparency and accuracy of the ViT model.

[0067] In some embodiments, the model processing module 130 is further used to generate a training sample data set corresponding to schizophrenia:

[0068] Acquire functional magnetic resonance (FMR) sample images of the brains of multiple sample subjects, and annotate each FMR sample image respectively; based on the preset position information of the brain region corresponding to schizophrenia, crop the interest region sample images from each FMR sample image respectively; wherein the annotation information of the interest region sample image is the same as the annotation information of the corresponding FMR sample image; flip each interest region sample image in the horizontal direction according to a first preset angle respectively to obtain a horizontally enhanced sample image; wherein the annotation information of the horizontally enhanced sample image is the same as the annotation information of the corresponding FMR sample image; flip each interest region sample image in the vertical direction according to a second preset angle respectively to obtain a vertically enhanced sample image; wherein the annotation information of the vertically enhanced sample image is the same as the annotation information of the corresponding FMR sample image; and form the training sample data set using the interest region sample image, the annotation information of the interest region sample image, the horizontally enhanced sample image, the annotation information of the horizontally enhanced sample image, the vertically enhanced sample image, and the annotation information of the vertically enhanced sample image.

[0069] The above embodiment adopts a data enhancement method, that is, horizontally and vertically flipping each region of interest sample image VOI to obtain a horizontally enhanced sample image and a vertically enhanced sample image, such as Figure 3 As shown in Figure 2, data augmentation techniques such as horizontal and vertical flipping can generate diverse training samples, helping the model learn brain features that remain unchanged under different transformations, thereby enhancing the generalization ability and robustness of the ViT model.

[0070] This publication includes 169 fMRI images, including 88 cases of schizophrenia (SZ) and 81 healthy subjects (Health). All subjects were scanned on a Siemens Verio 3-T scanner. The specific scanning parameters were: repetition time = 2000ms, echo time = 30ms, flip angle = 90°, matrix size = 64×64, axial slice resolution = 3.4×3.4mm 2, slice thickness = 4 mm, gap between slices = 0.6 mm. The resting state data were collected for 8 min, with a total of 240 time points.

[0071] All data were examined and annotated by experienced radiologists, and all schizophrenia patients were professionally diagnosed by experienced psychiatrists based on the criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) through the Structured Clinical Interview for the disease. For the screening of schizophrenia patients, the following clear inclusion criteria were set: (1) right-handed; (2) aged between 18 and 55 years; (3) the patient and his or her paternal or maternal relatives within three generations did not meet the diagnostic criteria for Axis I diseases based on DSM-IV and SCID. At the same time, strict exclusion criteria were set to ensure the accuracy and reliability of the method: (1) those who had received or were receiving psychiatric or psychotherapy; (2) those with other mental disorders, neurological diseases or serious physical diseases; (3) women who were pregnant or breastfeeding. For the selection of healthy subjects, corresponding inclusion criteria were also set: (1) right-handed; (2) aged between 18 and 55 years; (3) no history of any mental disorder, neurological disease or major physical disease. Ensure that the sample is highly homogeneous and representative, and improve the reliability and validity of the method.

[0072] Among the generated training sample datasets, 135 (80%) are used as training data, 17 (10%) are used as validation data, and 17 (10%) are used as test data, as shown in Table 1.

[0073] Table 1 Distribution of training and validation sample data

[0074]

[0075] like Figure 4As shown in the figure, the ViT model is a Transformer-like model applied to visual tasks. It directly applies the Transformer architecture to non-overlapping image patches for image classification. The ViT model mainly performs the following operations or has the following main processing modules: (1) Image patch embedding: The ViT model first crops the input image (region of interest image) into fixed-size image patches, and converts these image patches into embedding vectors through linear mapping, then adds position encoding and inputs them into the standard Transformer encoder; (2) Position encoding: Since the Transformer model itself does not have the ability to capture sequence order, position encoding (Positional Embedding) needs to be added to provide position information. The position encoding can be understood as a table, where each row represents a vector whose dimension is the same as the dimension of the input vector sequence Embedding; (3) Transformer encoder: The input embedding sequence is processed by the Transformer encoder, which includes modules such as multi-head self-attention mechanism, feedforward neural network, and normalization to capture global and local features in the image; (4) Classification tag: There is an additional category character (Class token) in the embedding vector of the input Transformer encoder. At the output of the Transformer encoder, this Class token needs to be extracted and fed into a multi-layer perceptron (MLP) for classification. The ViT model outputs the probability distribution of each category, thereby achieving accurate and efficient image classification tasks.

[0076] This paper used 5-fold cross-validation with case separation at the patient level. The experiment was conducted on GPUNVIDIA GeForce RTX 4060 using Python language. Deep learning was implemented on the Tensorflow framework. The optimizer was Adam (adaptive moment estimation), the weight decay was 1e-3, the epoch was 200, and the learning rate was 3e-4.

[0077] The technical method disclosed in the present invention effectively improves the detection accuracy of schizophrenia, as shown in Table 2 below, which compares the classification performance of the scheme disclosed in the present invention with other different existing schemes from the perspective of Acc (accuracy) and AUC (area under the curve). Among them, Acc is the classification accuracy, AUC is the area under the receiver operating characteristic (ROC) curve, the x-axis is (1-specificity), and the y-axis is sensitivity. The Data column in Table 2 represents the data used or based on the data set used to form the training model. As can be seen from Table 2, the scheme disclosed in the present invention obtains the highest Acc and AUC. This is mainly because the Vision Transformer model focuses on deep imaging genomics feature extraction. Therefore, from the comparison and predicted classification results, it can be easily found that the scheme disclosed in the present invention has a relative advantage in distinguishing schizophrenia patients from healthy people.

[0078] Table 2 Distribution of training and validation sample data

[0079]

[0080] The disclosed solution is superior to existing solutions mainly in terms of the accuracy of predicting schizophrenia and visual interpretation. (1) Prediction of schizophrenia focuses on deep imaging feature extraction; (2) Interpretability analysis is based on the attention feature map of the ViT model, visualizing multiple brain regions most relevant to schizophrenia.

[0081] Based on the same inventive concept, the present disclosure provides a Transformer-based schizophrenia identification and interpretation method, the execution steps of which are the same or similar to the modules in the above-mentioned system, so similar parts will not be repeated. Figure 5 As shown, the Transformer-based schizophrenia recognition and interpretation method of this embodiment includes:

[0082] S510: Acquire a functional magnetic resonance image of the target subject's brain.

[0083] S520: based on the preset position information of the brain region corresponding to schizophrenia, clipping the region of interest image from the functional magnetic resonance image.

[0084] S530, using the trained Vision Transformer model to perform a schizophrenia detection operation to output a predicted category and a probability corresponding to each predicted category: cropping the region of interest image into a plurality of image blocks of a preset size, using a linear mapping to convert each image block into an embedding vector, setting a learnable classification tag, adding a position code to each embedding vector, and forming an embedding sequence using all the embedding vectors with added position codes and the classification tags; inputting the embedding sequence into a Transformer encoder of the Vision Transformer model, and using the Transformer encoder to extract global features and local features from the embedding vector; extracting the classification tag from the output of the Transformer encoder, and inputting the extracted classification tag and the features output by the Transformer encoder into a multilayer perceptron; the multilayer perceptron uses the received classification tags and features to classify schizophrenia, and outputs a predicted category and a probability corresponding to each predicted category; wherein, during the training process, the Vision Transformer model is first pre-trained on a public dataset, and then the pre-trained Vision Transformer model is trained using a training sample dataset corresponding to schizophrenia.

[0085] S540: Perform binary classification based on the predicted categories and the probabilities corresponding to each predicted category to determine a first probability that the target object suffers from schizophrenia and a second probability that the target object is in a healthy state, and determine that the target object suffers from schizophrenia when the first probability is higher than the second probability.

[0086] S550. Obtain a feature map output by the Transformer encoder, and generate and display an attention feature map based on the feature map, wherein the attention feature map includes attention distribution information of each layer of the multi-head attention mechanism in the Transformer encoder.

[0087] S560: Obtain a feature map output by the Vision Transformer model, generate a heat map based on the feature map output by the Vision Transformer model, and display the heat map.

[0088] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a computer-readable storage medium.

[0089] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0090] like Figure 6 As shown, the device 600 includes a computing unit 610, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 620 or a computer program loaded from a storage unit 680 into a random access memory (RAM) 630. In the RAM 630, various programs and data required for the operation of the device 600 can also be stored. The computing unit 610, the ROM 620, and the RAM 630 are connected to each other via a bus 640. An input / output (I / O) interface 650 is also connected to the bus 640.

[0091] A number of components in the device 600 are connected to the I / O interface 650, including: an input unit 660, such as a keyboard, a mouse, etc.; an output unit 670, such as various types of displays, speakers, etc.; a storage unit 680, such as a disk, an optical disk, etc.; and a communication unit 690, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 690 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0092] The computing unit 610 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 610 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 610 performs the various methods and processes described above. For example, in some embodiments, any of the above methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 680. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 620 and / or a communication unit 690. When the computer program is loaded into RAM 630 and executed by the computing unit 610, one or more steps of any of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 610 may be configured to perform any of the methods described above in any other appropriate manner (e.g., by means of firmware).

[0093] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0094] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0095] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0096] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0097] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0098] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0099] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0100] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A Transformer-based schizophrenia recognition and interpretation system, characterized by: include: An image acquisition module, used for acquiring a functional magnetic resonance image of the brain of a target subject; An image preprocessing module, used for clipping a region of interest image from the functional magnetic resonance image based on preset position information of a brain region corresponding to schizophrenia, and inputting the region of interest image into an image recognition module; A model processing module, used to use the trained Vision Transformer model to perform the following operations: crop the image of the region of interest into multiple image blocks of preset sizes, convert each image block into an embedding vector using a linear mapping, set a learnable classification tag, add a position code to each embedding vector, and form an embedding sequence using all the embedding vectors with added position codes and the classification tags; input the embedding sequence into the Transformer encoder of the Vision Transformer model, and use the Transformer encoder to extract global features and local features of the embedding vector; extract the classification tag from the output of the Transformer encoder, and input the extracted classification tag and the features output by the Transformer encoder into a multilayer perceptron; the multilayer perceptron uses the received classification tags and features to classify schizophrenia, and outputs a predicted category and the probability corresponding to each predicted category; wherein, during the training process, the Vision Transformer model is first pre-trained on a public data set, and then the pre-trained Vision Transformer model is trained using a training sample data set corresponding to schizophrenia; an image classification decision module, configured to perform binary classification based on the predicted categories and the probabilities corresponding to each predicted category, determine a first probability that the target object suffers from schizophrenia and a second probability that the target object is in a healthy state, and determine that the target object suffers from schizophrenia when the first probability is higher than the second probability; A feature visualization module is used to obtain the feature map output by the Transformer encoder, and generate and display an attention feature map based on the feature map, wherein the attention feature map includes attention distribution information of each layer of the multi-head attention mechanism in the Transformer encoder; The key area visualization module is used to obtain the feature map output by the Vision Transformer model, generate a heat map based on the feature map output by the Vision Transformer model, and display the heat map.

2. The system according to claim 1, characterized in that The model processing module is also used to generate a training sample data set corresponding to schizophrenia: Acquire functional magnetic resonance imaging sample images of the brains of multiple sample subjects, and annotate each functional magnetic resonance imaging sample image respectively; Based on the preset location information of the brain region corresponding to schizophrenia, a sample image of the region of interest is obtained by clipping from each functional magnetic resonance imaging sample image; wherein the annotation information of the sample image of the region of interest is the same as the annotation information of the corresponding functional magnetic resonance imaging sample image; Flipping each region of interest sample image in the horizontal direction according to a first preset angle to obtain a horizontally enhanced sample image; wherein the annotation information of the horizontally enhanced sample image is the same as the annotation information of the corresponding functional magnetic resonance sample image; Flipping each sample image of the region of interest in the vertical direction according to a second set angle to obtain a vertically enhanced sample image; wherein the annotation information of the vertically enhanced sample image is the same as the annotation information of the corresponding functional magnetic resonance sample image; The training sample data set is formed by using the interest region sample image, the annotation information of the interest region sample image, the horizontal enhancement sample image, the annotation information of the horizontal enhancement sample image, the vertical enhancement sample image, and the annotation information of the vertical enhancement sample image.

3. The system according to claim 1, characterized in that The public data sets include MathVista data sets.

4. The system according to claim 1, characterized in that When the key area visualization module generates a heat map based on the feature map output by the Vision Transformer model, it is specifically used to: The Grad-CAM technique is used to generate a heat map based on the feature map output by the Vision Transformer model.

5. The system according to claim 1, characterized in that The Transformer encoder includes a multi-head attention mechanism layer, a feedforward neural network layer, and a normalization layer.

6. The system according to claim 1, characterized in that The Vision Transformer model includes: an embedding layer, a Transformer encoder, and a multi-layer perceptron.

7. The system according to claim 1, characterized in that The size of the interest region image is 120 pixels x 140 pixels x 120 pixels.

8. A Transformer-based schizophrenia recognition and interpretation method, characterized in that: include: Acquiring functional magnetic resonance images of the target subject's brain; A region of interest image is obtained by clipping from the functional magnetic resonance image based on preset location information of a brain region corresponding to schizophrenia; The trained Vision Transformer model is used to perform the following operations: the image of the region of interest is cropped into multiple image blocks of a preset size, each image block is converted into an embedding vector using a linear mapping, a learnable classification tag is set, a position code is added to each embedding vector, and an embedding sequence is formed using all the embedding vectors with added position codes and the classification tags; the embedding sequence is input into the Transformer encoder of the Vision Transformer model, and the Transformer encoder is used to extract global features and local features of the embedding vector; the classification tag is extracted from the output of the Transformer encoder, and the extracted classification tag and the features output by the Transformer encoder are input into a multilayer perceptron; the multilayer perceptron uses the received classification tags and features to classify schizophrenia, and outputs the predicted category and the probability corresponding to each predicted category; wherein, during the training process, the Vision Transformer model is first pre-trained on a public dataset, and then the pre-trained Vision Transformer model is trained using a training sample dataset corresponding to schizophrenia; Performing binary classification based on the predicted categories and the probabilities corresponding to each predicted category, determining a first probability that the target object suffers from schizophrenia and a second probability that the target object is in a healthy state, and determining that the target object suffers from schizophrenia when the first probability is higher than the second probability; Obtain a feature map output by the Transformer encoder, and generate and display an attention feature map based on the feature map, wherein the attention feature map includes attention distribution information of each layer of the multi-head attention mechanism in the Transformer encoder; A feature map output by the Vision Transformer model is obtained, and a heat map is generated based on the feature map output by the Vision Transformer model, and the heat map is displayed.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method of claim 8 when executing the computer program.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to claim 8 is implemented.

Images