Mental state data classification prediction system based on cavity convolution and consistency learning

Abstract

The invention discloses a mental state data classification prediction system based on cavity convolution and consistency learning, which fuses multi-modal data, extracts space and time sequence features, improves prediction accuracy, reduces dependence on high-cost detection, and provides objective and reliable decision support for clinic. An encoder in a Con4m framework is further improved, an attention mechanism is added into the encoder, so that the model can pay attention to a context mode in data, and the adaptability of the model to complex mental state time sequence changes is enhanced. And meanwhile, a time sequence feature dynamic adjustment mechanism is added into the consistency category prediction module, so that the model is ensured to always keep good adaptability to the senile mental state data in the training process.

Description

Technical Field

[0001] The present invention belongs to the field of medical artificial intelligence technology, and specifically relates to a system that uses deep learning technology to intelligently classify and predict the mental state data of the elderly. Background Art

[0002] Alzheimer's disease, a progressive neurodegenerative disease, is increasing in incidence as the global population ages. While early prediction and intervention are crucial for improving prognosis in the elderly, existing methods for classifying and predicting mental state data have numerous limitations. These include the difficulty of single-modality data in fully reflecting disease characteristics, the reliance on subjective experience in clinical assessments, and the high cost and invasive nature of imaging and biomarker testing. Furthermore, traditional computer-assisted prediction techniques also face challenges, including high data volume requirements, a lack of modeling for temporal dynamics, and insufficient cross-modal feature fusion.

[0003] Existing research has shortcomings, including over-reliance on static imaging features, high computational complexity, failure to consider the nonlinear nature of disease progression, and a lack of uncertainty quantification. These shortcomings indicate that traditional models still have significant room for improvement in accurately classifying and predicting mental state data. To address this, technical challenges include integrating multi-source heterogeneous data, extracting spatial and temporal features, improving model generalization, and achieving computational efficiency.

[0004] The trend of new technologies is to reduce dependence on labeled data through self-supervised learning, use graph neural networks to enhance brain area connection modeling, use federated learning to protect privacy data, and combine knowledge distillation to improve model deployment efficiency. In order to solve the current challenges, the proposed technical solution based on dilated convolution and consistency learning aims to improve prediction accuracy by fully utilizing multimodal data and extracting spatial and temporal features, and has good clinical application prospects. In response to the above problems, it is urgent to develop a mental state data classification and prediction system that can fully utilize multimodal data and achieve accurate predictions. The present invention proposes a technical solution based on dilated convolution and consistency learning, which can effectively extract spatial-temporal features and classify and predict the mental state data of the elderly. Summary of the Invention

[0005] This invention aims to overcome the shortcomings of existing technologies and provide a mental state data classification and prediction system based on dilated convolution and consistency learning. This system can effectively integrate multimodal medical data, significantly improving the accuracy and timeliness of mental state data classification and prediction, and providing a reliable and interpretable prediction basis for clinical decision-making.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] The mental state data classification and prediction system based on dilated convolution and consistency learning includes:

[0008] Data preprocessing module: used to receive and preprocess the mental state data of the elderly to obtain the fused multimodal vector.

[0009] Spatial feature extraction module: used to receive multimodal vectors and extract spatial feature representations through the spatial pyramid pooling module (ASPP).

[0010] Temporal feature extraction module: used to receive multimodal vectors and extract temporal feature representations through the temporal feature extraction framework.

[0011] Intelligent prediction module: Based on the spatial feature representation and temporal feature representation, it performs classification prediction and outputs the prediction results of the elderly's mental state data.

[0012] Training module: Comprehensive training of data preprocessing module, spatial feature extraction module, temporal feature extraction module and intelligent prediction module.

[0013] In one possible implementation, the data preprocessing module receives mental state data of an elderly person comprising medical imaging data, numerical data, and categorical data; performs brain extraction, registration, and intensity normalization on the medical imaging data; converts the numerical data and categorical data into embedded representations; extracts image embeddings from the processed medical imaging data using a pretrained medical image segmentation model; and merges the embedded representations of different modalities and inputs them into a Transformer encoder to obtain a fused multimodal vector.

[0014] In one possible implementation, the spatial feature extraction module receives the multimodal vector output by the Transformer encoder and inputs it into a spatial pyramid pooling module (ASPP) comprising a parallel-arranged atrous convolution layer and a global pooling layer with different expansion rates, extracts and fuses multi-scale spatial context information, and outputs a spatial feature representation.

[0015] In one possible implementation, the temporal feature extraction module receives the multimodal vector output by the Transformer encoder and inputs it into a temporal feature extraction framework comprising a context encoder and a consistency prediction module; the context encoder is used to capture local continuity and contextual information in the data; the consistency prediction module uses the output of the context encoder to perform consistency judgment; an attention mechanism is added to the temporal feature extraction framework; and finally, a temporal feature representation is output.

[0016] In one possible implementation, the intelligent prediction module receives the spatial feature representation and the temporal feature representation, fuses the spatial feature representation and the temporal feature representation through a gated attention mechanism, and obtains fused spatiotemporal features; performs classification prediction based on the spatiotemporal features, and outputs the predicted probability that the elderly person's mental state data belongs to the categories of normal cognition, mild cognitive impairment, or dementia.

[0017] In one possible implementation, the training module comprehensively trains the Transformer encoder, spatial feature extraction module, temporal feature extraction module, and intelligent prediction module of the data preprocessing module based on the cross-entropy loss function and mutual information constraints; during the training process, the generalization ability is enhanced by randomly masking some input features, and a dynamic adjustment mechanism of temporal features is added to adapt to non-stationary changes.

[0018] The beneficial effects of the present invention are as follows:

[0019] The present invention addresses the problems of single data, strong subjectivity, and high cost in predicting the mental state data of the elderly, and proposes an intelligent prediction system based on void convolution and consistency learning. The system integrates multimodal data, extracts spatial and temporal features, improves prediction accuracy, reduces dependence on high-cost testing, provides objective and reliable decision support for clinicians, and assists in early screening, prevention, and dynamic evaluation of diseases, with important clinical application value. The present invention also improves the encoder in the Con4m framework, adding an attention mechanism to it so that the model can pay attention to contextual patterns in the data, thereby enhancing the model's adaptability to complex temporal changes in mental states. At the same time, a dynamic adjustment mechanism for temporal features is added to the consistency category prediction module to ensure that the model always maintains good adaptability to the mental state data of the elderly during the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of the mental state data classification and prediction system according to an embodiment of the present invention.

[0021] Figure 2 Schematic diagram of the ASPP module's process of extracting spatial features using dilated convolution.

[0022] Figure 3 This is a schematic diagram of the Con4m encoder structure.

[0023] Figure 4 Schematic diagram of the Con4m consistency category prediction module structure. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0025] Alzheimer's disease is a progressive neurodegenerative disease whose incidence is increasing year by year as the population ages. This paper effectively improves the classification and prediction accuracy of elderly mental state data by extracting spatial and temporal features from multimodal medical data. It also compares the model's predictions with clinical predictions to ensure the interpretability of the model's predictions. This paper uses a dilated convolutional network and consistency learning to extract multi-scale spatial and temporal features from elderly mental state data, significantly improving the classification and prediction accuracy of elderly mental state data and providing assistance for early screening, prevention, and dynamic assessment of the disease.

[0026] The present application embodiment provides a mental state data classification and prediction method based on dilated convolution and consistency learning, including the following steps:

[0027] Step (1) Data preprocessing: receiving mental state data of the elderly comprising medical image data, numerical data and categorical data; performing brain extraction, registration and intensity normalization on the medical image data; converting the numerical data and categorical data into embedding representations; extracting image embeddings from the processed medical image data using a pre-trained medical image segmentation model; merging the embedding representations of different modalities and inputting them into a Transformer encoder to obtain a fused multimodal vector.

[0028] Step (2) Spatial feature extraction: The multimodal vector output by the Transformer encoder is input into a spatial pyramid pooling module (ASPP) comprising parallel atrous convolution layers with different dilation rates and global pooling layers to extract and fuse multi-scale spatial context information and output spatial feature representation.

[0029] Step (3) Temporal feature extraction: The multimodal vector output by the Transformer encoder is input into a temporal feature extraction framework comprising a context encoder and a consistency prediction module; the context encoder is used to capture local continuity and contextual information in the data; the consistency prediction module uses the output of the context encoder to make consistency judgments; an attention mechanism is added to the temporal feature extraction framework; and finally, a temporal feature representation is output.

[0030] Step (4) The classification prediction model fuses the spatial feature representation and the temporal feature representation through a gated attention mechanism to obtain a fused spatiotemporal feature; performs classification prediction based on the spatiotemporal feature, and outputs the predicted probability that the mental state data of the elderly belongs to the categories of normal cognition, mild cognitive impairment, or dementia;

[0031] Step (5) Based on the cross entropy loss function and mutual information constraint, the Transformer encoder, spatial pyramid pooling module (ASPP), temporal feature extraction framework and classification prediction model are comprehensively trained; during the training process, the generalization ability is enhanced by randomly masking some input features, and a dynamic adjustment mechanism of temporal features is added to adapt to non-stationary changes.

[0032] The present application also provides a mental state data classification and prediction system based on dilated convolution and consistency learning, which specifically includes the following modules:

[0033] Data preprocessing module: used to receive and preprocess the mental state data of the elderly to obtain the fused multimodal vector.

[0034] Spatial feature extraction module: used to receive multimodal vectors and extract spatial feature representations through the spatial pyramid pooling module (ASPP).

[0035] Temporal feature extraction module: used to receive multimodal vectors and extract temporal feature representations through the temporal feature extraction framework.

[0036] Intelligent prediction module: Based on the spatial feature representation and temporal feature representation, it performs classification prediction and outputs the prediction results of the elderly's mental state data.

[0037] Training module: Comprehensive training of data preprocessing module, spatial feature extraction module, temporal feature extraction module and intelligent prediction module.

[0038] In one possible implementation, the data preprocessing module utilizes a systematic approach to process multimodal medical data. Data preprocessing of acquired elderly mental state data involves two key stages: standardization of medical imaging data and feature embedding of multimodal data. Elderly mental state data primarily includes medical imaging data (MRI scan data), numerical data (such as the elderly's age, clinical physiological data, neuropsychological test results, etc.), and categorical data (such as the elderly's gender, disease stage, race, CDR scale, etc.). Specifically, the acquired multi-sequence MRI scan data is first comprehensively processed, followed by a unified embedding representation of different types of data. This approach converts heterogeneous data into standardized features that can be used by deep learning models.

[0039] Medical imaging data standardization: MRI scan data from various elderly individuals, including T1-weighted (T1w), T2-weighted (T2w), diffusion-weighted imaging (DWI), susceptibility-weighted imaging (SWI), and fluid-attenuated inversion recovery (FLAIR) sequences, were collected. The MRI scans were stored in the NIFTI file format and categorized by individual characteristics and disease stage. Skull stripping was performed using the SynthStrip tool, and the MRI scans were linearly registered to MNI space using the FSL "flirt" tool. The MRI scans were then intensity normalized, with grayscale values ​​mapped to the [0, 1] interval.

[0040] Feature Embedding for Multimodal Data: The collected numerical and categorical data are processed separately. Numerical data is embedded using a single linear layer; categorical data is converted to an embedded representation using a lookup table. A pre-trained Swin UNETR model is used to extract 3D image embeddings (feature embeddings) from MRI scans and downsampled to 256-dimensional vectors as their embedded representations. Finally, the embeddings from different modalities are merged and used as the input to the Transformer encoder. The Transformer's multi-head self-attention mechanism captures long-range dependencies in the image, ultimately outputting a fused multimodal vector.

[0041] In one possible implementation, in order to extract the spatial features of the data, the spatial feature extraction module inputs the output of the Transformer encoder into an ASPP (Atrous Spatial Pyramid Pooling) module, and extracts multi-scale contextual information through dilated convolutions with different dilation rates. ASPP is a deep learning module for extracting multi-scale features and is widely used in semantic segmentation tasks. It combines the ideas of dilated convolution and spatial pyramid pooling (SPP) and can capture contextual information of different scales without losing resolution. This example uses four parallel dilated convolution layers with dilation rates of 6, 12, 18, and 24, respectively, to extract spatial features at different scales, and then uses a global pooling layer to extract global information, such as Figure 2 As shown in Figure 2, the output of the Transformer encoder passes through four dilated convolutional layers and is then input into a global pooling layer to obtain the final extracted spatial features.

[0042] The output feature map size formula of dilated convolution is expressed as:

[0043]

[0044] in is the size of the input feature map, is the size of the padding, is the dilation rate of the dilated convolution, is the size of the convolution kernel, is the stride of the convolution.

[0045] In one possible implementation, the time series feature extraction module extends the Con4m framework to the field of cross-modal time series fusion, fully tapping the collaborative value of multi-source time series data. The Con4m framework includes two modules: an encoder for extracting continuous context representations and a consistency category prediction module based on context information. The encoder analyzes the time series segment by segment to identify important features in each segment of data; the consistency category prediction module uses these features to make consistency judgments, thereby extracting accurate time series features. The design of the Con4m framework enables Con4m to better capture the complex relationships in time series and improve the quality of extracted features.

[0046] First, the vector output by the Transformer encoder is input into the encoder for extracting continuous context representation, capturing the local continuity and context information in the data, such as Figure 3 As shown. The output vector of the encoder used to extract continuous context representation is then input into the consistency category prediction module to learn the category conversion and consistency relationship between data, as shown in Figure 4 As shown in the figure, the extracted temporal features are finally obtained. To adapt to the task of classifying and predicting mental state data of the elderly, the present invention adds an attention mechanism between the average pooling layer and the Con-Transformer in the encoder used to extract continuous context representations. This allows the model to automatically select which contextual pattern to focus on during prediction, enhancing the model's adaptability to complex temporal changes in mental states.

[0047] In one possible implementation, the intelligent prediction module first performs feature fusion. To better utilize the extracted spatiotemporal features, the present invention designs a gating mechanism to fuse spatial and temporal features and control and adjust the contribution of both features to the prediction results. Through learning, it can dynamically determine which information is important and which can be ignored. The formula is as follows:

[0048]

[0049]

[0050] in Refers to the extracted spatial features, Refers to the extracted temporal features, represents element-wise multiplication, Represents the sigmoid activation function, It stands for Feed Forward Neural Network. Finally, the fused spatiotemporal features are obtained. , the fused features The final prediction result is input into the softmax layer, and the predicted probability of the elderly's mental state data being in the three mental states of normal cognition, mild cognitive impairment, and dementia is obtained.

[0051] In one possible implementation, the training module combines the preprocessed data to perform comprehensive training on the Transformer encoder, spatial feature extraction module, temporal feature extraction module and intelligent prediction module of the data preprocessing module. The system uses the cross-entropy loss function to measure the difference between the prediction result and the true label, thereby guiding the optimization of the model. In addition, in order to fuse the features of different modalities, it is necessary to constrain the consistency of the features of different modalities. Therefore, the model also uses the mutual information between the spatial features of each modality and the mutual information between the temporal features of each modality as the loss function. The final loss function is the sum of the cross-entropy loss function, the mutual information between the spatial features of each modality and the mutual information between the temporal features of each modality.

[0052] In terms of optimizer, the present invention adopts Adam optimizer, and its initial learning rate is set to 0.001. Set to 0.9, the exponential decay rate of the second-order moment estimate It is set to 0.999, the constant ε to prevent division by zero is set to 1e-8, and the batchsize is set to 32. During the training process, in order to enhance the generalization ability of the model, the present invention randomly masks some input features to enhance the robustness of the model to missing data. In addition, based on the characteristics of non-stationary changes in the mental state of the elderly, the present invention also adopts a dynamic adjustment mechanism of time series features in the training process of the consistency category prediction module in the Con4m framework, and adjusts the parameters online during the training process. For example, when the model detects a sudden increase in the prediction error, it automatically reduces the window length to improve the ability to capture local time series features; when the data distribution changes, the parameters of the Con4m framework are fine-tuned to ensure that the model always maintains good adaptability to the mental state data of the elderly.

[0053] In one possible implementation, the probability output by the intelligent prediction module is fused with the clinician's prediction confidence to form an AI-assisted prediction confidence. This fusion method enables the AI ​​model to not only provide a separate prediction result, but also combine with the doctor's clinical judgment to form a more comprehensive and credible prediction suggestion. In order to comprehensively evaluate the performance of the model, the system also uses indicators such as AUROC and AUPR, which can quantify the accuracy, sensitivity and specificity of the model in different prediction tasks. Through these evaluation indicators, the actual performance of the model can be fully verified to ensure that it can provide reliable auxiliary decision support in clinical practice. The present invention can make full use of multimodal medical data, improve the accuracy of the prediction of the mental state of the elderly, realize early screening and dynamic evaluation of diseases, and provide an objective basis for clinical practice.

[0054] To evaluate the clinical impact of the model's predictions, five neurologists and three neuroradiologists were invited to participate in the prediction task. They were asked to provide their prediction impressions and original clinician confidence scores regarding the elderly patient's mental state. The clinician confidence scores and the model's prediction probabilities were averaged to form an AI-assisted prediction confidence score. The predictive accuracy of the clinician's original confidence scores and the AI-assisted confidence scores was then evaluated using the AUROC and AUPR metrics. AUROC is a commonly used metric for evaluating the performance of classification models. It describes the trade-off between the true positive rate and false positive rate at different thresholds. AUPR is another important evaluation metric that focuses on the trade-off between the model's precision and recall at different thresholds. Compared to AUROC, AUPR is more suitable for addressing class imbalance. The AI-assisted prediction confidence score outperformed the clinician's original confidence score in both AUROC and AUPR, demonstrating the clinical applicability of the proposed system and its potential as a valuable aid in physicians' predictions. By combining the system's output with the doctor's clinical judgment, a more realistic prediction support system is formed, helping doctors make more accurate and reliable decisions.

[0055] The embodiment of the present application further provides an electronic device, comprising a processor and a memory;

[0056] The memory is used to store computer programs;

[0057] The processor is used to implement any method described in this application when executing the program stored in the memory.

[0058] In a possible implementation, the electronic device of the embodiment of the present application further includes a communication interface and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus.

[0059] The communication bus mentioned in the electronic device mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0060] The communication interface is used for communication between the above electronic device and other devices.

[0061] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0062] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0063] In another embodiment provided in the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the methods described in the present application is implemented.

[0064] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the methods described in the present application.

[0065] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0066] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0067] Each embodiment in this specification is described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other.

[0068] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A mental state data classification and prediction system based on dilated convolution and consistency learning, characterized by: include: Data preprocessing module: used to receive and preprocess the mental state data of the elderly to obtain the fused multimodal vector; Spatial feature extraction module: used to receive multimodal vectors and extract spatial feature representations through the spatial pyramid pooling module (ASPP); Temporal feature extraction module: used to receive multimodal vectors and extract temporal feature representations through the temporal feature extraction framework; Intelligent prediction module: Based on the spatial feature representation and temporal feature representation, it performs classification prediction and outputs the prediction results of the elderly's mental state data; Training module: Comprehensive training of data preprocessing module, spatial feature extraction module, temporal feature extraction module and intelligent prediction module.

2. The mental state data classification and prediction system based on dilated convolution and consistency learning according to claim 1 is characterized in that: The data preprocessing module receives elderly mental state data including medical image data, numerical data, and categorical data; performs brain extraction, alignment, and intensity normalization on the medical image data; converts the numerical data and categorical data into embedded representations; uses a pre-trained medical image segmentation model to extract image embeddings from the processed medical image data; and merges the embedded representations of different modalities and inputs them into a Transformer encoder to obtain a fused multimodal vector.

3. The mental state data classification and prediction system based on dilated convolution and consistency learning according to claim 2 is characterized in that: The spatial feature extraction module receives the multimodal vector output by the Transformer encoder and inputs it into a spatial pyramid pooling module including a parallel set of hollow convolution layers and a global pooling layer with different expansion rates, extracts and fuses multi-scale spatial context information, and outputs a spatial feature representation.

4. The mental state data classification and prediction system based on dilated convolution and consistency learning according to claim 2 is characterized in that: The temporal feature extraction module receives the multimodal vector output by the Transformer encoder and inputs it into a temporal feature extraction framework including a context encoder and a consistency prediction module; the context encoder is used to capture local continuity and contextual information in the data; the consistency prediction module uses the output of the context encoder to make consistency judgments; an attention mechanism is added to the temporal feature extraction framework; and finally, a temporal feature representation is output.

5. The mental state data classification and prediction system based on dilated convolution and consistency learning according to any one of claims 1 to 4, characterized in that: The intelligent prediction module receives the spatial feature representation and the temporal feature representation, fuses the spatial feature representation and the temporal feature representation through a gated attention mechanism, and obtains a fused spatiotemporal feature; performs classification prediction based on the spatiotemporal feature, and outputs the predicted probability that the mental state data of the elderly belongs to the categories of normal cognition, mild cognitive impairment, or dementia.

6. The mental state data classification and prediction system based on dilated convolution and consistency learning according to claim 5 is characterized in that: The training module comprehensively trains the Transformer encoder, spatial feature extraction module, temporal feature extraction module and intelligent prediction module of the data preprocessing module based on the cross-entropy loss function and mutual information constraints; during the training process, the generalization ability is enhanced by randomly masking some input features, and a dynamic adjustment mechanism of temporal features is added to adapt to non-stationary changes.

Images