Cognition improvement method and system based on multi-modal data

Abstract

The invention discloses a cognitive improvement method and system based on multi-modal data. The cognition improvement method comprises the following steps: acquiring first multi-modal data of a patient before treatment; carrying out amyloid protein load grouping on the patients on the basis of amyloid protein loads in the brains of the patients so as to obtain corresponding initial cognitive training schemes; performing cognitive training based on the initial cognitive training scheme during the treatment period of the patient, and collecting second multi-modal data during the treatment period; based on the multi-modal data before and after the treatment of the patient, acquiring multi-modal change data of the patient in combination with the brain network group atlas; and inputting the multi-modal change data of the patient into a preset model to output a cognition improvement scheme for improving the cognition index of the patient to the greatest extent, and pushing the cognition improvement scheme to the patient for cognition training. According to the method, by integrating A beta-PET data, functional nuclear magnetic resonance imaging data and event-related potential data, the neuroplasticity change of a patient is evaluated in real time, so that a personalized cognition improvement scheme is dynamically generated.

Description

Technical Field

[0001] The present invention relates to a cognitive improvement method based on multimodal data, and also relates to a corresponding cognitive improvement system, belonging to the technical field of cognitive training. Background Art

[0002] Alzheimer's disease (AD) is a neurodegenerative disease characterized by a decline in cognitive function, and its main pathological manifestation is amyloid deposition. In recent years, the two anti-amyloid monoclonal antibody drugs approved by the US FDA, lencanezumab and donezumab, have been major breakthroughs in this field and have become the first drugs that can delay the progression of AD. However, although these monoclonal antibody drugs can delay disease progression (assessed by daily living ability), their improvement in cognitive function is very limited. For example, after 18 months of lencanezumab treatment, the difference in cognitive function observed by the ADAS-Cog14 scale (total score 90 points) between the treatment group and the placebo group was only 1.44 points, which is almost clinically unobservable. Therefore, further improving patients' cognitive function on the basis of the application of monoclonal antibody drugs is a clinical problem that needs to be solved urgently.

[0003] In clinical trials of monoclonal antibody drugs, patients with mild cognitive impairment due to AD were treated only with anti-amyloid monoclonal antibody drugs, while patients with mild dementia routinely used cholinesterase inhibitors, the first-line drug for AD, to improve cognitive function. Neither of these two types of patients used non-drug interventions (such as cognitive training) to improve cognitive function. Previously developed cognitive training programs were used in the natural course of patients, and the basis of cognitive training—changes in neuroplasticity—would not be subject to external intervention. However, monoclonal antibody drugs can change the course of the disease by rapidly clearing amyloid from the brains of AD patients, and amyloid deposition is closely related to neuroplasticity damage. Therefore, after using anti-amyloid monoclonal antibody drugs, new cognitive training programs and training models need to be designed to adapt to changes in the plasticity of the nervous system, so as to better improve patients' cognitive function.

[0004] Most of the existing non-drug interventions (such as cognitive training) are not designed to target the dynamic changes in neuroplasticity of patients after anti-amyloid drug treatment, and there is also a lack of integrated solutions that fully utilize multimodal markers to dynamically monitor neuroplasticity. The pathological changes of AD are closely related to the reorganization of brain functional networks, including functional abnormalities of the default mode network (DMN) and task-related networks revealed by functional magnetic resonance imaging (fMRI), and cognitive processing impairment reflected by event-related potentials (ERP). These changes provide objective indicators for the assessment of neuroplasticity. In addition, amyloid PET imaging (Aβ-PET) provides the possibility of quantitatively measuring the amyloid load in the brain. However, there is currently no dynamic monitoring technology based on these modalities in cognitive training for patients after antibody treatment. Therefore, there is an urgent need for a program that integrates fMRI, ERP and Aβ-PET data to personalize the content and dosage of cognitive training to adapt to changes in the patient's nervous system in real time and maximize the effect of cognitive training. Summary of the invention

[0005] The primary technical problem to be solved by the present invention is to provide a cognitive improvement method based on multimodal data.

[0006] Another technical problem to be solved by the present invention is to provide a cognitive improvement system based on multimodal data.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions: According to a first aspect of an embodiment of the present invention, a method for improving cognition based on multimodal data is provided, comprising the following steps: Acquiring first multimodal data of the patient before treatment; Based on the first multimodal data of the patient before treatment, obtaining the amyloid load of the patient to group the patient according to a preset value; Based on the grouping results of the patients, a corresponding initial cognitive training program is obtained; wherein different grouping results correspond to different initial cognitive training programs, and the training dose and adaptive coefficient of each initial cognitive training program are different; During the treatment of the patient, pushing the initial cognitive training program to the patient for cognitive training, and collecting second multimodal data during the treatment of the patient; Based on the first multimodal data of the patient before treatment and the second multimodal data of the patient during treatment, combined with the brain network group atlas, obtaining the difference in amyloid baseline load, functional connection strength, and cognitive ability assessment before and after treatment of the patient; Inputting the difference in the patient's amyloid baseline load before and after treatment, the difference in functional connectivity strength, and the difference in cognitive ability assessment into a preset model to output a cognitive improvement plan that can improve the patient's cognitive index the most; Pushing the cognitive improvement program to the patient for cognitive training to improve the patient's cognitive index; The first multimodal data and the second multimodal data include Aβ-PET data, fMRI data and ERP data, respectively.

[0008] Preferably, the difference in the baseline load of amyloid protein before and after treatment of the patient is obtained by: Based on the Aβ-PET data of the patient before treatment, the baseline load of amyloid protein of the patient before treatment is obtained and expressed as a centiloid value; Obtaining brain region distribution according to the brain network group atlas, and combining the patient's amyloid protein baseline load before treatment to obtain the patient's first brain region amyloid protein distribution matrix; wherein the number of columns of the first brain region amyloid protein distribution matrix is ​​the number of brain regions included in the brain network group atlas, the number of rows is 1, and the matrix elements are the centiloid values ​​of the patient before treatment; Based on the Aβ-PET data of the patient during the treatment, obtaining the baseline load of amyloid protein of the patient during the treatment, and expressing it as a centiloid value; Obtaining brain region distribution according to the brain network group atlas, and combining the patient's amyloid protein baseline load during treatment to obtain the patient's second brain region amyloid protein distribution matrix; wherein the number of columns of the second brain region amyloid protein distribution matrix is ​​the number of brain regions included in the brain network group atlas, the number of rows is at least 1, and the matrix elements are the centiloid values ​​of the patient during treatment; Compare the amyloid protein distribution matrix of the first brain region with the amyloid protein distribution matrix of the second brain region to obtain the patient's brain amyloid protein deposition change matrix; wherein the number of columns of the brain amyloid protein deposition change matrix is ​​the number of brain regions included in the brain network group atlas, the number of rows is 1, and the matrix elements are the differences in centiloid values ​​before and after treatment of the patient.

[0009] Preferably, the difference in functional connection strength before and after treatment of the patient is obtained by: Based on the fMRI data of the patient before treatment, obtaining the functional connection strength of the preset brain network of the patient before treatment; The distribution of brain regions is obtained according to the brain network group atlas, and the first functional connection matrix of the patient is obtained in combination with the functional connection strength of the preset brain network of the patient before treatment; wherein the number of rows and columns of the first functional connection matrix are the number of brain regions included in the brain network group atlas, and the elements of the matrix are the functional connection strengths between the brain regions represented by the rows and columns of the patient before treatment; Based on the fMRI data of the patient during the treatment, obtaining the functional connection strength of the preset brain network of the patient during the treatment; The distribution of brain regions is obtained according to the brain network group atlas, and the second functional connection matrix of the patient is obtained in combination with the functional connection strength of the preset brain network of the patient during the treatment; wherein the number of rows and columns of the second functional connection matrix are the number of brain regions included in the brain network group atlas, and the elements of the matrix are the functional connection strengths between the brain regions represented by the rows and columns during the treatment of the patient; Compare the first functional connectivity matrix with the second functional connectivity matrix to obtain the functional connectivity change matrix of the patient; wherein the number of rows and columns of the functional connectivity change matrix are the number of brain regions included in the brain network group atlas, and the elements of the matrix are the differences in functional connectivity strength between the brain regions represented by the rows and columns before and after treatment of the patient.

[0010] Preferably, the difference in cognitive ability assessment of the patient before and after treatment is obtained by the following method: Based on the ERP data of the patient before treatment, the EEG activity of the patient for cognitive training tasks before treatment is recorded, and preset key indicators are extracted based on the EEG activity; Based on the preset key indicators of the patient before treatment, evaluating multiple cognitive abilities of the user to obtain a first evaluation result; Based on the ERP data of the patient during the treatment, the patient's EEG activity for cognitive training tasks during the treatment is recorded, and preset key indicators are extracted based on the EEG activity; Re-evaluating the user's multiple cognitive abilities based on preset key indicators during the patient's treatment to obtain a second evaluation result; Comparing the first assessment result with the second assessment result to obtain the patient's cognitive ability change result; The patient's cognitive ability change results are combined with the corresponding preset key indicators and converted into an ERP plasticity improvement matrix; wherein the number of columns of the ERP plasticity improvement matrix is ​​the number of the preset key indicators; the number of rows is at least 2, representing the potential amplitude and the occurrence time, respectively.

[0011] Preferably, the preset model is constructed in the following manner: Obtain a large amount of multimodal data of historical users before and after treatment; For each historical user, based on the multimodal data of the historical user before and after treatment, a multimodal change matrix of each historical user is obtained; Obtain the cognitive index of each historical user at different time points; Taking the cognitive index as a reference variable, the multimodal change matrices of all historical users are input into the neural network model to output the cognitive training task combination that improves the cognitive index the most, thereby forming a mapping relationship from the multimodal change matrix to the cognitive training task combination and constructing the preset model.

[0012] Preferably, after the patient undergoes cognitive training based on the cognitive improvement program, the latest multimodal data of the patient is reacquired; Based on the latest multimodal data of the patient, the weights of different types of cognitive training tasks in the cognitive training task combination output by the preset model are adjusted to personalize the preset model.

[0013] Preferably, when the patient undergoes Aβ-PET examination, the patient's plasma is simultaneously collected for detection of AD biomarkers to obtain corresponding plasma test results; Based on the centiloid values ​​of multiple Aβ-PET examinations of the patient and the plasma test results corresponding to each Aβ-PET examination, the centiloid values ​​are input into a preset model framework for model training to output a mapping relationship from the plasma test results to the centiloid values ​​of the Aβ-PET examination, thereby forming a prediction model for predicting the total amount of amyloid protein load for the patient; The next amyloid baseline load of the patient is predicted by the prediction model.

[0014] Preferably, when data of any modality is missing in the first multimodal data and / or the second multimodal data input into the preset model, data of other modalities are supplemented by weighting.

[0015] According to a second aspect of an embodiment of the present invention, a cognitive improvement system based on multimodal data is provided, comprising: A data acquisition unit, used to acquire first multimodal data of a patient before treatment and / or second multimodal data during treatment; wherein the first multimodal data and the second multimodal data include Aβ-PET data, fMRI data and ERP data, respectively; an initial training program push unit connected to the data acquisition unit to obtain the amyloid load of the patient based on the first multimodal data of the patient before treatment, so as to group the patients according to a preset value; and based on the grouping results of the patients, obtain the corresponding initial cognitive training program; wherein different grouping results correspond to different initial cognitive training programs, and the training dose and adaptive coefficient of each initial cognitive training program are different; A data processing unit is connected to the data acquisition unit and the initial training program push unit to obtain a multimodal change matrix of the patient based on the first multimodal data of the patient before treatment and the second multimodal data of the patient during treatment, combined with a brain network group atlas; wherein the multimodal change matrix at least includes: a brain amyloid protein deposition change matrix, a functional connectivity change matrix, and an ERP plasticity improvement matrix; A cognitive improvement program push unit, connected to the data processing unit and having a preset model, for outputting a cognitive improvement program that can improve the cognitive index of the patient the most according to the multimodal change matrix of the patient; The interaction unit is connected to the cognitive improvement program pushing unit to push the cognitive improvement program to the patient for cognitive training to improve the patient's cognitive index.

[0016] According to a third aspect of an embodiment of the present invention, a cognitive improvement system based on multimodal data is provided, comprising a processor and a memory, wherein the processor reads a computer program in the memory to execute the above-mentioned cognitive improvement method based on multimodal data.

[0017] Compared with the prior art, the embodiment of the present invention forms a multimodal change matrix for patients based on the three modal data of amyloid load (detected by Aβ-PET imaging), fMRI and ERP before and after medication, fully utilizing the guiding role of multimodal data, evaluating the patient's neuroplasticity changes in real time, and dynamically generating personalized computerized cognitive training programs. On this basis, through the pre-constructed neuroplasticity prediction model of the historical user group, according to the patient's latest multimodal data, the weights of different types of cognitive training tasks in the cognitive training task combination output by the preset model are adjusted, thereby realizing personalized adjustment of the preset model. This method of combining drug intervention with personalized cognitive training can significantly improve the cognitive improvement effect of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 An overall flow chart of a cognitive improvement method based on multimodal data provided by the first embodiment of the present invention; Figure 2 A structural diagram of a cognitive improvement system based on multimodal data provided in the second embodiment of the present invention; Figure 3 A structural diagram of a cognitive improvement system based on multimodal data provided in the third embodiment of the present invention. DETAILED DESCRIPTION

[0019] The technical content of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The technical concept of the embodiment of the present invention is to evaluate the patient's neuroplasticity changes in real time by integrating Aβ-PET data, functional magnetic resonance imaging (fMRI) data and event-related potential (ERP) data, and dynamically generate personalized computerized cognitive training programs, so as to improve the cognitive function of AD patients who are receiving anti-amyloid monoclonal antibodies (such as lencanezumab) on the basis of drug treatment and in combination with personalized cognitive training.

[0021] It should be noted that early changes in AD often involve dysfunction of the default mode network (DMN), which mainly includes areas such as the posterior cingulate cortex, medial prefrontal cortex, and hippocampus. Studies have shown that a significant reduction in the functional connectivity of the DMN is closely related to cognitive decline. In addition, overactivation of task-positive networks (such as attention networks) is a manifestation of the brain's attempt to compensate for functional loss. These changes reflect insufficient plasticity of neural networks, and anti-amyloid drugs may have an indirect effect on the functional recovery of these networks by reducing the pathological load.

[0022] Amyloid PET imaging uses radioactive tracers (such as 18F-Florbetapir) to bind to amyloid plaques in the brain and assess the patient's pathological burden through quantitative analysis (centiloid value). The centiloid value can accurately reflect the accumulation level of amyloid in the patient's brain and decreases rapidly after antibody treatment. This dynamic change can serve as an indirect marker of improved neuroplasticity.

[0023] In one embodiment of the present invention, a brain network group atlas is used to evaluate the distribution of amyloid. The brain network group atlas is quoted from the atlas published on CerebralCortex in 2016 by Jiang Tianzai's team from the Institute of Automation, Chinese Academy of Sciences. The atlas divides the human brain into 246 fine regions. The brain network group atlas contains fine structures of cerebral cortical brain regions and subcortical nuclei subregions, and quantitatively depicts the anatomical and functional connection patterns of different brain subregions in vivo, and provides a detailed functional description of each subregion. Applying the atlas to PET data helps to accurately characterize the deposition pattern of amyloid.

[0024] Event-related potential (ERP) is a time-locked response of EEG signals to specific stimuli. The P300 wave is a commonly used ERP indicator, reflecting the process of attention allocation and working memory updating. Studies have shown that the increase in P300 amplitude and the shortening of latency are directly related to the improvement of neuroplasticity. Therefore, ERP can be used as a sensitive indicator to evaluate the effect of cognitive training.

[0025] like Figure 1 As shown, a first embodiment of the present invention provides a method for improving cognition based on multimodal data, which comprises at least the following steps: S1: Acquire the first multimodal data of the patient before treatment.

[0026] In one embodiment of the present invention, the multimodal data includes at least Aβ-PET data, fMRI data and ERP data, but is not limited to specific data types. It should be noted that in other embodiments, other types of data can also be selected for multimodal data fusion according to actual needs.

[0027] Specifically, for patients with mild cognitive impairment caused by AD, the baseline amyloid burden x is determined by Aβ-PET examination before the patients receive treatment and is calculated as a centiloid value.

[0028] In addition, functional magnetic resonance imaging (fMRI) data were collected from patients before the start of monoclonal antibody drug treatment to assess the functional connectivity characteristics of the patients' brain networks.

[0029] In addition, before the start of monoclonal antibody drug treatment, event-related potential (ERP) technology was used to collect ERP data from patients, thereby recording the patients' EEG activity in response to cognitive tasks and extracting key indicators such as P300 amplitude and latency.

[0030] S2: Based on the first multimodal data of the patient before treatment, the amyloid protein load in the patient's brain is obtained to group the patient into amyloid protein load groups according to preset values.

[0031] Specifically, the amyloid load in the patient's brain is quantified based on Aβ-PET data, expressed as centiloid values, and the patients are divided into high, medium and low load groups according to preset values. In one embodiment of the present invention, the grouping is performed in the following manner: 12<centiloid≤30: low load group; 30<centiloid≤50: medium load group; centiloid>50: high load group. It is understood that in other embodiments, the preset values ​​can be adjusted as needed, and other methods can be used to group patients, which are not specifically limited here.

[0032] In addition, preferably, when the patient undergoes an Aβ-PET examination, the patient's plasma is simultaneously collected for detection of AD biomarkers to obtain corresponding plasma test results. Then, based on the centiloid values ​​of multiple Aβ-PET examinations of the patient and the plasma test results corresponding to each Aβ-PET examination, the model training is input into the preset model framework to output a mapping relationship from the plasma test results to the centiloid values ​​of the Aβ-PET examination, thereby forming a prediction model for predicting the total amount of amyloid protein load for the patient.

[0033] Once the prediction model is formed, it can be used to predict the patient's next amyloid protein baseline load, thereby eliminating the need for the patient to go to the hospital for an Aβ-PET examination and improving the convenience of data collection (i.e., collection of amyloid protein baseline load).

[0034] S3: Based on the patient's grouping results, obtain the corresponding initial cognitive training plan.

[0035] Specifically, after the patients are grouped based on step S2, the corresponding initial cognitive training program is obtained according to the grouping results of the patients, so as to provide computerized cognitive training for the patients.

[0036] In one embodiment of the present invention, the initial cognitive training program is based on the adaptive cognitive training version, and three training doses K(x), high, medium and low, and three basic adaptive coefficients P(x) are selected according to the grouping results, corresponding to the high load group, medium load group and low load group, respectively.

[0037] S4: Pushing an initial cognitive training program to the patient for cognitive training during the patient's treatment, and collecting second multimodal data during the patient's treatment.

[0038] It is understandable that the type of multimodal data in step S4 is the same as that in step S1. The difference is that the data collection time in step S4 is during the patient's treatment, and cognitive training is also required during the patient's treatment according to the initial cognitive training program obtained in step S3.

[0039] S5: Based on the first multimodal data of the patient before treatment and the second multimodal data of the patient during treatment, combined with the brain network group map, the difference in amyloid baseline load, functional connection strength and cognitive ability assessment before and after treatment are obtained.

[0040] In one embodiment of the present invention, a multimodal change matrix is ​​formed using the difference in amyloid baseline load, functional connectivity strength, and cognitive ability assessment before and after treatment of the patient, and the multimodal change matrix at least includes: a brain amyloid deposition change matrix (corresponding to step S51), a functional connectivity change matrix (corresponding to step S52), and an ERP plasticity improvement matrix (corresponding to step S53). It is understandable that in one embodiment of the present invention, only the matrix is ​​used as an example for explanation, but it is not limited to the form of representing the difference in amyloid baseline load, functional connectivity strength, and cognitive ability assessment before and after treatment of the patient by a matrix.

[0041] The following is a detailed description of the acquisition process of each matrix: S51: Obtain the matrix of changes in amyloid protein deposition in the brain.

[0042] Specifically, it includes steps S511 to S515: S511: Based on the patient's Aβ-PET data before treatment, obtain the patient's amyloid protein baseline load before treatment and express it as a centiloid value.

[0043] S512: Obtain brain region distribution according to the brain network group atlas, and obtain the patient's first brain region amyloid protein distribution matrix in combination with the patient's amyloid protein baseline load before treatment. Specifically, the first brain region amyloid protein distribution matrix has the number of columns equal to the number of brain regions included in the brain network group atlas, the number of rows equal to 1, and the matrix elements are the centiloid values ​​of the patient before treatment.

[0044] S513: Based on the Aβ-PET data of the patient during treatment, the baseline amyloid protein load of the patient during treatment is obtained and expressed as a centiloid value.

[0045] S514: Obtain brain region distribution according to the brain network group atlas, and obtain the patient's second brain region amyloid protein distribution matrix in combination with the patient's amyloid protein baseline load during treatment. Specifically, the second brain region amyloid protein distribution matrix has the number of columns equal to the number of brain regions included in the brain network group atlas, the number of rows equal to 1, and the matrix elements are the centiloid values ​​of the patient during treatment.

[0046] S515: Compare the amyloid protein distribution matrix of the first brain region with the amyloid protein distribution matrix of the second brain region to obtain the patient's brain amyloid protein deposition change matrix.

[0047] After the first brain region amyloid protein distribution matrix and the second brain region amyloid protein distribution matrix are obtained based on step S512 and step S514, the corresponding matrix elements in the two matrices are difference calculated to form a brain amyloid protein deposition change matrix. That is, the number of columns of the brain amyloid protein deposition change matrix is ​​the number of brain regions included in the brain network group atlas, the number of rows is at least 1, and the matrix element is the difference between the centiloid values ​​of the patient before and after treatment.

[0048] S52: Obtain a functional connectivity change matrix.

[0049] Specifically, it includes steps S521 to S525: S521: Based on the fMRI data of the patient before treatment, the functional connection strength of the preset brain network of the patient before treatment is obtained. The functional connection strength can be obtained by calculating the Pearson correlation index, which is a conventional technical means in this field and will not be described in detail here.

[0050] S522: Obtain brain region distribution according to the brain network group atlas, and obtain the patient's first functional connection matrix in combination with the functional connection strength of the patient's preset brain network before treatment. Specifically, the number of rows and columns of the first functional connection matrix are the number of brain regions included in the brain network group atlas, and the elements of the matrix are the functional connection strengths between the brain regions represented by the rows and columns of the patient before treatment.

[0051] S523: Based on the fMRI data of the patient during treatment, the functional connection strength of the patient's preset brain network during treatment is obtained.

[0052] S524: Obtain brain region distribution according to the brain network group atlas, and obtain the second functional connection matrix of the patient in combination with the functional connection strength of the preset brain network of the patient during treatment. Specifically, the number of rows and columns of the second functional connection matrix are the number of brain regions included in the brain network group atlas, and the elements of the matrix are the functional connection strengths between the brain regions represented by the rows and columns during the treatment of the patient.

[0053] S525: Compare the first functional connectivity matrix with the second functional connectivity matrix to obtain a functional connectivity change matrix of the patient.

[0054] Specifically, after the first functional connection matrix and the second functional connection matrix are obtained based on step S512 and step S514, the corresponding matrix elements in the two matrices are calculated by difference, so as to obtain the functional connection change matrix. That is, the number of rows and columns of the functional connection change matrix are the number of brain regions included in the brain network group atlas, and the elements of the matrix are the difference in functional connection strength between the brain regions represented by the rows and columns before and after the treatment of the patient.

[0055] S53: Obtain ERP plasticity improvement matrix.

[0056] Specifically, it includes steps S531 to S536: S531: Based on the patient's ERP data before treatment, the patient's EEG activity for cognitive training tasks before treatment is recorded, and preset key indicators are extracted based on the EEG activity.

[0057] In one embodiment of the present invention, the preset key indicators include: P1, N1, P2, N2, P3. ERP-related event correlation is a response to the stimulus presented in the form of EEG. Generally speaking, the main components are P1, N1, P2, N2, P3. The numbers represent the time after the stimulus is presented, and P and N represent whether the wave is a positive wave or a negative wave. Therefore, P1 (or P100) is a positive wave 100ms after the stimulus is presented, and N2 is a negative wave 200ms after the stimulus is presented, and so on.

[0058] Moreover, in chronological order, the first three (i.e., P1, N1, and P2) are called exogenous components, which are affected by physical stimuli and are only related to the physical characteristics of the stimuli, while the last two (i.e., N2 and P3) are endogenous components, which reflect psychological activities, including but not limited to attention / discrimination / working memory, etc.

[0059] S532: Based on the preset key indicators of the patient before treatment, multiple cognitive abilities of the user are evaluated to obtain a first evaluation result; S533: Based on the ERP data of the patient during the treatment, the patient's EEG activity for cognitive training tasks during the treatment is recorded, and preset key indicators are extracted based on the EEG activity; S534: re-evaluating multiple cognitive abilities of the user based on preset key indicators during the patient's treatment to obtain a second evaluation result; S535: Compare the first assessment result with the second assessment result to obtain the patient's cognitive ability change result; S536: The patient's cognitive ability change results are combined with the corresponding preset key indicators and converted into an ERP plasticity improvement matrix to reflect the real-time changes in the patient's neural plasticity.

[0060] In one embodiment of the present invention, the ERP plasticity improvement matrix has 5 columns, corresponding to P1, N1, P2, N2, and P3 respectively, and at least 2 rows, representing potential amplitude and occurrence time respectively.

[0061] S6: Input the patient's multimodal change matrix into a preset model to output a cognitive improvement plan that can improve the patient's cognitive index the most.

[0062] In one embodiment of the present invention, a model needs to be constructed in advance, so that the multimodal change matrix of the patient is used as the model input, and then the model outputs a cognitive improvement plan that can improve the patient's cognitive index the most.

[0063] Specifically, the preset model is constructed in the following way: ① Obtain a large amount of multimodal data of historical users before and after treatment.

[0064] It can be understood that the multimodal data of the historical user before and after treatment respectively correspond to the data collected in the above-mentioned step S1 and step S4.

[0065] ② For each historical user, obtain the multimodal change matrix of each historical user based on the multimodal data of the historical user before and after treatment.

[0066] It can be understood that the multimodal change matrix of the historical user is the same as the process of obtaining the multimodal change matrix in the above step S5.

[0067] ③ Obtain the cognitive index of each historical user at different time points.

[0068] It is understandable that for each historical user, at different time points, the user cognitive index CI is different, which can reflect the difference in cognitive training effects.

[0069] ④ Build a preset model.

[0070] Specifically, the cognitive index CI is used as a reference variable, and the multimodal change matrix of all historical users is input into the neural network model to output the cognitive training task combination that improves the cognitive index the most, thereby forming a mapping relationship from the multimodal change matrix to the cognitive training task combination, and constructing the preset model. In addition, the loss function of the neural network model can use a cross entropy loss function or other loss functions, which are not specifically limited here.

[0071] S7: Push cognitive improvement programs to patients for cognitive training to improve their cognitive index.

[0072] In one embodiment of the present invention, preferably, after the patient has undergone cognitive training based on the cognitive improvement program, the patient's latest multimodal data is reacquired. Furthermore, based on the patient's latest multimodal data, the weights of different types of cognitive training tasks in the cognitive training task combination output by the preset model are adjusted to personalize the preset model.

[0073] Preferably, when data of any modality is missing in the multimodal data input into the preset model, it is supplemented by weighting the data of other modalities to ensure the stability and continuity of the cognitive improvement program.

[0074] like Figure 2 As shown, based on the above-mentioned first embodiment, the second embodiment of the present invention provides a cognitive improvement system based on multimodal data, including a data collection unit 1, an initial training program push unit 2, a data processing unit 3, a cognitive improvement program push unit 4 and an interaction unit 5.

[0075] Specifically, the data acquisition unit 1 is used to obtain the second multimodal data of the patient before or during treatment. The initial training program push unit 2 is connected to the data acquisition unit 1 to push the initial training program to the user. Among them, three different initial training programs are pre-stored in the initial training program push unit 2, with adaptive cognitive training as the basic version. According to the grouping results of the patients, three training doses K(x) of high, medium and low are selected, and three basic adaptive coefficients P(x) correspond to the high load group, medium load group and low load group respectively.

[0076] The data processing unit 3 is connected to the data acquisition unit 1 and the initial training program push unit 2 to obtain the patient's multimodal change matrix. In one embodiment of the present invention, the multimodal change matrix at least includes: a brain amyloid protein deposition change matrix, a functional connectivity change matrix, and an ERP plasticity improvement matrix. Accordingly, the data processing unit 3 at least includes a brain amyloid protein deposition assessment module 31, a functional connectivity assessment module 32, and an ERP plasticity assessment module 33 to respectively realize the acquisition of the above three matrices.

[0077] The cognitive improvement program push unit 4 is connected to the data processing unit 3 and has a preset model (the preset model is constructed in the manner in the above step S6). Thus, the preset model is used to output the cognitive improvement program that can improve the patient's cognitive index the most according to the multimodal change matrix obtained by the data processing unit 3.

[0078] The interaction unit 5 is connected to the cognitive improvement program pushing unit 4 to push the cognitive improvement program to the patient for cognitive training to improve the patient's cognitive index.

[0079] It is understandable that the above-mentioned functional units are functional modules corresponding to the steps in the above-mentioned first embodiment, but are not limited to the module type and connection form. In other embodiments, other types of modules can be selected as needed to implement the steps in the above-mentioned first embodiment.

[0080] like Figure 3As shown, based on the above first embodiment, the present invention further provides a cognitive enhancement system based on multimodal data. The cognitive enhancement system includes one or more processors 21 and a memory 22. The memory 22 is coupled to the processor 21 and is used to store one or more programs. When the program is executed by the processor 21, the processor 21 implements the cognitive enhancement method based on multimodal data in the above embodiment.

[0081] The processor 21 is used to control the overall operation of the cognitive enhancement system to complete all or part of the steps of the cognitive enhancement method based on multimodal data. The processor 21 can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processing (DSP) chip, etc. The memory 22 is used to store various types of data to support the operation of the cognitive enhancement system, which may include instructions for any application or method used to operate on the cognitive enhancement system, and application-related data. The memory 22 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, etc.

[0082] In an exemplary embodiment, the cognitive enhancement system can be implemented by a computer chip or entity, or by a product with a certain function, for executing the above-mentioned cognitive enhancement method based on multimodal data, and achieving the same technical effect as the above-mentioned method. A typical embodiment is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0083] In another exemplary embodiment, the present invention further provides a computer-readable storage medium including program instructions, which, when executed by a processor, implements the steps of the cognitive enhancement method based on multimodal data in any of the above embodiments. For example, the computer-readable storage medium may be the above-mentioned memory including program instructions, and the above-mentioned program instructions may be executed by a processor of a cognitive enhancement system to complete the above-mentioned cognitive enhancement method based on multimodal data, and achieve the same technical effect as the above-mentioned method.

[0084] In summary, the embodiment of the present invention provides a method and system for cognitive improvement based on multimodal data, which forms a multimodal change matrix for patients based on the three modal data of amyloid load (detected by Aβ-PET imaging), fMRI and ERP before and after medication, and makes full use of the guiding role of multimodal data to evaluate the patient's neuroplasticity changes in real time, and dynamically generate personalized computerized cognitive training programs. On this basis, through the pre-constructed neuroplasticity prediction model of the historical user group, according to the patient's latest multimodal data, the weights of different types of cognitive training tasks in the cognitive training task combination output by the preset model are adjusted, thereby realizing personalized adjustment of the preset model. This method of combining drug intervention with personalized cognitive training can significantly improve the cognitive improvement effect of patients.

[0085] It should be noted that the above embodiments are only examples, and the technical solutions of the various embodiments can be combined, all within the protection scope of the present invention.

[0086] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0087] The above is a detailed description of the cognitive improvement method and system based on multimodal data provided by the present invention. For those skilled in the art, any obvious changes made to it without departing from the essence of the present invention will constitute an infringement of the patent right of the present invention and will bear corresponding legal responsibilities.

Claims

1. A cognitive improvement method based on multimodal data, characterized in that The steps include: Acquiring first multimodal data of the patient before treatment; Based on the first multimodal data of the patient before treatment, obtaining the amyloid load of the patient to group the patient according to a preset value; Based on the grouping results of the patients, a corresponding initial cognitive training program is obtained; wherein different grouping results correspond to different initial cognitive training programs, and the training dose and adaptive coefficient of each initial cognitive training program are different; During the treatment of the patient, pushing the initial cognitive training program to the patient for cognitive training, and collecting second multimodal data during the treatment of the patient; Based on the first multimodal data of the patient before treatment and the second multimodal data of the patient during treatment, combined with the brain network group atlas, obtaining the difference in amyloid baseline load, functional connection strength, and cognitive ability assessment before and after treatment of the patient; Inputting the difference in the patient's amyloid baseline load before and after treatment, the difference in functional connectivity strength, and the difference in cognitive ability assessment into a preset model to output a cognitive improvement plan that can improve the patient's cognitive index the most; Pushing the cognitive improvement program to the patient for cognitive training to improve the patient's cognitive index; The first multimodal data and the second multimodal data include Aβ-PET data, fMRI data and ERP data, respectively.

2. The cognitive improvement method according to claim 1, characterized in that The difference in the patient's amyloid baseline load before and after treatment is obtained by: Based on the Aβ-PET data of the patient before treatment, the baseline load of amyloid protein of the patient before treatment is obtained and expressed as a centiloid value; Obtaining brain region distribution according to the brain network group atlas, and combining the patient's amyloid protein baseline load before treatment to obtain the patient's first brain region amyloid protein distribution matrix; wherein the number of columns of the first brain region amyloid protein distribution matrix is ​​the number of brain regions included in the brain network group atlas, the number of rows is 1, and the matrix elements are the centiloid values ​​of the patient before treatment; Based on the Aβ-PET data of the patient during the treatment, obtaining the baseline load of amyloid protein of the patient during the treatment, and expressing it as a centiloid value; Obtaining brain region distribution according to the brain network group atlas, and combining the patient's amyloid protein baseline load during treatment to obtain the patient's second brain region amyloid protein distribution matrix; wherein the number of columns of the second brain region amyloid protein distribution matrix is ​​the number of brain regions included in the brain network group atlas, the number of rows is at least 1, and the matrix elements are the centiloid values ​​of the patient during treatment; Compare the amyloid protein distribution matrix of the first brain region with the amyloid protein distribution matrix of the second brain region to obtain the patient's brain amyloid protein deposition change matrix; wherein the number of columns of the brain amyloid protein deposition change matrix is ​​the number of brain regions included in the brain network group atlas, the number of rows is 1, and the matrix elements are the differences in centiloid values ​​before and after treatment of the patient.

3. The cognitive enhancement method according to claim 2, characterized in that The functional connection strength difference of the patient before and after treatment is obtained by the following method: Based on the fMRI data of the patient before treatment, obtaining the functional connection strength of the preset brain network of the patient before treatment; The distribution of brain regions is obtained according to the brain network group atlas, and the first functional connection matrix of the patient is obtained in combination with the functional connection strength of the preset brain network of the patient before treatment; wherein the number of rows and columns of the first functional connection matrix are the number of brain regions included in the brain network group atlas, and the elements of the matrix are the functional connection strengths between the brain regions represented by the rows and columns of the patient before treatment; Based on the fMRI data of the patient during the treatment, obtaining the functional connection strength of the preset brain network of the patient during the treatment; The distribution of brain regions is obtained according to the brain network group atlas, and the second functional connection matrix of the patient is obtained in combination with the functional connection strength of the preset brain network of the patient during the treatment; wherein the number of rows and columns of the second functional connection matrix are the number of brain regions included in the brain network group atlas, and the elements of the matrix are the functional connection strengths between the brain regions represented by the rows and columns during the treatment of the patient; Compare the first functional connectivity matrix with the second functional connectivity matrix to obtain the functional connectivity change matrix of the patient; wherein the number of rows and columns of the functional connectivity change matrix are the number of brain regions included in the brain network group atlas, and the elements of the matrix are the differences in functional connectivity strength between the brain regions represented by the rows and columns before and after treatment of the patient.

4. The cognitive enhancement method according to claim 3, characterized in that The difference in cognitive ability assessment of the patient before and after treatment is obtained in the following way: Based on the ERP data of the patient before treatment, the EEG activity of the patient for cognitive training tasks before treatment is recorded, and preset key indicators are extracted based on the EEG activity; Based on the preset key indicators of the patient before treatment, evaluating multiple cognitive abilities of the user to obtain a first evaluation result; Based on the ERP data of the patient during the treatment, the patient's EEG activity for cognitive training tasks during the treatment is recorded, and preset key indicators are extracted based on the EEG activity; Re-evaluating the user's multiple cognitive abilities based on preset key indicators during the patient's treatment to obtain a second evaluation result; Comparing the first assessment result with the second assessment result to obtain the patient's cognitive ability change result; The patient's cognitive ability change results are combined with the corresponding preset key indicators and converted into an ERP plasticity improvement matrix; wherein the number of columns of the ERP plasticity improvement matrix is ​​the number of the preset key indicators; the number of rows is at least 2, representing the potential amplitude and the occurrence time, respectively.

5. The cognitive enhancement method according to claim 4, characterized in that The preset model is constructed in the following way: Obtain a large amount of multimodal data of historical users before and after treatment; For each historical user, based on the multimodal data of the historical user before and after treatment, a multimodal change matrix of each historical user is obtained; Obtain the cognitive index of each historical user at different time points; Taking the cognitive index as a reference variable, the multimodal change matrices of all historical users are input into the neural network model to output the cognitive training task combination that improves the cognitive index the most, thereby forming a mapping relationship from the multimodal change matrix to the cognitive training task combination and constructing the preset model.

6. The cognitive enhancement method according to claim 5, characterized in that: After the patient undergoes cognitive training based on the cognitive improvement program, reacquiring the latest multimodal data of the patient; Based on the latest multimodal data of the patient, the weights of different types of cognitive training tasks in the cognitive training task combination output by the preset model are adjusted to personalize the preset model.

7. The cognitive enhancement method according to claim 2, characterized in that: When the patient undergoes Aβ-PET examination, the patient's plasma is simultaneously collected for detection of AD biomarkers to obtain corresponding plasma test results; Based on the centiloid values ​​of multiple Aβ-PET examinations of the patient and the plasma test results corresponding to each Aβ-PET examination, the centiloid values ​​are input into a preset model framework for model training to output a mapping relationship from the plasma test results to the centiloid values ​​of the Aβ-PET examination, thereby forming a prediction model for predicting the total amount of amyloid protein load for the patient; The next amyloid baseline load of the patient is predicted by the prediction model.

8. The cognitive enhancement method according to claim 1, wherein: When data of any modality is missing in the first multimodal data and / or the second multimodal data input into the preset model, data of other modalities are supplemented by weighting.

9. A cognitive improvement system based on multimodal data, characterized in that include: A data acquisition unit, used to acquire first multimodal data of a patient before treatment and / or second multimodal data during treatment; wherein the first multimodal data and the second multimodal data include Aβ-PET data, fMRI data and ERP data, respectively; an initial training program push unit connected to the data acquisition unit to obtain the amyloid load of the patient based on the first multimodal data of the patient before treatment, so as to group the patients according to a preset value; and based on the grouping results of the patients, obtain the corresponding initial cognitive training program; wherein different grouping results correspond to different initial cognitive training programs, and the training dose and adaptive coefficient of each initial cognitive training program are different; A data processing unit is connected to the data acquisition unit and the initial training program push unit to obtain a multimodal change matrix of the patient based on the first multimodal data of the patient before treatment and the second multimodal data of the patient during treatment, combined with a brain network group atlas; wherein the multimodal change matrix at least includes: a brain amyloid protein deposition change matrix, a functional connectivity change matrix, and an ERP plasticity improvement matrix; A cognitive improvement program push unit, connected to the data processing unit and having a preset model, for outputting a cognitive improvement program that can improve the cognitive index of the patient the most according to the multimodal change matrix of the patient; The interaction unit is connected to the cognitive improvement program pushing unit to push the cognitive improvement program to the patient for cognitive training to improve the patient's cognitive index.

10. A cognitive improvement system based on multimodal data, characterized in that The invention comprises a processor and a memory, wherein the processor reads a computer program in the memory to execute the cognitive improvement method according to any one of claims 1 to 8.

Images