A multimodal data fusion modeling method for individualized cognitive ability assessment prediction

Abstract

Embodiments of the present application relate to a multi-modal data fusion modeling method for individual cognitive ability assessment prediction, comprising: first, the method groups the structured features according to semantics, and groups the brain region features extracted from brain structure images according to types, then maps them to a unified dimension, models global dependence through a multi-layer attention mechanism, and outputs a context representation containing inter-modal interaction information for each layer, and obtains an input feature sequence through average pooling of the sequence output. At the same time, three test result information is obtained, and various grade information is obtained through processing, and a three-dimensional soft label vector is constructed accordingly, wherein the three dimensions correspond to cognitive ability from low to high, and are used as target variables. Finally, the input feature sequence and the three-dimensional soft label vector are input into a neural network for training, the individual cognitive probability is predicted based on the trained network, and the individual cognitive score is obtained by combining the grade information, so as to realize more accurate individual cognitive ability assessment prediction.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multimodal data fusion modeling method for individualized cognitive ability assessment and prediction. Background Technology

[0002] In the field of cognitive ability assessment and prediction, existing models have several limitations. Firstly, they lack accurate modeling of continuous changes in cognitive ability, often providing only discrete assessment results and failing to reflect changes in cognitive ability across a continuous spectrum. Secondly, the labels used are relatively coarse, typically employing "hard labels" that simply categorize individuals to a specific cognitive level, ignoring the complexity and gradual nature of cognitive abilities and losing a significant amount of valuable information. Furthermore, existing models do not fully utilize data information, failing to fully explore the potential correlations and interactions between multimodal data, thus limiting the model's predictive accuracy and generalization ability. Therefore, developing a cognitive ability prediction model that can effectively address these problems is of significant practical importance. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by providing a multimodal data fusion modeling method for individualized cognitive ability assessment and prediction, thereby solving the problems existing in the prior art.

[0004] To achieve the above objectives, this invention provides a multimodal data fusion modeling method for individualized cognitive ability assessment and prediction, comprising:

[0005] The structured features are grouped according to semantics to obtain multiple groups of structured features;

[0006] Brain region features are extracted from brain structural images and grouped according to brain region type to obtain multiple groups of brain region features;

[0007] Multiple sets of structured features and multiple sets of brain region features are mapped to a unified dimension. Global dependencies are modeled through a multi-layer attention mechanism, and each layer outputs a contextual representation containing inter-modal interaction information. The sequence output is averaged and pooled to obtain the final feature vector, which is used as the input feature sequence.

[0008] Obtain three test results; process the three test results to obtain multiple levels of information, and construct a three-dimensional soft label vector based on the multiple levels of information; the cognitive abilities corresponding to the first, second, and third dimensions of the three-dimensional soft label vector are sequentially from low to high; use the three-dimensional soft label vector as the target variable;

[0009] The three-dimensional soft label vector is used as the target variable, combined with the input feature sequence, and input into the neural network to train the neural network. Based on the trained neural network, the individual cognitive probability is predicted.

[0010] An individual cognitive score is obtained based on the scores corresponding to the individual's cognitive probability and level information.

[0011] In one possible implementation, the structured features include: demographic information, anthropometric data, biochemical indicators, questionnaire survey information, and risk scoring information;

[0012] The demographic information includes age, gender, and education level; the body measurement data includes height, weight, and BMI; the biochemical indicators include fasting blood glucose and blood lipids; the questionnaire information includes psychological scores and lifestyle; and the risk score information includes risk values ​​for specific gene loci.

[0013] In one possible implementation, the step of grouping structured features semantically to obtain multiple groups of structured features specifically includes:

[0014] The structured features are grouped semantically, and each group is embedded separately; wherein, the grouping includes grouping by data source, grouping by feature type, and grouping by clinical relevance;

[0015] After standardizing the numerical features, they are mapped to a high dimension through a fully connected layer;

[0016] Assign an independent embedding vector to each category feature; average or concatenate the embeddings of all category features within the group;

[0017] Add a learnable group label to each set of features to represent the group;

[0018] Add a learnable bias vector to each feature within the group to enhance feature discriminative power.

[0019] In one possible implementation, the extraction of brain region features and grouping them according to brain region type to obtain multiple groups of brain region features specifically includes:

[0020] Brain features extracted based on volume and cortical thickness are grouped and embedded according to brain region type, with each group sharing embedding weights;

[0021] All features within the same brain region group share the same embedding layer;

[0022] Perform logarithmic transformation and standardization on the numerical features;

[0023] Encode the MNI coordinates or local spatial relationships of brain regions into learnable vectors.

[0024] In one possible implementation, obtaining the three test result information specifically includes:

[0025] The results of the fluid intelligence test, trail-making test, and symbol-number substitution test are obtained, resulting in the results of the three tests.

[0026] In one possible implementation, the process of processing the three test result information to obtain three levels of information, and constructing a three-dimensional soft label vector based on the three levels of information, specifically includes:

[0027] The scores of the three test results are graded to obtain three levels of information.

[0028] Based on the three levels of information, a three-dimensional soft label vector is constructed.

[0029] In one possible implementation, the method further includes:

[0030] The probability of the third level in the third dimension is no higher than the corresponding probability of the current cognitive state, and the probability of the first level in the first dimension is no lower than the corresponding probability of the current state.

[0031] Secondly, the present invention provides a multimodal data fusion modeling apparatus for individualized cognitive ability assessment and prediction, the apparatus comprising:

[0032] The first grouping module is used to group structured features according to semantics to obtain multiple groups of structured features;

[0033] The second grouping module is used to extract brain region features from brain structure images and group them according to brain region type to obtain multiple groups of brain region features.

[0034] The mapping module maps multiple sets of structured features and multiple sets of brain region features to a unified dimension. It models global dependencies through a multi-layer attention mechanism, and each layer outputs a contextual representation containing inter-modal interaction information. The sequence output is averaged and pooled to obtain the final feature vector, which serves as the input feature sequence.

[0035] The processing module is used to acquire three test result information; process the three test result information to obtain multiple levels of information, and construct a three-dimensional soft label vector based on the multiple levels of information; the cognitive abilities corresponding to the first, second and third dimensions of the three-dimensional soft label vector are sequentially from low to high; and the three-dimensional soft label vector is used as the target variable.

[0036] The input module is used to take the three-dimensional soft label vector as the target variable, combine it with the input feature sequence, input it into the neural network, train the neural network, and predict the individual cognitive probability based on the trained neural network.

[0037] The scoring determination module is used to obtain an individual cognitive score based on the scores corresponding to the individual's cognitive probability and level information.

[0038] Thirdly, the present invention provides a computer server, comprising: a memory, a processor, and a transceiver;

[0039] The processor is used to couple with the memory, read and execute instructions in the memory to implement the multimodal data fusion modeling method for individualized cognitive ability assessment and prediction as described in any of the first aspects;

[0040] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0041] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, the computer program being executed by a processor of the multimodal data fusion modeling method for individualized cognitive ability assessment and prediction as described in any one of the first aspects.

[0042] By applying the multimodal data fusion modeling method for individualized cognitive ability assessment and prediction provided in this invention, a soft labeling mechanism is introduced. Three-dimensional labels are used to achieve three-class classification of individual cognitive states and continuous score calculation, overcoming the limitations of traditional binary classification and significantly improving the ability to model cognitive ability spectrums. This facilitates the capture of early cognitive decline and subclinical abnormalities. In efficient multimodal fusion, multimodal data is integrated, combining Transformer and CNN structures to model complex nonlinear relationships and mine local and global feature interactions, enhancing the model's representation and robustness to high-dimensional heterogeneous data. Reasonable prediction constraints are implemented through post-processing design of prediction constraint strategies, adjusting the output based on the cognitive decline hypothesis to ensure that the prediction conforms to the "cognitive ability declines over time" pattern, improving the prediction's rationality, accuracy, clinical credibility, and interpretability. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the structure of the multimodal data fusion modeling method for individualized cognitive ability assessment and prediction provided in an embodiment of the present invention;

[0044] Figure 2 This is one of the schematic diagrams of a multimodal data fusion modeling method for individualized cognitive ability assessment and prediction provided in an embodiment of the present invention;

[0045] Figure 3 for Figure 2 Flowchart for step 110;

[0046] Figure 4This is the second schematic diagram of the multimodal data fusion modeling method for individualized cognitive ability assessment and prediction provided in this embodiment of the invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0048] Figure 1 This is a schematic diagram of the structure of the multimodal data fusion modeling method for individualized cognitive ability assessment and prediction provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the multimodal data fusion modeling method for individualized cognitive ability assessment and prediction provided in an embodiment of the present invention. The following is in conjunction with... Figure 1 and Figure 2 The technical solution of the present invention will be described with reference to specific embodiments. Figure 2 As shown, this application includes the following steps:

[0049] Step 110: Group the structured features according to semantics to obtain multiple groups of structured features;

[0050] Specifically, structured features include demographic information, anthropometric data, biochemical indicators, questionnaire survey information, and risk scoring information;

[0051] The demographic information includes age, gender, and education level; the anthropometric data includes height, weight, and BMI; the biochemical indicators include fasting blood glucose and blood lipids; the questionnaire information includes psychological scores and lifestyle; and the risk score information includes risk values ​​for specific gene loci. This structured information can constitute a certain number of structured information items, such as 3-dimensional demographic information and 3-dimensional anthropometric data, etc., thus the number of subcategories under each piece of information constitutes the items of the structured features of this application, for example, 245 items. Step 110 specifically includes:

[0052] Step 1101: Group the structured features according to semantics, and embed each group separately; wherein, the grouping includes grouping by data source, grouping by feature type, and grouping by clinical relevance;

[0053] Specifically, data can be categorized by source, such as demographic data (age, gender), biochemical indicators (blood sugar, blood lipids), and questionnaire data (psychological state, lifestyle). For example, categorical data can be categorically coded (e.g., gender, male 1, female 0).

[0054] They are processed separately according to feature type, such as numerical (e.g., BMI, blood glucose) and categorical (e.g., education level, medical history).

[0055] Based on clinical relevance, "blood glucose" and "insulin levels" were grouped into one group (metabolic group), while "anxiety score" and "depression score" were grouped into another group (psychological group).

[0056] Step 1102: After standardizing the numerical features, map them to a high dimension through a fully connected layer;

[0057] Specifically, after standardization, each set of numerical features can be mapped to a higher dimension, such as 512 dimensions.

[0058] Step 1103: Assign an independent embedding vector to each category feature; average or concatenate the embeddings of all category features within the group;

[0059] Specifically, for categorical features, in the learnable embedding layer, an independent embedding vector is assigned to each categorical feature, and the embeddings of all categorical features within the group are averaged or concatenated.

[0060] Step 1104: Add a learnable group label to each set of features as a representative of the group;

[0061] This involves adding learnable group tags to each group. These tags can serve as group-level representations, such as aggregate features of the group, and participate in subsequent Transformer encoding, such as interactions with other modalities. Furthermore, order preservation can be achieved by fixing the position of the group tags in the sequence, for example, the first token representing a demographic group, thus preserving the relative order between groups.

[0062] Step 1105: Add a learnable bias vector to each feature within the group to enhance feature discriminative power.

[0063] The purpose of intra-group bias is to distinguish features. For example, if features "blood sugar" and "cholesterol" in the same group have undergone the same processing, bias can separate their embedding space. It can also be dynamically adjusted, such as the bias parameters being learned automatically during training, without the need to manually design feature weights.

[0064] Step 120: Extract brain region features from brain structure images and group them according to brain region type to obtain multiple groups of brain region features;

[0065] Specifically, brain features extracted based on volume and cortical thickness are grouped and embedded according to brain region type, with each group sharing embedding weights; all features within the same brain region group share the same embedding layer; numerical features are subjected to logarithmic transformation and standardization; and the MNI coordinates or local spatial relationships of brain regions are encoded into learnable vectors.

[0066] This process utilizes FreeSurface to process brain MRI data, extracting multiple features such as volume, signal intensity, surface area, and curvature from different regions. The output features can be categorized by anatomical structure group, signal intensity group, such as average white matter intensity and gray matter intensity. Preprocessing of this data includes missing value handling, standardization, and type unification. An independent embedding layer is used for each feature group, but features within the same group share weights. Spatial information is encoded into learnable vectors, for example, by establishing brain region index embeddings, assigning a unique ID to each brain region, and learning spatial relationships. Alternatively, MNI coordinates can be directly used and encoded via MLP. The feature embeddings and spatial encoding are combined and fed into downstream tasks such as classification or regression.

[0067] Step 130: Map multiple sets of structured features and multiple sets of brain region features to a unified dimension, model global dependencies through a multi-layer attention mechanism, and output a contextual representation containing inter-modal interaction information in each layer; perform average pooling on the sequence output to obtain the final feature vector, which is used as the input feature sequence.

[0068] Specifically, features from different modalities are mapped to the same dimension through independent linear projection layers. A multi-head self-attention mechanism is used to model intermodal interactions and global dependencies, including: copying structured features to each brain region, concatenating structured features and brain region features, and encoding them using a Transformer. The attention output sequence is then average-pooled to generate a global feature vector.

[0069] Step 140: Obtain three test result information; process the three test result information to obtain multiple levels of information, and construct a three-dimensional soft label vector based on the multiple levels of information; the cognitive abilities corresponding to the first, second, and third dimensions of the three-dimensional soft label vector are sequentially from low to high; use the three-dimensional soft label vector as the target variable.

[0070] Specifically, the results of the fluid intelligence test, trail-making test, and symbol-number substitution test are obtained, resulting in three test score classifications. Based on these classifications, a three-dimensional soft-label vector is constructed. For each test, including the fluid intelligence, trail-making, and symbol-number substitution tests, the scores are converted into seven levels ranging from 3 to 9 points. These 3-9 point levels correspond to cognitive abilities from low to high, defining low, medium, and high cognitive levels. A three-dimensional soft-label vector is then constructed, where each dimension represents the probability of an individual being at a low cognitive level, the probability of being at a medium cognitive level, and the probability of being at a high cognitive level, corresponding to the three cognitive levels. Unlike traditional "hard labels," this label can simultaneously retain information from three cognitive dimensions, reflecting the complexity and gradual change of cognitive abilities. Finally, the soft labels of the three tests were merged with equal weights. The three tests were considered to represent an individual's cognitive level in different cognitive dimensions: reasoning ability, visual tracking, and information processing speed. Each dimension was considered equally. For example, the target variable was [0.33, 0.67, 0.0], and the three dimensions represented the probability that an individual was at the corresponding cognitive level, with a sum of 1.

[0071] The cognitive levels of the first, second, and third dimensions increase sequentially. Scores from 3 to 9 can be divided into 7 levels, and these 7 levels can be further subdivided into three cognitive levels, from lowest to highest: the first, second, and third dimensions.

[0072] Step 150: The three-dimensional soft label vector is used as the target variable, combined with the input feature sequence, and input into the neural network to train the neural network. Based on the trained neural network, the individual cognitive probability is predicted.

[0073] Specifically, after feature encoding is completed, the model building and optimization module begins. During model building, the input features are sequentially processed through convolutional layers to extract local features, gradually increasing the representation dimension. Then, a four-layer fully connected network (MLP) is used to achieve feature fusion and non-linear mapping, ultimately outputting a vector of dimension 3, corresponding to the probability values ​​of the three soft labels (low, medium, and high cognitive levels) of comprehensive cognitive ability. Local features are features extracted from local regions of the input data. In a convolutional neural network, convolutional layers slide convolutional kernels across the input data to extract features from local regions; gradually increasing the representation dimension involves multiple convolutional operations, each extracting higher-level features, progressively increasing the abstraction and representational power of the features; the output vector is like [0.33, 0.67, 0.0], where the three dimensions represent the probability of an individual being at the corresponding cognitive level, summing to 1. The output vector has already undergone softmax processing, and each dimension corresponds to the probability of each category, so no further calculation is needed.

[0074] Step 160: Based on the scores corresponding to the individual's cognitive probability and level information, obtain the individual's cognitive score.

[0075] Specifically, the output can be used not only for multi-class assessment but also, through weighted calculation with the grade score, to obtain a continuous cognitive score between 3 and 9, which can be used for subsequent cognitive level quantification and risk stratification. This design allows the model to have interpretable probabilistic outputs, ultimately outputting a 3-dimensional vector corresponding to the probability values ​​of the three soft labels (low, medium, and high cognitive levels) of comprehensive cognitive ability, while retaining the ability to continuously express cognitive ability.

[0076] To address potential errors in model output that contradict cognitive development trends, the module introduces an output optimization strategy based on the assumption of cognitive decline. Given the current cognitive state, the soft-label output of the model's predictions is adjusted: specifically, the probability of high cognitive categories (third dimension) in the predicted values ​​must not exceed the corresponding probability of the current cognitive state, while the probability of low cognitive categories (first dimension) must not be lower than the corresponding probability of the current state. The final output is a 3-dimensional vector, with each dimension corresponding to the probability of each category; no computation is required.

[0077] This strategy reflects the basic pattern of gradual decline in cognitive ability over time in the general population, which helps to improve the rationality of model predictions and clinical interpretability.

[0078] In a specific example, 14,942 participants with low general data loss rates and MRI image data were first selected from the UKB database, and imputation methods were used to fill in features with low loss rates. Then, a cognitive prediction model was built based on 6,690 participants after sample balancing, with 5,352 samples used for model training and 1,338 samples used for independent validation. The model input consisted of multimodal features collected at the 2010 baseline, and the prediction target was a soft label of the individual's comprehensive cognitive ability in 2021.

[0079] Considering that cognitive test results may be influenced by various subjective and accidental factors, a criterion of "difference between predicted score and actual score ≤ 1" is adopted to judge the accuracy of prediction. Based on this criterion, the model achieves an accuracy of 0.81 in quantifying cognitive scores. The final output is a 3-dimensional vector, which, through weighting with the level score, yields a continuous cognitive score between 3 and 9, calculated as 3 * first-dimensional probability + 6 * second-dimensional probability + 9 * third-dimensional probability. Furthermore, in multi-class classification assessment, ROC curves were plotted for the three cognitive states (low, medium, and high), with an average AUC of 0.80, demonstrating particularly significant discrimination ability in extreme cognitive categories.

[0080] By applying the multimodal data fusion modeling method for individualized cognitive ability assessment and prediction provided in this invention, a soft labeling mechanism is introduced. Three-dimensional labels are used to achieve three-class classification of individual cognitive states and continuous score calculation, overcoming the limitations of traditional binary classification and significantly improving the ability to model cognitive ability spectrums. This facilitates the capture of early cognitive decline and subclinical abnormalities. In efficient multimodal fusion, multimodal data is integrated, combining Transformer and CNN structures to model complex nonlinear relationships and mine local and global feature interactions, enhancing the model's representation and robustness to high-dimensional heterogeneous data. Reasonable prediction constraints are implemented through post-processing design of prediction constraint strategies, adjusting the output based on the cognitive decline hypothesis to ensure that the prediction conforms to the "cognitive ability declines over time" pattern, improving the prediction's rationality, accuracy, clinical credibility, and interpretability.

[0081] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0082] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0083] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal data fusion modeling method for individualized cognitive ability assessment and prediction, characterized in that, The method includes: The structured features are grouped semantically to obtain multiple groups of structured features; the structured features include: demographic information, anthropometric data, biochemical indicator information, questionnaire survey information, and risk score information; the questionnaire survey information includes psychological scores and lifestyle data, and the risk score information includes risk values ​​for specific gene loci; Brain region features are extracted from brain structural images and grouped according to brain region type to obtain multiple groups of brain region features; Multiple sets of structured features and multiple sets of brain region features are mapped to a unified dimension. Global dependencies are modeled through a multi-layer attention mechanism, and each layer outputs a contextual representation containing inter-modal interaction information. The sequence output is averaged and pooled to obtain the final feature vector, which is used as the input feature sequence. The system acquires three test results; processes these results to obtain multiple levels of information; and constructs a three-dimensional soft-label vector based on these levels. The cognitive abilities corresponding to the first, second, and third dimensions of the three-dimensional soft-label vector are sequentially ranked from low to high. The three-dimensional soft-label vector is used as the target variable. The three test results include those from a fluid intelligence test, a walkway construction test, and a symbol-number substitution test. The probability of the third level in the third dimension is not higher than the probability corresponding to the current cognitive state, and the probability of the first level in the first dimension is not lower than the probability corresponding to the current state. The three-dimensional soft label vector is used as the target variable, combined with the input feature sequence, and input into the neural network to train the neural network. Based on the trained neural network, the individual cognitive probability is predicted. An individual cognitive score is obtained based on the scores corresponding to the individual's cognitive probability and level information.

2. The method according to claim 1, characterized in that, The demographic information includes age, gender, and education level; the body measurement data includes height, weight, and BMI; and the biochemical information includes fasting blood glucose and blood lipids.

3. The method according to claim 1, characterized in that, The step of grouping structured features according to semantics to obtain multiple groups of structured features specifically includes: The structured features are grouped semantically, and each group is embedded separately; wherein, the grouping includes grouping by data source, grouping by feature type, and grouping by clinical relevance; After standardizing the numerical features, they are mapped to a high dimension through a fully connected layer; Assign an independent embedding vector to each category feature; average or concatenate the embeddings of all category features within the group; Add a learnable group label to each set of features to represent the group; Add a learnable bias vector to each feature within the group to enhance feature discriminative power.

4. The method according to claim 1, characterized in that, The extraction of brain region features and grouping them according to brain region type to obtain multiple groups of brain region features specifically includes: Brain features extracted based on volume and cortical thickness are grouped and embedded according to brain region type, with each group sharing embedding weights; All features within the same brain region group share the same embedding layer; Perform logarithmic transformation and standardization on the numerical features; Encode the MNI coordinates or local spatial relationships of brain regions into learnable vectors.

5. The method according to claim 1, characterized in that, The process of processing the three test results to obtain three levels of information, and constructing a three-dimensional soft label vector based on the three levels of information, specifically includes: The scores of the three test results are graded to obtain three levels of information. Based on the three levels of information, a three-dimensional soft label vector is constructed.

6. A multimodal data fusion modeling device for individualized cognitive ability assessment and prediction, characterized in that, The device includes: The first grouping module is used to group structured features semantically to obtain multiple groups of structured features; the structured features include: demographic information, anthropometric data, biochemical indicator information, questionnaire survey information, and risk score information; the questionnaire survey information includes psychological scores and lifestyle data, and the risk score information includes risk values ​​for specific gene loci; The second grouping module is used to extract brain region features from brain structure images and group them according to brain region type to obtain multiple groups of brain region features. The mapping module maps multiple sets of structured features and multiple sets of brain region features to a unified dimension. It models global dependencies through a multi-layer attention mechanism, and each layer outputs a contextual representation containing inter-modal interaction information. The sequence output is averaged and pooled to obtain the final feature vector, which serves as the input feature sequence. The processing module is used to acquire three test result information; process the three test result information to obtain multiple levels of information, and construct a three-dimensional soft label vector based on the multiple levels of information; the cognitive abilities corresponding to the first, second, and third dimensions of the three-dimensional soft label vector are sequentially from low to high; the three-dimensional soft label vector is used as the target variable; wherein, the three test result information includes the result information of the fluid intelligence test, the walkway construction test, and the symbol-number substitution test; the probability of the third level of the third dimension is not higher than the corresponding probability of the current cognitive state, and the probability of the first level of the first dimension is not lower than the corresponding probability of the current state; The input module is used to take the three-dimensional soft label vector as the target variable, combine it with the input feature sequence, input it into the neural network, train the neural network, and predict the individual cognitive probability based on the trained neural network. The scoring determination module is used to obtain an individual cognitive score based on the scores corresponding to the individual's cognitive probability and level information.

7. A computer server, characterized in that, include: Memory, processor, and transceiver; The processor is used to couple with the memory, read and execute instructions in the memory to implement the multimodal data fusion modeling method for individualized cognitive ability assessment and prediction as described in any one of claims 1-5; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is executed by a processor according to any one of claims 1-5, the multimodal data fusion modeling method for individualized cognitive ability assessment and prediction.

Images