Cognitive competence evaluation processing method and device based on multi-modal data

Abstract

The embodiment of the invention relates to a cognitive competence assessment processing method and device based on multi-modal data. The method comprises the following steps: setting two-stage data typing rules and typing feature coding rules of the multi-modal data; setting a corresponding score-classification relation table for the fluid intelligence scoring indexes of the two-stage data classification detailed rule; carrying out original data collection on the open biomedical database according to the two-stage data classification detailed rule, and carrying out training data arrangement according to the classification feature coding detailed rule, the score-classification relation table and the original data set; constructing a cognitive ability prediction model based on a LightGBM model; performing first-stage hyper-parameter optimization and full-data training on the cognitive competence prediction model, and then performing second-stage fault-tolerant training on the cognitive competence prediction model; and receiving current individual data after the two-stage training is finished, and performing cognitive ability evaluation and feature importance analysis based on the current individual data and the cognitive ability prediction model. According to the invention, the cognitive ability evaluation accuracy can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for cognitive ability assessment and processing based on multimodal data. Background Art

[0002] Traditional cognitive ability assessment methods rely on professionals to categorize subjects' cognitive abilities based on their subjective responses or behavioral observations, resulting in classifications such as "low cognitive ability" and "normal cognitive ability." The drawback of traditional assessment methods is their high degree of subjectivity. The accuracy of the assessment is closely tied to the evaluator's experience, making it prone to subjective bias.

[0003] Compared to the subjectivity of traditional assessment methods, a range of individual information about the subject (such as basic demographic information, lifestyle information, biochemical test results, medical imaging information, proteomics information, etc.) is also correlated with individual cognitive ability levels and possesses good objectivity. If cognitive ability assessment can be based on this series of objective data, it will inevitably reduce subjective bias and improve assessment accuracy. However, the correlation between this data and cognitive ability is not a simple one-to-many or many-to-one linear relationship; most of them are complex nonlinear coupling relationships, making it difficult to analyze and formulate qualitative correlations using conventional statistical methods.

[0004] The LightGBM (Light Gradient Boosting Machine) model is an improved version of the Gradient Boosting Decision Tree (GBDT) model. Its unique leaf growth strategy, histogram algorithm, unilateral gradient sampling, and mutually exclusive feature binding techniques enable faster training, lower memory usage, and faster model inference. It performs exceptionally well in classification and regression tasks and is well-suited for large-scale data inference.

[0005] If the multimodal data of the subjects (including basic demographic information, lifestyle information, biochemical test information, medical imaging information, proteomics information, etc.) can be segmented and integrated, and the LightGBM model training is used to establish a complex nonlinear association between these segmented data and cognitive ability levels (such as cognitive impairment, normal, etc.), the subjectivity of traditional assessment methods can be significantly reduced, and the objectivity and accuracy of the assessment can be improved. How to segment and integrate the multimodal data of the subjects and use the LightGBM model training to establish a complex nonlinear association between multimodal data and cognitive ability levels is the technical problem that the present invention needs to solve. Summary of the Invention

[0006] The purpose of the present invention is to provide a method, device, electronic device and computer-readable storage medium for cognitive ability assessment based on multimodal data in response to the defects of the prior art. The present invention first sets two-level data classification rules and classification feature coding rules for multimodal data, and sets a corresponding score-classification relationship table for the fluid intelligence scoring index, and collects original data from the biomedical database according to the two-level data classification rules, and constructs a training data set (i.e., a first data set) based on the classification feature coding rules, the score-classification relationship table and the original data set, and constructs a cognitive ability prediction model based on the LightGBM model; then, the cognitive ability prediction model is first subjected to a first-stage hyperparameter optimization and full-data training based on the first data set, and then the cognitive ability prediction model is subjected to a second-stage fault-tolerant training based on the first data set; finally, the cognitive ability prediction model and the SHAP algorithm are used to perform cognitive ability assessment and feature importance analysis based on the individual's multimodal data. The present invention can not only improve the accuracy of the individual's cognitive ability assessment, but also identify the strong / weak correlation between all the individual's segmented features and the cognitive ability level.

[0007] To achieve the above objectives, a first aspect of an embodiment of the present invention provides a method for cognitive ability assessment based on multimodal data, the method comprising:

[0008] Two-level data classification rules and classification feature coding rules for multimodal data are set; the first-level data types of the two-level data classification rules include five types of unimodal information groups and one type of evaluation information group, and each type of the unimodal information group includes one or more corresponding second-level data types; the five types of unimodal information groups include: a demographic information group, a lifestyle information group, a biochemical test information group, a medical imaging information group, and a proteomics information group; the second-level data types of the evaluation information group include at least a fluid intelligence scoring index; the classification feature coding rules configure corresponding feature coding rules for each type of the second-level data type of each type of the unimodal information group, and configure corresponding multimodal feature vector coding rules for multimodal data;

[0009] A corresponding score-classification relationship table is set for the fluid intelligence score indicator of the two-level data classification rules; the score-classification relationship table includes two relationship records; the relationship record includes a score field and a classification field; the score field is a score range for a fluid intelligence score; the classification field includes a cognitive impairment type and a cognitive normal type;

[0010] Collecting raw data from an open biomedical database according to the two-level data classification rules to obtain an original data set; and organizing training data according to the classification feature coding rules, the score-classification relationship table, and the original data set to obtain a first data set;

[0011] A binary classification model is constructed based on the LightGBM model as the corresponding cognitive ability prediction model; the cognitive ability prediction model is used to predict the cognitive ability level based on the multimodal feature vector input by the model and output the corresponding model prediction probability; the value range of the model prediction probability is between 0 and 1;

[0012] First, performing a first-stage hyperparameter optimization and full-data training on the cognitive ability prediction model based on the first data set; then performing a second-stage fault-tolerant training on the cognitive ability prediction model based on the first data set;

[0013] After the two-stage training is completed, the multimodal data input by the user that meets the requirements of the two-level data classification rules is received as the current individual data; and based on the current individual data and the cognitive ability prediction model, cognitive ability assessment and feature importance analysis are performed to obtain corresponding individual assessment results and individual analysis results and feedback to the current user.

[0014] Preferably, the secondary data types of the demographic information group include at least: gender, age, weight, waist circumference, height, leg fat percentage, arm fat percentage, body mass index, and body fat percentage;

[0015] The secondary data types of the lifestyle information group include at least: prenatal maternal smoking status, birth weight, breastfeeding during infancy, childhood economic status, drinking frequency, social activities, diastolic blood pressure, systolic blood pressure, pulse rate, peak expiratory flow, forced expiratory volume in 1 second, own smoking status, eating habits, environmental noise, and air pollution index;

[0016] The secondary data types of the biochemical test information group include at least: albumin index, C-reactive protein index, white blood cell count index, red blood cell distribution width index, urine potassium index, urine sodium index, fasting blood glucose index, total cholesterol index, and creatinine index;

[0017] The secondary data types of the medical imaging information group include at least: amygdala signal intensity, brain white matter volume, left ventricular ejection fraction, left ventricular myocardial mass, liver fat content, spleen length, and carotid intima-media thickness;

[0018] The secondary data types of the proteomics information group include at least: an apolipoprotein index, a glial fibrillary acidic protein index, an interleukin-6 index, a leptin protein index, an α-tumor necrosis factor index, and an adiponectin index;

[0019] The open biomedical database at least includes the UK Biobank database;

[0020] The original data set includes a plurality of original data records; the original data records include the demographic information group, the lifestyle information group, the biochemical test information group, the medical imaging information group, the proteomics information group and the assessment information group;

[0021] The first data set includes multiple first data records; the first data record includes a first training vector and a first label probability; the first label probability is 0 or 1; the first label probability is 0, indicating that the corresponding cognitive ability level type is a cognitive ability impairment type, and is 1, indicating that the corresponding cognitive ability level type is a cognitive ability normal type.

[0022] Preferably, the typing feature coding details are feature coding rules configured for the population information group including:

[0023] Set the gender feature coding rule to binary coding rule:

[0024]

[0025] Set the age feature encoding rule to the Z-score standardization rule:

[0026]

[0027] Among them, μ age , σ age is the preset age mean and age variance;

[0028] Set the waist circumference, height, weight, leg fat percentage, arm fat percentage, body mass index, and body fat percentage feature coding rules to Min-Max standardization rules:

[0029]

[0030] Among them, max waist 、min waist For the preset maximum and minimum waist circumference, max height 、min height For the preset maximum and minimum height, max weight 、min weight For the preset maximum and minimum weight, max lfp 、min lfp For the preset maximum and minimum leg fat rate, max afp 、min afp For the preset maximum and minimum arm fat rate, max BMI 、min BMI The preset maximum and minimum body mass index, max bfp 、min bfpThe preset maximum and minimum body fat percentages;

[0031] Setting the classification feature coding details as feature coding rules configured for the lifestyle information group includes:

[0032] The coding rules for prenatal maternal smoking status and infant breastfeeding characteristics are set to binary coding rules:

[0033]

[0034] The coding rules for childhood economic status, drinking frequency, social activities, smoking status, and dietary habits are set as multi-category coding rules. Specifically, the childhood economic status characteristics are coded according to multiple preset economic status levels; the drinking frequency characteristics are coded according to multiple preset drinking frequency levels; the social activity characteristics are coded according to multiple preset social activity levels; the smoking status characteristics are coded according to the preset three-level coding of never smoking, having a smoking history but also quitting smoking, and having a smoking history but not quitting smoking; and the dietary habits characteristics are coded according to multiple preset dietary habit types.

[0035] Set the birth weight, environmental noise, and air pollution index feature coding rules to Min-Max standardization rules:

[0036]

[0037] Among them, max b-weight 、min b-weight The maximum and minimum birth weights are preset. noise 、min noise For the preset maximum and minimum ambient noise, max air 、min air The preset maximum and minimum air pollution index;

[0038] Set the feature coding rules for diastolic blood pressure, systolic blood pressure, pulse rate, peak expiratory flow, and forced expiratory volume in one second to the Z-score standardization rule:

[0039]

[0040] Among them, μ dp , σ dp is the preset diastolic pressure mean and diastolic pressure variance, μ sp , σ sp is the preset systolic blood pressure mean and systolic blood pressure variance, μ pf , σ pf is the preset pulse rate mean and pulse rate variance, μ pef , σ pefis the preset mean and variance of peak expiratory flow, μ fev1 , σ fev1 The default forced expiratory volume in 1 second mean and forced expiratory volume in 1 second variance;

[0041] The typing feature coding details are feature coding rules configured for the biochemical test information group, including:

[0042] The characteristic coding rules for albumin index, C-reactive protein index, white blood cell count index, red blood cell distribution width index, urine potassium index, urine sodium index, fasting blood glucose index, total cholesterol index, and creatinine index are all set to Z-score standardization rules:

[0043]

[0044] Among them, μ bio1-9 is the mean of the preset albumin index, C-reactive protein index, white blood cell count index, red blood cell distribution width index, urine potassium index, urine sodium index, fasting blood glucose index, total cholesterol index, and creatinine index, σ bio1-9 The variance of the preset albumin index, C-reactive protein index, white blood cell count index, red blood cell distribution width index, urine potassium index, urine sodium index, fasting blood glucose index, total cholesterol index, and creatinine index;

[0045] The classification feature coding rules are feature coding rules configured for the medical image information group, including:

[0046] Set the feature encoding rules for amygdala signal intensity, white matter volume, left ventricular ejection fraction, left ventricular myocardial mass, liver fat content, spleen length, and carotid intima-media thickness to Min-Max standardization rules:

[0047]

[0048] Among them, max ass 、min ass is the preset maximum and minimum amygdala signal strength, max wvb 、min wvb is the preset maximum and minimum brain white matter volume, max lvef 、min lvef is the preset maximum and minimum left ventricular ejection fraction, max LVMI 、min LVMI The maximum and minimum left ventricular myocardial mass are preset, max LFC 、min LFC The maximum and minimum liver fat contents are preset, max sl 、min sl The maximum and minimum spleen lengths are preset.IMT 、min IMT The preset maximum and minimum carotid intima-media thickness;

[0049] The typing feature coding rules are feature coding rules configured for the proteomics information group, including:

[0050] The characteristic coding rules for apolipoprotein index, glial fibrillary acidic protein index, interleukin-6 index, leptin protein index, α-tumor necrosis factor index, and adiponectin index are all set to Z-score standardization rules:

[0051]

[0052]

[0053] Among them, μ APOE 、μ GFAP 、μ IL-6 、μ leptin 、μ TNF-α 、μ apn is the mean value of the preset apolipoprotein index, glial fibrillary acidic protein index, interleukin-6 index, leptin protein index, α tumor necrosis factor index, and adiponectin index, σ APOE , σ GFAP , σ IL-6 , σ leptin , σ TNF-α , σ apn The variances of the preset apolipoprotein index, glial fibrillary acidic protein index, interleukin-6 index, leptin protein index, α-tumor necrosis factor index, and adiponectin index;

[0054] The typing feature coding details are as follows: the multimodal characteristic vector coding rules corresponding to the multimodal data configuration are: the gender feature, the age feature, the weight feature, the waist circumference feature, the height feature, the leg fat rate feature, the arm fat rate feature, the body mass index feature, and the body fat rate feature corresponding to the demographic information group constitute a corresponding first single-modal feature vector; and the prenatal maternal smoking status feature, the birth weight feature, the infant breastfeeding feature, the childhood economic status feature, the drinking frequency feature, the social activity feature, the diastolic blood pressure feature, the systolic blood pressure feature, the pulse rate feature, the peak expiratory flow feature, the forced expiratory volume in 1 second feature, the smoking status feature, the eating habit feature, the environmental noise feature, and the air pollution index feature corresponding to the lifestyle information group constitute a corresponding second single-modal feature vector; and the albumin index feature, the C-reactive protein index feature, and the biochemical test information group correspond to the The characteristics, the white blood cell number index characteristics, the red blood cell distribution width index characteristics, the urine potassium index characteristics, the urine sodium index characteristics, the fasting blood glucose index characteristics, the total cholesterol index characteristics, and the creatinine index characteristics constitute a corresponding third unimodal feature vector; and the amygdala signal intensity characteristics, the brain white matter volume characteristics, the left ventricular ejection fraction characteristics, the left ventricular myocardial mass characteristics, the liver fat content characteristics, the spleen length characteristics, and the carotid intima-media thickness characteristics corresponding to the medical imaging information group constitute a corresponding fourth unimodal feature vector; and the apolipoprotein index characteristics, the glial fibrillary acidic protein index characteristics, the interleukin-6 index characteristics, the leptin protein index characteristics, the α tumor necrosis factor index characteristics, and the adiponectin index characteristics corresponding to the proteomics information group constitute a corresponding fifth unimodal feature vector; and the first, second, third, fourth and fifth unimodal feature vectors constitute the corresponding multimodal feature vector.

[0055] Preferably, the first data set is obtained by arranging training data according to the typing feature coding rules, the score-classification relationship table and the original data set, specifically including:

[0056] A round of traversal is performed on all the original data records of the original data set; and during this round of traversal, the original data record currently traversed is used as the corresponding current record; and according to the classification feature coding rules, the demographic information group, the lifestyle information group, the biochemical test information group, the medical imaging information group, and the proteomics information group of the current record are multimodally encoded to obtain the corresponding multimodal feature vector; and the current multimodal feature vector is used as the corresponding first training vector; and in the score-classification relationship table, the classification field of the relationship record whose score field satisfies the fluid intelligence scoring index of the assessment information group of the current record is used as the corresponding first label type; and the first label type is identified, and if the first label type is a cognitive impairment type, the corresponding first label probability is set to 0; if the first label type is a cognitive ability normal type, the corresponding first label probability is set to 1; and the first training vector corresponding to the current record and the first label probability are used to form a corresponding first data record; and at the end of this round of traversal, all the obtained first data records are used to form the corresponding first data set.

[0057] Preferably, the one-stage hyperparameter optimization and full-data training of the cognitive ability prediction model based on the first data set specifically includes:

[0058] Step 51: randomly dividing the first data set into two sub-data sets based on a preset first segmentation ratio and recording them as a corresponding first training set and a first evaluation set;

[0059] Wherein, both the first training set and the first evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio;

[0060] Step 52: setting the first optimization objective function to a cross entropy loss function; and setting the parameter ranges of the nine types of hyperparameters of the cognitive ability prediction model;

[0061] The nine types of hyperparameters include the number of leaf nodes, the maximum depth of the decision tree, the total number of decision trees, the learning rate, the minimum number of samples of leaf nodes, the L1 regularization coefficient, the L2 regularization coefficient, the feature sampling ratio, and the sample sampling ratio;

[0062] Step 53: performing a round of iterative optimization on the nine types of hyperparameters based on the TPE algorithm according to a preset first iteration number threshold, the first training set, and the first optimization objective function to obtain the corresponding current optimal hyperparameter combination;

[0063] Step 54: Under the premise that the nine types of hyperparameters of the cognitive ability prediction model are set as the current optimal hyperparameter combination, continue to perform a round of iterative optimization on the model parameters of the cognitive ability prediction model based on a preset second iteration number threshold, the first training set, and the first optimization objective function;

[0064] Step 55. After this round of iterative optimization is completed, the AUC indicator and the logarithmic loss indicator are calculated based on the first evaluation set and the cognitive ability prediction model to obtain the corresponding first AUC indicator and first logarithmic loss indicator; and it is identified whether the first AUC indicator and the first logarithmic loss indicator each meet the corresponding first AUC indicator range and first logarithmic loss indicator range; if not, return to step 53 to continue training; if so, it is confirmed that the full data training of the cognitive ability prediction model is completed.

[0065] Preferably, the two-stage fault-tolerant training of the cognitive ability prediction model based on the first data set specifically includes:

[0066] Step 61: randomly selecting a plurality of first data records from the first data set as second data records to form a corresponding second data set according to a preset first extraction ratio; and clearing a plurality of features in a second training vector of each second data record according to a preset first reset ratio;

[0067] The second data set includes a plurality of second data records; the ratio of the total number of records in the second data set to the total number of records in the first data set satisfies the first extraction ratio; the second data record includes the second training vector and the second label probability; the ratio of the total number of features reset to zero in the second training vector to the total number of vector features satisfies the first reset ratio; the second label probability is 0 or 1; a second label probability of 0 indicates that the corresponding cognitive ability level type is a cognitive impairment type, and a second label probability of 1 indicates that the corresponding cognitive ability level type is a normal cognitive ability type; the first extraction ratio is ≥80%, and the first reset ratio is ≤40%;

[0068] Step 62: randomly divide the second data set into two sub-data sets based on a preset second segmentation ratio, and record them as a corresponding second training set and a second evaluation set;

[0069] Wherein, both the second training set and the second evaluation set are composed of a plurality of second data records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second segmentation ratio;

[0070] Step 63, setting the second optimization objective function to a cross entropy loss function;

[0071] Step 64: performing a round of iterative optimization on the model parameters of the cognitive ability prediction model based on a preset third iteration number threshold, the second training set, and the second optimization objective function;

[0072] Step 65: After this round of iterative optimization is completed, the AUC indicator and the logarithmic loss indicator are calculated based on the second evaluation set and the cognitive ability prediction model to obtain the corresponding second AUC indicator and the second logarithmic loss indicator; and whether the second AUC indicator and the second logarithmic loss indicator each meet the corresponding second AUC indicator range and the second logarithmic loss indicator range is identified; if not, return to step 64 to continue training; if so, it is confirmed that the fault-tolerant training of the cognitive ability prediction model is completed.

[0073] Preferably, the cognitive ability assessment and feature importance analysis based on the current individual data and the cognitive ability prediction model are performed to obtain corresponding individual assessment results and individual analysis results and to feed back to the current user, specifically including:

[0074] Based on the classification feature coding rules, the current individual data is multimodal feature coded to obtain the corresponding multimodal feature vector; and the current multimodal feature vector is input into the cognitive ability prediction model to predict the cognitive ability level to obtain the corresponding model prediction probability as the current prediction probability;

[0075] Whether the current predicted probability exceeds a preset normal level probability threshold is determined; if so, the corresponding individual assessment result is set as normal cognitive ability; if not, the corresponding individual assessment result is set as low cognitive ability;

[0076] Calculating the SHAP values ​​of various secondary data features of the current multimodal feature vector based on the SHAP algorithm to obtain corresponding feature SHAP values; sorting all the secondary data types in descending order of the feature SHAP values ​​to obtain corresponding feature importance sequences; and using the feature importance sequences as the corresponding individual analysis results;

[0077] Among them, the secondary data features include the gender feature, the age feature, the weight feature, the waist circumference feature, the height feature, the leg fat rate feature, the arm fat rate feature, the body mass index feature, the body fat rate feature, the prenatal maternal smoking status feature, the birth weight feature, the infant breastfeeding feature, the childhood economic status feature, the drinking frequency feature, the social activity feature, the diastolic blood pressure feature, the systolic blood pressure feature, the pulse rate feature, the peak expiratory flow feature, the forced expiratory volume in 1 second feature, the smoking status feature, the eating habit feature, the environmental noise feature, the air pollution index feature, the albumin index feature The characteristics of the human body are as follows: the C-reactive protein index characteristic, the white blood cell count index characteristic, the red blood cell distribution width index characteristic, the urine potassium index characteristic, the urine sodium index characteristic, the fasting blood glucose index characteristic, the total cholesterol index characteristic, the creatinine index characteristic, the amygdala signal intensity characteristic, the brain white matter volume characteristic, the left ventricular ejection fraction characteristic, the left ventricular myocardial mass characteristic, the liver fat content characteristic, the spleen length characteristic, the carotid intima-media thickness characteristic, the apolipoprotein index characteristic, the glial fibrillary acidic protein index characteristic, the interleukin-6 index characteristic, the leptin protein index characteristic, the α-tumor necrosis factor index characteristic, and the adiponectin index characteristic.

[0078] A second aspect of an embodiment of the present invention provides a device for implementing the cognitive ability assessment and processing method based on multimodal data described in the first aspect, the device comprising: a data rule customization module, a first data preparation module, a second data preparation module, a model construction module, a model training module, and a model application module;

[0079] The data rule customization module is used to set two-level data typing rules and typing feature coding rules for multimodal data; the first-level data types of the two-level data typing rules include five types of unimodal information groups and one type of evaluation information group, and each type of unimodal information group includes one or more corresponding second-level data types; the five types of unimodal information groups include: demographic information group, lifestyle information group, biochemical test information group, medical imaging information group, and proteomics information group; the second-level data type of the evaluation information group includes at least a fluid intelligence scoring index; the typing feature coding rules configure corresponding feature coding rules for each type of second-level data type in each type of unimodal information group, and configure corresponding multimodal feature vector coding rules for multimodal data;

[0080] The first data preparation module is used to set a corresponding score-classification relationship table for the fluid intelligence score indicator of the two-level data classification rules; the score-classification relationship table includes two relationship records; the relationship record includes a score field and a classification field; the score field is a score range for a fluid intelligence score; the classification field includes a cognitive impairment type and a cognitive normal type;

[0081] The second data preparation module is used to collect raw data from an open biomedical database according to the two-level data typing rules to obtain a raw data set; and to organize training data according to the typing feature coding rules, the score-classification relationship table and the raw data set to obtain a first data set;

[0082] The model building module is used to build a binary classification model based on the LightGBM model as the corresponding cognitive ability prediction model; the cognitive ability prediction model is used to predict the cognitive ability level according to the multimodal feature vector input by the model and output the corresponding model prediction probability; the value range of the model prediction probability is between 0 and 1;

[0083] The model training module is used to first perform a first-stage hyperparameter optimization and full-data training on the cognitive ability prediction model based on the first data set; and then perform a second-stage fault-tolerant training on the cognitive ability prediction model based on the first data set;

[0084] The model application module is used to receive multimodal data input by the user that meets the requirements of the two-level data classification rules as current individual data after the two-stage training is completed; and perform cognitive ability assessment and feature importance analysis based on the current individual data and the cognitive ability prediction model to obtain corresponding individual assessment results and individual analysis results and feedback them to the current user.

[0085] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0086] The processor is configured to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in the first aspect above;

[0087] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0088] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a computer, the computer executes the instructions of the method described in the first aspect above.

[0089] The embodiment of the present invention provides a method, device, electronic device and computer-readable storage medium for cognitive ability assessment based on multimodal data. As can be seen from the above content, the embodiment of the present invention first sets two-level data classification rules and classification feature coding rules for multimodal data, and sets a corresponding score-classification relationship table for the fluid intelligence scoring index, and collects original data from the biomedical database according to the two-level data classification rules, and constructs a training data set (i.e., a first data set) based on the classification feature coding rules, the score-classification relationship table and the original data set, and constructs a cognitive ability prediction model based on the LightGBM model; then, the cognitive ability prediction model is first subjected to a first-stage hyperparameter optimization and full-data training based on the first data set, and then the cognitive ability prediction model is subjected to a second-stage fault-tolerant training based on the first data set; finally, the cognitive ability prediction model and the SHAP algorithm are used to perform cognitive ability assessment and feature importance analysis based on the individual's multimodal data. Through the embodiment of the present invention, not only the assessment accuracy of the individual's cognitive ability is improved, but also the ability to parse the correlation strength between the individual's multi-dimensional biological characteristics and cognitive ability is enhanced, and the specific weights of the impact of different characteristics on cognitive ability can be quantified and identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 A schematic diagram of a cognitive ability assessment and processing method based on multimodal data provided in Example 1 of the present invention;

[0091] Figure 2 This is a module structure diagram of a cognitive ability assessment and processing device based on multimodal data provided in Example 2 of the present invention;

[0092] Figure 3 This is a structural diagram of an electronic device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0093] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described herein are merely some, rather than all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0094] The first embodiment of the present invention provides a method for evaluating cognitive ability based on multimodal data. Figure 1 A method for evaluating cognitive ability based on multimodal data is shown in the following diagram, which is provided in the first embodiment of the present invention. The method mainly includes the following steps:

[0095] Step 1: Set the two-level data classification rules and classification feature coding rules for multimodal data.

[0096] Here, the primary data type of the two-level data classification rules of the embodiment of the present invention includes five types of single-modal information groups and one type of evaluation information group, and each type of single-modal information group also includes one or more corresponding secondary data types.

[0097] The secondary data type of the assessment information group includes at least a Fluid Intelligence scoring indicator, which is an integer score indicator.

[0098] The five types of single-modal information groups include: demographic information group, lifestyle information group, biochemical test information group, medical imaging information group, and proteomics information group; among them:

[0099] 1) The secondary data types of the demographic information group include at least: gender, age, weight, waist circumference, height, leg fat percentage, arm fat percentage, body mass index, and body fat percentage;

[0100] Here, gender differences lead to differences in social cognition, which in turn establish an indirect correlation with cognitive ability; there is an "inverted U-shaped" correlation between age and cognitive ability; leg fat percentage and arm fat percentage are positively correlated with cognitive function; the more stable the weight and body mass index, the lower the risk of cognitive impairment; an increased body fat percentage is associated with a risk of decreased cognitive ability; an increased waist circumference is associated with a risk of decreased cognitive ability; height growth in an individual's early developmental stage is associated with improved cognitive ability, and height in adulthood indirectly affects cognitive performance through self-efficacy;

[0101] 2) The secondary data types of the lifestyle information group include at least: prenatal maternal smoking status, birth weight, breastfeeding during infancy, childhood economic status, drinking frequency, social activities, diastolic blood pressure, systolic blood pressure, pulse rate, peak expiratory flow, forced expiratory volume in 1 second, own smoking status, dietary habits, environmental noise, and air pollution index;

[0102] Here, prenatal maternal smoking poses a risk of reduced offspring cognition; breastfeeding during infancy helps improve cognitive ability; both underweight and overweight birthweight are associated with a risk of cognitive impairment; childhood economic status, drinking frequency, smoking status, environmental noise, and air pollution index are negatively correlated with improved cognitive ability; social activities and eating habits can indirectly reflect the level of individual cognitive ability; and health status, as reflected by parameters such as diastolic blood pressure, systolic blood pressure, pulse rate, peak expiratory flow, and forced expiratory volume in one second, is negatively correlated with the rate of cognitive decline.

[0103] 3) The secondary data types of the biochemical test information group include at least: albumin index, C-reactive protein index, white blood cell count index, red blood cell distribution width index, urine potassium index, urine sodium index, fasting blood glucose index, total cholesterol index, and creatinine index;

[0104] Here, the health status reflected by all biochemical test information is negatively correlated with the rate of cognitive decline;

[0105] 4) The secondary data types of the medical imaging information group include at least: amygdala signal intensity, brain white matter volume, left ventricular ejection fraction, left ventricular myocardial mass, liver fat content, spleen length, and carotid intima-media thickness;

[0106] Here, the health level reflected by all medical imaging information is negatively correlated with the rate of cognitive decline;

[0107] 5) The secondary data types of the proteomics information group include at least: apolipoprotein index, glial fibrillary acidic protein index, interleukin-6 index, leptin protein index, α-tumor necrosis factor index, and adiponectin index.

[0108] Here, the health level reflected by all proteomic information is negatively correlated with the rate of cognitive decline.

[0109] The classification feature coding rules of the embodiment of the present invention configure corresponding feature coding rules for each secondary data type of each unimodal information group, and configure corresponding multimodal feature vector coding rules for multimodal data.

[0110] 1) The feature coding rules for the demographic information group include:

[0111] a. Set the gender feature coding rule to binary coding rule:

[0112]

[0113] b. Set the age feature coding rule to the Z-score standardization rule:

[0114]

[0115] Among them, μ age , σ age is the preset age mean and age variance;

[0116] It should be noted that the means and variances of the secondary data features corresponding to all secondary data types in the embodiments of the present invention can be obtained based on big data statistics of open biomedical databases, or based on big data statistics of the population of a certain region in a real scenario, or based on big data statistics of a specified population in a real scenario;

[0117] c. Set the waist circumference, height, weight, leg fat percentage, arm fat percentage, body mass index, and body fat percentage feature coding rules to Min-Max standardization rules:

[0118]

[0119]

[0120] Among them, max waist 、min waist For the preset maximum and minimum waist circumference, max height 、min height For the preset maximum and minimum height, max weight 、min weight For the preset maximum and minimum weight, max lfp 、min lfp For the preset maximum and minimum leg fat rate, max afp 、min afp For the preset maximum and minimum arm fat rate, max BMI 、min BMI The preset maximum and minimum body mass index, max bfp 、min bfp The preset maximum and minimum body fat percentages;

[0121] It should be noted that the maximum and minimum values ​​of the secondary data features corresponding to all secondary data types in the embodiments of the present invention can be obtained based on manually set thresholds, or based on big data statistics of open biomedical databases, or based on big data statistics of the population of a certain region in real scenarios, or based on big data statistics of a specified group of people in real scenarios;

[0122] d. Detailed rules for the classification feature coding: The feature coding rules configured for the lifestyle information group include:

[0123] e. Set the coding rules for maternal smoking status before delivery and breastfeeding characteristics during infancy to binary coding rules:

[0124]

[0125] f. Set the coding rules for childhood economic status, drinking frequency, social activities, smoking status, and dietary habits as multi-category coding rules. Specifically, set the childhood economic status characteristics to be coded according to multiple preset economic status levels; set the drinking frequency characteristics to be coded according to multiple preset drinking frequency levels; set the social activity characteristics to be coded according to multiple preset social activity levels; set the smoking status characteristics to be coded according to the preset three-level coding of never smoking, smoking history but also quitting, and smoking history but not quitting; set the dietary habits characteristics to be coded according to multiple preset dietary habit types;

[0126] Here, the multiple economic status levels are a set of pre-set classification information, such as economic status = poor / average / rich, corresponding to levels 0 / 1 / 2; the multiple drinking frequency levels are a set of pre-set classification information, such as drinking frequency = 0 days / week, corresponding to level 0, drinking frequency = 1-2 days / week, corresponding to level 1, drinking frequency = 1-2 days / week, etc.; the multiple social activity levels are a set of pre-set classification information, such as social activity = never participate, corresponding to level 0, social activity = rarely participate, corresponding to level 1, etc.; the multiple eating habits types are a set of pre-set classification information, such as eating habits = eating once a day, corresponding to level 0, eating habits = eating 2-3 times a day, corresponding to level 1, eating habits = eating 3-4 times a day, corresponding to level 2, etc.;

[0127] g. Set the birth weight, environmental noise, and air pollution index feature coding rules to Min-Max standardization rules:

[0128]

[0129] Among them, max b-weight 、min b-weight The maximum and minimum birth weights are preset. noise 、min noise For the preset maximum and minimum ambient noise, max air 、min air The preset maximum and minimum air pollution index;

[0130] h. Set the feature coding rules for diastolic pressure, systolic pressure, pulse rate, peak expiratory flow, and forced expiratory volume in one second to Z-score standardization rules:

[0131]

[0132] Among them, μ dp , σ dp is the preset diastolic pressure mean and diastolic pressure variance, μ sp , σ sp is the preset systolic blood pressure mean and systolic blood pressure variance, μ pf , σ pf is the preset pulse rate mean and pulse rate variance, μ pef , σ pef is the preset mean and variance of peak expiratory flow, μ fev1 , σ fev1 The default values ​​are the mean and variance of the forced expiratory volume in 1 second.

[0133] 2) Typing feature coding details The feature coding rules configured for the biochemical test information group include:

[0134] The characteristic coding rules for albumin index, C-reactive protein index, white blood cell count index, red blood cell distribution width index, urine potassium index, urine sodium index, fasting blood glucose index, total cholesterol index, and creatinine index are all set to Z-score standardization rules:

[0135]

[0136]

[0137] Among them, μ bio1-9 is the mean of the preset albumin index, C-reactive protein index, white blood cell count index, red blood cell distribution width index, urine potassium index, urine sodium index, fasting blood glucose index, total cholesterol index, and creatinine index, σ bio1-9 The variances are for the preset albumin index, C-reactive protein index, white blood cell count index, red blood cell distribution width index, urine potassium index, urine sodium index, fasting blood glucose index, total cholesterol index, and creatinine index.

[0138] 3) Classification feature coding details The feature coding rules configured for the medical imaging information group include:

[0139] Set the feature encoding rules for amygdala signal intensity, white matter volume, left ventricular ejection fraction, left ventricular myocardial mass, liver fat content, spleen length, and carotid intima-media thickness to Min-Max standardization rules:

[0140]

[0141] Among them, max ass 、min ass is the preset maximum and minimum amygdala signal strength, max wvb 、min wvb is the preset maximum and minimum brain white matter volume, max lvef 、min lvef is the preset maximum and minimum left ventricular ejection fraction, max LVMI 、min LVMI The maximum and minimum left ventricular myocardial mass are preset, max LFC 、min LFC The maximum and minimum liver fat contents are preset, max sl 、min sl The maximum and minimum spleen lengths are preset. IMT 、min IMT The preset maximum and minimum carotid intima-media thickness.

[0142] 4) Typing feature coding details The feature coding rules configured for the proteomics information group include:

[0143] The characteristic coding rules for apolipoprotein index, glial fibrillary acidic protein index, interleukin-6 index, leptin protein index, α-tumor necrosis factor index, and adiponectin index are all set to Z-score standardization rules:

[0144]

[0145] Among them, μ APOE 、μ GFAP 、μ IL-6 、μ leptin 、μ TNF-α 、μ apn is the mean value of the preset apolipoprotein index, glial fibrillary acidic protein index, interleukin-6 index, leptin protein index, α tumor necrosis factor index, and adiponectin index, σ APOE , σ GFAP , σ IL-6 , σ leptin , σ TNF-α , σ apn The variances are for the preset apolipoprotein index, glial fibrillary acidic protein index, interleukin-6 index, leptin protein index, α-tumor necrosis factor index, and adiponectin index.

[0146] The classification feature coding rules of the embodiment of the present invention are as follows:

[0147] The first unimodal feature vector corresponding to the demographic information group is composed of the gender feature, age feature, weight feature, waist circumference feature, height feature, leg fat rate feature, arm fat rate feature, body mass index feature, and body fat rate feature;

[0148] The second unimodal feature vector is composed of the prenatal maternal smoking status characteristics, birth weight characteristics, infant breastfeeding characteristics, childhood economic status characteristics, drinking frequency characteristics, social activity characteristics, diastolic blood pressure characteristics, systolic blood pressure characteristics, pulse rate characteristics, peak expiratory flow characteristics, forced expiratory volume in 1 second characteristics, smoking status characteristics, dietary habits characteristics, environmental noise characteristics, and air pollution index characteristics corresponding to the lifestyle information group;

[0149] The corresponding third single-mode feature vector is formed by the albumin index feature, C-reactive protein index feature, white blood cell count index feature, red blood cell distribution width index feature, urine potassium index feature, urine sodium index feature, fasting blood glucose index feature, total cholesterol index feature, and creatinine index feature corresponding to the biochemical test information group;

[0150] The amygdala signal intensity feature, brain white matter volume feature, left ventricular ejection fraction feature, left ventricular myocardial mass feature, liver fat content feature, spleen length feature, and carotid intima-media thickness feature corresponding to the medical imaging information group form a corresponding fourth single modality feature vector;

[0151] The corresponding fifth single modal feature vector is composed of the apolipoprotein index feature, glial fibrillary acidic protein index feature, interleukin-6 index feature, leptin protein index feature, α-tumor necrosis factor index feature, and adiponectin index feature corresponding to the proteomics information group;

[0152] The first, second, third, fourth and fifth unimodal feature vectors form a corresponding multimodal feature vector.

[0153] Step 2: Set up a corresponding score-classification relationship table for the fluid intelligence scoring indicators of the two-level data classification rules.

[0154] Here, the score-classification relationship table in this embodiment of the present invention includes two relationship records. The relationship record includes a score field and a classification field. The score field is a range of fluid intelligence scores, and the classification field includes cognitive impairment type and normal cognitive ability type. For example, when the fluid intelligence score range is [0, 5], the corresponding classification field is cognitive impairment type, and when the fluid intelligence score range is [8, 13], the corresponding classification field is normal cognitive ability type.

[0155] Step 3: Collect raw data from an open biomedical database according to the two-level data classification rules to obtain an original data set; and organize training data according to the classification feature coding rules, the score-classification relationship table, and the original data set to obtain a first data set;

[0156] Specifically comprising: step 31, collecting raw data from an open biomedical database according to the two-level data typing rules to obtain a raw data set;

[0157] Here, the open biomedical database of the embodiment of the present invention includes at least the UK Biobank database;

[0158] The original data set includes multiple original data records; the original data records include a demographic information group, a lifestyle information group, a biochemical test information group, a medical imaging information group, a proteomics information group, and an evaluation information group;

[0159] Step 32: organizing the training data according to the classification feature coding rules, the score-classification relationship table, and the original data set to obtain a first data set;

[0160] The first data set includes a plurality of first data records; the first data record includes a first training vector and a first label probability; the first label probability is 0 or 1; a first label probability of 0 indicates that the corresponding cognitive ability level type is a cognitive impairment type, and a first label probability of 1 indicates that the corresponding cognitive ability level type is a cognitive normal type;

[0161] Specifically, the method includes: performing a round of traversal on all original data records of the original data set; and in this round of traversal, taking the currently traversed original data record as the corresponding current record; and performing multimodal feature encoding on the population information group, lifestyle information group, biochemical test information group, medical imaging information group, and proteomics information group of the current record according to the classification feature coding rules to obtain the corresponding multimodal feature vector; and taking the current multimodal feature vector as the corresponding first training vector; and taking the classification field of the relationship record in the score-classification relationship table, whose score field satisfies the fluid intelligence score index of the assessment information group of the current record as the corresponding first label type; and identifying the first label type, and setting the corresponding first label probability to 0 if the first label type is a cognitive impairment type, and setting the corresponding first label probability to 1 if the first label type is a cognitive ability normal type; and forming a corresponding first data record by the first training vector and the first label probability corresponding to the current record; and at the end of this round of traversal, forming a corresponding first data set by all the obtained first data records.

[0162] Step 4: Build a binary classification model based on the LightGBM model as the corresponding cognitive ability prediction model.

[0163] Here, the cognitive ability prediction model of an embodiment of the present invention is used to predict the cognitive ability level based on the multimodal feature vector input by the model and output the corresponding model prediction probability; wherein the value range of the model prediction probability is between 0 and 1.

[0164] It should be noted that the predicted probability output by the cognitive ability prediction model defaults to the positive sample probability, that is, the probability of confirming that the individual's cognitive ability level is normal; the higher the model-predicted probability, the greater the likelihood that the individual's cognitive ability level is normal, and vice versa. To this end, the embodiment of the present invention also pre-sets a probability threshold for the model-predicted probability, which is referred to as the normal level probability threshold below. If the model-predicted probability exceeds this probability threshold, the individual's cognitive ability level is considered normal; otherwise, the individual's cognitive ability level is considered low.

[0165] Step 5: firstly, perform a first-stage hyperparameter optimization and full-data training on the cognitive ability prediction model based on the first data set; then, perform a second-stage fault-tolerant training on the cognitive ability prediction model based on the first data set;

[0166] Here, because the cognitive ability prediction model is implemented based on the LightGBM model, it is necessary to optimize its hyperparameters through a specified hyperparameter optimization method, so the first stage of training must first perform hyperparameter tuning / finalization; In addition, all the data of the first data set are used for model training / evaluation in the first stage, because all the multimodal feature vectors of the first data set, that is, the first training vectors, have no missing items, so it is called full data training; However, in actual applications, not every individual data can have all the data options of the two-level data classification rules of the embodiment of the present invention. In order to improve the generalization and compatibility of the model, some records of the first data set are extracted to form the second data set, and all records of the second data set are set to missing items, and then the model is trained based on the second data set. This stage of training is called fault-tolerant training;

[0167] Specifically, the method includes: step 51, firstly performing a phase of hyperparameter optimization and full data training on the cognitive ability prediction model based on the first data set;

[0168] Specifically, it includes: step 511, randomly dividing the first data set into two sub-data sets based on a preset first segmentation ratio and recording them as a corresponding first training set and a first evaluation set;

[0169] Here, the first split ratio is a preset ratio parameter, such as 8:2; the first training set and the first evaluation set both consist of a plurality of first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first split ratio;

[0170] Step 512: Setting the first optimization objective function to a cross entropy loss function; and setting the parameter ranges of the nine types of hyperparameters of the cognitive ability prediction model;

[0171] Among them, the nine types of hyperparameters include the number of leaf nodes, the maximum depth of the decision tree, the total number of decision trees, the learning rate, the minimum number of samples of leaf nodes, the L1 regularization coefficient, the L2 regularization coefficient, the feature sampling ratio, and the sample sampling ratio;

[0172] Step 513: Based on the TPE algorithm, perform one round of iterative optimization on the nine types of hyperparameters according to a preset first iteration number threshold, the first training set, and the first optimization objective function to obtain the corresponding current optimal hyperparameter combination;

[0173] Here, the first iteration threshold is a preset threshold parameter, such as 100, 200, 300 times, etc.; the TPE (Tree-structured Parzen Estimator) algorithm is one of the computational optimization algorithms for hyperparameter tuning of the LightGBM model; and the iterative parameter adjustment process of the LightGBM model belongs to the public LightGBM model technology, which will not be further described again;

[0174] Step 514: Under the premise that the nine types of hyperparameters of the cognitive ability prediction model are set as the current optimal hyperparameter combination, the model parameters of the cognitive ability prediction model are continuously iteratively optimized for one round based on the preset second iteration number threshold, the first training set, and the first optimization objective function.

[0175] Here, the second iteration number threshold is a preset threshold parameter, such as 100, 200, 300 times, etc.;

[0176] At step 515, after the current round of iterative optimization is completed, the AUC index and the logarithmic loss index are calculated based on the first evaluation set and the cognitive ability prediction model to obtain corresponding first AUC index and first logarithmic loss index; and it is determined whether the first AUC index and the first logarithmic loss index each meet the corresponding first AUC index range and first logarithmic loss index range; if not, the training is continued back to step 513; if so, it is confirmed that the full data training of the cognitive ability prediction model is completed;

[0177] Here, the AUC (Area Under the Curve) indicator is the area under the ROC curve and the coordinate axis, which is used to measure the performance of the binary classification model. The ROC curve represents the model performance by plotting the true positive rate (TPR) and the false positive rate (FPR), which can be obtained by calculating the true positive rate and the false positive rate. The logarithmic loss indicator is the loss value of the logarithmic loss function, which can be calculated by substituting the predicted probability and the label probability into the logarithmic loss function. The first AUC indicator range and the first logarithmic loss indicator range are two pre-set numerical ranges for model convergence.

[0178] Step 52: performing a second-stage fault-tolerant training on the cognitive ability prediction model based on the first data set;

[0179] Specifically, the method includes: step 521, randomly selecting a plurality of first data records from the first data set as second data records to form a corresponding second data set according to a preset first extraction ratio; and clearing a plurality of features in the second training vector of each second data record according to a preset first reset ratio;

[0180] Here, the first extraction ratio is a preset ratio parameter, and the first extraction ratio is ≥80%. For example, when the first extraction ratio is 80%, it means that 80% of the first data records are randomly extracted from the first data set as the second data records to form the corresponding second data set.

[0181] The first reset ratio is a preset ratio parameter, and the first reset ratio is ≤40%. For example, when the first reset ratio is 40%, it means that 40% of the features in the second data record are randomly cleared as missing items.

[0182] The second data set includes a plurality of second data records; the ratio of the total number of records in the second data set to the total number of records in the first data set satisfies the first extraction ratio; the second data record includes a second training vector and a second label probability; the ratio of the total number of features reset to zero in the second training vector to the total number of vector features satisfies the first reset ratio; the second label probability is 0 or 1; a second label probability of 0 indicates that the corresponding cognitive ability level type is a cognitive impairment type, and a second label probability of 1 indicates that the corresponding cognitive ability level type is a cognitive normal type;

[0183] Step 522: randomly divide the second data set into two sub-data sets based on a preset second segmentation ratio and record them as a corresponding second training set and a second evaluation set;

[0184] The second split ratio is a preset ratio parameter, such as 8:2; the second training set and the second evaluation set are both composed of a plurality of second data records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second split ratio;

[0185] Step 523, setting the second optimization objective function to a cross entropy loss function;

[0186] Step 524 , performing a round of iterative optimization on the model parameters of the cognitive ability prediction model based on a preset third iteration number threshold, the second training set, and the second optimization objective function;

[0187] Here, the third iteration number threshold is a preset threshold parameter;

[0188] Step 525: After this round of iterative optimization is completed, the AUC indicator and the logarithmic loss indicator are calculated based on the second evaluation set and the cognitive ability prediction model to obtain the corresponding second AUC indicator and the second logarithmic loss indicator; and it is identified whether the second AUC indicator and the second logarithmic loss indicator each meet the corresponding second AUC indicator range and the second logarithmic loss indicator range; if not, return to step 524 to continue training; if so, it is confirmed that the fault-tolerant training of the cognitive ability prediction model is completed.

[0189] Here, the second AUC indicator range and the second logarithmic loss indicator range are two pre-set numerical ranges for model convergence. The second AUC indicator range and the second logarithmic loss indicator range can be consistent with the corresponding first AUC indicator range and the first logarithmic loss indicator range, or they can be larger than the corresponding first AUC indicator range and the first logarithmic loss indicator range.

[0190] Step 6: After the two-stage training is completed, the multimodal data input by the user that meets the requirements of the two-level data classification rules is received as the current individual data; based on the current individual data and the cognitive ability prediction model, cognitive ability assessment and feature importance analysis are performed to obtain the corresponding individual assessment results and individual analysis results, which are fed back to the current user;

[0191] Specifically, step 61 includes: after the two-stage training is completed, receiving multimodal data input by the user that meets the requirements of the two-level data classification rules as current individual data;

[0192] Here, the multimodal data input by the user defaults to the multimodal data of a subject, and the data type of the multimodal data should meet the data type requirements of the two-level data classification rules of the embodiment of the present invention. It should be noted that the multimodal data may contain some missing data, but the missing ratio cannot exceed 40%. The embodiment of the present invention will perform a missing item check on the multimodal data input by the user according to the data type requirements of the two-level data classification rules. If it is found that the missing item ratio exceeds 40%, the subsequent step 62 will not be continued, and a data item missing error prompt will be directly fed back to the current user.

[0193] Step 62: Based on the current individual data and the cognitive ability prediction model, cognitive ability assessment and feature importance analysis are performed to obtain corresponding individual assessment results and individual analysis results, which are fed back to the current user;

[0194] Specifically, the method includes: step 621, performing multimodal feature encoding on the current individual data based on the classification feature encoding rules to obtain a corresponding multimodal feature vector; and inputting the current multimodal feature vector into the cognitive ability prediction model to predict the cognitive ability level to obtain a corresponding model prediction probability as the current prediction probability;

[0195] Step 622: Check whether the current predicted probability exceeds a preset normal level probability threshold; if so, set the corresponding individual assessment result as normal cognitive ability; if not, set the corresponding individual assessment result as low cognitive ability;

[0196] Here, the individual assessment results of the present invention include low cognitive ability and normal cognitive ability;

[0197] In step 623, the SHAP values ​​of various secondary data features of the current multimodal feature vector are calculated based on the SHAP algorithm to obtain corresponding feature SHAP values; and all secondary data types are sorted in descending order of feature SHAP values ​​to obtain corresponding feature importance sequences; and the feature importance sequences are used as the corresponding individual analysis results.

[0198] Here, the embodiment of the present invention will cache all the inference process data of the cognitive ability prediction model. When calculating the SHAP value of each type of secondary data feature of the current multimodal feature vector based on the SHAP algorithm: all decision tree nodes in the model are traversed by caching data, and the single-tree contribution of each type of secondary data feature to the final prediction result, that is, the current prediction probability, is calculated at each tree node according to the value of each type of secondary data feature and the current node splitting condition. Then, the single-tree contribution of the same type of feature on all trees is weighted averaged to obtain the SHAP value of each type of feature. The higher the SHAP value of each type of secondary data feature, the greater the correlation between the dimensional feature and the current prediction probability, the greater the corresponding cognitive ability correlation strength, and the greater the specific weight of the impact on cognitive ability.

[0199] It should be noted that the secondary data features of the embodiment of the present invention include at least: gender characteristics, age characteristics, weight characteristics, waist circumference characteristics, height characteristics, leg fat rate characteristics, arm fat rate characteristics, body mass index characteristics, body fat rate characteristics, prenatal maternal smoking status characteristics, birth weight characteristics, infant breastfeeding characteristics, childhood economic status characteristics, drinking frequency characteristics, social activity characteristics, diastolic blood pressure characteristics, systolic blood pressure characteristics, pulse rate characteristics, peak expiratory flow characteristics, forced expiratory volume in the first second characteristics, smoking status characteristics, eating habits characteristics, environmental noise characteristics, air pollution index characteristics, albumin index The characteristics of the blood vessels, tissues and organs are as follows: blood vessel marker characteristics, C-reactive protein index characteristics, white blood cell count index characteristics, red blood cell distribution width index characteristics, urine potassium index characteristics, urine sodium index characteristics, fasting blood glucose index characteristics, total cholesterol index characteristics, creatinine index characteristics, amygdala signal intensity characteristics, brain white matter volume characteristics, left ventricular ejection fraction characteristics, left ventricular myocardial mass characteristics, liver fat content characteristics, spleen length characteristics, carotid intima-media thickness characteristics, apolipoprotein index characteristics, glial fibrillary acidic protein index characteristics, interleukin-6 index characteristics, leptin protein index characteristics, α-tumor necrosis factor index characteristics, adiponectin index characteristics.

[0200] Figure 2 This is a module structure diagram of a cognitive ability assessment and processing device based on multimodal data provided in the second embodiment of the present invention. The device is a terminal device or server that implements the aforementioned method embodiment, and can also be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the device includes: a data rule customization module 201, a first data preparation module 202, a second data preparation module 203, a model construction module 204, a model training module 205, and a model application module 206.

[0201] The data rule customization module 201 is used to set the two-level data classification rules and classification feature coding rules for multimodal data; the first-level data types of the two-level data classification rules include five types of single-modal information groups and one type of evaluation information group, and each type of single-modal information group includes one or more corresponding second-level data types; the five types of single-modal information groups include: demographic information group, lifestyle information group, biochemical test information group, medical imaging information group, and proteomics information group; the second-level data type of the evaluation information group includes at least fluid intelligence scoring indicators; the classification feature coding rules configure corresponding feature coding rules for each type of second-level data type of each type of single-modal information group, and configure corresponding multimodal feature vector coding rules for multimodal data.

[0202] The first data preparation module 202 is used to set a corresponding score-classification relationship table for the fluid intelligence scoring indicators of the two-level data classification rules; the score-classification relationship table includes two relationship records; the relationship record includes a score field and a classification field; the score field is a score range for a fluid intelligence score; the classification field includes a cognitive impairment type and a cognitive normal type.

[0203] The second data preparation module 203 is used to collect raw data from an open biomedical database according to the two-level data classification rules to obtain a raw data set; and to organize training data according to the classification feature coding rules, the score-classification relationship table and the raw data set to obtain a first data set.

[0204] The model construction module 204 is used to construct a binary classification model based on the LightGBM model as the corresponding cognitive ability prediction model; the cognitive ability prediction model is used to predict the cognitive ability level according to the multimodal feature vector input by the model and output the corresponding model prediction probability; the value range of the model prediction probability is between 0 and 1.

[0205] The model training module 205 is used to first perform a first-stage hyperparameter optimization and full-data training on the cognitive ability prediction model based on the first data set; and then perform a second-stage fault-tolerant training on the cognitive ability prediction model based on the first data set.

[0206] The model application module 206 is used to receive the multimodal data that meets the requirements of the two-level data classification rules input by the user as the current individual data after the two-stage training is completed; and perform cognitive ability assessment and feature importance analysis based on the current individual data and the cognitive ability prediction model to obtain the corresponding individual assessment results and individual analysis results and feedback them to the current user.

[0207] An embodiment of the present invention provides a cognitive ability assessment and processing device based on multimodal data, which can execute the method steps in the above method embodiment. Its implementation principles and technical effects are similar and will not be repeated here.

[0208] It should be noted that the division of the various modules of the above devices is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into a single physical entity or physically separated. Furthermore, these modules can be implemented entirely in the form of software called by a processing element; or entirely in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the data rule customization module can be a separate processing element, or it can be integrated into a chip of the above device. In addition, it can be stored in the form of program code in the memory of the above device, and called by a processing element of the above device to perform the functions of the above-mentioned determination module. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed by hardware integrated logic circuits in the processor element or by software instructions.

[0209] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code through a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0210] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the above method embodiments are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) means. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, hard disk, tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0211] Figure 3 This is a schematic diagram of the structure of an electronic device provided in the third embodiment of the present invention. The electronic device can be a terminal device or server that implements the method of the aforementioned embodiment, or it can be a terminal device or server that implements the method of the aforementioned embodiment connected to the aforementioned terminal device or server. Figure 3 As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver 303's transceiver actions. Various instructions may be stored in the memory 302 for completing various processing functions and implementing the processing steps described in the aforementioned embodiment method. Preferably, the electronic device involved in the embodiment of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The above-mentioned communication port 306 is used for connecting and communicating between the electronic device and other peripherals.

[0212] exist Figure 3The system bus 305 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory (Non-Volatile Memory), such as at least one disk storage.

[0213] The above-mentioned processors can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0214] It should be noted that an embodiment of the present invention further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the methods and processing procedures provided in the above embodiments.

[0215] The embodiment of the present invention provides a method, device, electronic device and computer-readable storage medium for cognitive ability assessment based on multimodal data. As can be seen from the above content, the embodiment of the present invention first sets two-level data classification rules and classification feature coding rules for multimodal data, and sets a corresponding score-classification relationship table for the fluid intelligence scoring index, and collects original data from the biomedical database according to the two-level data classification rules, and constructs a training data set (i.e., a first data set) based on the classification feature coding rules, the score-classification relationship table and the original data set, and constructs a cognitive ability prediction model based on the LightGBM model; then, the cognitive ability prediction model is first subjected to a first-stage hyperparameter optimization and full-data training based on the first data set, and then the cognitive ability prediction model is subjected to a second-stage fault-tolerant training based on the first data set; finally, the cognitive ability prediction model and the SHAP algorithm are used to perform cognitive ability assessment and feature importance analysis based on the individual's multimodal data. Through the embodiment of the present invention, not only the assessment accuracy of the individual's cognitive ability is improved, but also the ability to parse the correlation strength between the individual's multi-dimensional biological characteristics and cognitive ability is enhanced, and the specific weights of the impact of different characteristics on cognitive ability can be quantified and identified.

[0216] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0217] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cognitive ability assessment and processing method based on multimodal data, characterized in that: The method comprises: Two-level data classification rules and classification feature coding rules for multimodal data are set; the first-level data types of the two-level data classification rules include five types of unimodal information groups and one type of evaluation information group, and each type of the unimodal information group includes one or more corresponding second-level data types; the five types of unimodal information groups include: a demographic information group, a lifestyle information group, a biochemical test information group, a medical imaging information group, and a proteomics information group; the second-level data types of the evaluation information group include at least a fluid intelligence scoring index; the classification feature coding rules configure corresponding feature coding rules for each type of the second-level data type of each type of the unimodal information group, and configure corresponding multimodal feature vector coding rules for multimodal data; A corresponding score-classification relationship table is set for the fluid intelligence score indicator of the two-level data classification rules; the score-classification relationship table includes two relationship records; the relationship record includes a score field and a classification field; the score field is a score range for a fluid intelligence score; the classification field includes a cognitive impairment type and a cognitive normal type; Collecting raw data from an open biomedical database according to the two-level data classification rules to obtain an original data set; and organizing training data according to the classification feature coding rules, the score-classification relationship table, and the original data set to obtain a first data set; A binary classification model is constructed based on the LightGBM model as the corresponding cognitive ability prediction model; the cognitive ability prediction model is used to predict the cognitive ability level based on the multimodal feature vector input by the model and output the corresponding model prediction probability; the value range of the model prediction probability is between 0 and 1; First, performing a first-stage hyperparameter optimization and full-data training on the cognitive ability prediction model based on the first data set; then performing a second-stage fault-tolerant training on the cognitive ability prediction model based on the first data set; After the two-stage training is completed, the multimodal data input by the user that meets the requirements of the two-level data classification rules is received as the current individual data; and based on the current individual data and the cognitive ability prediction model, cognitive ability assessment and feature importance analysis are performed to obtain corresponding individual assessment results and individual analysis results and feedback to the current user.

2. The cognitive ability assessment and processing method based on multimodal data according to claim 1, characterized in that: The secondary data types of the demographic information group include at least: gender, age, weight, waist circumference, height, leg fat percentage, arm fat percentage, body mass index, and body fat percentage; The secondary data types of the lifestyle information group include at least: prenatal maternal smoking status, birth weight, breastfeeding during infancy, childhood economic status, drinking frequency, social activities, diastolic blood pressure, systolic blood pressure, pulse rate, peak expiratory flow, forced expiratory volume in 1 second, own smoking status, eating habits, environmental noise, and air pollution index; The secondary data types of the biochemical test information group include at least: albumin index, C-reactive protein index, white blood cell count index, red blood cell distribution width index, urine potassium index, urine sodium index, fasting blood glucose index, total cholesterol index, and creatinine index; The secondary data types of the medical imaging information group include at least: amygdala signal intensity, brain white matter volume, left ventricular ejection fraction, left ventricular myocardial mass, liver fat content, spleen length, and carotid intima-media thickness; The secondary data types of the proteomics information group include at least: an apolipoprotein index, a glial fibrillary acidic protein index, an interleukin-6 index, a leptin protein index, an α-tumor necrosis factor index, and an adiponectin index; The open biomedical database at least includes the UK Biobank database; The original data set includes a plurality of original data records; the original data records include the demographic information group, the lifestyle information group, the biochemical test information group, the medical imaging information group, the proteomics information group and the assessment information group; The first data set includes multiple first data records; the first data record includes a first training vector and a first label probability; the first label probability is 0 or 1; the first label probability is 0, indicating that the corresponding cognitive ability level type is a cognitive ability impairment type, and is 1, indicating that the corresponding cognitive ability level type is a cognitive ability normal type.

3. The cognitive ability assessment and processing method based on multimodal data according to claim 2, characterized in that: The typing feature coding details are feature coding rules configured for the population information group, including: Set the gender feature coding rule to binary coding rule: Set the age feature encoding rule to the Z-score standardization rule: Among them, μ age , σ age is the preset age mean and age variance; Set the waist circumference, height, weight, leg fat percentage, arm fat percentage, body mass index, and body fat percentage feature coding rules to Min-Max standardization rules: Among them, max waist 、min waist For the preset maximum and minimum waist circumference, max height 、min height For the preset maximum and minimum height, max weight 、min weight For the preset maximum and minimum weight, max lfp 、min lfp For the preset maximum and minimum leg fat rate, max afp 、min afp For the preset maximum and minimum arm fat rate, max BMI 、min BMI The maximum and minimum body mass index are preset, max bfp 、min bfp The preset maximum and minimum body fat percentages; Setting the classification feature coding details as feature coding rules configured for the lifestyle information group includes: The coding rules for prenatal maternal smoking status and infant breastfeeding characteristics are set to binary coding rules: The coding rules for childhood economic status, drinking frequency, social activities, smoking status, and dietary habits are set as multi-category coding rules. Specifically, the childhood economic status characteristics are coded according to multiple preset economic status levels; the drinking frequency characteristics are coded according to multiple preset drinking frequency levels; the social activity characteristics are coded according to multiple preset social activity levels; the smoking status characteristics are coded according to the preset three-level coding of never smoking, having a smoking history but also quitting smoking, and having a smoking history but not quitting smoking; and the dietary habits characteristics are coded according to multiple preset dietary habit types. Set the birth weight, environmental noise, and air pollution index feature coding rules to Min-Max standardization rules: Among them, max b-weight 、min b-weight The maximum and minimum birth weights are preset. noise 、min noise For the preset maximum and minimum ambient noise, max air 、min air The preset maximum and minimum air pollution index; Set the feature coding rules for diastolic blood pressure, systolic blood pressure, pulse rate, peak expiratory flow, and forced expiratory volume in one second to the Z-score standardization rule: Among them, μ dp , σ dp is the preset diastolic pressure mean and diastolic pressure variance, μ sp , σ sp is the preset systolic blood pressure mean and systolic blood pressure variance, μ pf , σ pf is the preset pulse rate mean and pulse rate variance, μ pef , σ pef is the preset mean and variance of peak expiratory flow, μ fev1 , σ fev1 The default forced expiratory volume in 1 second mean and forced expiratory volume in 1 second variance; The typing feature coding details are feature coding rules configured for the biochemical test information group, including: The characteristic coding rules for albumin index, C-reactive protein index, white blood cell count index, red blood cell distribution width index, urine potassium index, urine sodium index, fasting blood glucose index, total cholesterol index, and creatinine index are all set to Z-score standardization rules: Among them, μ bio1-9 is the mean of the preset albumin index, C-reactive protein index, white blood cell count index, red blood cell distribution width index, urine potassium index, urine sodium index, fasting blood glucose index, total cholesterol index, and creatinine index, σ bio1-9 The variance of the preset albumin index, C-reactive protein index, white blood cell count index, red blood cell distribution width index, urine potassium index, urine sodium index, fasting blood glucose index, total cholesterol index, and creatinine index; The classification feature coding rules are feature coding rules configured for the medical image information group, including: Set the feature encoding rules for amygdala signal intensity, white matter volume, left ventricular ejection fraction, left ventricular myocardial mass, liver fat content, spleen length, and carotid intima-media thickness to Min-Max standardization rules: Among them, max ass 、min ass is the preset maximum and minimum amygdala signal strength, max wvb 、min wvb is the preset maximum and minimum brain white matter volume, max lvef 、min lvef is the preset maximum and minimum left ventricular ejection fraction, max LVMI 、min LVMI The maximum and minimum left ventricular myocardial mass are preset, max LFC 、min LFC The maximum and minimum liver fat contents are preset, max sl 、min sl The maximum and minimum spleen lengths are preset. IMT 、min IMT The preset maximum and minimum carotid intima-media thickness; The typing feature coding rules are feature coding rules configured for the proteomics information group, including: The characteristic coding rules for apolipoprotein index, glial fibrillary acidic protein index, interleukin-6 index, leptin protein index, α-tumor necrosis factor index, and adiponectin index are all set to Z-score standardization rules: Among them, μ APOE 、μ GFAP 、μ IL-6 、μ leptin 、μ TNF-α 、μ apn is the mean value of the preset apolipoprotein index, glial fibrillary acidic protein index, interleukin-6 index, leptin protein index, α tumor necrosis factor index, and adiponectin index, σ APOE , σ GFAP , σ IL-6 , σ leptin , σ TNF-α , σ apn The variances of the preset apolipoprotein index, glial fibrillary acidic protein index, interleukin-6 index, leptin protein index, α-tumor necrosis factor index, and adiponectin index; The typing feature coding details are as follows: the multimodal characteristic vector coding rules corresponding to the multimodal data configuration are: the gender feature, the age feature, the weight feature, the waist circumference feature, the height feature, the leg fat rate feature, the arm fat rate feature, the body mass index feature, and the body fat rate feature corresponding to the demographic information group constitute a corresponding first single-modal feature vector; and the prenatal maternal smoking status feature, the birth weight feature, the infant breastfeeding feature, the childhood economic status feature, the drinking frequency feature, the social activity feature, the diastolic blood pressure feature, the systolic blood pressure feature, the pulse rate feature, the peak expiratory flow feature, the forced expiratory volume in 1 second feature, the smoking status feature, the eating habit feature, the environmental noise feature, and the air pollution index feature corresponding to the lifestyle information group constitute a corresponding second single-modal feature vector; and the albumin index feature, the C-reactive protein index feature, and the biochemical test information group correspond to the The characteristics, the white blood cell number index characteristics, the red blood cell distribution width index characteristics, the urine potassium index characteristics, the urine sodium index characteristics, the fasting blood glucose index characteristics, the total cholesterol index characteristics, and the creatinine index characteristics constitute a corresponding third unimodal feature vector; and the amygdala signal intensity characteristics, the brain white matter volume characteristics, the left ventricular ejection fraction characteristics, the left ventricular myocardial mass characteristics, the liver fat content characteristics, the spleen length characteristics, and the carotid intima-media thickness characteristics corresponding to the medical imaging information group constitute a corresponding fourth unimodal feature vector; and the apolipoprotein index characteristics, the glial fibrillary acidic protein index characteristics, the interleukin-6 index characteristics, the leptin protein index characteristics, the α tumor necrosis factor index characteristics, and the adiponectin index characteristics corresponding to the proteomics information group constitute a corresponding fifth unimodal feature vector; and the first, second, third, fourth and fifth unimodal feature vectors constitute the corresponding multimodal feature vector.

4. The cognitive ability assessment and processing method based on multimodal data according to claim 2, characterized in that: The first data set is obtained by arranging training data according to the typing feature coding rules, the score-classification relationship table, and the original data set, specifically comprising: A round of traversal is performed on all the original data records of the original data set; and during this round of traversal, the original data record currently traversed is used as the corresponding current record; and according to the classification feature coding rules, the demographic information group, the lifestyle information group, the biochemical test information group, the medical imaging information group, and the proteomics information group of the current record are multimodally encoded to obtain the corresponding multimodal feature vector; and the current multimodal feature vector is used as the corresponding first training vector; and in the score-classification relationship table, the classification field of the relationship record whose score field satisfies the fluid intelligence scoring index of the assessment information group of the current record is used as the corresponding first label type; and the first label type is identified, and if the first label type is a cognitive impairment type, the corresponding first label probability is set to 0; if the first label type is a cognitive ability normal type, the corresponding first label probability is set to 1; and the first training vector corresponding to the current record and the first label probability are used to form a corresponding first data record; and at the end of this round of traversal, all the obtained first data records are used to form the corresponding first data set.

5. The cognitive ability assessment and processing method based on multimodal data according to claim 2, characterized in that: The step of performing one-stage hyperparameter optimization and full-data training on the cognitive ability prediction model based on the first data set specifically includes: Step 51: randomly dividing the first data set into two sub-data sets based on a preset first segmentation ratio and recording them as a corresponding first training set and a first evaluation set; Wherein, both the first training set and the first evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio; Step 52: setting the first optimization objective function to a cross entropy loss function; and setting the parameter ranges of the nine types of hyperparameters of the cognitive ability prediction model; The nine types of hyperparameters include the number of leaf nodes, the maximum depth of the decision tree, the total number of decision trees, the learning rate, the minimum number of samples of leaf nodes, the L1 regularization coefficient, the L2 regularization coefficient, the feature sampling ratio, and the sample sampling ratio; Step 53: performing a round of iterative optimization on the nine types of hyperparameters based on the TPE algorithm according to a preset first iteration number threshold, the first training set, and the first optimization objective function to obtain the corresponding current optimal hyperparameter combination; Step 54: Under the premise that the nine types of hyperparameters of the cognitive ability prediction model are set as the current optimal hyperparameter combination, continue to perform a round of iterative optimization on the model parameters of the cognitive ability prediction model based on a preset second iteration number threshold, the first training set, and the first optimization objective function; Step 55. After this round of iterative optimization is completed, the AUC indicator and the logarithmic loss indicator are calculated based on the first evaluation set and the cognitive ability prediction model to obtain the corresponding first AUC indicator and first logarithmic loss indicator; and it is identified whether the first AUC indicator and the first logarithmic loss indicator each meet the corresponding first AUC indicator range and first logarithmic loss indicator range; if not, return to step 53 to continue training; if so, it is confirmed that the full data training of the cognitive ability prediction model is completed.

6. The cognitive ability assessment and processing method based on multimodal data according to claim 2, characterized in that: The performing two-stage fault-tolerant training on the cognitive ability prediction model based on the first data set specifically includes: Step 61: randomly selecting a plurality of first data records from the first data set as second data records to form a corresponding second data set according to a preset first extraction ratio; and clearing a plurality of features in a second training vector of each second data record according to a preset first reset ratio; The second data set includes a plurality of second data records; the ratio of the total number of records in the second data set to the total number of records in the first data set satisfies the first extraction ratio; the second data record includes the second training vector and the second label probability; the ratio of the total number of features reset to zero in the second training vector to the total number of vector features satisfies the first reset ratio; the second label probability is 0 or 1; a second label probability of 0 indicates that the corresponding cognitive ability level type is a cognitive impairment type, and a second label probability of 1 indicates that the corresponding cognitive ability level type is a normal cognitive ability type; the first extraction ratio is ≥80%, and the first reset ratio is ≤40%; Step 62: randomly divide the second data set into two sub-data sets based on a preset second segmentation ratio, and record them as a corresponding second training set and a second evaluation set; Wherein, both the second training set and the second evaluation set are composed of a plurality of second data records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second segmentation ratio; Step 63, setting the second optimization objective function to a cross entropy loss function; Step 64: performing a round of iterative optimization on the model parameters of the cognitive ability prediction model based on a preset third iteration number threshold, the second training set, and the second optimization objective function; Step 65: After this round of iterative optimization is completed, the AUC indicator and the logarithmic loss indicator are calculated based on the second evaluation set and the cognitive ability prediction model to obtain the corresponding second AUC indicator and the second logarithmic loss indicator; and whether the second AUC indicator and the second logarithmic loss indicator each meet the corresponding second AUC indicator range and the second logarithmic loss indicator range is identified; if not, return to step 64 to continue training; if so, it is confirmed that the fault-tolerant training of the cognitive ability prediction model is completed.

7. The cognitive ability assessment and processing method based on multimodal data according to claim 2, characterized in that: The cognitive ability assessment and feature importance analysis based on the current individual data and the cognitive ability prediction model are performed to obtain corresponding individual assessment results and individual analysis results and to feed back to the current user, specifically including: Based on the classification feature coding rules, the current individual data is multimodal feature coded to obtain the corresponding multimodal feature vector; and the current multimodal feature vector is input into the cognitive ability prediction model to predict the cognitive ability level to obtain the corresponding model prediction probability as the current prediction probability; Whether the current predicted probability exceeds a preset normal level probability threshold is determined; if so, the corresponding individual assessment result is set as normal cognitive ability; if not, the corresponding individual assessment result is set as low cognitive ability; Calculating the SHAP values ​​of various secondary data features of the current multimodal feature vector based on the SHAP algorithm to obtain corresponding feature SHAP values; sorting all the secondary data types in descending order of the feature SHAP values ​​to obtain corresponding feature importance sequences; and using the feature importance sequences as the corresponding individual analysis results; Among them, the secondary data features include the gender feature, the age feature, the weight feature, the waist circumference feature, the height feature, the leg fat rate feature, the arm fat rate feature, the body mass index feature, the body fat rate feature, the prenatal maternal smoking status feature, the birth weight feature, the infant breastfeeding feature, the childhood economic status feature, the drinking frequency feature, the social activity feature, the diastolic blood pressure feature, the systolic blood pressure feature, the pulse rate feature, the peak expiratory flow feature, the forced expiratory volume in 1 second feature, the smoking status feature, the eating habit feature, the environmental noise feature, the air pollution index feature, the albumin index feature The characteristics of the human body are as follows: the C-reactive protein index characteristic, the white blood cell count index characteristic, the red blood cell distribution width index characteristic, the urine potassium index characteristic, the urine sodium index characteristic, the fasting blood glucose index characteristic, the total cholesterol index characteristic, the creatinine index characteristic, the amygdala signal intensity characteristic, the brain white matter volume characteristic, the left ventricular ejection fraction characteristic, the left ventricular myocardial mass characteristic, the liver fat content characteristic, the spleen length characteristic, the carotid intima-media thickness characteristic, the apolipoprotein index characteristic, the glial fibrillary acidic protein index characteristic, the interleukin-6 index characteristic, the leptin protein index characteristic, the α-tumor necrosis factor index characteristic, and the adiponectin index characteristic.

8. A device for executing the cognitive ability assessment and processing method based on multimodal data according to any one of claims 1 to 7, characterized in that: The device includes: a data rule customization module, a first data preparation module, a second data preparation module, a model construction module, a model training module, and a model application module; The data rule customization module is used to set two-level data typing rules and typing feature coding rules for multimodal data; the first-level data types of the two-level data typing rules include five types of unimodal information groups and one type of evaluation information group, and each type of unimodal information group includes one or more corresponding second-level data types; the five types of unimodal information groups include: demographic information group, lifestyle information group, biochemical test information group, medical imaging information group, and proteomics information group; the second-level data type of the evaluation information group includes at least a fluid intelligence scoring index; the typing feature coding rules configure corresponding feature coding rules for each type of second-level data type in each type of unimodal information group, and configure corresponding multimodal feature vector coding rules for multimodal data; The first data preparation module is used to set a corresponding score-classification relationship table for the fluid intelligence score indicator of the two-level data classification rules; the score-classification relationship table includes two relationship records; the relationship record includes a score field and a classification field; the score field is a score range for a fluid intelligence score; the classification field includes a cognitive impairment type and a cognitive normal type; The second data preparation module is used to collect raw data from an open biomedical database according to the two-level data typing rules to obtain a raw data set; and to organize training data according to the typing feature coding rules, the score-classification relationship table and the raw data set to obtain a first data set; The model building module is used to build a binary classification model based on the LightGBM model as the corresponding cognitive ability prediction model; the cognitive ability prediction model is used to predict the cognitive ability level according to the multimodal feature vector input by the model and output the corresponding model prediction probability; the value range of the model prediction probability is between 0 and 1; The model training module is used to first perform a first-stage hyperparameter optimization and full-data training on the cognitive ability prediction model based on the first data set; and then perform a second-stage fault-tolerant training on the cognitive ability prediction model based on the first data set; The model application module is used to receive multimodal data input by the user that meets the requirements of the two-level data classification rules as current individual data after the two-stage training is completed; and perform cognitive ability assessment and feature importance analysis based on the current individual data and the cognitive ability prediction model to obtain corresponding individual assessment results and individual analysis results and feedback them to the current user.

9. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is configured to be coupled to the memory, read and execute instructions in the memory, so as to implement the method according to any one of claims 1 to 7; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images