A cognitive impairment auxiliary screening method based on a frozen large language model

Abstract

The application discloses a cognitive impairment auxiliary screening method based on a frozen large language model, and belongs to the field of medical information technology and artificial intelligence auxiliary screening. The method obtains writing trajectory and facial behavior data of a target object performing a preset cognitive task, extracts corresponding features and constructs a multi-modal supervised learning sample, and obtains writing and facial numerical features through statistical aggregation; the numerical features are generated into a template text, input into a parameter frozen large language model encoder to complete semantic coding, and a semantic enhanced representation is obtained; the semantic enhanced representation and the numerical representation are fused, and a cognitive state auxiliary screening result is output through a classification module. The application retains the semantic ability of the pre-trained language model, avoids the high cost of large model fine-tuning and the risk of small sample overfitting, mines complementary information through multi-modal feature fusion, improves the accuracy and generalization of cognitive state recognition, and can realize high-precision cognitive impairment auxiliary screening.

Description

Technical Field

[0001] This invention relates to the fields of medical information technology, computer-aided screening, multimodal behavior recognition, and deep learning technology, specifically to a cognitive impairment-assisted screening method based on a frozen large language model. Background Technology

[0002] Because cognitive impairment is characterized by its insidious onset and slow progression, accurate screening of a subject's cognitive state in the early stages is crucial for early intervention, treatment planning, and slowing disease progression. In recent years, artificial intelligence-based methods for assisting in the screening of cognitive impairment have become a research hotspot. Among these methods, intelligent assessment of cognitive status by analyzing behavioral data generated during cognitive tasks such as drawing and writing can effectively improve screening efficiency and reduce the cost of manual assessment.

[0003] In existing technologies, cognitive impairment screening methods typically rely on single-modal data such as writing trajectories, speech, images, or facial behaviors, or employ simple feature concatenation to achieve multimodal fusion, followed by deep learning models to classify cognitive states. However, due to significant differences in expression, feature space, and semantic level among different modalities such as writing and facial behaviors, traditional multimodal fusion methods struggle to fully uncover the semantic relationships between different modalities. Furthermore, most existing methods directly utilize numerical features for classification, failing to fully leverage the rich semantic knowledge inherent in pre-trained large language models, resulting in limited semantic understanding of behavioral data. Moreover, directly fine-tuning large language models usually requires a large number of labeled samples; in medical scenarios such as cognitive impairment screening, small-sample training can easily lead to model overfitting and increase training costs.

[0004] Therefore, how to fully utilize the semantic representation capabilities of pre-trained large language models under limited sample conditions, achieve deep semantic fusion of multimodal information such as writing behavior and facial behavior, and improve the ability of cognitive impairment auxiliary screening models to express multimodal behavioral information and classify accuracy has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] To address the above technical problems, this invention provides a cognitive impairment-assisted screening method based on a frozen large language model, the method comprising the following steps: S100: Obtain writing trajectory data and facial behavior data generated by the target object during the execution of a preset cognitive task; extract features from the writing trajectory data and facial behavior data to obtain corresponding writing features and facial features. S200. Construct multimodal supervised learning samples based on the writing features and facial features, and statistically aggregate the writing features and facial features to obtain writing numerical features and facial numerical features. S300. Generate templated text based on the written numerical features and facial numerical features, and perform semantic encoding on the large language model encoder with the input parameters of the templated text frozen to obtain a semantically enhanced representation; S400. Input the semantically enhanced representation and the numerical representations corresponding to the written numerical features and facial numerical features into the multimodal fusion module for fusion processing to obtain the multimodal fusion representation; S500: Input the multimodal fusion representation into the classification module, and determine the cognitive state auxiliary screening result corresponding to the target object based on the classification output result.

[0006] As a further improvement of the present invention, step S100 specifically includes the following steps: S110. Collect writing trajectory data generated by the target object during the execution of a preset cognitive task, wherein the writing trajectory data includes at least one of writing trajectory coordinates, time information, pressure information, and air writing state information and paper writing state information; S120. Collect facial behavior data generated by the target object during the execution of the preset cognitive task, wherein the facial behavior data includes at least one of facial action unit activation information, head posture information, and facial key point change information. S130. Preprocess the writing trajectory data, the preprocessing including at least one of abnormal data removal, time correction, coordinate normalization and writing state recognition. S140. Extract writing features based on the preprocessed writing trajectory data. The writing features include at least one of writing speed features, pressure features, writing time features, stroke features, and trajectory space features. S150. The facial behavior data is preprocessed, and facial features are extracted based on the preprocessed facial behavior data; wherein the facial features include at least one of facial action unit features, head posture features, and facial key point change features.

[0007] As a further improvement of the present invention, step S200 includes the following steps: S210. After completing the extraction of the writing features and facial features, a unified multimodal supervised learning sample is constructed, wherein the multimodal supervised learning sample is represented as follows: in, This represents a multimodal supervised learning dataset; Indicates the total number of subjects; Indicates the subject's number; Indicates the first The written feature sequences obtained by feature extraction from the subjects; Indicates the first Facial behavioral feature sequences obtained from subject samples through feature extraction; This indicates the corresponding cognitive state label.

[0008] S220. Perform statistical aggregation on the writing feature data and facial feature data respectively to obtain writing numerical features and facial numerical features, which are represented as follows: in, Indicates the first The written numerical feature vector obtained by statistical aggregation of 100 subjects; Indicates the first Facial numerical feature vectors obtained by statistical aggregation of data from 10 subjects; and These represent the statistical aggregation operators for the writing modality and the facial modality, respectively.

[0009] The statistical aggregation operator includes at least one of the following: mean, standard deviation, maximum value, minimum value, quantile, or rate of change. The obtained sample can be represented as: S230. When using a segment-level sample construction method, the continuous writing task is divided into multiple segments according to task segments, writing units, or fixed time windows, and segment-level samples are constructed based on the writing numerical features and facial numerical features corresponding to each segment, as shown below: in, Indicates the first The first subject The written numerical feature vector obtained by statistically aggregating the fragments; Indicates the first The first subject The facial numerical feature vector obtained by statistically aggregating the fragments; Indicates the first Cognitive state labels corresponding to each subject; Indicates the segment number; Indicates the first The number of segments per subject.

[0010] S240. The written numerical features and facial numerical features are respectively input into the templated text generation branch and the numerical feature representation construction branch. The templated text generation branch is used to generate corresponding templated text based on the written numerical features and facial numerical features, and the numerical feature representation construction branch is used to construct the corresponding numerical representation.

[0011] As a further improvement of the present invention, step S300 specifically includes the following steps: S310. Construct corresponding templated texts based on the written numerical features and facial numerical features respectively, wherein the written modality templated text is generated based on the numerical features corresponding to the written modality, and the facial modality templated text is generated based on the numerical features corresponding to the facial modality. S320. The large language model encoder, which freezes the input parameters of the written modality templated text and the facial modality templated text, obtains the semantic embeddings of the corresponding modalities respectively: in, A semantic encoding function for a large language model where parameters remain frozen; This represents templated text representing a writing modality generated from writing numerical features; This represents templated text representing facial modalities generated from numerical facial features. This indicates semantic embedding of writing modality; This represents the semantic embedding of facial modalities.

[0012] S330. The writing modality semantic embedding and the facial modality semantic embedding are concatenated or mapped to obtain the overall semantic embedding: in, This represents the overall semantic embedding obtained by fusing the written modality semantic embedding and the facial modality semantic embedding. The overall semantic embedding serves as a semantic enhancement representation; the parameters of the large language model encoder remain frozen during training and do not participate in gradient updates.

[0013] As a further improvement of the present invention, step S400 specifically includes the following steps: S410. Obtain the written numerical features and facial numerical features, and perform feature mapping processing on them respectively to map the numerical features of different modalities to a unified feature space. S420. Construct written numerical representations and facial numerical representations based on the mapped written numerical features and facial numerical features, respectively; S430. Associate the numerical representation with the overall semantic embedding and use it as the corresponding input of the multimodal fusion module.

[0014] The numerical feature representation construction branch is used to retain the numerical feature information corresponding to the writing modality and the facial modality, and the templated text generation branch is used to extract the corresponding semantic information using the parameter-frozen large language model encoder.

[0015] As a further improvement of the present invention, the multimodal fusion module in step S400 adopts a gated fusion method, specifically including the following steps: S440, The joint numerical representation formed by the written numerical representation and the facial numerical representation, as well as the overall semantic embedding, are used as inputs to the multimodal fusion module; S450. Determine the corresponding gating weights based on the information contribution relationship between different modal features, and adaptively weight the different modal features according to the gating weights; S460. Obtain the gated fusion representation based on the weighted modal features, and input the gated fusion representation as the multimodal fusion representation into the classification module.

[0016] As a further improvement of the present invention, the multimodal fusion module in step S400 adopts a cross-modal Transformer fusion method, specifically including the following steps: S470. Map the written numerical representation, facial numerical representation, and overall semantic embedding to a unified latent space to construct corresponding modal tags, which are represented as follows: in, , and These represent written modal tags, facial modal tags, and semantic modal tags, respectively. , and They represent the linear mapping parameters, respectively. , and These represent the corresponding bias terms; This indicates the written representation of numerical values; This represents facial numerical values; This indicates the overall semantic embedding.

[0017] S480. The writing modality markers, facial modality markers, and semantic modality markers are arranged into a marker sequence according to a preset order: The labeled sequence is then input into a cross-modal Transformer coding layer for multi-head self-attention encoding to obtain an updated labeled sequence: S490. Perform mean pooling or attention pooling on the updated labeled sequence to obtain a cross-modal fusion representation: in, This represents a tag sequence consisting of three modal tags; This represents the updated label sequence; This represents cross-modal fusion representation; This indicates mean pooling or attention pooling operations.

[0018] As a further improvement of the present invention, step S500 includes the following steps: S510. Input the multimodal fusion representation into the classification module, wherein the multimodal fusion representation is a gated fusion representation or a cross-modal Transformer fusion representation; S520. Calculate the classification output based on the multimodal fusion representation, which is expressed as: in, This represents a multimodal fusion representation; Indicates the classifier weight parameters; This represents the classifier bias term; This indicates the output value for the category.

[0019] S530, The classification output value conduct The calculation yields the predicted probability that the target object belongs to each cognitive state category: in, This represents the predicted probability vector for each cognitive state category. The cognitive state-assisted screening result corresponding to the target object is determined based on the predicted probabilities.

[0020] As a further improvement of the present invention, step S500 further includes: S540. Construct training sample batches, obtain the predicted probability corresponding to each cognitive state category based on the output of the classification module, and calculate the model training loss based on the predicted probability and the true label. S550. Calculate the model training loss using the cross-entropy loss function or the category-weighted cross-entropy loss function, and update the parameters of the numerical feature representation construction branch, multimodal fusion module, and classification module in reverse based on the model training loss, while keeping the parameters of the large language model encoder frozen; wherein, the category-weighted cross-entropy loss function is expressed as: in, This indicates the number of samples included in the current training batch; Indicates the sample number in the training batch; Indicates the number of cognitive state categories; Indicates the first Does the sample belong to the ? The label indicating the class; The model represents the first The predicted probability of a class; Indicates the first The weight corresponding to the class.

[0021] Based on this, the present invention also provides a computer-readable storage medium storing a computer program that, when executed, implements the method described in any of the above embodiments.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention constructs multimodal supervised learning samples of writing trajectory data and facial behavior data, statistically aggregates behavioral features of different modalities, and uses a large language model encoder with frozen parameters to semantically encode templated text. While preserving the semantic expressive power of the pre-trained language model, it avoids the increased training cost caused by fine-tuning of large language model parameters and the overfitting problem in small sample scenarios. Furthermore, it combines numerical representation and semantically enhanced representation for fusion modeling, fully explores the complementary information and higher-order semantic associations between different modalities, improves the expressive power of multimodal features, and enhances the model's accuracy and generalization ability in recognizing cognitive states. Thus, it achieves high-precision assisted screening of cognitive impairment and has good practical value and application prospects. Attached Figure Description

[0023] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the synchronous acquisition of digital writing motion trajectory and facial behavior signals in an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this invention should have the meanings commonly understood by those skilled in the art to which this invention pertains. Terms such as "first" and "second" are used only to distinguish different objects and do not indicate order, quantity, or importance.

[0027] Large language model encoder with frozen parameters: refers to a large language model encoder whose parameters remain frozen during model training and do not participate in gradient updates.

[0028] An Action Unit (AU) is the smallest facial muscle movement unit defined in the Facial Action Coding System (FACS). It corresponds to the independent contraction action of a single facial muscle or muscle group, such as frowning, raising eyebrows, or turning up the corners of the mouth.

[0029] Gated fusion: An adaptive feature fusion mechanism that uses a learnable gating network to generate element-wise weight coefficients, dynamically adjusting the contribution ratio of different input features in the fusion result.

[0030] Cross-modal Transformer: A cross-modal feature fusion model based on the Transformer self-attention architecture. By transforming features from different modalities into modal tokens in a unified format, it utilizes a multi-head self-attention mechanism to model pairwise interactions and higher-order associations between modalities. This enables in-depth mining of the intrinsic linkage patterns between features from different modalities, generating fusion representations with higher information density.

[0031] Templated text: refers to standardized text descriptions that are automatically generated by converting structured numerical features according to preset fixed formats and sentence rules.

[0032] Statistical aggregation operators: These are sets of calculation rules used to perform dimensionality reduction statistics on time series features and generate fixed-dimensional statistical features. Common types include mean, standard deviation, maximum value, minimum value, quantile, rate of change, etc.

[0033] Cognitive impairment assisted screening: This refers to a technical approach that utilizes digital behavioral data collection technology and artificial intelligence algorithms to perform risk stratification and preliminary screening of the cognitive status of test subjects. The output results serve only as preliminary screening prompts and follow-up management references, and are not directly used as independent medical diagnostic conclusions. Its core function is to improve the efficiency and accessibility of large-scale population cognitive screening, and to assist professionals in conducting early cognitive health management.

[0034] Example 1 Please see Figure 1 and Figure 2 This invention provides a cognitive impairment assisted screening method based on a frozen large language model. The method adopts a complete process of bimodal behavioral data collection and feature extraction, multimodal supervised sample construction and numerical feature aggregation, frozen large language model semantic enhancement, multimodal adaptive fusion, and cognitive state risk stratification output. It integrates numerical features and semantic priors of writing motor behavior and facial behavior in small sample scenarios to achieve stable and reliable cognitive impairment assisted screening.

[0035] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The method includes the following steps.

[0036] S100. Obtain writing trajectory data and facial behavior data generated by the target object during the execution of a preset cognitive task, extract features from the writing trajectory data and facial behavior data, and obtain corresponding writing features and facial features.

[0037] This step proceeds step by step from raw signal acquisition and data cleaning to feature extraction, providing a standardized feature foundation for subsequent modeling. Specifically, it includes the following sub-steps: S110. Collect writing trajectory data generated by the target object during the execution of a preset cognitive task, wherein the writing trajectory data includes at least one of writing trajectory coordinates, time information, pressure information, and air writing state information and paper writing state information.

[0038] In actual deployment, it is preferable to use a dot matrix digital pen with special dot matrix paper to achieve high-precision acquisition of writing trajectory. Alternatively, depending on the application scenario, a stylus pen, digitizer, or other acquisition device with equivalent functions can be selected.

[0039] The pre-defined cognitive writing tasks preferably include the following five types: First, a random number copying task, such as copying irregular number strings; second, an intersecting pentagon tracing task, used to induce visual constructivist and fine motor behavior; third, a graded Chinese word copying task, such as copying Chinese words with different stroke complexities, such as "son," "daughter," "evening," "key," "umbrella," and "supermarket"; fourth, a spontaneous sentence generation task, such as generating and writing a sentence around the word "umbrella"; and fifth, a continuous subtraction of 7 writing task, such as subtracting 7 consecutively from 100 and writing the results 100, 93, 86, etc. These tasks can induce behavioral signals under different cognitive loads from multiple aspects, including visual-spatial attention, graphic constructivist behavior, lexical processing, language generation, working memory, and executive function.

[0040] By collecting standardized writing trajectories, the fine motor control information of subjects during the cognitive task can be captured objectively and completely without the need for subjective manual annotation. This effectively avoids the bias and information omission caused by manual recording and provides a reliable data foundation for the accurate extraction of subsequent kinematic features.

[0041] S120. Collect facial behavior data generated by the target object during the execution of a preset cognitive task, wherein the facial behavior data includes at least one of facial action unit activation information, head posture information, and facial key point change information.

[0042] During implementation, it is preferable to place a camera unit directly in front of the subject, and start recording synchronously with the writing trajectory acquisition device to acquire video of the subject's facial behavior throughout the process. Subsequently, mature facial behavior analysis tools such as OpenFace can be used to process the video frame by frame to extract information such as the activation intensity, occurrence confidence, head pitch angle, yaw angle, roll angle, and facial key point coordinates of each facial action unit.

[0043] Facial behavior data collected synchronously with writing trajectory data can supplement cognitive-related behavioral signals from the dimensions of facial muscle activity and head posture changes, breaking through the information limitations of a single writing modality. By utilizing multi-dimensional behavioral representations, it can more comprehensively reflect the cognitive load and neuromotor control status of the subjects, effectively improving the robustness and anti-interference ability of the screening method and reducing the impact of single-modal noise on the final results.

[0044] S130. Preprocess the writing trajectory data. The preprocessing includes at least one of the following: abnormal data removal, time correction, coordinate normalization, and writing state recognition.

[0045] To address potential anomalies such as jump points, repetition points, and device drift in the original writing trajectory data, the system first removes and corrects abnormal data through threshold judgment and interpolation algorithms. Secondly, it uniformly calibrates the timestamps of different devices to eliminate clock deviations. Subsequently, it normalizes the coordinate data to eliminate the scale effects caused by differences in writing size. Finally, it automatically identifies writing states such as pen start, pen end, and pause in the air based on pressure thresholds and status indicators, thus dividing the data into valid paper writing segments and air pen movement segments.

[0046] Through multi-stage preprocessing, noise and interference in the original data can be effectively filtered out, the temporal and spatial benchmarks of the data can be unified, the effective writing segments can be accurately divided, the accuracy and comparability of subsequent feature extraction can be guaranteed, and the deviation of the subsequent analysis results can be avoided due to the quality problems of the original data.

[0047] S140. Extract writing features from the preprocessed writing trajectory data. The writing features include at least one of the following: writing speed features, pressure features, writing time features, stroke features, and trajectory space features.

[0048] Based on the preprocessed temporal trajectory data, multi-dimensional writing features can be automatically calculated using preset calculation rules. These features include: writing speed features (instantaneous speed, average speed, speed variance, frequency of speed reduction events, etc.); pressure features (average pressure, pressure standard deviation, maximum pressure, minimum pressure, etc.); writing time features (total writing time on paper, total pause time in the air, average duration of a single stroke, etc.); stroke features (total number of strokes, average stroke length, percentage of horizontal and vertical strokes, etc.); and trajectory space features (coordinate variance, trajectory envelope area, stroke inclination angle distribution, etc.).

[0049] Taking instantaneous velocity and average pressure as examples, the calculation method is as follows: Given the first The coordinates and timestamps of each sampling point, and the instantaneous writing speed are: in, , They represent the first The x and y coordinates of each sampling point; Indicates the corresponding timestamp; This indicates the instantaneous writing speed between adjacent sampling points.

[0050] Instantaneous acceleration can be expressed as: in, Indicates the first The instantaneous acceleration corresponding to each sampling point.

[0051] Mean pressure can be expressed as: in, Indicates average pressure; Indicates the first Pressure values ​​at each sampling point; This indicates the number of valid sampling points. The time spent in the air and the time spent writing on paper can be obtained by summing the durations of each air trajectory segment and the pen stroke trajectory segment, respectively.

[0052] By automatically extracting multi-dimensional writing features, continuous temporal motion signals can be transformed into structured and quantifiable indicators, which can characterize the subject's fine motor abilities from multiple levels such as movement rhythm, force control, and spatial construction, providing a unified feature foundation for subsequent numerical modeling and semantic enhancement.

[0053] The aforementioned writing speed features, pressure features, writing time features, stroke features, and trajectory space features are associated with the target object and its corresponding cognitive task to form a writing feature sequence corresponding to the target object, which serves as the input for multimodal supervised learning sample construction and statistical aggregation in subsequent step S200. The writing features obtained in step S140 are used to characterize the writing behavior state at the sampling point, stroke, or writing segment level. Subsequent step S220 further performs subject-level or segment-level statistical aggregation on the writing feature sequence to form fixed-dimensional writing numerical features.

[0054] S150. Preprocess the facial behavior data and extract facial features based on the preprocessed facial behavior data; wherein, the facial features include at least one of facial action unit features, head posture features, and facial key point change features.

[0055] First, the facial video data is preprocessed with face detection, key point localization and inter-frame alignment to exclude invalid frames with face offset or occlusion. Then, the activation intensity and occurrence marker of facial action units, the angle value of head pose and the coordinate changes of key points are estimated frame by frame. Statistics are performed within the effective time window to obtain the mean, standard deviation, maximum value and minimum value of each feature.

[0056] With the first Taking the average intensity and standard deviation of a facial motion unit as an example, the calculation method is as follows: in, and They represent the first Mean and standard deviation of activation intensity of each action unit; Indicates the first The first frame The activation intensity of each action unit; This indicates the number of valid video frames. Correspondingly, the mean, maximum, minimum, and variance of the confidence scores for action units can also be statistically analyzed.

[0057] Standardized extraction of facial features can transform continuous video signals into objective numerical features. The entire process is based on automatic algorithm calculation, without relying on subjective human judgment of facial expressions and emotions, ensuring the objectivity and reproducibility of the features. It can capture subtle changes in facial micro-expressions and head postures of subjects in cognitive tasks, supplementing behavioral information that is difficult to cover in writing modalities.

[0058] S200. Construct multimodal supervised learning samples based on writing features and facial features, and statistically aggregate the writing features and facial features to obtain writing numerical features and facial numerical features.

[0059] This step organizes and aggregates features from a single frame and segment into standardized supervised samples, providing a unified sample unit for subsequent two-branch modeling. Specifically, it includes the following sub-steps: S210. After completing the extraction of writing features and facial features, construct a unified multimodal supervised learning sample, which is represented as follows: in, This represents a multimodal supervised learning dataset; Indicates the total number of subjects; Indicates the subject's number; Indicates the first The written feature sequences obtained by feature extraction from the subjects; Indicates the first Facial behavioral feature sequences obtained from subject samples through feature extraction; This indicates the corresponding cognitive state label.

[0060] During the dataset construction phase, subjects were used as the basic unit. Each subject's writing feature sequence and facial behavior feature sequence were bound to corresponding cognitive state labels to form a complete supervised learning sample set. The labeling system can be set to a binary classification format according to the screening objectives, for example, 0 corresponds to normal cognition and 1 corresponds to a risk of mild cognitive impairment, or 0 corresponds to normal cognition and 2 corresponds to a risk of Alzheimer's disease-related cognitive decline.

[0061] By constructing unified multimodal supervised learning samples, feature sequences from two different sources, writing and facial features, can be associated with a unified supervision target, providing standardized sample units for subsequent joint modeling, ensuring that bimodal data share the same label supervision signal, and guaranteeing the effectiveness of multimodal modeling.

[0062] S220. Perform statistical aggregation on the handwriting feature data and facial feature data respectively to obtain the handwriting numerical features and facial numerical features, which are represented as follows: in, Indicates the first The written numerical feature vector obtained by statistical aggregation of 100 subjects; Indicates the first Facial numerical feature vectors obtained by statistical aggregation of data from 10 subjects; and These represent the statistical aggregation operators for the writing modality and the facial modality, respectively.

[0063] Statistical aggregation operators include at least one of the following: mean, standard deviation, maximum, minimum, quantile, or rate of change. The resulting sample can be represented as: For each subject's temporal feature sequence, dimensionality reduction aggregation is performed using corresponding statistical aggregation operators to transform fine-grained features at the fragment and frame levels into fixed-dimensional numerical feature vectors at the subject level. During the aggregation process, an appropriate aggregation method can be selected based on the physical meaning of the features. For example, summation or mean can be taken for duration-type features, maximum or mean can be taken for intensity-type features, and standard deviation can be taken for fluctuation-type features.

[0064] Statistical aggregation can transform time-series features of varying lengths into numerical feature vectors of uniform dimensions, adapting to the input requirements of subsequent machine learning models. At the same time, the aggregated features can reflect the overall behavioral performance of the subjects throughout the task, reducing the interference of local noise and improving the stability and representativeness of the features.

[0065] Specifically, step S220 uses the writing feature sequence obtained in step S140 as input to the writing modality and performs corresponding statistical aggregation according to the data form of different writing features. For example, for writing speed features, the average value, standard deviation, maximum value, or rate of change in the complete cognitive task or corresponding task segment can be statistically analyzed; for pressure features, the average pressure, pressure standard deviation, maximum pressure, or minimum pressure can be statistically analyzed; for writing time features, the paper writing time, air pen movement time, or corresponding time statistics can be statistically analyzed; and corresponding statistics are performed on stroke features and trajectory space features, thereby converting the temporal or local writing features obtained in step S140 into fixed-dimensional writing numerical features.

[0066] S230. When using a segment-level sample construction method, the continuous writing task is divided into multiple segments according to task segments, writing units, or fixed time windows, and segment-level samples are constructed based on the writing numerical features and facial numerical features corresponding to each segment, as shown below: in, Indicates the first The first subject The written numerical feature vector obtained by statistically aggregating the fragments; Indicates the first The first subject The facial numerical feature vector obtained by statistically aggregating the fragments; Indicates the first Cognitive state labels corresponding to each subject; Indicates the segment number; Indicates the first The number of segments per subject.

[0067] In scenarios requiring more granular analysis, a complete cognitive task can be broken down into multiple sub-task segments, individual writing units, or equal-length time windows. Features can then be extracted and aggregated for each segment to form segment-level samples. During the model inference phase, subject-level auxiliary screening results can be generated based on the prediction results corresponding to each segment.

[0068] The fragment-level sample construction method can capture dynamic behavioral changes during the task and uncover more granular behavioral patterns. At the same time, data augmentation is achieved through sample segmentation, which can effectively expand the number of training samples in small sample scenarios and improve the generalization ability of the model. The multi-fragment result aggregation method can also further improve the stability of the final screening results.

[0069] S240. Input the written numerical features and facial numerical features into the templated text generation branch and the numerical feature representation construction branch, respectively. The templated text generation branch is used to generate corresponding templated text based on the written numerical features and facial numerical features, and the numerical feature representation construction branch is used to construct the corresponding numerical representation.

[0070] After obtaining standardized written numerical features and facial numerical features, a dual-branch parallel representation architecture is adopted, which simultaneously feeds the same set of features into two independent processing branches: one is a templated text generation branch, which is used to complete the conversion of numerical features into semantic representations; the other is a numerical feature representation construction branch, which is used to complete the spatial mapping of numerical features and the preservation of numerical representations.

[0071] Through the dual-branch parallel architecture design, semantic representation and numerical representation can be generated simultaneously based on the same set of numerical features. This fully utilizes the semantic prior of the large language model for feature enhancement while completely preserving the quantitative information of the original numerical features, avoiding information loss caused by a single representation method, and providing dual information support for subsequent multimodal deep fusion.

[0072] Therefore, the handwriting numerical features extracted in step S140 and statistically aggregated in step S220 enter two subsequent processing paths: one is the input templated text generation branch, which generates handwriting modality templated text in step S300, and obtains the corresponding semantic representation through the large language model encoder with frozen parameters; the other is the input numerical feature representation construction branch, which forms the handwriting numerical representation in step S400. The semantic representation and numerical representation then jointly participate in multimodal fusion, enabling the handwriting behavior information obtained in step S140 to continuously participate in subsequent cognitive state-assisted screening along the semantic representation path and the numerical representation path.

[0073] S300. Generate templated text based on written numerical features and facial numerical features. Semantically encode the large language model encoder with frozen input parameters of the templated text to obtain a semantically enhanced representation.

[0074] This step injects semantic priors into behavioral features through numerical textification and frozen encoding, and specifically includes the following sub-steps: S310. Construct corresponding templated texts based on the numerical features of writing and facial features respectively. Specifically, generate templated text for writing modality based on the numerical features corresponding to writing modality, and generate templated text for facial modality based on the numerical features corresponding to facial modality.

[0075] The templated text generation preferably uses a structured format of feature name = feature value. Alternatively, it can be generated by discretizing the numerical features based on a preset threshold, resulting in a natural language description where the feature name is at a low / medium / high level. The generation process is entirely automated, based on numerical feature mapping, without incorporating subjective human handwriting evaluation, facial expression judgment, or clinical diagnostic conclusions.

[0076] Among them, the writing numerical features used in the writing modality templated text are obtained by statistical aggregation of the writing features extracted in step S140 in step S220.

[0077] For example, the writing modal text can be expressed as "mean pressure = 0.56; time in the air = 2350ms; time on paper = 4180ms; mean velocity = 12.30mm / s; velocity deceleration events = 5", and the facial modal text can be expressed as "AU04 mean intensity = 1.20; AU06 maximum intensity = 2.10; AU12 mean intensity = 1.80; head posture change rate = 0.15".

[0078] Through templated text transformation, structured numerical features can be mapped into descriptive text with natural language semantics, providing content that conforms to the input format for the semantic encoding of subsequent large language models. Without introducing subjective information, numerical features are given clear semantic meanings, enabling large language models to enhance feature representations using semantic knowledge accumulated during pre-training.

[0079] S320. Freeze the input parameters of the written modality templated text and the facial modality templated text in the large language model encoder to obtain the semantic embeddings of the corresponding modalities: in, A semantic encoding function for a large language model where parameters remain frozen; This represents templated text representing a writing modality generated from writing numerical features; This represents templated text representing facial modalities generated from numerical facial features. This indicates semantic embedding of writing modality; This represents the semantic embedding of facial modalities.

[0080] The preferred large language model encoder is the pre-trained Qwen-3.5 encoder, but other pre-trained language model encoders with equivalent text encoding capabilities can also be used. Throughout the model training process, the parameters of this encoder remain frozen and do not participate in gradient updates; they serve only as a fixed feature extractor to map text into semantic embedding vectors.

[0081] By leveraging the large language model encoder with parameter freezing, we can fully utilize the general semantic knowledge accumulated through large-scale pre-training, inject semantic priors into writing and facial behavior features, and improve the richness of feature representation. At the same time, the parameter freezing design significantly reduces the number of trainable parameters of the model, effectively reducing the risk of overfitting in small sample scenarios, and balancing the effect of semantic enhancement with the stability of model training.

[0082] S330. Concatenate or map the semantic embeddings of the writing modality and the facial modality to obtain the overall semantic embedding: in, This represents the overall semantic embedding obtained by fusing the written modality semantic embedding and the facial modality semantic embedding. The overall semantic embedding serves as a semantic enhancement representation; the parameters of the large language model encoder remain frozen during training and do not participate in gradient updates.

[0083] Two modally independent semantic embedding vectors are concatenated along their dimensions to form a unified overall semantic embedding vector, which serves as the final output of the semantic enhancement branch. In some implementations, a linear mapping layer can be used to transform the dimensions of the concatenated vector to match the output dimension of the numerical branch, facilitating subsequent fusion processing.

[0084] By obtaining the overall semantic embedding through modal concatenation, the semantic information of writing and facial behaviors can be integrated into a unified semantic representation, fully preserving the semantic enhancement results of the two modalities, and providing complete semantic information input for subsequent fusion with numerical representation.

[0085] S400. Input the semantic enhancement representation, i.e. the overall semantic embedding, and the numerical representations corresponding to the written numerical features and facial numerical features into the multimodal fusion module for fusion processing to obtain the multimodal fusion representation.

[0086] This step first completes the spatial mapping of numerical features, and then achieves deep complementarity between semantics and numerical representation through an adaptive fusion mechanism. Specifically, it includes the following sub-steps: S410. Obtain written numerical features and facial numerical features, and perform feature mapping processing on them respectively to map the numerical features of different modalities to a unified feature space.

[0087] The numerical feature representation construction branch contains independent writing projection blocks and facial projection blocks, which respectively receive writing numerical features and facial numerical features as input.

[0088] Each projection block preferably adopts a lightweight multilayer perceptron structure, including at least one of linear mapping layer, layer normalization layer, nonlinear activation layer and random deactivation layer, which maps two types of features with different dimensions and distributions to the same dimension of latent feature space through learnable parameters.

[0089] By using an independent projection module to spatially map the numerical features of different modalities, the dimensional and distributional differences between handwriting and facial features can be effectively eliminated, aligning the two types of features into a unified feature space and laying the foundation for subsequent joint representation and cross-modal fusion. At the same time, the lightweight network structure ensures computational efficiency and facilitates deployment and operation on edge devices.

[0090] S420. Construct written numerical representations and facial numerical representations based on the mapped written numerical features and facial numerical features, respectively.

[0091] In a preferred embodiment, the written numerical representation output by the writing projection module and the facial numerical representation output by the facial projection module can be concatenated to form a joint numerical representation. The process can be represented as follows: in, and These represent the writing projection parameters and the facial projection parameters, respectively. and These represent written numerical features and facial numerical features, respectively. and These represent written numerical representation and facial numerical representation, respectively. This represents a joint numerical representation. This process maps numerical features of different dimensions and sources to a unified latent space, facilitating subsequent fusion with the overall semantic embedding.

[0092] By constructing a joint numerical representation through concatenation, the numerical representations of two modalities can be integrated into a unified numerical feature vector, fully preserving the quantitative information in the original numerical features. This complements the output of the semantic enhancement branch, providing complete numerical evidence support for subsequent fusion.

[0093] S430. Associate the numerical representation with the overall semantic embedding and use it as the corresponding input to the multimodal fusion module.

[0094] Among them, the numerical feature representation construction branch is used to retain the numerical feature information corresponding to the writing modality and the facial modality, and the templated text generation branch is used to extract the corresponding semantic information using the large language model encoder with frozen parameters.

[0095] The joint numerical representation is mapped one-to-one with the overall semantic embedding to construct the corresponding joint input, which is then fed into the subsequent multimodal fusion module. The two branches each have their own functions: the numerical branch focuses on preserving the accuracy of the original quantized features, while the semantic branch focuses on injecting prior knowledge to enhance the semantic richness of the representation. The two form a complementary dual representation system.

[0096] By associating and pairing the dual-branch representations, numerical evidence and semantic priors can be mapped to the same sample, providing matching input for subsequent adaptive fusion. The parallel design of the dual branches also ensures that the two representations do not interfere with each other, each retaining its own information advantage, thus achieving a balance between quantification accuracy and semantic richness.

[0097] In a preferred embodiment, the multimodal fusion module employs a gated fusion method, specifically including the following steps: S440, The joint numerical representation formed by the written numerical representation and the facial numerical representation, as well as the overall semantic embedding, are used as inputs to the multimodal fusion module.

[0098] The gated fusion module receives two inputs: joint numerical representation and overall semantic embedding. First, it maps the two to the same feature dimension through a linear layer to ensure dimension matching in subsequent weighted operations.

[0099] S450. Determine the corresponding gating weights based on the information contribution relationship between different modal features, and adaptively weight different modal features according to the gating weights.

[0100] First, the dimension-aligned numerical representation is concatenated with the semantic embedding and input into the gating network to calculate the element-wise gating weight vector. The weight values ​​range from 0 to 1, and the calculation process is as follows: The gating vector is then calculated based on the concatenation result: in, This represents the joint numerical representation after dimension alignment. This represents the overall semantic embedding after dimension alignment; , , Represents the linear mapping parameters; , , Indicates the corresponding bias term; This represents the Sigmoid activation function; This represents the gating weight vector.

[0101] The gating weights can be adaptively adjusted according to the feature quality of different samples. When the signal-to-noise ratio of numerical features is higher, the gating weights are tilted towards numerical representation. When the semantic association is more significant, the contribution of semantic information is strengthened. This achieves sample-level adaptive adjustment of the fusion weights and avoids the problem of insufficient adaptability caused by fixed fusion weights.

[0102] The final weighted fusion representation is obtained as follows: in, This represents gating fusion representation.

[0103] The gating fusion method has a simple logic, high computational efficiency, and adaptive weighting capability. It can maintain a stable fusion effect in different groups of people and different task scenarios, making it suitable for screening scenarios with high requirements for real-time performance and ease of deployment.

[0104] In another preferred embodiment, the multimodal fusion module adopts a cross-modal Transformer fusion method, specifically including the following steps: S470. Map the written numerical representation, facial numerical representation, and overall semantic embedding to a unified latent space to construct corresponding modal tags, which are represented as follows: in, , and These represent written modal tags, facial modal tags, and semantic modal tags, respectively. , and They represent the linear mapping parameters, respectively. , and These represent the corresponding bias terms; This indicates the written representation of numerical values; This represents facial numerical values; This indicates the overall semantic embedding.

[0105] The three independent representations—written numerical representation, facial numerical representation, and overall semantic embedding—are mapped to modal tokens of the same dimension through a linear layer, giving them a unified vector dimension and enabling them to form a sequence input Transformer encoding layer.

[0106] S480. Arrange the written modal markers, facial modal markers, and semantic modal markers into a marker sequence according to a preset order: The labeled sequence is then input into a cross-modal Transformer encoding layer for multi-head self-attention encoding to obtain the updated labeled sequence: Three modality labels are concatenated in a fixed order to form a label sequence of length 3, which is then fed into a multi-head self-attention encoding layer. Through the self-attention mechanism, the three modality labels can interact pairwise, capturing high-order correlations between different modality features and outputting an updated label sequence.

[0107] Compared to the weighted combination of gated fusion, the cross-modal Transformer can explicitly model the paired and higher-order interaction relationships between the three modalities of writing, face and semantics, deeply explore the intrinsic connections between features of different modalities, capture richer cross-modal fusion patterns, and further improve the discriminativeness of fusion representations.

[0108] S490. Perform mean pooling or attention pooling on the updated labeled sequence to obtain a cross-modal fusion representation: in, This represents a tag sequence consisting of three modal tags; This represents the updated label sequence; This represents cross-modal fusion representation; This indicates mean pooling or attention pooling operations.

[0109] The three updated labels are aggregated into a single fixed-dimensional vector through pooling operations, serving as the final cross-modal fusion representation. Mean pooling averages the three labels, offering advantages such as simplicity and high computational efficiency; attention pooling, on the other hand, adaptively aggregates information from different modalities using learnable attention weights, providing greater flexibility.

[0110] By pooling aggregation, a unified dimensional fusion representation is obtained, which can adapt to the input requirements of subsequent classification modules; the cross-modal Transformer fusion method can realize deeper multimodal interaction and is suitable for auxiliary screening scenarios with higher requirements for classification accuracy.

[0111] S500: Input the multimodal fusion representation into the classification module, and determine the cognitive state auxiliary screening result corresponding to the target object based on the classification output result.

[0112] This step outputs standardized risk stratification results through classification modeling and completes the training and optimization of the model. Specifically, it includes the following sub-steps: S510. Input the multimodal fusion representation into the classification module, wherein the multimodal fusion representation is a gated fusion representation or a cross-modal Transformer fusion representation.

[0113] The classification module preferably adopts a fully connected classifier structure, receiving a fixed-dimensional multimodal fusion representation as input. Regardless of whether gated fusion or cross-modal Transformer fusion is used, the output fusion representation is a vector of the same dimension, which can be directly connected to the same classification module.

[0114] S520. Calculate the classification output based on the multimodal fusion representation, which is expressed as follows: in, This represents a multimodal fusion representation; Indicates the classifier weight parameters; This represents the classifier bias term; This indicates the output value for the category.

[0115] Multimodal fusion represents a linear transformation through a fully connected layer, yielding the classification output values ​​for each category. This reflects the original score of the sample belonging to each category.

[0116] S530, Output values ​​for classification conduct The calculation yields the predicted probability that the target object belongs to each cognitive state category: in, This represents the predicted probability vector for each cognitive state category. The cognitive state-assisted screening result corresponding to the target object is determined based on the predicted probabilities.

[0117] pass The function will categorize the output values. Normalized to probability values ​​between 0 and 1, with the sum of probabilities for all categories being 1. Based on the category corresponding to the maximum predicted probability, the corresponding cognitive state risk stratification result is output, such as risk propensity for normal cognition, risk propensity for mild cognitive impairment, etc.

[0118] By standardizing probability outputs and risk stratification, the abstract representation of the model can be transformed into valuable screening prompts. The results are clear and intuitive, making them easy for non-professionals to understand and use. At the same time, clear usage boundaries ensure the compliance and rigor of the technology application.

[0119] S540. Construct training sample batches, obtain the predicted probability corresponding to each cognitive state category based on the output of the classification module, and calculate the model training loss based on the predicted probability and the true label.

[0120] During model training, samples are input into the network in batches. After forward propagation to obtain the predicted probabilities, these probabilities are compared with the corresponding true labels of the samples to calculate the loss value, which is then used for backpropagation to update the model parameters. Throughout the training process, the parameters of the large language model encoder are kept frozen, and only the parameters of the numerical mapping module, multimodal fusion module, and classification module are updated.

[0121] S550. Calculate the model training loss using the cross-entropy loss function or the category-weighted cross-entropy loss function, and update the parameters of the numerical feature representation construction branch, multimodal fusion module, and classification module in reverse based on the model training loss, while keeping the parameters of the large language model encoder frozen; wherein, the category-weighted cross-entropy loss function is expressed as: in, This indicates the number of samples included in the current training batch; Indicates the sample number in the training batch; Indicates the number of cognitive state categories; Indicates the first Does the sample belong to the ? The label indicating the class; The model represents the first The predicted probability of a class; Indicates the first The weight corresponding to the class.

[0122] Ordinary cross-entropy loss is suitable for class-balanced datasets, while class-weighted cross-entropy loss adapts to the class imbalance problem commonly encountered in cognitive screening scenarios by assigning differentiated weights to different classes. Class weights are typically inversely proportional to the number of samples in each class, increasing the contribution of minority class samples in the loss calculation.

[0123] By using a class-weighted loss optimization strategy, we can effectively alleviate the model bias problem caused by sample class imbalance and improve the recognition accuracy of high-risk minority classes. At the same time, the training strategy of freezing large model parameters not only retains the support of semantic priors, but also avoids the problem of overfitting when fine-tuning large models with small samples, thus ensuring the stability and generalization ability of model training.

[0124] This invention constructs multimodal supervised learning samples of writing trajectory data and facial behavior data, statistically aggregates behavioral features of different modalities, and uses a large language model encoder with frozen parameters to semantically encode templated text. While preserving the semantic expressive power of the pre-trained language model, it avoids the increased training cost caused by fine-tuning of large language model parameters and the overfitting problem in small sample scenarios. Furthermore, it combines numerical representation and semantically enhanced representation for fusion modeling, fully explores the complementary information and higher-order semantic associations between different modalities, improves the expressive power of multimodal features, and enhances the model's accuracy and generalization ability in recognizing cognitive states. Thus, it achieves high-precision assisted screening of cognitive impairment and has good practical value and application prospects.

[0125] It should be noted that the method of this embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In this distributed scenario, any one of the multiple devices can execute only one or more steps of the method of this embodiment, and the multiple devices will interact with each other to complete the method of this embodiment.

[0126] Example 2 Based on the same inventive concept, and corresponding to any of the above method embodiments, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed, implements the methods described in any of the above embodiments.

[0127] The computer-readable storage medium may include phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, read-only optical disc memory (CD-ROM), digital multifunction optical disc (DVD), magnetic tape, disk storage devices, or other media capable of storing computer programs and being accessed by computing devices.

[0128] When the computer program stored in the aforementioned computer-readable storage medium is executed, it implements the cognitive impairment assisted screening method based on the frozen large language model described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0129] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Modifications, equivalent substitutions, or improvements made by those skilled in the art without departing from the spirit and principles of the present invention should all fall within the scope of protection of the present invention.

Claims

1. A cognitive impairment screening method based on a frozen large language model, characterized in that, The method includes the following steps: S100: Obtain writing trajectory data and facial behavior data generated by the target object during the execution of a preset cognitive task; extract features from the writing trajectory data and facial behavior data to obtain corresponding writing features and facial features. S200. Construct multimodal supervised learning samples based on the writing features and facial features, and statistically aggregate the writing features and facial features to obtain writing numerical features and facial numerical features. S300. Generate templated text based on the written numerical features and facial numerical features, and perform semantic encoding on the large language model encoder with the input parameters of the templated text frozen to obtain a semantically enhanced representation; S400. Input the semantic enhancement representation and the numerical representations corresponding to the written numerical features and facial numerical features into the multimodal fusion module for fusion processing to obtain the multimodal fusion representation; S500: Input the multimodal fusion representation into the classification module, and determine the cognitive state auxiliary screening result corresponding to the target object based on the classification output result.

2. The cognitive impairment assisted screening method based on a frozen large language model according to claim 1, characterized in that, S100 specifically includes the following steps: S110. Collect writing trajectory data generated by the target object during the execution of a preset cognitive task, wherein the writing trajectory data includes at least one of writing trajectory coordinates, time information, pressure information, and air writing state information and paper writing state information; S120. Collect facial behavior data generated by the target object during the execution of the preset cognitive task, wherein the facial behavior data includes at least one of facial action unit activation information, head posture information, and facial key point change information. S130. Preprocess the writing trajectory data, the preprocessing including at least one of abnormal data removal, time correction, coordinate normalization and writing state recognition. S140. Extract writing features based on the preprocessed writing trajectory data. The writing features include at least one of writing speed features, pressure features, writing time features, stroke features, and trajectory space features. S150. The facial behavior data is preprocessed, and facial features are extracted based on the preprocessed facial behavior data; wherein the facial features include at least one of facial action unit features, head posture features, and facial key point change features.

3. The cognitive impairment assisted screening method based on a frozen large language model according to claim 1, characterized in that, S200 includes the following steps: S210. After completing the extraction of the writing features and facial features, a unified multimodal supervised learning sample is constructed, wherein the multimodal supervised learning sample is represented as follows: in, This represents a multimodal supervised learning dataset; Indicates the total number of subjects; Indicates the subject's number; Indicates the first The written feature sequences obtained by feature extraction from the subjects; Indicates the first Facial behavioral feature sequences obtained from subject samples through feature extraction; Indicates the corresponding cognitive state label; S220. Perform statistical aggregation on the writing features and facial features respectively to obtain writing numerical features and facial numerical features, which are represented as follows: in, Indicates the first The written numerical feature vector obtained by statistical aggregation of 100 subjects; Indicates the first Facial numerical feature vectors obtained by statistical aggregation of data from 10 subjects; and These represent statistical aggregation operators for the writing modality and the facial modality, respectively. The statistical aggregation operator includes at least one of the following: mean, standard deviation, maximum value, minimum value, quantile, or rate of change. The obtained sample can be represented as: S230. When using a segment-level sample construction method, the continuous writing task is divided into multiple segments according to task segments, writing units, or fixed time windows, and segment-level samples are constructed based on the writing numerical features and facial numerical features corresponding to each segment, as shown below: in, Indicates the first The first subject The written numerical feature vector obtained by statistically aggregating the fragments; Indicates the first The first subject The facial numerical feature vector obtained by statistically aggregating the fragments; Indicates the first Cognitive state labels corresponding to each subject; Indicates the segment number; Indicates the first The number of segments per subject; S240. Input the written numerical features and facial numerical features into the templated text generation branch and the numerical feature representation construction branch, respectively; the templated text generation branch is used to generate corresponding templated text based on the written numerical features and facial numerical features, and the numerical feature representation construction branch is used to construct the corresponding numerical representation.

4. The cognitive impairment assisted screening method based on a frozen large language model according to claim 1, characterized in that, S300 specifically includes the following steps: S310. Construct corresponding templated texts based on the written numerical features and facial numerical features respectively, wherein the written modality templated text is generated based on the numerical features corresponding to the written modality, and the facial modality templated text is generated based on the numerical features corresponding to the facial modality. S320. The large language model encoder, which freezes the input parameters of the written modality templated text and the facial modality templated text, obtains the semantic embeddings of the corresponding modalities respectively: in, A semantic encoding function for a large language model where parameters remain frozen; This represents templated text representing a writing modality generated from writing numerical features; This represents templated text representing facial modalities generated from facial numerical features. This indicates semantic embedding of writing modality; Represents facial modality semantic embedding; S330. The writing modality semantic embedding and the facial modality semantic embedding are concatenated or mapped to obtain the overall semantic embedding: in, The overall semantic embedding is obtained by fusing the written modality semantic embedding and the facial modality semantic embedding; the overall semantic embedding serves as a semantic enhancement representation; the parameters of the large language model encoder are kept frozen during training and do not participate in gradient updates.

5. The cognitive impairment assisted screening method based on a frozen large language model according to claim 4, characterized in that, The S400 specifically includes the following steps: S410. Obtain the written numerical features and facial numerical features, and perform feature mapping processing on them respectively to map the numerical features of different modalities to a unified feature space. S420. Construct written numerical representations and facial numerical representations based on the mapped written numerical features and facial numerical features, respectively; S430. Associate the numerical representation with the overall semantic embedding and use it as the corresponding input of the multimodal fusion module.

6. The cognitive impairment assisted screening method based on a frozen large language model according to claim 5, characterized in that, The multimodal fusion module in S400 adopts a gated fusion method, which specifically includes the following steps: S440, The joint numerical representation formed by the written numerical representation and the facial numerical representation, as well as the overall semantic embedding, are used as inputs to the multimodal fusion module; S450. Determine the corresponding gating weights based on the information contribution relationship between different modal features, and adaptively weight the different modal features according to the gating weights; S460. Obtain the gated fusion representation based on the weighted modal features, and input the gated fusion representation as the multimodal fusion representation into the classification module.

7. The cognitive impairment assisted screening method based on a frozen large language model according to claim 5, characterized in that, The multimodal fusion module in S400 adopts a cross-modal Transformer fusion method, which specifically includes the following steps: S470. Map the written numerical representation, facial numerical representation, and overall semantic embedding to a unified latent space to construct corresponding modal tags, which are represented as follows: in, , and These represent written modal tags, facial modal tags, and semantic modal tags, respectively. , and They represent the linear mapping parameters, respectively. , and These represent the corresponding bias terms; This indicates the written representation of numerical values; This represents facial numerical values; Indicates overall semantic embedding; S480. The writing modality markers, facial modality markers, and semantic modality markers are arranged into a marker sequence according to a preset order: The labeled sequence is then input into a cross-modal Transformer coding layer for multi-head self-attention encoding to obtain an updated labeled sequence: S490. Perform mean pooling or attention pooling on the updated labeled sequence to obtain a cross-modal fusion representation: in, This represents a tag sequence consisting of three modal tags; This represents the updated label sequence; This represents cross-modal fusion representation; This indicates mean pooling or attention pooling operations.

8. The cognitive impairment assisted screening method based on a frozen large language model according to claim 6 or 7, characterized in that, Step S500 includes the following steps: S510. Input the multimodal fusion representation into the classification module, wherein the multimodal fusion representation is a gated fusion representation or a cross-modal Transformer fusion representation; S520. Calculate the classification output based on the multimodal fusion representation, which is expressed as: in, This represents a multimodal fusion representation; Indicates the classifier weight parameters; This represents the classifier bias term; Indicates the output value for the category; S530, The classification output value conduct The calculation yields the predicted probability that the target object belongs to each cognitive state category: in, This represents the predicted probability vector for each cognitive state category; the cognitive state-assisted screening result corresponding to the target object is determined based on the predicted probabilities.

9. The cognitive impairment assisted screening method based on a frozen large language model according to claim 8, characterized in that, Step S500 further includes the following steps: S540. Construct training sample batches, obtain the predicted probability corresponding to each cognitive state category based on the output of the classification module, and calculate the model training loss based on the predicted probability and the true label. S550. Calculate the model training loss using the cross-entropy loss function or the category-weighted cross-entropy loss function, and update the parameters of the numerical feature representation construction branch, multimodal fusion module, and classification module in reverse based on the model training loss, while keeping the parameters of the large language model encoder frozen; wherein, the category-weighted cross-entropy loss function is expressed as: in, This indicates the number of samples included in the current training batch; Indicates the sample number in the training batch; Indicates the number of cognitive state categories; Indicates the first Does the sample belong to the ? The label indicating the class; The model represents the first The predicted probability of a class; Indicates the first The weight corresponding to the class.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 9.

Images