An intelligent evaluation method for comprehensive performance of examinees based on multi-modal fusion

Abstract

The application discloses a kind of based on multimodal fusion's examinee comprehensive performance intelligent evaluation method, comprising the following steps: generating standardized multimodal data;Examinee baseline response data to conventional question is constructed;Form continuous answering process;Collect the disturbance response data of examinee to the set of antagonistic problems;Baseline response data and disturbance response data are aligned in cross-modal time sequence, and difference characteristics are extracted;Tensor voting field fusion processing is carried out, and multimodal response offset representation is generated;Using cognitive response dynamics evaluation model, the cognitive stability of examinee, anti-interference ability, true understanding degree and decision consistency are parameter inversion, and ability state vector is obtained;Examinee comprehensive performance evaluation result is generated, and the application can be in conventional answering situation and disturbed answering situation to the multimodal response change of examinee is carried out collaborative analysis, improves the objectivity and accuracy of comprehensive performance evaluation.

Description

Technical Field

[0001] This invention relates to the field of intelligent assessment and human-computer interaction, and in particular to an intelligent assessment method for the comprehensive performance of test takers based on multimodal fusion. Background Technology

[0002] With the rapid development of online exams, remote interviews, professional qualification assessments, classroom formative assessments, and intelligent education platforms, the demand for automated and objective evaluation of test takers' overall performance is constantly increasing. Traditional test assessment methods mainly rely on manual grading, human interview observation, or scoring rules based on final answer results. When applied to scenarios such as large-scale exams, remote recruitment, and online training assessments, these methods typically suffer from low evaluation efficiency, high subjectivity, insufficient utilization of process information, and weak interpretability of results, thus failing to meet the requirements of current intelligent assessment scenarios for real-time performance, accuracy, and comprehensiveness.

[0003] To improve the automation of assessment, existing technologies have begun to incorporate speech recognition, text analysis, facial expression recognition, eye-tracking analysis, and behavior detection. These technologies collect speech data, text data, facial image data, eye-tracking data, and body behavior data from test takers during the question-and-answer process, and then classify or score this data using machine learning or deep learning models. Compared to traditional assessment methods that rely solely on the final answer, these technologies have expanded the assessment dimensions to some extent, allowing the system to focus not only on "whether the answer is correct" but also on "how well the test takers performed during the question-and-answer process." However, most existing technologies still remain at the level of direct recognition, simple concatenation, and static scoring of single-modal or multi-modal data, lacking systematic modeling of the cognitive changes experienced by test takers in both normal and disturbed action-based response scenarios. Summary of the Invention

[0004] One objective of this invention is to propose an intelligent assessment method for the comprehensive performance of test takers based on multimodal fusion. This invention constructs standardized multimodal data, baseline response data, and perturbation response data, introduces an adversarial question set and forms a continuous answering process, combines cross-modal temporal alignment and differential feature extraction mechanisms, utilizes tensor voting fields to fuse differential features, establishes a multimodal response offset representation, and performs parameter inversion on cognitive stability, anti-interference ability, true comprehension level, and decision consistency based on a cognitive response dynamics assessment model. This achieves a systematic assessment of the comprehensive abilities of test takers, possessing the advantages of comprehensive assessment dimensions, detailed process characterization, accurate anti-interference ability analysis, and strong objectivity of results.

[0005] According to an embodiment of the present invention, a method for intelligent assessment of test takers' comprehensive performance based on multimodal fusion includes the following steps: Obtain and preprocess the raw data of test takers during the answering process to generate standardized multimodal data; Baseline response data of test takers to routine questions were constructed based on standardized multimodal data; Construct a set of adversarial questions and insert it into the regular question process to form a continuous answering process; During the continuous response process, data on the test takers' perturbation responses to a set of adversarial questions are collected. Cross-modal temporal alignment of baseline response data and perturbation response data was performed to extract differential features characterizing changes in test takers' responses; Tensor voting field fusion processing is performed on the differential features, and the differential features are used as sparse voting points. Tensor voting and sparse point densification are then performed to generate a multimodal response shift representation. The multimodal response shift characterization is input into the cognitive response dynamics assessment model, and the parameters of the test taker’s cognitive stability, anti-interference ability, true comprehension level and decision consistency are inverted to obtain the ability state vector. The overall performance evaluation results of the test takers are generated based on the ability state vector.

[0006] Optionally, the raw data includes question text data, answer text data, speech data, facial image data, eye movement trajectory data, and body behavior data. The preprocessing includes word segmentation and vectorization encoding of question text data and answer text data, noise reduction, frame segmentation, and speech feature extraction of speech data, face detection, key point localization, and expression feature extraction of facial image data, gaze point extraction and trajectory smoothing of eye movement trajectory data, and posture estimation and action sequence extraction of body behavior data.

[0007] Optionally, the construction of the baseline response data specifically includes: Extract test taker text data, answer text data, voice data, facial image data, eye movement data, and body behavior data from standardized multimodal data for routine questions; Based on the presentation time, start time, and end time of each regular question, the question text data, answer text data, voice data, facial image data, eye movement trajectory data, and body behavior data are segmented at the question level to obtain the data segments corresponding to each regular question. Based on the data segments corresponding to each regular question, we construct the regular question input sequence, the corresponding answer text sequence, the voice response sequence, the facial expression sequence, the eye movement trajectory sequence, and the body behavior sequence, respectively. For the same regular question, the corresponding regular question input sequence, corresponding answer text sequence, voice response sequence, facial expression sequence, eye movement trajectory sequence, and body behavior sequence are correlated to generate a single question baseline response unit; The baseline response units for each question in all regular questions are sequentially summarized to construct baseline response data.

[0008] Optionally, the formation of the continuous response process specifically includes: Extract semantic information, logical relationship information, and answer target information of questions from the question text data and corresponding answer text data of regular questions. Based on the extracted semantic information, logical relationship information, and answer target information of the questions, semantic interference questions, information conflict questions, cognitive switching questions, and induced deviation questions are constructed. Semantic interference problems, information conflict problems, cognitive switching problems, and induced deviation problems are labeled with question numbers and question types to generate an adversarial problem set; Following the question numbering order in the regular question process, the set of adversarial questions is configured after the corresponding regular questions; The configured set of adversarial questions is written into the regular question process, forming a continuous answering process that includes both regular questions and adversarial questions.

[0009] Optionally, in the continuous answering process, after the adversarial questions in the adversarial question set are presented to the test taker, the time of presentation of the corresponding adversarial question is used as the starting node for data collection. The time interval for data collection is determined by the start and end times of the test taker's answer to the adversarial question. Within the time interval, the test taker's answer text data, voice data, facial image data, eye movement trajectory data, and body behavior data are collected simultaneously. The collected answer text data, voice data, facial image data, eye movement trajectory data, and body behavior data are bound according to the question number of the corresponding adversarial question and a correspondence is established with the content of the adversarial question to form perturbation response data that characterizes the test taker's response under the adversarial question conditions.

[0010] Optionally, the extraction of the differential features specifically includes: Cross-modal temporal alignment was performed on the baseline response data and the perturbation response data, and text semantic feature vectors, speech prosody feature vectors, facial expression feature vectors, eye movement distribution feature vectors, and body behavior feature vectors were extracted respectively. Calculate text semantic offset features based on text semantic feature vectors; Calculate the prosodic variation features based on the prosodic feature vector; Calculate facial expression fluctuation features based on facial expression feature vectors; Calculate eye movement distribution offset features based on eye movement distribution feature vectors; Calculate the characteristics of changes in limb behavior based on limb behavior feature vectors; By combining text semantic shift features, speech prosody change features, facial expression fluctuation features, eye movement distribution shift features, and body behavior change features, a differential feature vector corresponding to each question number is generated. The differential feature vectors corresponding to all question numbers are then summarized to form differential features.

[0011] Optionally, the generation of the multimodal response shift characterization specifically includes: Tensor voting field fusion processing is performed on the differential features, and the differential features are organized according to the question number to generate sparse voting points corresponding to each question number. Tensor encoding is performed on each sparse voting point to generate tensor voting data types corresponding to each question number; Perform tensor voting processing on the tensor voting data type corresponding to each question number to obtain the cumulative tensor voting result corresponding to each question number; Based on the degree of difference between the sparse voting points corresponding to each question number, calculate the voting weight between the tensor voting data types corresponding to each question number, and update the tensor voting cumulative results based on the voting weight; Based on the updated tensor voting accumulation results, the sparse voting points corresponding to each question number are densified to generate a local dense tensor field corresponding to each question number. The local dense tensor fields corresponding to each problem number are fused and migrated to obtain the fused migration field, and a multimodal response migration characterization is generated.

[0012] Optionally, obtaining the capability state vector specifically includes: The multimodal response offset representation sequence is input into the cognitive response dynamics evaluation model, and the state recursion calculation is performed according to the question number order to obtain the cognitive state vector corresponding to each question number. The cognitive stability parameters are calculated based on the cognitive state vector corresponding to each question number, and the cognitive stability parameters corresponding to each question number are obtained. The anti-interference ability parameters are calculated based on the cognitive state vector corresponding to each question number, and the anti-interference ability parameters corresponding to each question number are obtained. The true comprehension level parameter is calculated based on the cognitive state vector corresponding to each question number. The decision consistency parameters are calculated based on the cognitive state vector corresponding to each question number, and the decision consistency parameters corresponding to each question number are obtained. A capability state vector is constructed based on cognitive stability parameters, anti-interference ability parameters, true comprehension level parameters, and decision consistency parameters.

[0013] Optionally, the cognitive stability parameter, anti-interference ability parameter, true comprehension level parameter, and decision consistency parameter in the ability state vector are used as the evaluation basis for the corresponding sub-ability. The parameters are numerically mapped according to a unified scoring range to obtain the cognitive stability assessment result, anti-interference ability assessment result, true comprehension level assessment result, and decision consistency assessment result, forming the sub-ability assessment result. The cognitive stability parameter, anti-interference ability parameter, true comprehension level parameter, and decision consistency parameter are weighted and summarized to obtain the comprehensive score result. Based on the dispersion of the cognitive stability parameter, anti-interference ability parameter, true comprehension level parameter, and decision consistency parameter corresponding to each question number, the fluctuation range of each parameter across all question numbers is statistically calculated, and the stability of the comprehensive score result is measured according to the fluctuation range to obtain the confidence level of the result, forming the comprehensive performance assessment result of the examinee.

[0014] The beneficial effects of this invention are: This invention does not rely solely on static scoring based on the examinee's final answers, nor does it simply concatenate voice, text, facial expression, or behavioral data to directly output evaluation results. Instead, it constructs a complete technical chain around the examinee's actual response process during the question-answering process: "standardized multimodal data—baseline response data—perturbation response data—difference features—multimodal response shift representation—ability state vector—comprehensive performance evaluation result." Through this technical chain, this invention can first generate the examinee's baseline response data in a normal question-answering context, and then generate perturbation response data after the introduction of adversarial questions. This allows the evaluation process to be based on comparability of "the same examinee, the same question association, and different answering contexts," effectively improving the data organization of the evaluation process and the reliability of subsequent analysis results.

[0015] This invention introduces an adversarial question set into the conventional question process, enabling the assessment system to move beyond passively recording test-takers' routine responses. Instead, it proactively introduces perturbation factors such as semantic interference, information conflict, cognitive switching, and induced biases into the continuous answering process, thereby collecting data on the test-taker's perturbation response under these conditions. Based on the correspondence between baseline response data and perturbation response data, this invention can more accurately identify changes in the test-taker's textual semantics, speech rhythm, facial expressions, eye movement distribution, and body language, effectively distinguishing between the test-taker's surface-level responses and their true cognitive state. Therefore, this invention not only reflects whether the test-taker answered correctly but also whether they maintained stable understanding and decision-making under perturbation, making the overall performance assessment results closer to the test-taker's true ability level.

[0016] This invention establishes a correspondence between baseline response data and perturbation response data across modal time series alignment, enabling textual, speech, facial image, eye-tracking, and body behavior data to be correlated under a unified time reference. This solves the problems of different sampling frequencies, temporal granularities, and difficulty in direct comparison of data from different modalities in existing technologies. The differential features extracted based on this cross-modal time series alignment mechanism can accurately characterize the multidimensional response changes of test takers under adversarial questions, providing a unified, computable, and clearly correlated data foundation for subsequent fusion processing, thereby improving the accuracy and stability of the differential analysis results.

[0017] This invention does not employ common simple weighting, direct concatenation, or ordinary attention mechanisms to fuse differential features. Instead, it introduces a tensor voting field fusion mechanism, treating differential features as sparse voting points for tensor encoding, tensor voting, and sparse point densification, ultimately generating a multimodal response shift representation. Through this approach, the invention can simultaneously maintain the directionality, saliency, and structural continuity of differential features in the feature space. This allows the multimodal response shift representation to reflect not only local changes in a single modality but also the overall shift trend and structural consistency among multiple modalities. Compared to the simple vector-level fusion methods in existing technologies, the multimodal response shift representation formed by this invention has stronger structural expressive power and higher noise resistance, which is beneficial for the accurate subsequent cognitive state modeling.

[0018] This invention utilizes a cognitive response dynamics assessment model to perform state recursion and parameter inversion on the representation of multimodal response shifts, obtaining an ability state vector that characterizes cognitive stability, resistance to interference, true comprehension, and decision consistency. In other words, this invention goes beyond simply detecting multimodal behavioral changes; it maps these changes to quantifiable results reflecting the examinee's internal ability state, achieving a progressive inference from surface-level performance to deep cognitive parameters. This approach enables a more comprehensive evaluation of the examinee's stability, depth of comprehension, and decision-making quality during continuous responses, improving the interpretability and discriminative power of the overall performance assessment results.

[0019] This invention enables collaborative analysis of test takers' routine and disrupted responses within a unified data organization framework. It accurately characterizes multimodal response changes through cross-modal temporal alignment and tensor voting field fusion processing. Based on a cognitive response dynamics assessment model, it can inversely derive ability state vectors with clear ability meanings, ultimately generating a comprehensive performance assessment result that includes sub-ability assessment results, a comprehensive score, and result confidence levels. Therefore, this invention effectively overcomes the problems of existing technologies, such as single assessment dimensions, insufficient process utilization, inability to identify anti-interference capabilities, difficulty in characterizing true understanding, and insufficient objectivity of results. It offers the advantages of a complete assessment process, objective assessment results, detailed ability characterization, and strong application adaptability. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of an intelligent assessment method for comprehensive test performance based on multimodal fusion proposed in this invention. Figure 2 This is a schematic diagram illustrating the construction of differential features in an intelligent assessment method for comprehensive test performance based on multimodal fusion proposed in this invention. Figure 3 This is a schematic diagram illustrating the construction of a multimodal response shift characterization for an intelligent assessment method of test takers' comprehensive performance based on multimodal fusion proposed in this invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0022] refer to Figures 1-3 A multimodal fusion-based intelligent assessment method for comprehensive test taker performance includes the following steps: Obtain and preprocess the raw data of test takers during the answering process to generate standardized multimodal data; Baseline response data of test takers to routine questions were constructed based on standardized multimodal data; Construct a set of adversarial questions and insert it into the regular question process to form a continuous answering process; During the continuous response process, data on the test takers' perturbation responses to a set of adversarial questions are collected. Cross-modal temporal alignment of baseline response data and perturbation response data was performed to extract differential features characterizing changes in test takers' responses; Tensor voting field fusion processing is performed on the differential features, and the differential features are used as sparse voting points. Tensor voting and sparse point densification are then performed to generate a multimodal response shift representation. The multimodal response shift characterization is input into the cognitive response dynamics assessment model, and the parameters of the test taker’s cognitive stability, anti-interference ability, true comprehension level and decision consistency are inverted to obtain the ability state vector. The overall performance evaluation results of the test takers are generated based on the ability state vector.

[0023] In this embodiment, the raw data includes question text data, answer text data, speech data, facial image data, eye movement trajectory data, and body behavior data. The preprocessing includes word segmentation and vectorization encoding of question text data and answer text data, noise reduction, frame segmentation, and speech feature extraction of speech data, face detection, key point localization, and expression feature extraction of facial image data, gaze point extraction and trajectory smoothing of eye movement trajectory data, and pose estimation and action sequence extraction of body behavior data.

[0024] In this embodiment, the construction of baseline response data specifically includes: Extract test taker text data, answer text data, voice data, facial image data, eye movement data, and body behavior data from standardized multimodal data for routine questions; Based on the presentation time, start time, and end time of each regular question, the question text data, answer text data, voice data, facial image data, eye movement trajectory data, and body behavior data are segmented at the question level to obtain the data segments corresponding to each regular question. The question-level segmentation process is as follows: the presentation time of each regular question is taken as the starting time point of the regular question. The answering time interval of the regular question is determined by the corresponding answering start time and answering end time. According to the answering time interval, the data content corresponding to the regular question is extracted from the question text data, answer text data, voice data, facial image data, eye movement trajectory data, and body behavior data. The question text data is extracted according to the question number to extract the corresponding question content. The answer text data is extracted according to the answering time interval to extract the corresponding text segment. The voice data is extracted according to the answering time interval to extract the corresponding voice segment. The facial image data is extracted according to the answering time interval to extract the corresponding image frame segment. The eye movement trajectory data is extracted according to the answering time interval to extract the corresponding gaze trajectory segment. The body behavior data is extracted according to the answering time interval to extract the corresponding action segment, forming a data segment that corresponds one-to-one with a single regular question. Based on the data segments corresponding to each regular question, we construct the regular question input sequence, the corresponding answer text sequence, the voice response sequence, the facial expression sequence, the eye movement trajectory sequence, and the body behavior sequence, respectively. The construction process is as follows: the data fragments corresponding to each regular question are serialized and organized according to the data type; the question text data corresponding to the regular questions are constructed into a regular question input sequence according to the question content and presentation order; the answer text data corresponding to the regular questions are constructed into a corresponding answer text sequence according to the generation order of the answer content; the voice data corresponding to the regular questions are constructed into a voice response sequence according to the time sequence; the facial image data corresponding to the regular questions are constructed into a facial expression sequence according to the time sequence of the image frames; the eye movement trajectory data corresponding to the regular questions are constructed into an eye movement trajectory sequence according to the time sequence of the fixation point or trajectory point; and the body behavior data corresponding to the regular questions are constructed into a body behavior sequence according to the time sequence of the action. For the same regular question, the corresponding regular question input sequence, corresponding answer text sequence, voice response sequence, facial expression sequence, eye movement trajectory sequence, and body behavior sequence are correlated to generate a single question baseline response unit; The single-question baseline response unit generation process is as follows: the input sequence of the regular question, the corresponding answer text sequence, the voice response sequence, the facial expression sequence, the eye movement trajectory sequence, and the body behavior sequence corresponding to the same regular question are uniformly bound according to the same question number and the same answer time interval. Taking a single regular question as the associated object, the above sequences are integrated to form a combined data unit containing regular question input information, text answer information, voice answer information, facial expression information, eye movement behavior information, and body behavior information, thereby generating a single-question baseline response unit that represents the test taker's complete baseline response to the regular question. The baseline response units for each question in all regular questions are sequentially summarized to construct baseline response data.

[0025] In this embodiment, the formation of the continuous response process specifically includes: Extract semantic information, logical relationship information, and answer target information of questions from the question text data and corresponding answer text data of regular questions. The extraction process is as follows: First, read the text data of regular questions, perform word segmentation, syntactic analysis, and semantic encoding on the text data, and identify keywords, limiting words, conditional words, relational words, and target words in the questions to obtain semantic information representing the core content of the questions. Second, based on the sentence structure, conditional constraints, causal relationships, adversative relationships, parallel relationships, progressive relationships, and comparative relationships in the question text data, establish the association relationships between various semantic units within the questions to obtain logical relationship information representing the reasoning path and structural relationships of the questions. Third, read the answer text data corresponding to the regular questions, perform word segmentation, syntactic analysis, and semantic encoding on the answer text data, and perform semantic matching between the answer text data and the question text data to identify the answer content, reasoning content, and conclusion content corresponding to the requirements of the questions in the answer text data. Fourth, based on the task instructions, question format, constraints, and scoring points in the question text data, combined with the actual answer content and reasoning content covered in the answer text data, determine the answering tasks, answer direction, and output content required by the regular questions for the test takers to complete, and obtain the question answering target information. Based on the extracted semantic information, logical relationship information, and answer target information of the questions, semantic interference questions, information conflict questions, cognitive switching questions, and induced deviation questions are constructed. The construction process is as follows: First, using the semantic information of the question as the basis for reorganizing the question content, keywords, limiting words, conditional words, relational words, and target words in the question are replaced, deleted, rearranged, or added to generate semantically disruptive questions that deviate from the original question's semantic direction but maintain a superficial connection. Second, using the logical relationship information of the question as the basis for constructing the logical structure, causal relationships, adversative relationships, parallel relationships, progressive relationships, comparative relationships, and conditional constraints in the original question are reversed, crossed, or misplaced to generate information-conflicting questions that are inconsistent with the original question's logical structure. Third, using the target information for answering the question as... Based on task switching, the original question's required answering task is switched from one type of task (fact identification, relationship judgment, causal analysis, conditional reasoning, viewpoint induction, or conclusion generation) to another type of task, generating cognitive switching questions that require test takers to change their original thinking paths and processing methods. Using a combination of semantic information, logical relationship information, and answering target information as the basis for generating inductive content, while maintaining the relevance of the question's theme, directional information, additional conditional information, or alternative judgment paths that differ from the original question's target direction are introduced to generate inductive deviation questions that guide test takers away from the original answering target. Semantic interference problems, information conflict problems, cognitive switching problems, and induced deviation problems are labeled with question numbers and question types to generate an adversarial problem set; The process of generating the adversarial question set is as follows: semantic interference questions, information conflict questions, cognitive switching questions, and induced deviation questions are associated one by one with the question numbers of their corresponding regular questions, and each adversarial question is assigned the same question number as its corresponding regular question. At the same time, each adversarial question is marked as a semantic interference question, information conflict question, cognitive switching question, or induced deviation question according to its construction method, thus forming an adversarial question set. Following the question numbering order in the regular question process, the set of adversarial questions is configured after the corresponding regular questions; The configured set of adversarial questions is written into the regular question process, forming a continuous answering process that includes both regular questions and adversarial questions.

[0026] In this embodiment, during the continuous answering process, after the adversarial questions in the adversarial question set are presented to the test taker, the time when the corresponding adversarial question is presented is used as the data collection start node. The data collection time interval is determined by the start and end times of the test taker's answer to the adversarial question. Within the data collection time interval, the test taker's answer text data, voice data, facial image data, eye movement trajectory data, and body behavior data are collected simultaneously. The collected answer text data, voice data, facial image data, eye movement trajectory data, and body behavior data are bound according to the question number of the corresponding adversarial question and a correspondence is established with the content of the adversarial question to form perturbation response data that characterizes the test taker's response under the adversarial question conditions.

[0027] In this embodiment, the extraction of differential features specifically includes: Cross-modal temporal alignment was performed on the baseline response data and the perturbation response data, and text semantic feature vectors, speech prosody feature vectors, facial expression feature vectors, eye movement distribution feature vectors, and body behavior feature vectors were extracted respectively. The extraction process of text semantic feature vectors, speech prosody feature vectors, facial expression feature vectors, eye movement distribution feature vectors, and body behavior feature vectors specifically involves: performing cross-modal temporal alignment on the response text data, speech data, facial image data, eye movement trajectory data, and body behavior data in the baseline response data and perturbation response data. Cross-modal temporal alignment includes reading the question number, question presentation time, response start time, and response end time, using the response time interval corresponding to the same question number as a unified time reference, and performing time localization and matching on the response text data, speech data, facial image data, eye movement trajectory data, and body behavior data according to their respective acquisition times. This completes the temporal alignment... The system performs word segmentation, semantic encoding, and vectorization on the response text data to extract semantic feature vectors. It also performs frame segmentation, fundamental frequency extraction, energy extraction, speech rate extraction, and pause duration extraction on the temporally aligned speech data to extract prosodic feature vectors. Furthermore, it performs face region localization, facial expression key point localization, and facial expression state encoding on the temporally aligned facial image data to extract facial expression feature vectors. Finally, it performs gaze point recognition, saccade trajectory statistics, and gaze region distribution encoding on the temporally aligned eye movement trajectory data to extract eye movement distribution feature vectors. Finally, it performs posture key point recognition, movement amplitude statistics, and movement frequency encoding on the temporally aligned limb behavior data to extract limb behavior feature vectors. The text semantic offset feature is calculated based on the text semantic feature vector. The specific calculation process of the text semantic offset feature is as follows: calculate the inner product between the text semantic feature vector corresponding to the baseline response data and the text semantic feature vector corresponding to the perturbation response data, then calculate the magnitude of the text semantic feature vector corresponding to the baseline response data and the magnitude of the text semantic feature vector corresponding to the perturbation response data respectively, divide the inner product by the product of the magnitudes, and finally subtract the calculation result from 1 to obtain the text semantic offset feature. The speech prosodic variation features are calculated based on the speech prosodic feature vector. The specific calculation process of the speech prosodic variation features is as follows: take the absolute value of the difference between the corresponding components of each dimension in the speech prosodic feature vector, and sum the absolute values ​​of all dimensions and divide by the number of dimensions of the speech prosodic feature vector to obtain the speech prosodic variation features. The facial expression fluctuation feature is calculated based on the facial expression feature vector. The specific calculation process of the facial expression fluctuation feature is as follows: take the absolute value of the difference between the corresponding components of each dimension in the facial expression feature vector, and sum the absolute values ​​of all dimensions and divide by the number of dimensions of the facial expression feature vector to obtain the facial expression fluctuation feature. The eye movement distribution offset feature is calculated based on the eye movement distribution feature vector. The specific calculation process of the eye movement distribution offset feature is as follows: take the absolute value of the difference between the corresponding components of each dimension in the eye movement distribution feature vector, and sum the absolute values ​​of all dimensions and divide by the number of dimensions of the eye movement distribution feature vector to obtain the eye movement distribution offset feature. The calculation process of the limb behavior change features is as follows: take the absolute value of the difference between the corresponding components of each dimension in the limb behavior feature vector, and sum the absolute values ​​of all dimensions and divide by the number of dimensions of the limb behavior feature vector to obtain the limb behavior change features. By combining text semantic shift features, speech prosody change features, facial expression fluctuation features, eye movement distribution shift features, and body behavior change features, a differential feature vector corresponding to each question number is generated. The differential feature vectors corresponding to all question numbers are then summarized to form differential features.

[0028] In this embodiment, the generation of the multimodal response shift characterization specifically includes: Tensor voting field fusion processing is performed on the differential features, and the differential features are organized according to the question number to generate sparse voting points corresponding to each question number. The sparse voting point generation process is as follows: The text semantic shift features, speech prosody variation features, facial expression fluctuation features, eye movement distribution shift features, and body behavior change features corresponding to the same question number are combined from the difference features. The combined difference feature values ​​are then arranged in a unified feature dimension order to form the difference feature coordinates corresponding to the question number. The position of the difference feature coordinates in the feature space is used as the point position, and the combined result of the text semantic shift features, speech prosody variation features, facial expression fluctuation features, eye movement distribution shift features, and body behavior change features corresponding to the question number is used as the point attribute to construct the feature point corresponding to the question number. All feature points corresponding to all question numbers are then summarized to generate sparse voting points distributed in the feature space. Tensor encoding is performed on each sparse voting point to generate tensor voting data types corresponding to each question number; The generation of the tensor voting data type is as follows: taking the position of each sparse voting point as the tensor encoding center, calculating the directional components of the sparse voting point in each feature dimension, and constructing directional information representing the main directional information based on each directional component; determining the uncertainty information of the sparse voting point based on the distribution dispersion and numerical variation of each difference feature, and combining the uncertainty information with the directional information to generate the tensor encoding result of the corresponding sparse voting point; binding the tensor encoding result with the corresponding question number to obtain the tensor voting data type corresponding to each question number; Perform tensor voting processing on the tensor voting data type corresponding to each question number to obtain the cumulative tensor voting result corresponding to each question number; The tensor voting process is as follows: Read the tensor voting data type corresponding to each question number, and use each tensor voting data type as a voting source in the feature space; Using the location of each tensor voting data type as the center, determine the voting propagation range and path for the locations corresponding to the remaining question numbers according to the distribution relationship of the difference features corresponding to the question numbers in the feature space, and transmit the directional information and uncertainty information contained in the tensor voting data type to the surrounding locations along the voting propagation path; Superimpose and accumulate the multiple voting results received at each receiving location to form the cumulative tensor result corresponding to the receiving location; Use all the cumulative tensor results received at the locations corresponding to each question number as the tensor voting cumulative result corresponding to each question number. Based on the degree of difference between the sparse voting points corresponding to each question number, calculate the voting weight between the tensor voting data types corresponding to each question number, and update the tensor voting cumulative results based on the voting weight; The update process is as follows: Read the degree of difference between the sparse voting points corresponding to each question number, and calculate the voting weight between any two question numbers based on the degree of difference between the sparse voting points corresponding to any two question numbers in the feature space. Iterate through the tensor voting accumulation results corresponding to each question number, extract the received voting contributions from the tensor voting data types corresponding to other question numbers, and multiply each voting contribution by its corresponding voting weight to obtain the weighted voting contribution. Then, accumulate and merge all the weighted voting contributions corresponding to the same question number to generate the updated tensor voting accumulation result corresponding to the question number. Based on the updated tensor voting accumulation results, the sparse voting points corresponding to each question number are densified to generate a local dense tensor field corresponding to each question number. The densification process is as follows: the updated tensor voting accumulation results are mapped to the feature space locations of the corresponding sparse voting points; the feature space location of each sparse voting point is taken as the center location, and tensor propagation and tensor interpolation are performed on the neighborhood space around the center location according to the directional information, saliency information and uncertainty information contained in the updated tensor voting accumulation results; the tensor information received at each location within the neighborhood is continuously accumulated and locally smoothed; and the tensor information continuously distributed within the center location and neighborhood of each question number is taken as the local dense tensor field corresponding to the question number. The local dense tensor fields corresponding to each question number are fused and migrated to obtain the fused migration field, and a multimodal response migration characterization is generated. The multimodal response migration characterization generation process is as follows: extract the direction information, uncertainty information, and spatial distribution information from each local dense tensor field; perform fusion migration calculation on the local dense tensor fields corresponding to each question number, and uniformly map and accumulate the direction migration and uncertainty migration of the local dense tensor fields corresponding to different question numbers in the feature space to obtain the fusion migration field of the overall migration trend of the differences in features corresponding to all question numbers; extract the overall migration direction, local migration intensity, structural continuity, and migration concentration based on the fusion migration field, and combine and encode the overall migration direction, local migration intensity, structural continuity, and migration concentration to form a multimodal response migration characterization.

[0029] In this embodiment, obtaining the capability state vector specifically includes: The multimodal response offset representation sequence is input into the cognitive response dynamics evaluation model, and the state recursion calculation is performed according to the question number order to obtain the cognitive state vector corresponding to each question number. The process of obtaining the cognitive state vector is as follows: the multimodal response offset representation sequence arranged in order of question number is input into the cognitive response dynamics evaluation model item by item. During the state recursion calculation, the multimodal response offset representation corresponding to the first question number is input into the state initialization unit of the cognitive response dynamics evaluation model to generate the initial cognitive state vector. For each subsequent question number, the multimodal response offset representation corresponding to the current question number and the cognitive state vector corresponding to the previous question number are jointly input into the state update unit of the cognitive response dynamics evaluation model. The state update unit performs joint mapping, weighted update and nonlinear transformation on the historical state information in the cognitive state vector corresponding to the previous question number and the current offset information in the multimodal response offset representation corresponding to the current question number, and outputs the cognitive state vector corresponding to the current question number. The cognitive stability parameters are calculated based on the cognitive state vector corresponding to each question number, and the cognitive stability parameters corresponding to each question number are obtained. The specific process for calculating the cognitive stability parameter is as follows: extract the magnitude and direction of change of the cognitive state vector in the continuous sequence of question numbers; calculate the difference between the cognitive state vectors corresponding to adjacent question numbers to obtain the result of cognitive state change, and perform cumulative statistics on the result of cognitive state change to generate the degree of state fluctuation corresponding to each question number; associate the degree of state fluctuation with the cognitive state vector of the corresponding question number, perform reverse mapping on the degree of state fluctuation, and obtain the cognitive stability parameter corresponding to each question number. The anti-interference ability parameters are calculated based on the cognitive state vector corresponding to each question number, and the anti-interference ability parameters corresponding to each question number are obtained. The specific process for calculating the anti-interference capability parameter is as follows: Read the cognitive state vector corresponding to each question number, and mark the question number according to the insertion position of the adversarial question. The question number containing the adversarial question is taken as the interference range. Extract the cognitive state vector corresponding to each question number within the interference range and compare it with the cognitive state vectors corresponding to adjacent regular questions to obtain the cognitive state change results. Perform statistical processing on the cognitive state change results to obtain the anti-interference change amplitude corresponding to each question number, and associate the anti-interference change amplitude with the cognitive state vector of the corresponding question number to generate the anti-interference capability parameter corresponding to each question number. The true comprehension level parameter is calculated based on the cognitive state vector corresponding to each question number. The specific process for calculating the true comprehension level parameter is as follows: Based on the question type labels of regular questions and adversarial questions with question numbers, the response consistency of the cognitive state vector under different question type conditions is extracted; the cognitive state vectors of regular questions and corresponding adversarial questions under the same question semantic goal are matched to calculate the consistency between the cognitive state vectors, and the semantic consistency result corresponding to each question number is obtained; the semantic consistency result is normalized and associated with the cognitive state vector of the corresponding question number to generate the true comprehension level parameter corresponding to each question number; The decision consistency parameters are calculated based on the cognitive state vector corresponding to each question number, and the decision consistency parameters corresponding to each question number are obtained. The specific process for calculating the decision consistency parameter is as follows: extract the cognitive state vector change relationship between consecutive question numbers according to the question number order; group question numbers with the same answering goal, match the cognitive state vectors corresponding to each question number within the same group, calculate the degree of consistency in direction and the degree of consistency in change trend between each cognitive state vector, and obtain the decision consistency result corresponding to each question number; normalize the decision consistency result and associate it with the cognitive state vector of the corresponding question number to generate the decision consistency parameter corresponding to each question number; A capability state vector is constructed based on cognitive stability parameters, anti-interference ability parameters, true comprehension level parameters, and decision consistency parameters.

[0030] In this implementation, the cognitive stability parameter, anti-interference ability parameter, true comprehension level parameter, and decision consistency parameter in the ability state vector are used as the evaluation basis for the corresponding sub-abilities. Each parameter is numerically mapped according to a unified scoring range to obtain the cognitive stability assessment result, anti-interference ability assessment result, true comprehension level assessment result, and decision consistency assessment result, forming the sub-ability assessment result. The cognitive stability parameter, anti-interference ability parameter, true comprehension level parameter, and decision consistency parameter are weighted and summarized to obtain the comprehensive score result. Based on the dispersion of the cognitive stability parameter, anti-interference ability parameter, true comprehension level parameter, and decision consistency parameter corresponding to each question number, the fluctuation range of each parameter across all question numbers is statistically calculated. The stability of the comprehensive score result is measured based on the fluctuation range to obtain the result confidence level, forming the comprehensive performance evaluation result of the examinee.

[0031] Example 1: This example illustrates an online structured interview assessment scenario for a professional qualification. In this scenario, the participants are applicants for a professional qualification certification. The assessment platform simultaneously collects the applicant's written answers, voice data, facial image data, eye movement trajectory data, and body language data during the answering process using a camera, microphone, text input terminal, eye-tracking module, and posture acquisition module. Traditional online interview assessments typically rely on whether the answer is relevant, the fluency of the language, and the examiner's subjective impression as primary criteria. While these methods can evaluate the applicant's expressive ability to some extent, they lack effective means to identify key issues such as whether the applicant truly understands the question, whether the applicant can maintain stable judgment after being disturbed, and whether the applicant's expression is superficially fluent but actually flawed. This invention is applied in this scenario, focusing on solving the technical problems of existing technologies that rely solely on static answer results, lack adversarial question perturbation mechanisms, lack multimodal difference analysis, and are difficult to quantify cognitive stability.

[0032] The system first acquires raw test data after the test taker enters the answer interface. This raw data includes question text data, answer text data, speech data, facial image data, eye-tracking data, and body behavior data. The platform preprocesses the raw test data to generate standardized multimodal data. For question and answer text data, the system performs word segmentation, syntactic analysis, and semantic encoding; for speech data, it performs noise reduction, frame segmentation, fundamental frequency extraction, energy extraction, and pause detection; for facial image data, it performs face region localization, key point recognition, and facial expression encoding; for eye-tracking data, it performs gaze point recognition, saccade path organization, and distribution mapping; and for body behavior data, it performs posture key point extraction, movement amplitude statistics, and movement frequency encoding. After preprocessing, all modal data are uniformly mapped to the same time reference, forming standardized multimodal data suitable for subsequent analysis.

[0033] After standardized multimodal data is generated, the system first constructs baseline response data around routine questions. For example, under a routine question such as "Please analyze the personnel coordination mechanism in a certain emergency," the system records the candidate's input content, written response, verbal response, facial expression changes, eye movement changes, and body language changes during normal answering. In this way, the system does not simply save "what the candidate said," but organizes the entire response process corresponding to that question into a complete baseline response unit. After performing the same processing on multiple routine questions throughout the interview, baseline response data for the candidate in the routine question process can be generated.

[0034] The system constructs an adversarial question set based on the semantic information, logical relationships, and answer objectives of regular questions. This adversarial question set is then inserted into the regular question flow to form a continuous answering process. Taking this embodiment as an example, after the test-taker answers the regular question "How to coordinate resources," the system inserts semantic interference questions, such as replacing key conditions in the question with synonymous but different conditions; information conflict questions, such as adding contradictory limiting information to the original logical relationships; cognitive switching questions, such as switching a question requiring factual analysis to one requiring reverse reasoning; and deflection-inducing questions, such as adding additional judgment paths that easily lead the test-taker away from the core conclusion while maintaining relevance to the topic. With this setup, the test-taker is not simply answering questions on a familiar, normal answering track, but rather entering a disturbed environment containing interference, conflict, switching, and inducement, thereby exposing a more authentic cognitive state.

[0035] After presenting the adversarial question set, the system continues to collect data on the test takers' perturbation responses to the set. This time, the collected data still includes written responses, speech, facial images, eye movements, and body language; however, this data no longer represents performance under normal conditions but rather performance under perturbed conditions. Since the system has already constructed baseline response data under normal questions, the perturbed response data collected here can be strictly compared with the previous baseline response data.

[0036] After collecting baseline and perturbation response data, the system performs cross-modal temporal alignment and extracts differential features. These differential features are not simply scores, but include at least text semantic shift features, speech prosody variation features, facial expression fluctuation features, eye movement distribution shift features, and body behavior variation features. For example, in the text dimension, the system can identify whether the test taker's retention of core concepts has decreased; in the speech dimension, the system can identify whether pause duration, speech rate fluctuations, and volume changes have significantly increased; in the facial expression dimension, the system can identify whether facial states such as tension, hesitation, and avoidance have increased; in the eye movement dimension, the system can identify whether there is significant drift and repeated jumping of the gaze point; and in the body behavior dimension, the system can identify whether there are significant changes in movement frequency, movement amplitude, and postural stability.

[0037] This embodiment does not use ordinary weighted averaging or simple concatenation methods to process differential features. Instead, it employs a tensor voting field to fuse the differential features. The system organizes the differential features according to the question number, generating sparse voting points corresponding to each question number. Then, it performs tensor encoding on each sparse voting point to form a corresponding tensor voting data type. Next, the system performs tensor voting processing and voting weight update processing on the tensor voting data type, and then densifies the sparse voting points to generate a locally dense tensor field, ultimately obtaining the fused offset field. The multimodal response offset representation generated by the fused offset field can not only reflect the variation amplitude of a single mode, but also the structural consistency and overall offset trend between different modes. In this way, the system's understanding of "change" is no longer discrete and isolated, but continuous and structured.

[0038] The system establishes a cognitive response dynamics assessment model based on multimodal response shift representation. It recursively calculates the cognitive state of test-takers in a continuous sequence of questions, obtaining a cognitive state vector. Further, it calculates parameters such as cognitive stability, anti-interference ability, true comprehension, and decision consistency. Finally, the system constructs an ability state vector based on these parameters and generates a comprehensive performance evaluation result. This comprehensive performance evaluation result includes sub-ability assessment results, a comprehensive score, and corresponding result confidence levels. For examiners, this result not only provides information on "how many points the test-taker scored," but also provides conclusions on "in which dimensions the test-taker performed consistently," "whether the test-taker is prone to deviating from the core questions under distracting conditions," and "whether the test-taker truly understood the question requirements."

[0039] To verify the effectiveness of this invention, 120 candidates were selected for testing in the same online structured interview scenario for professional qualification. 60 candidates were evaluated using the traditional multimodal static scoring method, and the remaining 60 were evaluated using the method of this invention. After the evaluation, five senior reviewers with over three years of interviewing experience manually reviewed all candidates' work. The results of this manual review were used as a reference standard to compare the performance of the two methods in terms of authenticity identification, anti-interference ability judgment, consistency of comprehensive scoring, and stability of result confidence. The test results are shown in Table 1.

[0040] Table 1. Comparison of the effectiveness of different assessment methods in online structured interview scenarios for professional qualifications.

[0041] As shown in Table 1, this invention significantly outperforms traditional multimodal static scoring methods in all key indicators. Regarding the consistency rate with the overall conclusions of manual review, this invention achieves 91.7%, a 13.4 percentage point increase compared to the traditional method's 78.3%, indicating that the overall performance evaluation results generated by this invention are more consistent with the final judgments of senior reviewers. This demonstrates that this invention does not simply output a model score, but rather portrays the candidate's performance more closely to the actual review logic.

[0042] In terms of accuracy rates for identifying true comprehension, resistance to interference, decision consistency, and cognitive stability, this invention achieves over 86%, significantly higher than the approximately 70% level of traditional methods. Among these, the accuracy rate for identifying resistance to interference increased most significantly, from 68.4% to 88.6%. This indicates that after the introduction of adversarial questions, by comparing baseline response data with perturbation response data, this invention can more effectively distinguish between "stable comprehension" and "perturbed collapse," a feat difficult to achieve with traditional methods that only consider conventional response results.

[0043] Regarding the misjudgment rate under highly distracting questions, this invention achieves a misjudgment rate of only 8.4%, significantly lower than the 21.6% of traditional methods. This indicates that this invention is more accurate in data organization, temporal alignment, and differential feature extraction under disturbed scenarios, effectively reducing evaluation bias caused by high-pressure, conflicting, and leading questions. Particularly noteworthy is the "recognition rate of samples that appear fluent but have poor comprehension," where this invention reaches 84.1%, compared to only 54.8% for traditional methods. This demonstrates that this invention can effectively identify test-takers who appear fluent, speak at a stable pace, and outwardly perform well, but actually suffer from insufficient depth of comprehension and inadequate logical retention.

[0044] Regarding the correlation coefficient between the overall score and manual review, this invention achieves 0.91, indicating a high degree of consistency between the system's output score and the manual review results. In terms of result confidence fluctuation, this invention has a fluctuation of only 0.07, significantly lower than the 0.18 of the traditional method, indicating that the results output by this invention are more stable. Although the average evaluation time per person increases from 4.8 minutes to 5.4 minutes, the additional 0.6 minutes yields higher authenticity recognition capability, lower false positive rate, and more stable result confidence. From an engineering application perspective, this cost is acceptable.

[0045] This embodiment demonstrates the feasibility and effectiveness of the present invention in online structured interview scenarios for professional qualifications. By constructing baseline and perturbation response data, introducing an adversarial question set, extracting cross-modal difference features, generating multimodal response shift representations using tensor voting fields, and further establishing a cognitive response dynamics evaluation model, the present invention effectively solves the problems of existing technologies that rely solely on static answer results, struggle to identify true comprehension levels, quantify anti-interference capabilities, and maintain objective and stable evaluation results. This enables a more refined, reliable, and interpretable intelligent evaluation of candidates' overall performance in application scenarios.

[0046] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal fusion-based intelligent assessment method for comprehensive test taker performance, characterized in that, Includes the following steps: Obtain and preprocess the raw data of test takers during the answering process to generate standardized multimodal data; Baseline response data of test takers to routine questions were constructed based on standardized multimodal data; Construct a set of adversarial questions and insert it into the regular question process to form a continuous answering process; During the continuous response process, data on the test takers' perturbation responses to a set of adversarial questions are collected. Cross-modal temporal alignment of baseline response data and perturbation response data was performed to extract differential features characterizing changes in test takers' responses; Tensor voting field fusion processing is performed on the differential features, and the differential features are used as sparse voting points. Tensor voting and sparse point densification are then performed to generate a multimodal response shift representation. The multimodal response shift characterization is input into the cognitive response dynamics assessment model, and the parameters of the test taker’s cognitive stability, anti-interference ability, true comprehension level and decision consistency are inverted to obtain the ability state vector. The overall performance evaluation results of the test takers are generated based on the ability state vector.

2. The intelligent assessment method for comprehensive test taker performance based on multimodal fusion according to claim 1, characterized in that, The raw data includes question text data, answer text data, speech data, facial image data, eye movement trajectory data, and body behavior data. The preprocessing includes word segmentation and vectorization encoding of question text data and answer text data, noise reduction, frame segmentation, and speech feature extraction of speech data, face detection, key point localization, and expression feature extraction of facial image data, gaze point extraction and trajectory smoothing of eye movement trajectory data, and posture estimation and action sequence extraction of body behavior data.

3. The intelligent assessment method for comprehensive test taker performance based on multimodal fusion according to claim 1, characterized in that, The construction of the baseline response data specifically includes: Extract test taker text data, answer text data, voice data, facial image data, eye movement data, and body behavior data from standardized multimodal data for routine questions; Based on the presentation time, start time, and end time of each regular question, the question text data, answer text data, voice data, facial image data, eye movement trajectory data, and body behavior data are segmented at the question level to obtain the data segments corresponding to each regular question. Based on the data segments corresponding to each regular question, we construct the regular question input sequence, the corresponding answer text sequence, the voice response sequence, the facial expression sequence, the eye movement trajectory sequence, and the body behavior sequence, respectively. For the same regular question, the corresponding regular question input sequence, corresponding answer text sequence, voice response sequence, facial expression sequence, eye movement trajectory sequence, and body behavior sequence are correlated to generate a single question baseline response unit; The baseline response units for each question in all regular questions are sequentially summarized to construct baseline response data.

4. The intelligent assessment method for comprehensive test taker performance based on multimodal fusion according to claim 1, characterized in that, The formation of the continuous response process specifically includes: Extract semantic information, logical relationship information, and answer target information of questions from the question text data and corresponding answer text data of regular questions. Based on the extracted semantic information, logical relationship information, and answer target information of the questions, semantic interference questions, information conflict questions, cognitive switching questions, and induced deviation questions are constructed. Semantic interference problems, information conflict problems, cognitive switching problems, and induced deviation problems are labeled with question numbers and question types to generate an adversarial problem set; Following the question numbering order in the regular question process, the set of adversarial questions is configured after the corresponding regular questions; The configured set of adversarial questions is written into the regular question process, forming a continuous answering process that includes both regular questions and adversarial questions.

5. The intelligent assessment method for comprehensive test taker performance based on multimodal fusion according to claim 1, characterized in that, In the continuous response process, once the adversarial questions in the adversarial question set are presented to the test taker, the time of presentation of the corresponding adversarial question is used as the starting node for data collection. The time interval for data collection is determined by the start and end times of the test taker's response to the adversarial question. Within the time interval, the test taker's text data, voice data, facial image data, eye movement trajectory data, and body behavior data are collected simultaneously. The collected text data, voice data, facial image data, eye movement trajectory data, and body behavior data are bound according to the question number of the corresponding adversarial question and a correspondence is established with the content of the adversarial question to form perturbation response data that characterizes the test taker's response under adversarial question conditions.

6. The intelligent assessment method for comprehensive test taker performance based on multimodal fusion according to claim 1, characterized in that, The extraction of the differential features specifically includes: Cross-modal temporal alignment was performed on the baseline response data and the perturbation response data, and text semantic feature vectors, speech prosody feature vectors, facial expression feature vectors, eye movement distribution feature vectors, and body behavior feature vectors were extracted respectively. Calculate text semantic offset features based on text semantic feature vectors; Calculate the prosodic variation features based on the prosodic feature vector; Calculate facial expression fluctuation features based on facial expression feature vectors; Calculate eye movement distribution offset features based on eye movement distribution feature vectors; Calculate the characteristics of changes in limb behavior based on limb behavior feature vectors; By combining text semantic shift features, speech prosody change features, facial expression fluctuation features, eye movement distribution shift features, and body behavior change features, a differential feature vector corresponding to each question number is generated. The differential feature vectors corresponding to all question numbers are then summarized to form differential features.

7. The intelligent assessment method for comprehensive test taker performance based on multimodal fusion according to claim 1, characterized in that, The generation of the multimodal response shift characterization specifically includes: Tensor voting field fusion processing is performed on the differential features, and the differential features are organized according to the question number to generate sparse voting points corresponding to each question number. Tensor encoding is performed on each sparse voting point to generate tensor voting data types corresponding to each question number; Perform tensor voting processing on the tensor voting data type corresponding to each question number to obtain the cumulative tensor voting result corresponding to each question number; Based on the degree of difference between the sparse voting points corresponding to each question number, calculate the voting weight between the tensor voting data types corresponding to each question number, and update the tensor voting cumulative results based on the voting weight; Based on the updated tensor voting accumulation results, the sparse voting points corresponding to each question number are densified to generate a local dense tensor field corresponding to each question number. The local dense tensor fields corresponding to each problem number are fused and migrated to obtain the fused migration field, and a multimodal response migration characterization is generated.

8. The intelligent assessment method for comprehensive test taker performance based on multimodal fusion according to claim 1, characterized in that, The acquisition of the capability state vector specifically includes: The multimodal response offset representation sequence is input into the cognitive response dynamics evaluation model, and the state recursion calculation is performed according to the question number order to obtain the cognitive state vector corresponding to each question number. The cognitive stability parameters are calculated based on the cognitive state vector corresponding to each question number, and the cognitive stability parameters corresponding to each question number are obtained. The anti-interference ability parameters are calculated based on the cognitive state vector corresponding to each question number, and the anti-interference ability parameters corresponding to each question number are obtained. The true comprehension level parameter is calculated based on the cognitive state vector corresponding to each question number. The decision consistency parameters are calculated based on the cognitive state vector corresponding to each question number, and the decision consistency parameters corresponding to each question number are obtained. A capability state vector is constructed based on cognitive stability parameters, anti-interference ability parameters, true comprehension level parameters, and decision consistency parameters.

9. The intelligent assessment method for comprehensive test taker performance based on multimodal fusion according to claim 1, characterized in that, The cognitive stability parameter, anti-interference ability parameter, true comprehension parameter, and decision consistency parameter in the ability state vector are used as the evaluation criteria for the corresponding sub-abilities. Each parameter is numerically mapped according to a unified scoring range to obtain the cognitive stability assessment result, anti-interference ability assessment result, true comprehension assessment result, and decision consistency assessment result, forming the sub-ability assessment result. The cognitive stability parameter, anti-interference ability parameter, true comprehension parameter, and decision consistency parameter are weighted and summarized to obtain the comprehensive score result. Based on the dispersion of the cognitive stability parameter, anti-interference ability parameter, true comprehension parameter, and decision consistency parameter corresponding to each question number, the fluctuation range of each parameter across all question numbers is statistically calculated. The stability of the comprehensive score result is measured based on the fluctuation range to obtain the result confidence level, forming the comprehensive performance evaluation result of the examinee.

Images