Intelligent evaluation method and system for individual situation awareness ability based on multi-modal data fusion and dynamic task adjustment

Abstract

The invention relates to the technical field of artificial intelligence and human factor engineering, and particularly discloses an individual situation awareness ability intelligent evaluation method and system based on multi-modal data fusion and dynamic task adjustment. The method comprises the following steps: constructing a standardized multi-channel task script, and configuring task stages and information parameters; operation behavior data and eye movement video streams in the task execution process are collected, and a unified time sequence data set is generated; multi-modal features are extracted and fused, a task execution state is judged, and task difficulty and an information presentation mode are dynamically adjusted; and reasoning individual ability levels and factor scores based on the fusion features and the task adjustment scheme, generating an evaluation report, and updating a task script and a model weight. According to the invention, the dynamic, intelligent and sustainable evaluation of the individual situation awareness can be realized, and the accuracy and adaptability of the evaluation result are improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and human factors engineering, and particularly to an intelligent evaluation method and system for individual situation awareness ability based on multi-modal data fusion and dynamic task adjustment. Background Art

[0002] With the continuous improvement of the degree of human-machine collaboration in complex combat systems, intelligent driving platforms and emergency command scenarios, the situation awareness ability of individuals in dynamic task environments has become a key factor affecting the overall system efficiency. Traditional ability evaluation methods mostly rely on static questionnaires, offline reviews or single-modal data analysis, and it is difficult to truly reflect the real-time perception, judgment and response abilities of individuals under the conditions of multi-source information input and continuous task changes.

[0003] In practical applications, the situation awareness ability of individuals is not only reflected in the response time and task execution order at the behavioral operation level, but also in multi-modal manifestation forms such as visual attention distribution, information channel selection strategies and task stage adaptability. Therefore, how to extract key features from heterogeneous data such as operation behavior data and eye movement video streams, and fuse multi-modal data to comprehensively model the perception state of individuals has become the key difficulty in the development of current intelligent evaluation systems.

[0004] On the other hand, existing evaluation systems generally lack a dynamic response mechanism for individual task performance, and are unable to adjust task difficulty and control information channels according to real-time status, resulting in biased evaluation results and being unable to support the continuous optimization and tracking analysis of the subsequent individual ability growth trajectory.

[0005] Therefore, there is an urgent need for an intelligent evaluation method and system that can fuse operation behavior features and gaze trajectory features, support dynamic adjustment of task stages and model adaptive updates, so as to achieve accurate scoring and sustainable optimization feedback of individual situation awareness ability, thereby improving the scientificity, objectivity and practicality of the evaluation system. Summary of the Invention

[0006] The present invention provides an intelligent evaluation method and system for individual situation awareness ability based on multi-modal data fusion and dynamic task adjustment, so as to solve the problem of how to construct multi-modal fusion features based on operation behavior data and eye movement video streams, and combine a dynamic task adjustment mechanism to achieve intelligent scoring and adaptive evaluation update of individual situation awareness ability.

[0007] To solve the above technical problems, the present invention provides an intelligent evaluation method for individual situation awareness ability based on multi-modal data fusion and dynamic task adjustment, including: Construct a standardized multi-channel task scenario, configure task stages, information types, presentation methods and time window parameters, and generate a task configuration structure; Obtain the operation behavior data and eye movement video stream during task execution, perform timestamp synchronization and format standardization processing, and generate a unified time-series dataset; Extract operation behavior features and fixation trajectory features from the unified time-series dataset, perform multi-modal feature fusion and structure encoding, and generate a fusion feature set; Judge the task execution status according to the fusion feature set, adjust the task difficulty level, information presentation channel and time limit parameter, and generate a dynamic task adjustment plan; The construction of the dynamic task adjustment plan is as follows: Among them, is the next-stage dynamic adjustment plan output by the system; is the next-stage task difficulty after adjustment according to the status feedback; is the number of adjusted information channels; is the task allowance time for the individual in the next stage; The "question type switching mark" identifies whether to switch the task type, which is used to maintain challenge and diversity; Take the fusion feature set and the dynamic task adjustment plan as inputs, perform the inference of the ability level classification model, and generate individual score labels and factor dimension scores; Generate an evaluation report structure from the individual score labels and the factor dimension scores, and update the task script configuration parameters and model weights for subsequent task execution and model iteration.

[0008] Furthermore, in the step of constructing the standardized multi-channel task script, the information types include graphical status information, text prompt information, fixed-position radar alarm voice information and dynamic-position drone alarm voice information.

[0009] Furthermore, the operation behavior data includes answer click trajectory, response time, task switching times and information access path, and the eye movement video stream includes fixation points, fixation duration, saccade path and region jump frequency.

[0010] Furthermore, the steps of the multi-modal feature fusion and structure encoding include: performing time-series alignment on the operation behavior features and the fixation trajectory features; performing normalization and dimensionality reduction processing on the time-series aligned features; constructing a fusion vector based on the multi-channel feature structure to generate the fusion feature set.

[0011] Furthermore, the steps of judging the task execution status include: calculating the response characteristic index of each stage task in the fusion feature set; performing distribution fitting on the response characteristic index to identify high-load or low-load states; determining the corresponding task difficulty level and information presentation channel number according to the identification result.

[0012] Further, the dynamic task adjustment scheme includes: a task difficulty level adjustment parameter, a presentation channel quantity setting parameter, a time limit setting parameter, and a question type switching flag.

[0013] Further, the ability level classification model is a classification model constructed based on logistic regression, and the classification model is trained using the labeled data in the training stage and the expert scoring data.

[0014] Further, the factor dimension scores are based on the results of exploratory factor analysis. The features in the fusion feature set are attributed to multiple ability-related factors, and the scores of each factor are generated by summing after Z-score normalization.

[0015] Further, after the step of generating the evaluation report structure, it further includes: inputting the factor dimension scores and individual rating labels in the evaluation report structure into the report template generation module to output a visual evaluation report; feeding back the behavior deviation features included in the report to the model weight update module for self-iterative training of the model.

[0016] Further, an intelligent evaluation system for individual situation awareness ability based on multi-modal data fusion and dynamic task adjustment includes: A task scenario construction module, used to construct a standardized multi-channel task scenario, configure task stages, information types, presentation methods, and time window parameters, and generate a task configuration structure; A data collection and standardization module, used to obtain operation behavior data and eye movement video streams during task execution, perform timestamp synchronization and format standardization processing, and generate a unified time series data set; A feature fusion modeling module, used to extract operation behavior features and fixation trajectory features from the unified time series data set, perform multi-modal feature fusion and structure encoding, and generate a fusion feature set; A task state recognition and adjustment module, used to judge the task execution state according to the fusion feature set, adjust task difficulty levels, information presentation channels, and time limit parameters, and generate a dynamic task adjustment scheme; An ability score inference module, used to take the fusion feature set and the dynamic task adjustment scheme as inputs, perform ability level classification model inference, and generate individual rating labels and factor dimension scores; An evaluation report generation and model update module, used to generate an evaluation report structure from the individual rating labels and the factor dimension scores, update task scenario configuration parameters and model weights for subsequent task execution and model iteration.

[0017] The following are its main beneficial effects: (1) In terms of data collection, the present invention integrates multi-modal behavioral data such as click trajectories, response times, and fixation paths, and forms a unified time series structure through standardized processing, avoiding the problems of feature loss and time series mismatch caused by heterogeneous multi-source data in traditional methods, and improving the feature alignment efficiency and the integrity of behavioral characterization.

[0018] (2) In terms of feature fusion and ability inference, by constructing a fusion feature set and cooperating with an ability level division model and a factor analysis structure, automatic inference of multi-dimensional ability scoring labels can be realized, supporting the mapping from micro-behaviors to macro-cognitive levels, and enhancing the interpretability of the evaluation and the generalization ability of the model.

[0019] (3) In terms of system update and closed-loop feedback, the present invention supports the reverse feedback of the evaluation output results to the script configuration and model parameter update processes, forming a self-evolving closed-loop optimization mechanism, breaking through the limitations of traditional evaluation means being static, one-time, and non-feedback. Description of the Drawings

[0020] Figure 1 It is a schematic flowchart of an intelligent evaluation method for individual situation awareness ability based on multi-modal data fusion and dynamic task adjustment provided by an embodiment of the present application; Figure 2 It is a structural block diagram of an intelligent evaluation system for individual situation awareness ability based on multi-modal data fusion and dynamic task adjustment provided by an embodiment of the present application. Detailed Embodiments

[0021] Embodiment 1: Refer to Figure 1 , which is a schematic flowchart of an intelligent evaluation method for individual situation awareness ability based on multi-modal data fusion and dynamic task adjustment provided by an embodiment of the present invention. This process can at least include steps S100 - S600: S100. Construct a standardized multi-channel task script, configure task stages, information types, presentation methods, and time window parameters, and generate a task configuration structure; S200. Obtain operation behavior data and eye movement video streams during task execution, perform timestamp synchronization and format standardization processing, and generate a unified time series data set; S300. Extract operation behavior features and fixation trajectory features from the unified time series data set, perform multi-modal feature fusion and structure encoding, and generate a fusion feature set; S400. Judge the task execution status according to the fusion feature set, adjust task difficulty levels, information presentation channels, and time limit parameters, and generate a dynamic task adjustment plan; S500. Use the fusion feature set and the dynamic task adjustment plan as inputs, perform ability level division model inference, and generate individual scoring labels and factor dimension scores; S600. Generate an evaluation report structure from the individual score tags and factor dimension scores, and update the task scenario configuration parameters and model weights for subsequent task execution and model iteration.

[0022] Step S100 includes at least steps S110 - S130: S110. Obtain the scenario parameters in the task design library, configure the number of task phases, task order, and task execution objectives, and generate a task phase configuration list.

[0023] In this step, the system first reads the scenario parameters from a pre - set task design library. The task design library consists of multiple groups of verified standardized situation task scenarios, which include corresponding task structures, operation indicators, feedback logics, and scenario identifiers for different test objectives and cognitive load levels. Specifically, the system selects a matching evaluation scenario template from the task design library according to the evaluation objective of the current evaluation object and reads the meta - parameter information of the scenario.

[0024] Among them, the scenario parameters include the number of task phases, the order of task phases, the target categories of each phase, the task structure identifier, and its logical configuration relationship. Further, the system encodes and arranges the sequence and inter - group relationship of the task phases in combination with the task path rules preset by experts to generate a task phase configuration list.

[0025] In the task phase configuration list, each phase entry corresponds to an independent evaluation unit. The system records the evaluation objectives to be completed in this phase, such as information screening, attention allocation, anomaly recognition, task switching, or risk judgment, etc., and presets the number of answering rounds, feedback frequencies, and the maximum task completion time for each phase for subsequent information channel configuration and data presentation.

[0026] S120. Select the information types for each phase from the task phase configuration list, and configure graphical status information, text prompt information, fixed - position radar alarm voice information, and dynamic - position drone alarm voice information.

[0027] After the task phase configuration list is generated, the system traverses each task phase in sequence and allocates corresponding multi - channel information source types according to the evaluation objective of the current phase by invoking the information type configuration rules.

[0028] Specifically, the system combines and configures from graphical status information, text prompt information, fixed - position radar alarm voice information, and dynamic - position drone alarm voice information according to the interaction complexity control model to form the multi - channel information stimulus structure required for the current phase.

[0029] The graphical status information includes static / dynamic image expression contents such as a map panel, instrument status, and personnel location; the text prompt information includes task progress prompts, warning summaries, or key text descriptions; the fixed-position radar alarm voice information simulates environmental warnings provided by a fixed base station sound source, such as obstacle alarms and route changes; the dynamic-position UAV alarm voice information simulates the enemy situation dynamics provided by a mobile information source, such as detection reports or task interference factors.

[0030] Furthermore, the system configures different information source combination forms for each stage according to the difficulty levels marked in the script. For example, in the low-load stage, a combination of graphics and text prompts is used; in the medium-load stage, fixed voice is superimposed; in the high-load stage, dynamic voice is superimposed and interference factors are increased.

[0031] The system writes the multi-channel information structure configured for each stage into the stage configuration record, and marks the presentation area, activation order, and duration of each information type, providing a unified structure template for subsequent presentation scheduling and data recording.

[0032] S130. Based on the information type and task stage configuration, set the presentation method and time window parameters for each stage, and generate a task configuration structure body for subsequent task presentation and behavior recording.

[0033] After completing the binding of the stage configuration and information type, the system sets the corresponding information presentation method and time window parameters according to the content volume and interaction complexity of each stage.

[0034] The presentation method includes the graphical layout strategy of the information window, layer priority, content switching method, channel activation order, etc. For example, when presenting two visual information in a task, the system arranges the information windows to be centered and there is no occlusion between layers; when adding voice prompt information, the system plays the voice through an external speaker and synchronizes the time axis to record the trigger time point.

[0035] The time window parameters include time sequence control parameters such as the task answering limit time, minimum information switching interval, interference information duration, and answer submission window. The time parameters are set according to the experimental configuration rules in the disclosure letter. For example, there is no limit in the first stage, a maximum of 3 minutes is set in the third stage, and all answering behaviors are required to be completed within 4 minutes in the seventh stage.

[0036] Furthermore, the system integrates the above information type, stage sequence, presentation method, and time control parameters to generate a structured task configuration structure body. The task configuration structure body is used for presentation invocation, synchronization marking of behavior data, and information alignment processing during the subsequent task execution process.

[0037] The task configuration structure will be used as the input of the task controller to drive the task engine to schedule the information presentation units of the corresponding channels within the specified time window, ensuring that the occurrence time point and duration of the information are consistent with the task event trajectory formed by the behavior data of the tested person.

[0038] Technical effects of this paragraph: Through the task script construction process set in S100 of the present invention, a standardized structure construction of task stages, information types, presentation methods, and time parameters is achieved. Combining the existing stage configuration rules in the script library, the system can combine and finely control task trigger events and information channel configurations as needed. In subsequent steps, the task configuration structure can be used to guide data collection and standardization operations, forming a fusion basis for behavior events and information events under a unified time sequence, providing a unified task context framework support for subsequent multi-modal modeling and ability evaluation. This step constructs the core of task and information control in the entire intelligent evaluation process and is the key fulcrum for the high reliability of the system and the high coordination of modules.

[0039] Step S200 at least includes steps S210 - S230: S210. Obtain the operation behavior data and eye movement video stream generated by the tested person during the task execution process, and record the original data such as the answer click trajectory, response time, fixation point, and saccade path.

[0040] After completing the construction of the task configuration structure in S100, the system starts the task presentation module to present a multi-channel situation awareness task to the tested person according to the preset task stage sequence, information channel type, and time window parameters in the task configuration structure during the task execution stage. During this process, the system needs to synchronously record various multi-source behavior data generated by the tested person in the interaction environment as the basic input for subsequent evaluation modeling.

[0041] Specifically, the operation behavior data includes but is not limited to: Answer click trajectory: Refers to the interaction paths such as mouse clicks and option selections performed by the tested person in the graphical task interface; Response time: Refers to the time consumption from the start of task information presentation to the completion of the feedback by the tested person; Information access path: Refers to the switching frequency between information areas, the content flipping order, and the operation mode of the tested person; Task switching flag: Refers to whether the tested person actively or passively switches the information focus or task module in the current task stage.

[0042] The eye movement video stream is captured in real time by a non-contact eye movement tracking device connected to the system and includes the following data dimensions: Fixation point: Refers to the coordinate of the fixation position of the tested person on the screen; Fixation duration: Refers to the length of time staying at the same fixation point; Saccade path: refers to the jumping trajectory of the eyeball between different fixation areas; Region jumping frequency: refers to the number of jumps of the subject between preset regions of interest per unit time.

[0043] The system records the corresponding task stage number, trigger information type, task scenario identifier, and timestamp for the above raw data at each moment, constructs a preliminary data record form, and lays a data foundation for the next data time synchronization and standardization processing.

[0044] S220. Align and synchronize the timestamps of the operation behavior data and the eye movement video stream to construct a unified time reference framework.

[0045] After obtaining the original multi-source data, the system needs to construct a unified time reference for the operation behavior data and the eye movement video stream to align different modal data on the same time axis. This step provides a basic support for subsequent feature fusion and modeling with strict temporal consistency.

[0046] Specifically, the system first extracts various task event trigger nodes based on the task stage start time, information channel activation time, and time window parameters marked in the task configuration structure in step S130, and constructs a task event timeline.

[0047] The system uses the above task event timeline as an alignment reference point and performs the following data synchronization processing operations: Perform timestamp standardization processing on the click events, access path switching, task submission marks, and other behavior data in the operation behavior data, so that the event tags can be accurately mapped to the task stage and information presentation period; Resample and interpolate the fixation events and saccade paths in the eye movement video stream according to the timestamp and screen refresh rate to align the fixation point sequence with the operation events within the same task stage; In view of the possible time drift phenomenon, the system combines the synchronization anchor points in the task event timeline to perform sliding calibration and boundary correction on the original timestamp sequence to ensure that the time deviation between different data streams is controlled within an acceptable range.

[0048] After completing the alignment operation, the system constructs a unified time reference framework and marks auxiliary fields such as task stage number, task event label, and information channel identifier for each modal data, which is convenient for subsequent processing steps to perform stage segmentation and modal differentiation on the data.

[0049] Description of front - back connection: The time alignment operation depends on the key fields such as the task - phase time window and the information presentation order in the task configuration structure generated by S130; the unified time - reference framework output by this operation will be used as the reference benchmark for the standardization process of S230 and will also provide phase labels and synchronization information for the feature extraction module of S300.

[0050] S230. Perform format conversion, field unification, and noise elimination processing on the aligned multi - source data to generate a standardized unified time - series data set.

[0051] After completing the time synchronization processing of the multi - source data, the system needs to further perform standardized structural conversion on the aligned data, including operations such as format conversion, field normalization, and anomaly elimination, so as to generate a unified and standardized input data structure for subsequent fusion modeling.

[0052] Specifically, the system performs the following steps: Format conversion processing: Convert the data into a unified event format according to the task phase and data type. The operation behavior data is encoded into a standard behavior event record unit, which includes fields such as behavior type, event time, screen coordinates, event label, response duration, etc.; the eye - movement data is encoded into a continuous fixation event stream, which includes fields such as fixation point position, fixation duration, jump path number, task - phase number, etc.

[0053] Field normalization processing: Perform standardized conversion on the field names, field structures, and numerical ranges in each modality data. The response time is unified into milliseconds; the fixation coordinates are standardized to the relative screen coordinate system (normalized to 0 to 1); the fixation duration is smoothed at a fixed period; the saccade jump path is replaced with a discrete number instead of a text description to construct a standard behavior label dictionary.

[0054] Noise elimination processing. The system, based on the multi - modality joint denoising strategy, identifies the following abnormal data for elimination or repair: Missing or repeated fixation events; Abnormal click areas or abnormally high - frequency clicks; Data records with a response time of zero or much higher than the set time window; Records with timestamp chaos or missing phase numbers.

[0055] Construct a unified time - series data set: Finally, the system organizes the data in a task - phase structure and integrates all data structures into a unified task behavior sequence list. Each row unit in this data table corresponds to a response record of a subject to specific task information at a specific time point and includes fields such as operation data, fixation data, phase label, channel source, and task identifier.

[0056] The unified time-series data set is cached in the feature processing module and used as the direct input of S300 for performing the extraction and fusion processing of operation behavior features and gaze trajectory features.

[0057] Technical effects of this section: Through the data acquisition and standardization process set by S200 of the present invention, the unified acquisition, time synchronization, and structure standardization of operation behavior data and eye movement video streams are realized. It not only ensures the time-series consistency between multi-source data, but also significantly improves the structural integrity and processing robustness of the data through means such as field specification and noise elimination. The unified time-series data set generated by S200 provides a highly consistent and directly usable basic data format for subsequent feature fusion modeling, ensuring the consistency and interpretability of the overall system modeling accuracy and ability evaluation, and is the key intermediate link supporting the full-process situation awareness evaluation system.

[0058] Step S300 at least includes steps S310 - S330: S310. Extract operation behavior features and gaze trajectory features from the unified time-series data set to generate structured behavior feature vectors and eye movement feature vectors.

[0059] After completing the standardization processing of operation behavior data and eye movement data in step S230, the system extracts key behavior indicators and gaze trajectory indicators with evaluation significance based on the unified time-series data set, and constructs operation behavior feature vectors and eye movement feature vectors respectively.

[0060] Specifically, the construction of the operation behavior feature vector includes the following steps: Based on the fields "response duration", "click position", and "operation behavior type", statistically analyze the response speed distribution, click area offset value, and operation event frequency in each task stage; Divide the behavior in each stage into time periods, and calculate the number of behaviors per unit time, behavior diversity index, and repeated click frequency; Summarize the stage-level behavior sequences to generate stage-level behavior feature vectors , where is the number of the individual being tested, represents the task stage.

[0061] The operation behavior feature vector is defined as: where, : stage average response time; : click area offset distance; : number of task switches; : total number of dimensions of operation behavior features.

[0062] Meanwhile, based on information such as "fixation point position", "fixation duration", and "saccade path number", gaze trajectory features are extracted to construct an eye movement feature vector. , and its calculation process includes: Statistical proportion of the total fixation duration in the key information area in each stage; Calculate the area jump frequency, average fixation path length, and fixation stability index; Extract the fixation hot zone distribution matrix and compress it into a feature dimension vector form.

[0063] The eye movement feature vector is expressed as: where, : Average fixation duration; : Fixation area jump frequency; : Line-of-sight stability; : Total number of eye movement feature dimensions.

[0064] The above feature items are the key innovation variables of this system. Different from traditional single behavioral indicators, they emphasize the joint representation ability of process behavior and perceptual response, providing higher-dimensional information input for subsequent multimodal modeling.

[0065] S320. Perform temporal alignment, normalization processing, and dimensionality reduction encoding on the behavioral feature vector and the eye movement feature vector to construct a fused feature data structure.

[0066] After obtaining the structured behavioral feature vector and the eye movement feature vector , to achieve effective fusion of multimodal data, the system performs the following three processing operations: First, perform temporal alignment processing. Since there are differences in the acquisition frequency and temporal granularity between the operation behavior features and the eye movement features, the system aligns the two types of features according to the stage number and time window set in the task configuration structure at the stage granularity. After alignment, the behavioral features and the eye movement features form a feature pair with synchronous task labels: where, is the feature pair with synchronous task labels; is the behavioral feature vector; is the eye movement feature vector.

[0067] Furthermore, perform normalization processing. To eliminate the interference caused by the scale differences between feature dimensions to the training of the fusion model, the system uses the Z-score normalization method to perform normalization processing on each dimension feature to unify its distribution: where, : Original eigenvalue; 、 : Feature mean and standard deviation in the sample set; : Eigenvalue after standardization.

[0068] Further, dimensionality reduction encoding is performed. Since the dimensions of behavior and eye movement features may be high, the system reduces the dimensions of the standardized feature vectors through principal component analysis (PCA) or an autoencoder network, compresses them into a unified embedding space, and constructs a fused feature data structure : Among them, is the fused feature data structure of individual i at stage t, which is the unified feature representation after dimensionality reduction; Encode(⋅): Dimensionality reduction encoding function, which can be principal component analysis (PCA) or an autoencoder network, compressing the original high-dimensional feature pair to a unified vector space, retaining the maximum discriminant information.

[0069] The generated in this step is the standard fusion unit for subsequent modeling input, with dimensional consistency, time series labels, and normalized encoding characteristics, providing a structurally stable input for representation learning.

[0070] S330. Input the fused feature data structure into the multi-modal structure modeling module, perform multi-channel feature aggregation and representation learning, and generate a fused feature set.

[0071] After completing the construction of the fused feature structure, the system inputs it into the multi-modal structure modeling module, uses the multi-channel attention mechanism to realize the information interaction between operation behavior features and eye movement features, and forms a fused feature set with context awareness ability.

[0072] The system constructs a multi-modal attention fusion function , defined as follows: Among them, : Linear transformation matrices of query, key, and value respectively, and the parameters are from model training; is the fused feature representation of individual i at stage t, including the information aggregation result of cross-modal semantic context; is the fusion mapping function, indicating projecting the input features to the attention context space; is the multi-head attention mechanism function, used to capture the deep interaction between operation behavior and eye movement features.

[0073] The fused feature set is represented as a set of sequences of fused representations for all stage tasks: Among them, represents the fused feature set of individual i under all T task stages, that is, the core input finally used for subsequent task state recognition and ability scoring; is the fused feature representation at the t-th stage, which already contains dynamic context semantics and multi-modal cross-semantic information.

[0074] Technical contribution description: Compared with traditional average pooling or concatenation fusion methods, the present invention proposes a feature fusion modeling method based on a multi-head attention mechanism, which can dynamically capture the interaction intensity and temporal context relationship between operation behaviors and eye movements, improve the modeling ability of individual multi-channel response features, and significantly enhance the interpretability and prediction accuracy of the situation awareness state.

[0075] Technical effects of this paragraph: Through the multi-modal feature extraction, normalization and alignment, structure fusion and attention mechanism modeling processes set by S300 of the present invention, the full-process conversion from raw data to modelable fusion vectors is realized. This step significantly improves the expression synergy and feature interaction ability between operation behaviors and eye movement data, forms a fused feature set with temporal structure, semantic association and multi-channel perception ability, provides core data support for the overall situation state recognition, ability level division and individual evaluation accuracy of the system, and is an important innovative link for realizing intelligent modeling in the system.

[0076] Step S400 at least includes steps S410 - S430: S410. Calculate task state indicators such as response time and information jump frequency according to the behavior performance of each stage task in the fused feature set.

[0077] This step takes the fused feature set generated by S300 as the input basis, and for each task stage , extracts key behavior response and eye movement pattern indicators, and constructs a task state indicator vector. Specific indicators include: Stage average response time : Represents the average feedback speed of the test subject to the task stimulus of this stage; Information channel jump frequency : The number of times the test subject switches different information channels per unit time; Fixation jump frequency : Refers to the ratio of the eye movement trajectory jumping between multiple attention areas; Behavior deviation rate : Measures whether the user's operation deviates from the task target area; Saccade path return ratio : represents the frequency of returning to the visited area in the saccade path; Number of stage misoperations : The number of click errors or sluggish behaviors that occur during the task stage.

[0078] The above indicators can be uniformly expressed as a stage status indicator vector: Among them, represents the status indicator vector of individual i under task stage t; represents the number of the measured individual; represents the task stage.

[0079] The original numerical value of each indicator comes from the structured behavior and eye movement statistics in the fusion feature set.

[0080] S420. Conduct a statistical distribution analysis on the task status indicators, identify whether it is in a high-load or low-load state, and mark the perceptual response level of the current stage.

[0081] The system performs statistical distribution modeling on the status indicator vector obtained in step S410 to construct a cross-sample load discrimination model, and then identify the perceptual load level of the individual in each stage.

[0082] First, perform normalization processing on each dimension indicator and set weight coefficients in combination with the load sensitivity of different indicators , and obtain the weighted load score: Among them, The load score of individual i in task stage t; is the normalized value of the th indicator; is the weight coefficient corresponding to the indicator, such as response time (0.4), fixation jump frequency (0.2), etc., which comes from the load discrimination analysis of the training samples; is the weighted summation operation of the six indicators, representing the overall load level.

[0083] Furthermore, based on the normal distribution fitting results of in all training samples, the system sets the load status boundary threshold: Among them: and are the mean and standard deviation of the load score respectively; is the label of the stage perceptual response level, used for task adjustment.

[0084] The status of each stage is based on its is marked with the corresponding "perceived response level".

[0085] Furthermore, the discriminated status label is attached to the phase identifier to form the following structure: where is a unified structure aggregating the status indicator, load score, and status label, serving as the input to S430.

[0086] This step completes the mapping from quantified behavioral response data to the individual's state cognitive level and is the core basis for implementing personalized adjustment of the assessment content. Using a load classification method based on distribution fitting ensures the robustness and generalization ability of the state division across individuals.

[0087] S430. According to the perceived response level, set the task difficulty level, the number of presentation channels, and the time limit parameters to generate a dynamic task adjustment plan for subsequent assessment execution.

[0088] Based on the phase perceived response level label generated in step S420, the system dynamically adjusts the task parameters in combination with the preset adjustment logic rules in the task script to generate a task adjustment plan structure: Task difficulty level setting. Set the task difficulty level according to the status result : where is the current task difficulty level; is the next-stage task difficulty adjusted according to the status feedback.

[0089] Number of channels and information richness setting. Set the number of information channels , according to the following strategy: where is the current number of information channels; is the adjusted number of information channels, ensuring no less than 1; is the step value adjusted based on the perceived state (−1, 0, +1).

[0090] Time window parameter adjustment. Set the stage time limit according to the task difficulty and status : where is the default time window, is the adjustment coefficient, usually set in the range of , derived from the individual response ability assessment; The allowed time for the individual's task in the next stage.

[0091] Finally, construct a dynamic task adjustment scheme: Among them, is the dynamic adjustment scheme for the next stage output by the system; the "question type switching flag" identifies whether to switch the task type, which is used to maintain challenge and diversity.

[0092] The technical effect of this paragraph: Through the operation of S400, the mapping from the fused feature vector to the multi-dimensional task state index, the accurate identification of the perceived load state, and the generation of task adjustment parameters for individual dynamic feedback are realized. This step not only improves the real-time perception and feedback ability of the evaluation system for the user's state, but also realizes the dynamic closed-loop adjustment between the evaluation content and the performance of the tested person, with good task adaptability and generalization ability, which is the key to realizing personalized and adaptive regulation of intelligent evaluation.

[0093] Step S500 at least includes steps S510 - S530: S510. Take the fused feature set and the dynamic task adjustment scheme as inputs, perform the inference of the ability level classification model, and output an individual score label.

[0094] In this step, the system takes the fused feature set output by S300 and the dynamic task adjustment scheme generated by S400 as joint inputs, and inputs them into the ability level classification model to perform the automatic inference of the individual score label.

[0095] Specifically, the fused feature set contains the structured vector after multi-modal data alignment, representing the comprehensive behavioral feature pattern shown by the tested person in multiple task stages, including behavioral decision-making rhythm, gaze stability, task deviation features, etc.; the dynamic task adjustment scheme reflects the task load strategy adjusted by the system according to the individual state change, including task difficulty level, number of information channels, and presentation time window.

[0096] The system splices the above two input structures into a joint feature vector and inputs it into the ability level classification model. The ability level classification model is constructed using a logistic regression model, and the model structure has been fitted based on expert-annotated data in the training stage.

[0097] The system generates a score label based on the output result of the model. The score label is a multi-class label set, representing the comprehensive ability performance level of the individual in the situation awareness task, including different level marks such as "beginner", "intermediate", and "advanced".

[0098] The output of this step is: Individual score label: Represents the ability level of the individual after completing the current task; Scoring confidence: Represents the internal confidence value of the model's determination of the label, which is used for subsequent optimization of the model's credibility.

[0099] The input structure and output structure of this step strictly correspond to the subsequent factor analysis process and the visualization report generation process.

[0100] S520. Input the fusion feature set into the factor analysis module, assign it to multiple factor structures related to the situation awareness ability, and generate factor dimension scores.

[0101] Based on the output of the individual's overall score label, this step further conducts multi-factor attribution analysis on the fusion feature set, extracts multiple dimensional factors related to the situation awareness ability, and normalizes the scores of each factor.

[0102] Specifically, the system constructs a factor variable matrix based on the fusion feature set. This matrix is a set of standardized high-dimensional feature vectors. Subsequently, the system calls a pre-trained factor analysis model to perform factor attribution processing. The factor analysis model uses the exploratory factor analysis method to automatically generate a factor loading matrix based on the covariance structure between features and perform clustering mapping on the fusion features.

[0103] The factor structure includes but is not limited to the following dimensions: Information reception efficiency factor; Attention stability factor; Multi-source information integration factor; Decision-making and judgment ability factor; Operation accuracy factor; Task load coping factor, etc.

[0104] The system projects the fusion features according to the factor loadings and calculates the score values of each dimension. The score of each factor dimension is output after being normalized by the Z-score, which is used to reflect the specific performance of the individual in each sub-ability dimension.

[0105] Finally, this step outputs the following structure: Factor dimension label structure body; The set of score values corresponding to each factor dimension.

[0106] This output will be used as the direct input for generating the evaluation report structure and updating the model weights in S600.

[0107] S530. Calibrate the scoring model using the expert scoring data in the training phase, and the calibration result is used to optimize the ability level division model.

[0108] To ensure the accuracy and generalization ability of the model inference results, this step calibrates the ability level division model based on the expert scoring data in the training phase.

[0109] Specifically, the system first constructs an expert scoring dataset, which includes the qualitative evaluation results of experts on the individual under test in multiple real task scenarios, including grade judgment, typical behavior description, abnormal state annotation, etc.

[0110] Then, the system constructs a mapping relationship between the fusion feature set and the expert scoring labels, compares the model output with the expert scores, and constructs model performance evaluation indicators (such as accuracy, recall, consistency coefficient, etc.) based on the confusion matrix.

[0111] According to the difference index, the system performs model weight update to optimize the internal parameter coefficients of the logistic regression model to minimize the difference between the predicted output and the expert scores.

[0112] This model calibration process provides an optimization basis for subsequent new sample evaluations, can be continuously updated during the evaluation iteration process, and has good self-evolution ability.

[0113] After the calibration process is completed, the system will generate: The calibrated scoring model parameter structure; The performance index log during the calibration process; The updated model version information.

[0114] The above structure will be used as an important input condition for the model update module in S600 to form a dynamic closed-loop mechanism between the scoring model and expert knowledge.

[0115] The technical effect of this paragraph: Through the operation of S500, an intelligent scoring mechanism under the condition of feature fusion and dynamic adjustment is realized. It can not only output the overall ability level label of the individual, but also construct a multi-dimensional factor dimension score structure, realizing the organic combination of scoring interpretability and individualized control analysis. At the same time, the system has a mechanism for dynamic calibration using expert knowledge to ensure that the scoring model has the consistency of industry standards and long-term evolution ability, which is the technical support for the intelligent scoring core module in the evaluation system of the present invention.

[0116] Step S600 at least includes steps S610 - S630: S610. Obtain the individual scoring label and the factor dimension score, match the visualization report template, and generate an evaluation report structure including the ability level and the feature dimension distribution.

[0117] Specifically, the individual rating labels and factor dimension scores output in S500 constitute the input for this step. Among them, the individual rating labels are inferred and generated by the ability level division model based on the fusion feature set and the dynamic task adjustment scheme, and are used to represent the overall situation awareness ability level of the measured individual; the factor dimension scores are structured scores generated after the fusion feature set is attributed to multiple factor structures related to the situation awareness ability by the factor analysis module, and are used to characterize the performance of the individual in various specific ability factors.

[0118] In this step, first, the visualization report template built in the report generation module is called, and the above two key indicators are input into this template. The report template includes, but is not limited to, the following dimension structures: overall level evaluation structure, factor dimension bar chart structure, task phase feature difference chart structure, behavior deviation feature summary form structure, etc. The system fills in the formats of the above structures and generates graphics to construct a complete visualization evaluation report structure. This evaluation report structure is not only output as the final evaluation result, but also serves the feedback and optimization process of subsequent modules.

[0119] During this process, to ensure the personalization and adaptability of template selection, different template versions with different styles or content focuses can be further selected according to the group characteristics of the measured individual (such as job type, training stage, previous evaluation records, etc.).

[0120] Connection description: The content in the visualization report structure, such as behavior deviation indicators, scores of each dimension, etc., will be extracted and utilized in step S620 to complete the deviation correction and parameter optimization of the model, forming a front-back closed loop.

[0121] S620: Input the behavior deviation indicators in the evaluation report structure into the model training unit to optimize and update the model parameters.

[0122] Specifically, the system automatically extracts the characteristic indicators marked as "abnormal deviation behaviors" from the evaluation report structure. The deviation indicators are obtained by comparing the model with the expert calibration reference standard, and include, but are not limited to, the frequency of task response time exceeding the threshold, abnormal increase in the fixation duration in non-target areas, inconsistent number of task switches and information backtracking paths, and other behavior - type abnormalities.

[0123] These deviation indicators will be converted into training inputs in S620 and fed back to the model training unit. Based on the relationship between the behavior deviation characteristics and the known rating labels, the training unit uses a fine - tuning mechanism to locally optimize and update the internal weight parameters of the ability level division model and the factor analysis module. During the update process, the system will refer to the expert scoring data used in the historical training stage and combine the latest feedback to perform weight re - distribution and classification boundary fine - tuning to enhance the discrimination accuracy of the model for potential ability marginal samples.

[0124] Understandably, the optimization process can adopt the form of incremental learning, enabling the model to retain old knowledge while integrating the information gain brought by new evaluation data to form a continuously evolving ability analysis mechanism.

[0125] Relevant description: The optimized model weight parameters will be used in S630 to update the task configuration structure to achieve adaptive adjustment of the script content.

[0126] S630. Update the task configuration structure based on the optimized model weights and task execution data to drive the next round of evaluation execution tasks.

[0127] In this step, the system combines the unified time-series data set generated during the previous stage task execution (from S230) with the optimized model parameters in the current step to jointly update the relevant parameters in the task script configuration structure.

[0128] Specifically, it includes: adjusting the difficulty threshold settings of task stages at each ability level according to the classification boundary adjustment of the scoring model; Based on the factor dimension score distribution, re-set the combination structure and occurrence frequency of various information types (such as graphical status information, text prompt information, fixed-position radar alarm voice information, and dynamic-position drone alarm voice information) in the next round of tasks; Use the characteristic deviation shown by the individual in the previous round of tasks as an input index to adjust the script parameter settings such as the corresponding time window parameters and information presentation methods in the next round of tasks.

[0129] The updated task configuration structure will be re-input to S100 to drive the next round of task script construction process, realizing a parameter feedback closed-loop from model output to task input.

[0130] Understandably, through the dynamic update of the task configuration structure, the system not only completes the evaluation output of the individual's ability state but also can adaptively adjust the next evaluation content according to the evaluation results, thereby supporting continuous evaluation and ability improvement in multiple rounds, across tasks, and multiple ability dimensions.

[0131] Technical effects of this paragraph: In S600 of the present invention, an adaptive feedback closed-loop is established by integrating the evaluation output results and the model update mechanism. Through the extraction and utilization of the evaluation report structure and behavior deviation indicators, the system can continuously optimize the ability level classification model and further promote the automatic update of the task script configuration, enabling the system to have the capabilities of long-term availability, cross-stage adaptation, and self-evolution of evaluation content, significantly improving the accuracy, adaptability, and individual ability characterization level of intelligent evaluation.

[0132] The key innovation points of the present invention include: (1)Structured construction mechanism of multi-channel task scenarios, which combines information types, presentation methods, and time window parameters to form a controllable and adjustable task configuration structure for the first time, supporting complex situation simulation and dynamic configuration of task dimensions.

[0133] (2)Fusion modeling method based on multi-modal time series data, which fuses time series data such as click behavior and eye movement trajectories, constructs a fusion vector through time series alignment, normalization, and structure encoding, and realizes the deep fusion of operation behavior and attention features.

[0134] (3)Intelligent ability scoring model driven by dynamic task adjustment, which combines task status judgment and individual behavior response, dynamically adjusts task parameters, and drives the inference ability level and factor dimension scores of the scoring model, constructing an adaptive and closed-loop updatable evaluation system.

[0135] The following are its main beneficial effects: (1)In terms of data collection, the present invention fuses multi-modal behavior data such as click trajectories, response times, and fixation paths, and forms a unified time series structure through standardized processing, avoiding the problems of feature loss and time series mismatch caused by heterogeneous multi-source data in traditional methods, and improving the feature alignment efficiency and the integrity of behavior characterization.

[0136] (2)In terms of feature fusion and ability inference, by constructing a fusion feature set and cooperating with an ability level classification model and a factor analysis structure, automatic inference of multi-dimensional ability scoring labels can be realized, supporting the mapping from micro-behaviors to macro-cognitive levels, and enhancing the interpretability of the evaluation and the generalization ability of the model.

[0137] (3)In terms of system update and closed-loop feedback, the present invention supports the reverse feedback of the evaluation output results to the scenario configuration and model parameter update processes, forming a self-evolving closed-loop optimization mechanism, breaking through the limitations of traditional evaluation means being static, one-time, and non-feedback.

[0138] Embodiment 2: Figure 2 The structural block diagram of an intelligent evaluation system for individual situation awareness ability based on multi-modal data fusion and dynamic task adjustment according to an embodiment of the present invention is shown. As Figure 2 shown, the structure may include: A task scenario construction module 10, which is used to construct a standardized multi-channel task scenario, configure task stages, information types, presentation methods, and time window parameters, and generate a task configuration structure. Specifically, it includes: Obtain the script parameters in the task design library, configure the number of task stages, task order, and task execution objectives, and generate a task stage configuration list; select the information type for each stage from the task stage configuration list, and configure graphical status information, text prompt information, fixed-position radar alarm voice information, and dynamic-position drone alarm voice information; based on the information type and task stage configuration, set the presentation method and time window parameters for each stage, and generate a task configuration structure for subsequent task presentation and behavior recording.

[0139] The data acquisition and standardization module 20, which is used to obtain the operation behavior data and eye movement video stream during the task execution, and perform timestamp synchronization and format standardization processing to generate a unified time-series dataset. Specifically, it includes: Obtain the original data such as the answer click trajectory, response time, task switching times, information access path, fixation points, fixation duration, saccade path, and region jump frequency generated by the subject during the task execution; perform timestamp alignment and synchronization on the operation behavior data and the eye movement video stream to construct a unified time reference framework; perform format conversion, field unification, and noise elimination processing on the aligned multi-source data to generate a standardized unified time-series dataset, providing basic data support for subsequent feature extraction and analysis.

[0140] The multi-modal feature fusion and modeling module 30, which is used to extract operation behavior features and fixation trajectory features from the unified time-series dataset, perform multi-modal feature fusion and structure encoding, and generate a fusion feature set. Specifically, it includes: Extract structured behavior feature vectors and eye movement feature vectors from the unified time-series dataset, including indicators such as answer click rate, access path depth, fixation duration, and fixation density; perform time-series alignment, normalization processing, and dimensionality reduction encoding on the behavior feature vectors and the eye movement feature vectors to construct a fusion feature data structure; input the fusion feature data structure into the multi-modal structure modeling module to perform multi-channel feature aggregation and representation learning to generate a fusion feature set, which is used to characterize the task response characteristics of an individual in a multi-modal interaction scenario.

[0141] The task status recognition and adjustment module 40, which is used to judge the task execution status according to the fusion feature set, adjust the task difficulty level, information presentation channel, and time limit parameters, and generate a dynamic task adjustment plan. Specifically, it includes: Calculate task status metrics such as response time and information jump frequency based on the behavioral performance of each stage task in the fusion feature set; perform statistical distribution analysis on the task status metrics to identify whether it is in a high-load or low-load state, and mark the perceived response level of the current stage; set task difficulty levels, the number of presentation channels, and time limit parameters according to the perceived response level to generate a dynamic task adjustment plan for adjusting subsequent task loads.

[0142] The ability score inference module 50, which is used to take the fusion feature set and the dynamic task adjustment plan as inputs, perform the inference of the ability level classification model, and generate individual score labels and factor dimension scores. Specifically, it includes: Input the fusion feature set and the dynamic task adjustment plan into the ability level classification model, perform the model inference process, and output individual score labels; input the fusion feature set into the factor analysis module, classify it into multiple factor structures related to the situation awareness ability, and generate factor dimension scores for multi-dimensional ability analysis; calibrate the scoring model using the labeled data and expert scoring data in the training stage, and the calibration results are used to continuously optimize the structure and parameter configuration of the ability level classification model.

[0143] The evaluation report generation and model update module 60, which is used to generate an evaluation report structure from the individual score labels and factor dimension scores, and update the task scenario configuration parameters and model weights for subsequent task execution and model iteration. Specifically, it includes: Obtain the individual score labels and the factor dimension scores, match the visualization report template, and generate an evaluation report structure including the ability level and the feature dimension distribution; input the behavior deviation index in the evaluation report structure into the model training unit to optimize and update the model parameters, and improve the adaptability of the scoring model to different individual characteristics; update the task configuration structure based on the optimized model weights and task execution data to drive the next round of evaluation execution tasks and realize the collaborative evolution of task content and evaluation models.

[0144] Beneficial effects of the embodiment: (1) Achieved the efficient fusion and modeling of multi-modal behavioral data and eye movement data, which can comprehensively reflect the dynamic response patterns of individuals under complex tasks, and improve the comprehensiveness and objectivity of situation awareness ability assessment; (2) Constructed a closed-loop control mechanism for dynamically adjusting task loads, which can adjust task difficulty levels and presentation parameters in real time according to the individual state recognition results, and ensure the individual adaptability and continuous effectiveness of the evaluation process; (3)An optimized mechanism of the cyclic model of evaluation - scoring - feedback - update is implemented, enabling the system to continuously evolve the task scenario and scoring algorithm in multiple rounds of evaluation, and improving the evaluation accuracy, applicability, and model generalization ability; (4)The system structure has a high degree of modularity, is easy to expand and deploy, and is applicable to individual ability intelligent evaluation tasks in various high-cognitive load environments such as pilot selection, combat training, and unmanned system operation evaluation.

[0145] Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all embodiments. The drawings show preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosed content of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements for some of the technical features. Any equivalent structure directly or indirectly using the content of the specification and drawings of the present application in other related technical fields shall be equally within the scope of the patent protection of the present application.

Claims

1. An intelligent evaluation method for individual situation awareness ability based on multimodal data fusion and dynamic task adjustment, characterized in that, It includes the following steps: Construct a standardized multi-channel task script, configure task phases, information types, presentation methods, and time window parameters to generate a task configuration structure; Obtain operation behavior data and eye movement video streams during task execution, perform timestamp synchronization and format standardization processing to generate a unified time-series dataset; Extract operation behavior features and fixation trajectory features from the unified time-series dataset, perform multi-modal feature fusion and structure encoding to generate a fusion feature set; Judge the task execution status according to the fusion feature set, adjust task difficulty levels, information presentation channels, and time limit parameters to generate a dynamic task adjustment plan; The construction of the dynamic task adjustment plan is as follows: Among them, is the next-stage dynamic adjustment plan output by the system; is the task difficulty of the next stage adjusted according to the state feedback; is the number of adjusted information channels; is the task allowance time of the individual in the next stage; the "question type switching mark" indicates whether to switch the task type, which is used to maintain challenge and diversity; Use the fusion feature set and the dynamic task adjustment plan as inputs, perform inference on the ability level classification model to generate individual scoring labels and factor dimension scores; Generate an evaluation report structure from the individual scoring labels and factor dimension scores, update task script configuration parameters and model weights for subsequent task execution and model iteration.

2. The evaluation method according to claim 1, characterized in that, In the step of constructing the standardized multi-channel task script, the information types include graphical status information, text prompt information, fixed-position radar alarm voice information, and dynamic-position drone alarm voice information.

3. The evaluation method according to claim 1, wherein The operation behavior data includes answer click trajectories, response times, task switching frequencies, and information access paths, and the eye movement video stream includes fixation points, fixation durations, saccade paths, and region jump frequencies.

4. The evaluation method according to claim 1, wherein The steps of the multi-modal feature fusion and structure encoding include: Align the operation behavior features and the fixation trajectory features in time series; Perform normalization and dimensionality reduction processing on the features after time series alignment; Construct a fusion vector based on the multi-channel feature structure to generate the fusion feature set.

5. The evaluation method according to claim 1, characterized in that The steps of judging the task execution status include: Calculate the response characteristic indicators of each stage task in the fusion feature set; Perform distribution fitting on the response characteristic indicators to identify high-load or low-load states; Determine the corresponding task difficulty level and the number of information presentation channels according to the identification results.

6. The evaluation method according to claim 1, wherein The dynamic task adjustment plan includes: task difficulty level adjustment parameters, presentation channel number setting parameters, time limit setting parameters, and question type switching marks.

7. The evaluation method according to claim 1, wherein The ability level classification model is a classification model constructed based on logistic regression, and the classification model is trained using labeled data and expert scoring data in the training stage.

8. The evaluation method according to claim 1, wherein The factor dimension scores are based on the results of exploratory factor analysis. The features in the fusion feature set are attributed to multiple ability-related factors, and the scores of each factor are generated by summing after normalization by Z-score.

9. The evaluation method according to claim 1, characterized in that, After the step of generating the evaluation report structure, it further includes: Input the factor dimension scores and individual scoring labels in the evaluation report structure into the report template generation module to output a visual evaluation report; Feed back the behavior deviation features included in the report to the model weight update module for self-iterative training of the model.

10. An intelligent evaluation system for individual situation awareness ability based on multimodal data fusion and dynamic task adjustment, which is applied to an intelligent evaluation method for individual situation awareness ability based on multimodal data fusion and dynamic task adjustment according to any one of claims 1-9, characterized in that It includes: A task script construction module for constructing a standardized multi-channel task script, configuring task phases, information types, presentation methods, and time window parameters to generate a task configuration structure; The data acquisition and standardization module is used to obtain the operation behavior data and eye movement video stream during the task execution process, perform timestamp synchronization and format standardization processing, and generate a unified time-series dataset; The feature fusion and modeling module is used to extract operation behavior features and fixation trajectory features from the unified time-series dataset, perform multi-modal feature fusion and structure encoding, and generate a fusion feature set; The task status recognition and adjustment module is used to judge the task execution status according to the fusion feature set, adjust the task difficulty level, information presentation channel and time limit parameters, and generate a dynamic task adjustment plan; The ability score inference module is used to take the fusion feature set and the dynamic task adjustment plan as inputs, perform the inference of the ability level division model, and generate individual score labels and factor dimension scores; The evaluation report generation and model update module is used to generate the evaluation report structure from the individual score labels and the factor dimension scores, update the task script configuration parameters and model weights, and be used for subsequent task execution and model iteration.

Images