Psychomotor ability training method and system based on mixed reality and behavior analysis

Abstract

The present invention relates to the fields of human-computer interaction training, virtual reality, and intelligent assessment, and specifically discloses a psychomotor ability training method and system based on mixed reality and behavioral analysis. The method includes: constructing a multi-scenario psychological training environment and presenting it in a mixed reality device, collecting the pilot's action behavior data, eye movement path data, and operation reaction time data during the training process, constructing multimodal original behavioral features, and then constructing a set of psychomotor ability labels; achieving ability-task modeling by matching ability labels with task structures, generating personalized training paths and ability difference maps based on difference analysis with preset ability standards, adjusting task parameter configuration data and dynamically updating training scenarios to generate ability evolution trajectories and output training analysis results. This can achieve accurate ability modeling, dynamic task adaptation, and visual tracking of training effects, significantly improving the personalization and intelligence level of psychomotor ability training.

Description

Technical Field

[0001] The present invention relates to the fields of human-computer interaction training, virtual reality and intelligent assessment, and in particular to a psychomotor ability training method and system based on mixed reality and behavior analysis. Background Art

[0002] With the increasing complexity of modern combat mission environments and the continuous increase in flight operational workloads, the psychomotor abilities required of pilots during mission execution (including at least operational reaction speed, visual attention control, and movement coordination) have become key indicators for measuring their mission adaptability and execution efficiency. Existing technologies, on the one hand, often rely on static simulation training or the collection of single behavioral characteristics, making it difficult to effectively combine task structure with multimodal behavioral data for capability modeling and training assessment. This leads to problems such as poor adaptability to training scenarios, coarse granularity in capability modeling, and a lack of personalized training path development.

[0003] On the other hand, although mixed reality technology has been gradually applied to the field of flight training, the current system focuses more on the immersive reproduction of operational skills, and has not yet formed a dynamic ability assessment mechanism driven by behavioral data. It also lacks a system closed-loop design such as correlation analysis between behavioral characteristics and task structure, ability evolution trend modeling and personalized task adjustment. As a result, training feedback lacks specificity and is difficult to support the full process tracking and optimization of individual ability growth.

[0004] Therefore, it is urgent to propose a psychomotor ability training method and system that integrates multimodal behavioral feature collection and in-depth analysis, combined with dynamic adjustment of mixed reality environment, so as to realize the personalization, intelligence and visualization of the training process, and provide more scientific and efficient technical support for pilot ability improvement and mission adaptability training. Summary of the Invention

[0005] The present invention provides a psychomotor ability training method and system based on mixed reality and behavioral analysis. The method addresses the problem of how to achieve accurate modeling of pilot psychomotor abilities, generation of personalized training paths, and growth trajectory assessment based on the fusion analysis of multimodal behavioral data (including action behavior data, eye movement path data, and operation reaction time data) combined with the dynamic adaptation mechanism of the mixed reality training environment.

[0006] In order to solve the above technical problems,

[0007] In a first aspect, the present invention provides a method for training psychomotor skills based on mixed reality and behavioral analysis, comprising:

[0008] S100: Build a multi-scenario psychomotor training environment, generate a task structure diagram and environment configuration parameters, and push them to a mixed reality device for scene presentation;

[0009] S200, collecting the pilot's action behavior data, eye movement path data, and operation reaction time data during training, and constructing multimodal original behavior features;

[0010] S300, performing a fusion analysis on the multimodal original behavioral features, constructing a psychomotor ability label set, and associating it with the task structure diagram to complete ability-task matching modeling;

[0011] S400, generating an ability difference map for task adjustment based on a difference analysis result between the ability labels in the psychomotor ability label set and a preset ability standard, and a weight aggregation function, and determining a personalized training path based on the difficulty adjustment function and the ability difference map;

[0012] The weight aggregation function is:

[0013]

[0014] in, Represents the aggregated capability difference value of the i-th task node among multiple task nodes; α k is the difference weight of the k-th ability dimension; represents the capability difference value of the kth capability dimension on the i-th task node;

[0015] The difficulty adjustment function is:

[0016]

[0017] Among them, λ i Indicates the training difficulty adjustment coefficient corresponding to the i-th task node;

[0018] S500: Adjusting task parameter configuration data based on the personalized training path and the capability difference map, and pushing the adjusted task scenario configuration data to the mixed reality device, wherein the adjusted task scenario configuration data is used to replace the current training task and execute a new round of training process;

[0019] S600: Record and organize the pilot's psychomotor ability tag sets during multiple training cycles, perform time series modeling on all psychomotor ability tag sets, generate an ability evolution trajectory, and generate a training process analysis report and an individual ability growth map based on the ability evolution trajectory.

[0020] Furthermore, the task structure diagram includes at least: task process nodes, task logical relationships and task content categories; the environmental configuration parameters include at least: weather condition parameters, background change speed parameters, visual change range parameters and limb operation frequency parameters; the S100, constructing a multi-scenario psychomotor training environment, generating a task structure diagram and environmental configuration parameters, and pushing them to the mixed reality device for scene presentation, includes: S110, obtaining a training task template and training target type information, selecting a corresponding training scene type based on flight mission requirements and setting a training level; S120, based on the training task structure composition information extracted from the training task template, constructing the task process nodes, the task logical relationships and the task content categories; S130, based on the task process nodes, the task logical relationships, the task content categories and the training levels, setting the weather condition parameters, the background change speed parameters, the visual change range parameters and the limb operation frequency parameters required for the training scene type, generating a training task configuration file and pushing it to the mixed reality device for scene presentation.

[0021] Furthermore, the motion behavior data at least includes: push-pull operation behavior, hand clicking behavior and posture adjustment interaction behavior; the S200, collecting the pilot's motion behavior data, eye movement path data and operation reaction time data during the training process, and constructing a multimodal original behavior feature, includes: S210, obtaining the push-pull operation behavior, the hand clicking behavior and the posture adjustment interaction behavior of the pilot in the mixed reality task during the training process; S220, constructing eye trajectory information and time interval sequence based on the pilot's eye movement path data and the operation reaction time data during the training process; S230, aligning and integrating the push-pull operation behavior, the hand clicking behavior, the posture adjustment interaction behavior, the eye trajectory information and the time interval sequence to construct the multimodal original behavior feature data.

[0022] Furthermore, the S300 performs a fusion analysis on the multimodal original behavioral features, constructs a psychomotor ability label set, and associates it with the task structure diagram to complete the ability-task matching modeling, including: S310, extracting key behavioral indicator data from the multimodal original behavioral feature data, and the key behavioral indicator data at least includes: operation accuracy, reaction time, gaze reaction time and gaze time; S320, performs a fusion analysis on the operation accuracy, the reaction time, the gaze reaction time and the gaze time, and constructs a psychomotor ability label set corresponding to the training task; S330, matches the psychomotor ability label set with the training process nodes in the task structure diagram, and constructs a mapping relationship between ability labels and task nodes to complete the ability-task matching modeling.

[0023] Furthermore, the S400 generates an ability difference map for task adjustment based on the difference analysis results between the ability labels in the psychomotor ability label set and the preset ability standards, and a weight aggregation function, and determines a personalized training path based on the difficulty adjustment function and the ability difference map, including: S410, calculating the difference analysis results based on the ability labels in the psychomotor ability label set and the preset ability standards, and the difference analysis results are a difference numerical set between the label values; S420, generating the ability difference map according to the difference numerical set and the weight aggregation function; S430, generating the personalized training path based on the ability difference map and the training process structure in combination with the difficulty adjustment function.

[0024] Furthermore, the task parameter configuration data includes at least: task type and capability label adaptation range; the S500, based on the personalized training path and the capability difference map, adjusts the task parameter configuration data, and pushes the adjusted task scene configuration data to the mixed reality device, including: S510, based on the training task node parameters in the personalized training path, extracts the task type and the capability label adaptation range; S520, based on the capability difference map and the environment configuration parameters, adjusts the task type and the capability label adaptation range, and generates the adjusted task type and the adjusted capability label adaptation range; S530, pushes the adjusted task type and the adjusted capability label adaptation range to the mixed reality device.

[0025] Furthermore, the S600, recording and organizing the pilot's psychomotor ability label sets in multiple training cycles, performing time series modeling on all psychomotor ability label sets to generate an ability evolution trajectory, and generating a training process analysis report and an individual ability growth map based on the ability evolution trajectory, includes: S610, constructing a chronologically arranged ability label change sequence based on the pilot's psychomotor ability label sets in the multiple training cycles; S620, performing time series modeling on the ability label change sequence to extract the ability evolution trajectory during the training process; S630, generating the training process analysis report and the individual ability growth map based on the ability evolution trajectory.

[0026] Furthermore, when performing time series modeling on the capability label change sequence in S620, a sliding window mechanism is adopted, and the window length is multiple consecutive training cycles. The evolution speed and fluctuation amplitude of each capability dimension are calculated within the window by difference, slope and curve fitting, and a multi-dimensional capability growth curve map is constructed. The evolution trend is mapped to the corresponding node in the training task structure diagram to analyze the correlation between capability dynamics and task types. The evolution trend is used to characterize the capability evolution trajectory.

[0027] Furthermore, when adjusting the task parameter configuration data based on the personalized training path and the ability difference map in S500, if the i-th task node is marked as "enhancement", at least one of the background change speed parameter, the visual change range parameter, and the limb operation frequency parameter is increased; if the i-th task node is marked as "adjustment", the target level environmental parameter load is maintained; if the i-th task node is marked as "adaptation", the original environmental configuration parameters are retained.

[0028] In a second aspect, the present invention provides a psychomotor ability training system based on mixed reality and behavior analysis, comprising:

[0029] The mixed reality task generation module is used to build a multi-scenario psychomotor training environment, generate task structure diagrams and environment configuration parameters, and push them to the mixed reality device for scene presentation;

[0030] The data acquisition module is used to collect the pilot's action behavior data, eye movement path data and operation reaction time data during the training process;

[0031] A behavior analysis module is configured to construct the multimodal original behavior features based on the action behavior data, the eye movement path data, and the operation reaction time data; and to perform a fusion analysis on the multimodal original behavior features to construct a psychomotor ability label set, and to associate the labels with the task structure diagram to complete ability-task matching modeling;

[0032] The ability reasoning and task planning module is used to generate an ability difference map for task adjustment based on the difference analysis results between the ability labels in the psychomotor ability label set and the preset ability standards, and a weight aggregation function, and determine a personalized training path based on the difficulty adjustment function and the ability difference map; the weight aggregation function is: in, Represents the aggregated capability difference value of the i-th task node among multiple task nodes; α k is the difference weight of the k-th ability dimension; represents the ability difference value of the kth ability dimension on the i-th task node; the difficulty adjustment function is: Among them, λ i Indicates the training difficulty adjustment coefficient corresponding to the i-th task node;

[0033] A task scheduling module, configured to adjust task parameter configuration data based on the personalized training path and the capability difference map, and push the adjusted task scenario configuration data to the mixed reality device, wherein the adjusted task scenario configuration data is used to replace the current training task and execute a new round of training process;

[0034] The training evaluation module is used to record and organize the pilots' psychomotor ability label sets over multiple training cycles, perform time series modeling on all psychomotor ability label sets, generate ability evolution trajectories, and based on the ability evolution trajectories, generate training process analysis reports and individual ability growth maps.

[0035] The following are its main beneficial effects:

[0036] (1) Achieve precise adaptation of training tasks to pilots' ability status, and improve training personalization and targeting. The present invention integrates mixed reality environment modeling technology with a multimodal behavioral data acquisition mechanism, constructs psychomotor ability labels based on key indicators such as action behavior data, eye movement path data, and operation reaction time data, and forms an ability-task mapping relationship by matching it with the task structure diagram. Based on the pilot's dynamic ability status, it can automatically generate personalized training paths and task difficulty adjustment plans, effectively avoiding the problems of fixed task intensity and delayed feedback in traditional training.

[0037] (2) Construct a mathematical modeling system that integrates behavioral data to achieve quantifiable evolution tracking of capability status. This paper introduces a high-dimensional feature fusion algorithm and a capability difference calculation function, innovatively uses a weighted regression model and a time-series capability evolution function to model capability labels and conduct dynamic difference analysis. Combining training task nodes with environmental configuration parameters, it promotes closed-loop regulation between task configuration and capability progress, and improves the accuracy of capability growth identification and regulation sensitivity during training.

[0038] (3) Forming an interpretable training effect evaluation map to achieve training decision visualization and closed-loop optimization. The present invention constructs a capability label change sequence in multiple training cycles, extracts the capability growth trajectory based on time series modeling, generates a capability growth map and process analysis report, and provides a visual basis for subsequent training task recommendations and strategy adjustments, significantly improving the tracking transparency and tuning efficiency of training effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of a psychomotor ability training method based on mixed reality and behavioral analysis provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of the structure of a psychomotor ability training system based on mixed reality and behavior analysis provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0042] The following describes in detail the psychomotor ability training method based on mixed reality and behavior analysis provided by the embodiment of the present invention, taking the psychomotor ability training system based on mixed reality and behavior analysis (hereinafter referred to as the system) as the execution body:

[0043] Example 1: Reference Figure 1 , is a flow chart of a psychomotor ability training method based on mixed reality and behavior analysis provided by an embodiment of the present invention. The method may at least include S100-S600:

[0044] S100: Build a multi-scenario psychomotor training environment, generate a task structure diagram and environment configuration parameters, and push them to a mixed reality device for scene presentation.

[0045] S200: Collect the pilot's action behavior data, eye movement path data, and operation reaction time data during the training process to construct multimodal original behavior characteristics.

[0046] S300: Fusion analysis of multimodal raw behavioral features is performed to construct a psychomotor ability label set, which is then associated with a task structure diagram to complete ability-task matching modeling.

[0047] S400, based on the difference analysis results between the ability labels in the psychomotor ability label set and the preset ability standards, and the weight aggregation function, generate an ability difference map for task adjustment, and determine a personalized training path based on the difficulty adjustment function and the ability difference map.

[0048] S500. Based on the personalized training path and the capability difference map, adjust the task parameter configuration data, and push the adjusted task scene configuration data to the mixed reality device. The adjusted task scene configuration data is used to replace the current training task and execute a new round of training process.

[0049] S600: Record and organize the pilot's psychomotor ability label set during multiple training cycles, perform time series modeling on the psychomotor ability label set, generate an ability evolution trajectory, and generate a training process analysis report and an individual ability growth map based on the ability evolution trajectory.

[0050] Among them, the above weight aggregation function is:

[0051]

[0052] Represents the aggregated capability difference value of the i-th task node among multiple task nodes; α k is the difference weight of the k-th ability dimension; Represents the capability difference value of the kth capability dimension on the i-th task node;

[0053] The difficulty adjustment function is:

[0054]

[0055] λ i Indicates the training difficulty adjustment coefficient corresponding to the i-th task node.

[0056] In some embodiments, the task structure diagram includes at least: task process nodes, task logical relationships, and task content categories; the environmental configuration parameters include at least: weather condition parameters, background change speed parameters, visual change range parameters, and limb operation frequency parameters; the above-mentioned S100 includes at least S110-S130:

[0057] S110: Obtain a training mission template and training target type information, select a corresponding training scenario type based on flight mission requirements, and set a training level.

[0058] In this step, the system first connects to the training mission database to retrieve standardized training mission templates corresponding to the pilot's training phase. At least one training mission template is required to construct a training mission template dataset. Optionally, each training mission template includes at least the mission objective type, mission execution elements, mission control logic, and mission evaluation elements. The system then retrieves the required standardized training mission template from four categories: flight capability maintenance, stress response, multi-tasking, and perception interpretation.

[0059] Specifically, the system receives a training purpose profile input by the user into the system. Optionally, the profile includes at least the flight mission phase (such as takeoff, stable flight, landing), the current capability assessment score, and recommended training dimensions. Then, the system automatically matches the target type of the above-mentioned training task template according to the flight mission phase in the profile, and extracts the standard task category associated with the training task template. Furthermore, the system can select a scene type suitable for the current training cycle. Optionally, the scene type includes at least feature dimensions such as the degree of scene abstraction (high abstraction / low abstraction), interaction complexity (single task / multi-task), and physical perception channel (visual dominant / cross-perception).

[0060] Subsequently, the system sets the training level (abbreviated as: training level) based on the pilot's training level, past training data and task recommendation tags. The setting result of the training level will be used in the environmental parameter configuration of S130 as one of the input conditions for scene difficulty control.

[0061] This step outputs three items: the training task template data set, the training target type identifier, and the set training level grade, and passes them as input to step S120 for constructing the task structure diagram.

[0062] S120. Based on the training task structure composition information extracted from the training task template, construct task process nodes, task logical relationships and task content categories.

[0063] Following the training task template and training target type generated in S110, this step performs a structural analysis on the internal structure of the training task template, extracts task process elements (i.e., task process nodes, task logical relationships, and task content categories), and constructs a task structure diagram.

[0064] It should be noted that the above task structure diagram is a directed task execution diagram, wherein each node in the task structure diagram represents a task step, and each edge represents the logical sequence between tasks.

[0065] During this week, the system first analyzes the task phase information in the training task template, identifying and dividing the task process nodes, such as "identify target logo - perform simulated operation - complete visual confirmation - perform complex action - return to baseline state" and other typical node content. Each task node is annotated with the task function type (perception, decision-making, operation), the associated task objectives (such as visual accuracy, body control), and the required user response time limit.

[0066] Secondly, based on the time logic, causal trigger relationship and task fusion requirements between nodes, the system configures the task path sequence in the task structure diagram to form a complete task flow diagram structure, and further marks the task logical relationship, such as "parallel execution", "serial switching", "conditional jump" and other relationship modes.

[0067] Finally, the system categorizes the tasks at each node and generates task category labels, which serve as the basis for subsequent capability tag matching. Task categories include operational tasks (such as hand movements and posture changes), reaction tasks (such as button confirmation and steering adjustments), and decision-making tasks (such as target determination and information screening).

[0068] The output of this step includes a task structure diagram, a task process node definition table, a task logic topology diagram, and a task content category label table. This task structure diagram will be directly used to match psychomotor ability labels to training process nodes in the S300 module. The task content category labels will be used to classify ability differences in S400, forming a multi-level task-ability label mapping relationship.

[0069] S130. Based on the task process nodes, task logical relationships, task content categories, and training levels, set the weather condition parameters, background change speed parameters, visual change range parameters, and limb operation frequency parameters required for the training scenario type, generate a training task configuration file, and push it to the mixed reality device for scene presentation.

[0070] Specifically, the system calls the environmental parameter model that matches the training level and extracts the control parameter indicators of the corresponding level. The model defines the following four types of environmental parameters for each training level based on the training goals and ability types: the weather condition parameters, background change speed parameters, visual change range parameters, and limb operation frequency parameters. Specifically,

[0071] Weather condition parameters include at least visual clarity, brightness contrast, and weather type (e.g., sunny, rainy, nighttime, etc.);

[0072] Background change speed parameter: controls the background visual flow rate, used to simulate different flight speeds or interference states during flight;

[0073] Visual change range parameters: used to adjust the range of the visual attention area during training, such as whether a large head turn is required and the ability to pay attention to the edge;

[0074] Limb manipulation frequency parameters: Set the number of actions required in the task and the frequency of action switching, and define the interaction density and response requirements.

[0075] The system matches the above four types of environmental parameters with the task path nodes in the task structure diagram one by one, and combines the task content category labels to perform numerical mapping of the operation complexity, perception requirements and response time limit of each node to complete the personalized configuration of the scene.

[0076] The system then generates a training mission configuration file, which optionally includes at least core elements such as a mission structure diagram, a set of environmental parameters, a training target type identifier, and a mission content category table. This configuration file is packaged into a standard mission data format and pushed to the mixed reality device. Upon receiving this configuration file, the mixed reality device generates a 3D training scene based on the mission definition and parameter settings in the configuration file, supporting subsequent data collection of the pilot's training behavior.

[0077] Through the organic connection of the three steps S110, S120 and S130, the present invention realizes a complete process from calling the training task template, constructing the task structure diagram to the parameterized generation of the mixed reality training scene. The entire process provides a unified and interpretable training task map and environmental parameter basis for subsequent behavior collection, ability label extraction, path reasoning and dynamic adjustment, thereby effectively supporting the closed-loop logic of "structure-parameter-behavior-label-feedback" in the entire psychomotor ability training system. The implementation of this module ensures the standardized output and dynamic adaptability of the system's personalized training tasks, and is a key link in the structure of the present invention that connects the previous and the next.

[0078] In some embodiments, the action behavior data includes at least: push-pull operation behavior, hand click behavior, and gesture adjustment interaction behavior; S200 includes at least S210-S230:

[0079] S210. Acquire the pilot's push-pull operation behavior, hand clicking behavior, and posture adjustment interaction behavior in the mixed reality task during training.

[0080] This step enables the action behavior data acquisition module in real time during the mixed reality training process to collect and analyze the operational data generated by the pilot when performing the training mission.

[0081] Specifically, the collection of action behavior data is based on the inertial measurement unit (IMU) and spatial positioning module carried by the mixed reality device, and combined with the hand sensor, handle control device or gesture recognition module to obtain the pilot's interactive behavior in the mixed reality task. Optionally, the key action types collected include at least: push-pull operation behavior, hand click behavior and posture adjustment interaction behavior. Specifically,

[0082] Push / pull operation behavior: indicates performing spatial directional operations such as forward / backward / upward / downward in the mixed reality space;

[0083] Hand click behavior: indicates an operation confirmation event triggered by a specific gesture or device;

[0084] Attitude adjustment behavior: refers to the pilot's head rotation, perspective repositioning, or body posture change behavior.

[0085] Action behavior data must be synchronized with the task flow nodes preset in the training task structure diagram. To this end, the system calls the task node information in the training task configuration file generated in step S130 of S100, performs task matching on the collected operation events, and records the training task number and trigger time corresponding to the operation behavior, thereby dynamically binding the behavior data to the task stage.

[0086] In addition, during the collection process, the system will call the "limb operation frequency parameter" indicator in the environmental configuration parameters set by S130 as a judgment reference to identify whether the current action frequency is within the task target setting range, and record high-frequency operation or delayed operation behavior as a subsequent capability analysis feature indicator.

[0087] The output of this step is a standardized action behavior data sequence, each of which includes at least fields such as behavior type, behavior timestamp, task node number, operation frequency, and execution duration.

[0088] S220: Constructing gaze trajectory information and time interval sequence based on the pilot's eye movement path data and operation reaction time data during the training process.

[0089] In this step, the system obtains the pilot's line of sight change behavior (i.e., eye movement path data) and operation reaction time data through the eye tracking device and the operation trigger recording module.

[0090] First, the system uses an eye-tracking module, either built into the mixed reality device or plugged in, to acquire the pilot's 2D / 3D spatial coordinates in real time. Specifically, this eye-tracking data is recorded frame by frame, reflecting behavioral indicators such as the pilot's gaze focus area, scan path, and gaze duration during the training mission.

[0091] To improve the task relevance of eye movement path data, the system performs boundary screening on the collected gaze paths based on the "visual change range parameter" generated in S130. Specifically, the system sets the spatial range of the target area in the training task and marks whether the pilot's gaze moves within the target area, forming the basis for judging "effective gaze interval" and "deviated gaze behavior." This visual parameter is one of the key indicators defined in the environmental configuration parameters, and its value is derived from the flight mission scenario setting level.

[0092] Secondly, the system concurrently collects action response time data. This data represents the time interval between the pilot receiving a mixed reality task prompt and completing their first action. This data is obtained by comparing the difference between the triggering time of the mixed reality system task prompt frame and the triggering time of the action data. It accurately reflects the pilot's ability to respond to sudden tasks or dynamic operations.

[0093] The output content of this step includes at least: sight track information sequence and time interval sequence. Specifically,

[0094] Gaze trajectory information sequence: including eye movement point coordinates, gaze duration, gaze point density and other fields;

[0095] Time interval sequence: includes the response time difference of each task triggering node, task number and operation time stamp.

[0096] The above output results will serve as input for the S230 behavioral data fusion and as the key basis for generating capability labels such as “gaze reaction time” and “reaction time” in the S300 module.

[0097] S230: Align and integrate the push-pull operation behavior, hand click behavior, posture adjustment interaction behavior, gaze trajectory information, and time interval sequence to construct multimodal original behavior feature data.

[0098] Optionally, after S230 , the method may further include: performing structured storage on the multimodal original behavior feature data.

[0099] This step is the core fusion step of the S200 module. Its goal is to align the task phases, synchronize the time axis, and organize the data of the aforementioned multiple modalities into structures to form a collection of raw behavioral feature data in a unified format.

[0100] Specifically, the system first aligns the action data collected in S210 with the eye movement path data and action reaction time data in S220 based on the training node information in the task structure diagram. For each training task node, the system constructs a "task action unit," integrating all action, gaze, and reaction time data for that task node into a single structure.

[0101] Secondly, the system synchronizes the timeline of multimodal data. Because motion data and eye movement data are sampled at different frequencies, the system uses a unified timestamp as the synchronization benchmark, interpolating and aligning the sampling intervals to ensure that all behavioral data can be accessed based on the three-way relationship of "task node-timeline-data modality."

[0102] Next, the system executes the behavior feature vector construction process. Based on the aligned data set, the system generates original behavior feature entries for each task behavior unit. Optionally, the original behavior feature entries include at least: operation behavior dimension, eye movement behavior dimension and reaction time dimension. Specifically,

[0103] The operation behavior dimensions include at least: operation type, operation amplitude, operation frequency, and average operation duration;

[0104] Eye movement behavior dimensions include at least: average gaze time, scan span, gaze distribution density, and gaze point concentration area;

[0105] The reaction time dimension includes at least: initial operation delay, average reaction time, and extreme reaction interval ratio.

[0106] The above original features do not include any capability labels and are only behavioral performance indicators. In the S300 module, a capability label mapping model will be constructed through fusion analysis.

[0107] Finally, the system stores the multimodal raw behavioral feature data in a structured manner and indexes it according to multi-dimensional labels such as pilot number, training task number, and training time period, for subsequent training evaluation, capability modeling, and training adjustment path reasoning tasks.

[0108] Through the streamlined operation of S210, S220, and S230, the system achieves precise collection and task-aligned integration of pilots' multimodal behaviors during training missions, providing a solid data foundation for subsequent capability modeling and training adjustments. The constructed multimodal raw behavioral feature data not only covers the full range of operational and visual response data, but also structurally binds behavioral performance to the task structure diagram, ensuring the mappability and traceability between behavioral features, capability labels, and task content, effectively supporting the high-precision input and label semantic closed loop of the capability modeling process in the S300 module.

[0109] In some embodiments, S300 includes at least S310-S330:

[0110] S310 , extracting key behavior indicator data from the multimodal original behavior feature data, where the key behavior indicator data at least includes: operation accuracy, reaction time, gaze reaction time, and gaze time.

[0111] This step aims to mathematically analyze the multimodal raw behavioral feature data to construct the basic indicator data upon which psychomotor ability labels are based. This process is based on high-order behavioral functional modeling and temporal feature extraction strategies, primarily involving temporal reconstruction, geometric mapping, and partial derivative mining of action, reaction, and eye movement behaviors.

[0112] Specifically, suppose that at time t, the kth type of multimodal behavior feature data collected is X k (t), where k∈{0,1,2,3}, corresponding to:

[0113] X0(t): represents the sequence of operation accuracy behavior;

[0114] X1(t): represents the action sequence of operation reaction time data;

[0115] X2(t): represents the behavioral sequence of gaze reaction duration;

[0116] X3(t): represents the gaze time behavior sequence.

[0117] In order to further describe the dynamic change characteristics of behavioral features during the training process, the following functional transformation is introduced:

[0118] ①High-order behavioral functional transformation function:

[0119] Ψ k (t) = ∫G(X k (t))·f(t)dt

[0120] in:

[0121] Ψ k (t): represents the dynamic response functional result of the k-th type of behavior characteristics in the time domain;

[0122] X k (t): represents the observed value of the k-th category of original behavior data at time t;

[0123] G(X k (t)):indicates the k (t) Trajectory function after geometric mapping, such as trajectory rotation, scaling, and other transformation results;

[0124] f(t): represents the task prompt function, which activates the behavioral response at a specific moment and represents the weight factor of the behavior during that period;

[0125] t: represents the current time variable, acting on the time integral;

[0126] k: represents the behavior type subscript, with a total of 4 types of behavior characteristics.

[0127] The system applies the above transformation to X0(t) to X3(t), extracts the time domain response features of X0(t) to X3(t), and aligns the i-th training process node N in the task structure diagram. i , and finally form a behavioral indicator quadruple:

[0128]

[0129] Among them, B i Represents training process node N i The set of key behavioral indicators extracted from the above provides a data basis for capability fusion analysis and label generation; Indicates that at training process node N i Above, the dynamic response eigenvalues ​​are obtained after functional modeling of the operation accuracy behavior sequence; Indicates that at training process node N i Above, the dynamic response eigenvalues ​​obtained after functional modeling of the reaction time behavior sequence; Indicates that at training process node N i Above: Dynamic response eigenvalues ​​obtained after functional modeling of the gaze response time series; Indicates that at training process node Ni Above, the dynamic response eigenvalues ​​obtained after functional modeling of the gaze time series.

[0130] S320: Perform a fusion analysis on the operation accuracy, reaction time, gaze reaction time, and gaze time to construct a psychomotor ability label set corresponding to the training task.

[0131] This step is based on the indicator quadruple B extracted in S310 i ,Right now A nonlinear weighted fusion model is used to construct a set of psychomotor ability labels corresponding to training task nodes.

[0132] ② Multimodal fusion capability calculation formula:

[0133]

[0134] in: Represents training process node N i The pilot's overall psychomotor ability score;

[0135] W k : represents the fusion weight of the k-th type of behavior features, satisfying For example, it can be set to W0=0.35, W1=0.25, W2=0.20, and W3=0.20, which are obtained based on empirical fitting of historical training data;

[0136] As a nonlinear mapping function, the example uses the Log-Sigmoid function:

[0137]

[0138] Mapping can standardize functional results to a unified scale and enhance the comparability between different behavioral characteristics.

[0139] The system is based on comprehensive rating Further extract four subdivision dimension capability labels:

[0140] M1 (i) :Indicates operational control capability, mainly derived from

[0141] M2 (i) :Indicates reaction and decision-making ability, mainly derived from

[0142] M3 (i) : Indicates visual perception sensitivity, mainly derived from

[0143] M4 (i) : Indicates gaze stability, mainly derived from

[0144] Constructing psychomotor ability label set T (i) for:

[0145] T (i) ={M1 (i) ,M2 (i) ,M3 (i) ,M4 (i)}

[0146] The label value of each capability label is obtained by weighted decomposition of the corresponding behavioral characteristics and comprehensive score, and serves as the basis for capability label mapping.

[0147] S330: Match the psychomotor ability label set with the training process nodes in the task structure diagram, and construct a mapping relationship between the ability label and the task node to complete the ability-task matching modeling.

[0148] Among them, the training process node is the task node / task process node.

[0149] This step is based on the task structure diagram in S120 and the capability label set T output in S320. (i) ,construct the semantic mapping relationship between the training task nodes and the capability labels, and form a capability-task comparison table that can be used for training scheduling and ,difference analysis.

[0150] ③Capability label mapping function:

[0151]

[0152] in: represents a mapping function that performs label matching based on the task process node number i, node task type, and capability strength score;

[0153] If the training process node N i If the type is "operation class", the matching weight will be given priority to M1 (i) ;

[0154] If the training process node N i If the type is "perception type" or "decision type", then connect to M3 respectively (i) or M2 (i) 、M4 (i) .

[0155] The system combines the task flow logic and capability label semantics in the task structure diagram to construct the following mapping table:

[0156] Map Task ={N i →T (i)}

[0157] Among them, Map Task Represents the task-ability label mapping table, which is used to describe each training task node and the corresponding matching psychomotor ability label set T (i) The corresponding relationship between N i It represents the i-th training process node in the task structure diagram and is the basic unit in the task structure with a clear order and content category.

[0158] It should be noted that the above task-capability label mapping table serves as an input index for difference analysis and path generation in S400, supporting subsequent capability gap identification and training configuration optimization.

[0159] The present invention proposes a behavior-capability modeling mechanism based on high-order functional analysis and multimodal nonlinear fusion in the S310 analysis and path generation module, which has the following significant technical effects:

[0160] The integration of multi-source behavioral characteristics forms a standardized capability label structure, solving the problem of fuzzy capability indicators and unclear structure in traditional training.

[0161] The introduction of nonlinear weight modeling methods makes label expression more interpretable and discriminative, significantly improving the targetedness of training results;

[0162] Build a dynamic mapping mechanism between labels and task nodes to support the structural closed loop of subsequent training path adjustment and capability difference analysis, and realize the data-driven task adjustment logic closed loop.

[0163] In some embodiments, S400 includes at least S410-S430:

[0164] S410 , calculating a difference analysis result based on the ability labels in the psychomotor ability label set and the preset ability standard, where the difference analysis result is a set of difference values ​​between the label values.

[0165] This step is based on the psychomotor ability label set T constructed in S320 (i) ={M1 (i) ,M2 (i) ,M3 (i) ,M4 (i)}, combined with the system preset standard capability tag set T standard ={S1,S2,S3,S4}, calculate the numerical difference between the capabilities of each dimension.

[0166] Specifically, let the kth capability dimension be at task node N i The fusion label value under The corresponding preset standard capability value is The difference measure between the two is expressed as:

[0167] ① Difference value calculation formula:

[0168]

[0169] in:

[0170] Represents task node N i The ability difference value of the k-th ability dimension;

[0171] The actual capability tag value calculated by the S320 module;

[0172] The system sets the standard ability label value based on the training level, which can be specifically set as: 0.60 for elementary level, 0.75 for intermediate level, and 0.90 for advanced level. The value comes from the standard ability evaluation model.

[0173] By performing the above calculations for four dimensions k = 1 to 4, the system can construct a complete capability difference vector:

[0174]

[0175] Difference set D (i) It will serve as the core input indicator for constructing the capability difference map and optimizing the training path.

[0176] S420: Generate a capability difference map based on the difference value set and the weight aggregation function.

[0177] Optionally, after S420 , the method may further include: classifying capability gaps between task types and task requirements.

[0178] This step is to process the capability difference set D output by S410. (i) Perform weighted aggregation processing to generate an overall capability difference strength index for task nodes to support training path scheduling and capability gap classification.

[0179] ② Weight aggregation function of capability difference map:

[0180]

[0181] in:

[0182] Represents the i-th task node N among multiple task nodes i (i.e. training process node N i )’s aggregation ability difference value;

[0183] α k : represents the difference weight of the k-th ability dimension, satisfying For example, α1 = 0.4, α2 = 0.3, α3 = 0.2, and α4 = 0.1 can be set. These difference weights are derived from the empirical weight configuration model;

[0184] The kth label difference value from S410.

[0185] The system will then The capacity gap levels are divided into the following three ranges according to the corresponding numerical values:

[0186] Too low a difference range (e.g. 0.0–0.3): No adjustment necessary;

[0187] Medium difference range (e.g. 0.3–0.6): moderate reinforcement;

[0188] High variance range (e.g. 0.6–1.0): requires key adjustments.

[0189] This clustering basis will serve as the priority basis for the next step of personalized path optimization.

[0190] S430. Generate a personalized training path based on the ability difference map and the training process structure, combined with the difficulty adjustment function.

[0191] In this step, the system calculates the task node capability difference strength based on the task node capability difference strength obtained in S420. In combination with the task node dependency relationship of the training process structure diagram in S120, the training path is optimized and configured, and the training difficulty parameters are dynamically allocated.

[0192] ③Training path difficulty adjustment function:

[0193]

[0194] Among them, λ i : Indicates task node N i Corresponding training difficulty adjustment coefficient;

[0195] when When the value is large, the system assigns a higher training intensity and dynamically adjusts scene parameters such as prompt frequency, task response time limit, interference intensity, etc. through the mixed reality device;

[0196] At the same time, the system follows the node dependencies in the structure diagram and prioritizes task nodes with large capability gaps to improve the targeted nature of training.

[0197] Finally, the system outputs a table of personalized training paths:

[0198] Path personalized ={(N i ,λ i), (i=1,...,N)}

[0199] Among them, Path personalized A table representing a personalized training path is used to describe the personalized training task sequence and its corresponding difficulty adjustment parameters generated based on the ability difference analysis and task structure diagram; the training path table will serve as the core input data structure for task generation and mixed reality device push in S500.

[0200] N represents the total number of task nodes, which is an integer greater than or equal to 2.

[0201] This invention achieves difference analysis and personalized path recommendation based on behavior tag fusion by building a structured ability difference map and a training task process linkage mechanism.

[0202] The difference quantization formula ② is used to achieve the aggregation modeling of multi-label differences, significantly improving the training accuracy and target matching;

[0203] The proposed task difficulty adjustment function ③ can flexibly control the mixed reality training scenario and effectively support the dynamic adjustment of task intensity;

[0204] A mapping path structure between training tasks and capability gaps has been established, which has improved the system's ability to adapt to individual differences and solved the problem of homogeneous and lack of targeted training in traditional systems.

[0205] In some embodiments, S500 includes at least S510-S530:

[0206] S510: Extract task types and capability tag adaptation ranges based on training task node parameters in the personalized training path.

[0207] In this step, the system calls the table of personalized training paths output by module S430 and reads the path node information one by one. Personalized training paths are constructed using task nodes as the basic unit. Each node is associated with a corresponding capability label adjustment factor and training task attributes, including task objectives, task types, training priorities, and scenario constraints.

[0208] Specifically, the system first parses each training task node in the path structure, retrieves the matching task process node number in the corresponding training task structure diagram, and extracts the basic attribute parameters of the task node.

[0209] Optionally, the above basic attribute parameters include at least:

[0210] Task node number;

[0211] Task category (e.g., visual perception, hand-eye coordination, spatial memory, etc.);

[0212] Training mission objectives (e.g., completing specific flight instruction simulations, determining designated indicator responses, etc.);

[0213] Operation frequency and mission duration (derived from the training level setting in S130);

[0214] The set of capability labels corresponding to this node during historical training (supported by the capability label-task node mapping relationship generated by S330);

[0215] For each task node, the system will match the subset of capability labels that the node needs to have.

[0216] Furthermore, the system needs to obtain the difference degree information of the corresponding label dimension in the capability difference map output by the S400 module, and quantify the adaptation degree of the task node capability label. The adaptation degree is divided into preset capability label intervals, and combined with historical training results, the following adaptation mapping structure is established:

[0217] If the capability tag required by the current mission node is basically consistent with the pilot's capability level, it will be marked as "adaptive";

[0218] If a label has a moderate deviation, it is marked as “adjusted”;

[0219] If a label is severely biased, it is marked as "enhanced".

[0220] The system saves the above adaptation results and path nodes synchronously for differential adjustment of task parameter configuration data in the next step.

[0221] This step extracts task node information that is highly correlated with capability status from the training path structure, and constructs a three-layer structure of training task-capability label-adaptability, providing an accurate reference for subsequent scenario generation.

[0222] S520. Based on the capability difference map and the environment configuration parameters, adjust the task type and the capability label adaptation range to generate an adjusted task type and an adjusted capability label adaptation range.

[0223] Based on the task node capability adaptation information extracted in this step, the system further combines the environmental configuration parameters in the training task generated in S130 to make differentiated adjustments to the core parameters of each training task and form the final task scenario configuration data set.

[0224] Specifically, the system first retrieves the environmental parameter information in the previous training task configuration file, including: weather condition parameters, background change speed parameters, visual change range parameters, and limb operation frequency parameters.

[0225] The above four parameters have been associated with the training level and preliminarily configured in S130. In the current step, the system adaptively adjusts these environmental parameters based on the task type and capability adaptation degree of each training node. The adjustment method depends on the label difference degree classification in the capability difference map in S420:

[0226] If the node is in the "enhanced" mark, appropriately increase the background change speed parameter, reduce the visual prompt range, and increase the operation frequency to increase the scene difficulty;

[0227] If the node is in the "regulation" mark, the target level environmental parameter load is maintained, specifically, the medium environmental load is maintained;

[0228] If the node is marked as "adaptive", the original environment configuration parameters are maintained to avoid interfering with normal capability migration.

[0229] The system will also dynamically fine-tune the system based on historical training success rates for similar tasks. For example, if a pilot performs poorly on a "spatial tracking task" during a previous training session, the system will automatically reduce the density of visual clutter elements in that node or adjust the cadence of interactive prompts.

[0230] During this process, the system constructs a three-dimensional parameter mapping structure of "task type - capability difference - environmental adjustment" to ensure the accuracy and adaptability of task scenario generation. Optionally, the final updated task scenario configuration data includes at least the following fields:

[0231] Scene ID;

[0232] Task node number;

[0233] Updated values ​​of four types of environmental parameters;

[0234] Adaptability label and current status;

[0235] Task execution is time-limited;

[0236] Scene background and interaction logic script ID.

[0237] This configuration data structure will be used for the next step of mixed reality scene push and replacement.

[0238] S530: Push the adjusted task type and the adjusted capability tag adaptation range to the mixed reality device.

[0239] In this step, the system receives the updated task scenario configuration data generated by S520 and pushes it to the target device through the mixed reality training control interface.

[0240] Specifically, the system first completes the following operations through the task synchronization module:

[0241] Detect the task status currently running on the mixed reality device;

[0242] If the task is completed or the training is paused, loading a new configuration is allowed;

[0243] If the current task is still in progress, select the wait-for-replacement or interrupt-execution-after-replacement mode according to the system settings.

[0244] Next, the system performs a structured analysis of the configuration data, including:

[0245] Scene background rendering resources loading;

[0246] Interaction logic controls node mapping updates;

[0247] Capability tags hint at the generated text binding of semantics;

[0248] Visual distraction and task response module parameter injection.

[0249] The above operations are all performed based on the standard interface of the mixed reality system SDK.

[0250] After the configuration is loaded, the system will push execution instructions to the mixed reality device, enter a new round of training task process, and initialize the scene operation log structure to facilitate the collection of subsequent training process data (interfacing with the S600 module).

[0251] At the same time, the system stores the mission configuration number, mission type, capability label range and adjustment information into the training record database, forming "mission-label-environment-reaction" structure data for training strategy analysis and capability growth curve modeling.

[0252] Through the implementation of this module, the system has realized an adaptive adjustment mechanism for mission parameter configuration data based on the capability difference map, and combined with environmental configuration parameter data to update the mixed reality mission scene in real time. Compared with traditional training programs with fixed difficulty and path, the dynamic scheduling mechanism of this module effectively improves the adaptability between training tasks and individual capability status, and enhances the responsiveness and personalization of the training system. This module implements the full-process dynamic mission control logic from "ability status identification" to "environmental parameter update" to "training mission scene push", breaking through the problems of rigid mission configuration and difficulty in dynamic intervention in existing technologies, and enhancing the practicality and intelligence level of the system in complex flight training scenarios.

[0253] In some embodiments, S600 includes at least S610-S630:

[0254] S610: Construct a chronologically ordered sequence of ability label changes based on the pilot's psychomotor ability label sets during multiple training cycles.

[0255] This step is performed after completing mission parameter adjustment and dynamic mission scenario push (i.e., S530). After each training cycle, the system extracts the pilot's ability tag set after completing the current mission process from the behavior analysis module. The ability tag set is the training mission capability representation result constructed through S320 based on the fused feature data.

[0256] Specifically, the system first sets a training cycle index, sequentially numbers the capability tag sets for each training cycle, and arranges them into a capability tag change sequence according to the training execution order. The capability tag set includes structured tag items composed of multiple capability dimensions. Each tag item has a task node association identifier, a corresponding indicator dimension, an evaluation value, and a timestamp.

[0257] Furthermore, the system uniformly structures and encodes the capability label values ​​for all pilots across each capability dimension during the training cycle, forming a time series data matrix. The rows of this matrix represent the training cycle sequence, and the columns represent various capability label items, such as operational accuracy, reaction time, gaze response time, and gaze duration. Each cell value in the matrix corresponds to the capability label value for a specific capability dimension of a training node and is annotated with the corresponding timestamp for subsequent module access.

[0258] For example, for the training cycle numbered as round n, the corresponding capability label set is recorded as:

[0259] Capability label sequence {A label sequence, capability label set corresponding to A label sequence i}, where i represents the number of ability dimensions involved in the current training task.

[0260] The capability label change sequence will be directly used as an input parameter for the subsequent capability time series modeling (S620), running through the input process of capability evolution trajectory modeling.

[0261] S620: Perform time series modeling on the capability label change sequence to extract the capability evolution trajectory during the training process.

[0262] After obtaining the chronologically arranged sequence of capability label changes, the system introduces a time series modeling method to identify the capability evolution trajectory of pilots based on the dynamic change trends of different capability label values ​​in multiple consecutive training cycles.

[0263] Optionally, when performing time series modeling on the capability label change sequence in S620, a sliding window mechanism is adopted, and the window length is multiple consecutive training cycles. The evolution speed and fluctuation amplitude of each capability dimension are calculated within the window by difference, slope and curve fitting, and a multi-dimensional capability growth curve map is constructed. The evolution trend is mapped to the corresponding node in the training task structure diagram to analyze the correlation between capability dynamics and task types. The evolution trend is used to characterize the capability evolution trajectory.

[0264] Specifically, the system employs a long- and short-term feature modeling strategy based on a sliding window mechanism. After segmenting the sequence of capability label changes across multiple training cycles into windows, it extracts short-term capability fluctuation trends and long-term evolutionary directions, integrating the synergistic relationships between the various capability dimensions within the window. The sliding window length is set based on the training cadence and capability label sampling frequency, typically spanning multiple consecutive training cycles (e.g., 3 to 5), with window overlap permitted.

[0265] Furthermore, during the modeling process, the system aggregates and calculates statistics such as the rate, magnitude, and frequency of change across multiple capability dimensions. For metrics like operational accuracy and gaze response time, the system calculates their evolutionary speed and fluctuation magnitude through methods such as difference, slope, and curve fitting, thereby constructing a multidimensional curve map of pilot capability evolution. If the label value of a particular capability dimension maintains an upward trend over multiple consecutive rounds, the system identifies it as positive capability growth. If there are significant fluctuations or downward trends over multiple consecutive training cycles, this is marked as a key fluctuation dimension and recorded for training strategy adjustments.

[0266] For example, for the change sequence of the ability label "reaction time":

[0267] {Rization sequence::ization sequence::time" changes, where each item is the label value under the corresponding training cycle,

[0268] The system constructs a trend line for this sequence and performs differential approximation to capture its changing growth rate and fluctuation points. This trend line is used as part of the capability growth curve to reconstruct the capability trajectory.

[0269] Ultimately, the system maps the evolution curves of all capability dimensions uniformly into the pilot training task structure diagram, corresponding to specific task nodes, so as to analyze the corresponding capability growth dynamics with task types.

[0270] S630. Based on the capability evolution trajectory, generate a training process analysis report and an individual capability growth map for subsequent training strategy optimization.

[0271] After completing the capability evolution trajectory modeling, the system enters the training process analysis and capability growth map generation phase. This step, based on the capability evolution results output from S620, conducts individual performance analysis and trend statistics at the task node dimension of the training structure diagram.

[0272] Specifically, the system constructs a structured training process analysis report, which includes the following key contents:

[0273] Table of changes in ability label values ​​under different training cycles;

[0274] Statistics of key capability dimensions with large capability growth;

[0275] Capability degradation trend dimensions and their corresponding task types;

[0276] Judgment of correlation between high volatility dimensions and task difficulty;

[0277] Capability growth node mapping diagram (task flow chart + capability dimension label weight);

[0278] Simulate and deduce personalized adjustment suggestions for the next round of training tasks.

[0279] Based on the generated training process analysis report, the system simultaneously constructs a pilot's individual capability growth graph. This graph uses graphical visualization to present the pilot's growth trajectory across various capability dimensions in a time-series coordinate format, with each capability dimension corresponding to an evolution curve. Nodes in the graph are composed of training cycle numbers and capability labels, while edge weights are a weighted combination of capability growth rate and training task complexity, providing a visual representation of the capability evolution process.

[0280] For example, when generating a graph, the system maps the training cycle number to the horizontal axis coordinate, and the label value of each capability dimension as the vertical axis value, and displays it in the form of a line graph; and automatically adds analysis annotations where the capability fluctuates dramatically, prompting the system whether the capability dimension needs to be retrained.

[0281] Finally, the system packages and outputs the training process analysis report and the capability growth map as input to the training level setting logic and training task template parameter optimization entry of the S100 module, realizing the capability feedback loop and providing the original basis for subsequent adaptive iteration of training strategies.

[0282] By executing the steps described in this module, it is possible to dynamically track and time-series analyze pilots' psychomotor abilities over multiple training cycles, construct a continuous ability evolution trajectory, and output an ability growth map and training process analysis report based on structured data, thereby achieving reverse drive of ability feedback to the training structure and providing a quantitative basis for personalized adjustment of the training process and adaptive control of the training difficulty level.

[0283] The key innovations of the present invention include:

[0284] (1) Constructing a capability-task mapping model based on mixed reality task structure diagrams and multimodal behavior data. Innovatively designing the task structure diagram and the environment configuration parameter construction mechanism, and combining the multimodal raw behavior data to establish a capability label set, the automatic adaptation and dynamic adjustment of training tasks are achieved through the one-to-one mapping of labels to task nodes.

[0285] (2) Integrating the ability difference map and dynamic feedback adjustment algorithm to build a personalized training path. This paper designs a fusion ability difference calculation and task adjustment mechanism, introduces a weighted regression model for dimensions such as operation accuracy and gaze duration, combines environmental parameters to build a training path and adjust task configuration, and realizes the push of dynamically updated mixed reality scenes.

[0286] (3) A temporal capability growth modeling method based on the change sequence of capability labels in the psychomotor capability label set. The present invention constructs a temporally ordered capability label change sequence and introduces a capability evolution trajectory function and a visual graph generation module to achieve full-process tracking, modeling, and report output of training effects, providing a personalized and quantifiable decision support mechanism for pilot training.

[0287] The following are its main beneficial effects:

[0288] (1) Achieve precise adaptation of training tasks to pilots' ability status, and improve training personalization and targeting. The present invention integrates mixed reality environment modeling technology with a multimodal behavioral data acquisition mechanism, constructs psychomotor ability labels based on key indicators such as action behavior data, eye movement path data, and operation reaction time data, and forms an ability-task mapping relationship by matching it with the task structure diagram. Based on the pilot's dynamic ability status, it can automatically generate personalized training paths and task difficulty adjustment plans, effectively avoiding the problems of fixed task intensity and delayed feedback in traditional training.

[0289] (2) Construct a mathematical modeling system that integrates behavioral data to achieve quantifiable evolution tracking of capability status. This paper introduces a high-dimensional feature fusion algorithm and a capability difference calculation function, innovatively uses a weighted regression model and a time-series capability evolution function to model capability labels and conduct dynamic difference analysis. Combining training task nodes with environmental configuration parameters, it promotes closed-loop regulation between task configuration and capability progress, and improves the accuracy of capability growth identification and regulation sensitivity during training.

[0290] (3) Forming an interpretable training effect evaluation map to achieve training decision visualization and closed-loop optimization. The present invention constructs a capability label change sequence in multiple training cycles, extracts the capability growth trajectory based on time series modeling, generates a capability growth map and process analysis report, and provides a visual basis for subsequent training task recommendations and strategy adjustments, significantly improving the tracking transparency and tuning efficiency of training effects.

[0291] The following is a detailed explanation of the psychomotor training system based on mixed reality and behavioral analysis:

[0292] Example 2: Figure 2 FIG. 1 is a schematic diagram showing a structure of a psychomotor ability training system based on mixed reality and behavior analysis according to an embodiment of the present invention. Figure 2 As shown, the structure may include:

[0293] The mixed reality task generation module 10 is used to construct a multi-scenario psychomotor training environment, generate a task structure diagram and environment configuration parameters, and push them to the mixed reality device for scene presentation.

[0294] The data acquisition module 20 is used to collect the pilot's action behavior data, eye movement path data and operation reaction time data during the training process.

[0295] The behavior analysis module 30 is used to construct the multimodal original behavior characteristics based on the action behavior data, the eye movement path data and the operation reaction time data; and to perform a fusion analysis on the multimodal original behavior characteristics, construct a psychomotor ability label set, and associate it with the task structure diagram to complete the ability-task matching modeling.

[0296] The ability reasoning and task planning module 40 is configured to generate an ability difference map for task adjustment based on the difference analysis results between the ability labels in the psychomotor ability label set and the preset ability standards, and a weight aggregation function, and to determine a personalized training path based on the difficulty adjustment function and the ability difference map; the weight aggregation function is: in, Represents the aggregated capability difference value of the i-th task node among multiple task nodes; α k is the difference weight of the k-th ability dimension; represents the ability difference value of the kth ability dimension on the i-th task node; the difficulty adjustment function is: Among them, λ i Indicates the training difficulty adjustment coefficient corresponding to the i-th task node.

[0297] The task scheduling module 50 is used to adjust the task parameter configuration data based on the personalized training path and the ability difference map, and push the adjusted task scene configuration data to the mixed reality device. The adjusted task scene configuration data is used to replace the current training task and execute a new round of training process.

[0298] The training evaluation module 60 is used to record and organize the pilot's psychomotor ability label set during multiple training cycles, perform time series modeling on the psychomotor ability label set, generate an ability evolution trajectory, and generate a training process analysis report and an individual ability growth map based on the ability evolution trajectory.

[0299] Optionally, the task structure diagram includes at least: task process nodes, task logical relationships and task content categories; the environment configuration parameters include at least: weather condition parameters, background change speed parameters, visual change range parameters and limb operation frequency parameters; the mixed reality task generation module 10 is specifically used to obtain training task templates and training target type information, select corresponding training scene types based on flight mission requirements and set training levels; based on the training task structure composition information extracted from the training task template, construct the task process nodes, the task logical relationships and the task content categories; based on the task process nodes, the task logical relationships, the task content categories and the training levels, set the weather condition parameters, the background change speed parameters, the visual change range parameters and the limb operation frequency parameters required for the training scene type, generate a training task configuration file and push it to the mixed reality device for scene presentation.

[0300] Optionally, the behavior analysis module 30 is specifically used to construct gaze trajectory information and a time interval sequence based on the push-pull operation behavior, the hand clicking behavior, and the posture adjustment interaction behavior of the pilot in the mixed reality task during the training process obtained by the data acquisition module 20; based on the eye movement path data and the operation reaction time data of the pilot during the training process obtained by the data acquisition module 20; align and integrate the push-pull operation behavior, the hand clicking behavior, the posture adjustment interaction behavior, the gaze trajectory information and the time interval sequence to construct the multimodal original behavior feature data.

[0301] Optionally, the behavior analysis module 30 is specifically used to extract key behavior indicator data from the multimodal original behavior feature data, and the key behavior indicator data includes at least: operation accuracy, reaction time, gaze reaction time and gaze time; perform a fusion analysis on the operation accuracy, the reaction time, the gaze reaction time and the gaze time to construct a set of psychomotor ability labels corresponding to the training task; match the set of psychomotor ability labels with the training process nodes in the task structure diagram, and construct a mapping relationship between the ability labels and the task nodes to complete the ability-task matching modeling.

[0302] Optionally, the ability reasoning and task planning module 40 is specifically used to calculate the difference analysis result based on the ability label in the psychomotor ability label set and the preset ability standard, and the difference analysis result is a set of difference values ​​between the label values; based on the difference value set and the weight aggregation function, the ability difference map is generated; based on the ability difference map and the training process structure, combined with the difficulty adjustment function, the personalized training path is generated.

[0303] Optionally, the task parameter configuration data includes at least: task type and capability label adaptation range; a task scheduling module 50, specifically used to extract the task type and capability label adaptation range based on the training task node parameters in the personalized training path; based on the capability difference map and the environment configuration parameters, adjust the task type and capability label adaptation range to generate the adjusted task type and the adjusted capability label adaptation range; push the adjusted task type and the adjusted capability label adaptation range to the mixed reality device.

[0304] Optionally, the training evaluation module 60 is specifically configured to construct a chronologically arranged sequence of ability label changes based on the pilot's psychomotor ability label set during the multiple training cycles; perform time series modeling on the ability label change sequence to extract the ability evolution trajectory during the training process; and generate an analysis report of the training process and an individual ability growth map based on the ability evolution trajectory.

[0305] Optionally, when performing time series modeling on the sequence of capability label changes, a sliding window mechanism is adopted, and the window length is multiple consecutive training cycles. The evolution speed and fluctuation amplitude of each capability dimension are calculated within the window by difference, slope and curve fitting, and a multi-dimensional capability growth curve map is constructed. The evolution trend is mapped to the corresponding node in the training task structure diagram to analyze the correlation between capability dynamics and task types. The evolution trend is used to characterize the capability evolution trajectory.

[0306] Optionally, when adjusting the task parameter configuration data based on the personalized training path and the ability difference map, if the i-th task node is marked as "enhancement", at least one of the background change speed parameter, the visual change range parameter, and the limb operation frequency parameter is increased; if the i-th task node is marked as "adjustment", the target level environmental parameter load is maintained; if the i-th task node is marked as "adaptation", the original environmental configuration parameters are retained.

[0307] Beneficial effects of the embodiment:

[0308] High accuracy of multimodal behavior capture: The system can obtain multi-dimensional data such as action behavior data, eye movement path data, and operation reaction time data from mixed reality devices in real time, forming a high-precision behavioral feature foundation and providing data support for subsequent modeling.

[0309] Intelligent capability-task matching: By modeling the mapping relationship between capability labels and training task process nodes, precise adaptation between training tasks and individual capabilities can be achieved.

[0310] Closed-loop training task scheduling: Combining personalized training paths with the aforementioned capability difference map, the system can dynamically adjust task parameters and mixed reality scenarios, ensuring that training content is always aligned with individual capability development.

[0311] Intelligent growth trend modeling: The system can intuitively depict the pilot's ability growth process through continuous modeling of the ability label change sequence, providing intelligent support for subsequent training planning.

[0312] Closed-loop parameter transfer throughout the entire process: Closed-loop transfer of key data such as capability labels, task structure diagrams, capability difference maps, and environmental parameters is achieved between modules to form a complete intelligent training system.

[0313] In summary, the system integrates mixed reality technology and intelligent analysis of behavioral data, breaking through the limitations of traditional flight training evaluation methods. It has broad practical application value and promotion prospects in improving training accuracy, adaptability and intelligent evaluation.

[0314] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A psychomotor ability training method based on mixed reality and behavioral analysis, characterized in that: The following steps are involved: S100: Build a multi-scenario psychomotor training environment, generate a task structure diagram and environment configuration parameters, and push them to a mixed reality device for scene presentation; S200, collecting the pilot's action behavior data, eye movement path data and operation reaction time data during the training process, and constructing a multimodal original behavior feature; wherein the multimodal original behavior feature includes k types of multimodal behavior feature data X at time t k (t), k∈{0,1,2,3}, are the operation accuracy behavior sequence X0(t), operation reaction time data behavior sequence X1(t), gaze reaction duration behavior sequence X2(t) and gaze time behavior sequence X3(t), respectively; S300, performing fusion analysis on the multimodal original behavior features, constructing a psychomotor ability label set, and associating it with the task structure diagram to complete the ability-task matching modeling; wherein, the psychomotor ability label set is T (i) ={M1 (i) ,M2 (i) ,M3 (i) ,M4 (i) }; M1 (i) Indicates operational control capability, derived from Represents the task node N in the task structure diagram i Above, the dynamic response eigenvalues ​​are obtained after functional modeling of the operation accuracy behavior sequence X0(t); M2 (i) Indicates reaction decision-making ability, derived from Indicates that at the task node N i The dynamic response characteristic value obtained by performing functional modeling on the operation reaction time data behavior sequence X1(t); M3 (i) Indicates visual perception sensitivity, derived from Indicates that at the task node N i Above, the dynamic response characteristic value obtained after functional modeling of the gaze reaction duration behavior sequence X2(t); M4 (i) Indicates gaze stability, derived from Indicates that at the task node N i , the dynamic response characteristic value obtained after functional modeling of the gaze time behavior sequence X3(t); S400, generating an ability difference map for task adjustment based on a difference analysis result between the ability labels in the psychomotor ability label set and a preset ability standard, and a weight aggregation function, and determining a personalized training path based on the difficulty adjustment function and the ability difference map; The weight aggregation function is: in, Represents the i-th task node N i The aggregation ability difference value; α k is the difference weight of the k-th ability dimension; Represents the i-th task node N i The ability difference value of the k-th ability dimension; The difficulty adjustment function is: Among them, λ i Represents the i-th task node N i Corresponding training difficulty adjustment coefficient; S500: Adjusting task parameter configuration data based on the personalized training path and the capability difference map, and pushing the adjusted task scenario configuration data to the mixed reality device, wherein the adjusted task scenario configuration data is used to replace the current training task and execute a new round of training process; S600: Record and organize the pilot's psychomotor ability tag sets during multiple training cycles, perform time series modeling on all psychomotor ability tag sets, generate an ability evolution trajectory, and generate a training process analysis report and an individual ability growth map based on the ability evolution trajectory.

2. The training method according to claim 1, characterized in that The task structure diagram includes at least: task nodes, task logical relationships and task content categories; the environmental configuration parameters include at least: weather condition parameters, background change speed parameters, visual change range parameters and limb operation frequency parameters; The step S100 of constructing a multi-scenario psychomotor training environment, generating a task structure diagram and environment configuration parameters, and pushing them to a mixed reality device for scene presentation includes: S110: Obtain training mission template and training target type information, select corresponding training scenario type based on flight mission requirements, and set training level; S120, constructing the task nodes, the task logical relationships, and the task content categories based on the training task structure composition information extracted from the training task template; S130. Based on the task nodes, the task logical relationships, the task content categories, and the training levels, the weather condition parameters, the background change speed parameters, the visual change range parameters, and the limb operation frequency parameters required for the training scene type are set, a training task configuration file is generated, and the configuration file is pushed to the mixed reality device for scene presentation.

3. The training method according to claim 1, characterized in that The action behavior data at least includes: push-pull operation behavior, hand click behavior and posture adjustment interaction behavior; The step S200 of collecting the pilot's action behavior data, eye movement path data, and operation reaction time data during the training process to construct multimodal original behavior features includes: S210, acquiring the push-pull operation behavior, the hand clicking behavior, and the posture adjustment interaction behavior of the pilot in the mixed reality task during the training process; S220, constructing gaze trajectory information and time interval sequence based on the pilot's eye movement path data and the operation reaction time data during the training process; S230: Align and integrate the push-pull operation behavior, the hand clicking behavior, the posture adjustment interaction behavior, the eye tracking information, and the time interval sequence to construct the multimodal original behavior feature data.

4. The training method according to claim 2, characterized in that S300, performing a fusion analysis on the multimodal original behavioral features, constructing a psychomotor ability label set, and associating it with the task structure diagram to complete ability-task matching modeling, includes: S310, extracting key behavior indicator data from the multimodal original behavior feature data, the key behavior indicator data at least including: operation accuracy, reaction time, gaze reaction time and gaze time; S320, performing a fusion analysis on the operation accuracy, the reaction time, the gaze reaction time, and the gaze time to construct a psychomotor ability label set corresponding to the training task; S330 , matching the psychomotor ability label set with the task nodes in the task structure diagram, and constructing a mapping relationship between the ability labels and the task nodes to complete ability-task matching modeling.

5. The training method according to claim 1, wherein: The step S400, generating an ability difference map for task adjustment based on a difference analysis result between the ability labels in the psychomotor ability label set and a preset ability standard, and a weight aggregation function, and determining a personalized training path based on the difficulty adjustment function and the ability difference map, includes: S410, calculating a difference analysis result based on the ability labels in the psychomotor ability label set and the preset ability standard, the difference analysis result being a set of difference numerical values ​​between label values; S420: Generate the capability difference map according to the difference value set and the weight aggregation function; S430: Based on the ability difference map and the training process structure, combined with the difficulty adjustment function, generate the personalized training path.

6. The training method according to claim 1, characterized in that: The task parameter configuration data includes at least: task type and capability tag adaptation range; The step S500 of adjusting task parameter configuration data based on the personalized training path and the capability difference map, and pushing the adjusted task scenario configuration data to the mixed reality device, includes: S510: Extracting the task type and the capability tag adaptation range based on the training task node parameters in the personalized training path; S520: Based on the capability difference map and the environment configuration parameters, adjust the task type and the capability tag adaptation range to generate an adjusted task type and an adjusted capability tag adaptation range; S530: Push the adjusted task type and the adjusted capability tag adaptation range to the mixed reality device.

7. The training method according to claim 1, characterized in that S600 records and organizes the pilot's psychomotor ability tag sets during multiple training cycles, performs time series modeling on all psychomotor ability tag sets, generates an ability evolution trajectory, and generates a training process analysis report and an individual ability growth map based on the ability evolution trajectory, including: S610: constructing a chronologically arranged ability label change sequence based on the pilot's psychomotor ability label set during the multiple training cycles; S620: Perform time series modeling on the capability label change sequence to extract the capability evolution trajectory during the training process; S630: Generate the training process analysis report and the individual capability growth graph based on the capability evolution trajectory.

8. The training method according to claim 7, characterized in that: When performing time series modeling on the capability label change sequence in S620, a sliding window mechanism is adopted, and the window length is multiple consecutive training cycles. The evolution speed and fluctuation amplitude of each capability dimension are calculated within the window by difference, slope and curve fitting, and a multi-dimensional capability growth curve map is constructed. The evolution trend is mapped to the corresponding node in the training task structure diagram to analyze the correlation between capability dynamics and task types. The evolution trend is used to characterize the capability evolution trajectory.

9. The training method according to claim 1, characterized in that When adjusting the task parameter configuration data based on the personalized training path and the ability difference map in S500, if the i-th task node N i Marked as enhanced, at least one of the parameters of background change speed, visual change range reduction and limb operation frequency is increased; if the i-th task node N i Marked as adjustment, the target level environmental parameter load is maintained; if the i-th task node N i If marked as adapted, the original environment configuration parameters are retained.

10. A psychomotor ability training system based on mixed reality and behavior analysis, characterized in that: include: The mixed reality task generation module is used to build a multi-scenario psychomotor training environment, generate task structure diagrams and environment configuration parameters, and push them to the mixed reality device for scene presentation; The data acquisition module is used to collect the pilot's action behavior data, eye movement path data and operation reaction time data during the training process; The behavior analysis module is used to construct a multimodal original behavior feature based on the action behavior data, the eye movement path data and the operation reaction time data; wherein the multimodal original behavior feature includes k types of multimodal behavior feature data X at time t k (t), k∈{0,1,2,3}, which are respectively the operation accuracy behavior sequence X0(t), the operation reaction time data behavior sequence X1(t), the gaze reaction duration behavior sequence X2(t) and the gaze time behavior sequence X3(t); and the multimodal original behavior features are fused and analyzed to construct a psychomotor ability label set, and the task structure diagram is associated to complete the ability-task matching modeling; wherein, the psychomotor ability label set is T (i) ={M1 (i) ,M2 (i) ,M3 (i) ,M4 (i) };M1 (i) Indicates operational control capability, derived from Represents the task node N in the task structure diagram i The dynamic response eigenvalues ​​obtained by functional modeling of the operation accuracy behavior sequence X0(t); M2 (i) Indicates reaction decision-making ability, derived from Indicates that at the task node N i The dynamic response characteristic value obtained by functional modeling the operation reaction time data behavior sequence X1(t); M3 (i) Indicates visual perception sensitivity, derived from Indicates that at the task node N i The dynamic response characteristic value obtained after functional modeling of the gaze reaction duration behavior sequence X2(t); M4 (i) Indicates gaze stability, derived from Indicates that at the task node N i , the dynamic response characteristic value obtained after functional modeling of the gaze time behavior sequence X3(t); The ability reasoning and task planning module is used to generate an ability difference map for task adjustment based on the difference analysis results between the ability labels in the psychomotor ability label set and the preset ability standards, and a weight aggregation function, and determine a personalized training path based on the difficulty adjustment function and the ability difference map; the weight aggregation function is: in, Represents the i-th task node N i The aggregation ability difference value; α k is the difference weight of the k-th ability dimension; Represents the i-th task node N i The ability difference value of the k-th ability dimension above; the difficulty adjustment function is: Among them, λ i Represents the i-th task node N i Corresponding training difficulty adjustment coefficient; A task scheduling module, configured to adjust task parameter configuration data based on the personalized training path and the capability difference map, and push the adjusted task scenario configuration data to the mixed reality device, wherein the adjusted task scenario configuration data is used to replace the current training task and execute a new round of training process; The training evaluation module is used to record and organize the pilots' psychomotor ability label sets over multiple training cycles, perform time series modeling on all psychomotor ability label sets, generate ability evolution trajectories, and based on the ability evolution trajectories, generate training process analysis reports and individual ability growth maps.

Images