Psychological exercise ability training method and system based on mixed reality and behavior analysis

Abstract

The invention relates to the field of man-machine interaction training, virtual reality and intelligent evaluation, and particularly discloses a psychological exercise ability training method and system based on mixed reality and behavior analysis. The method comprises the steps that a multi-scene psychological training environment is constructed and presented in mixed reality equipment, action behavior data, eye movement path data and operation response time data of a pilot in the training process are collected, multi-modal original behavior characteristics are constructed, and then a psychological exercise ability label set is constructed; and realizing capability-task modeling through matching of a capability label and a task structure, generating a personalized training path and a capability difference atlas based on difference analysis with a preset capability standard, adjusting task parameter configuration data, dynamically updating a training scene, generating a capability evolution trajectory, and outputting a training analysis result. In this way, accurate ability modeling, task dynamic adaptation and visual tracking of the training effect can be achieved, and the individuation and intelligence level of psychological exercise ability training is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the fields of human-computer interaction training, virtual reality, and intelligent evaluation, and particularly to a method and system for training psychomotor ability based on mixed reality and behavior analysis. Background Art

[0002] With the complication of the modern combat mission environment and the continuous increase of the flight operation load, the psychomotor ability required by pilots during mission execution (including at least operation reaction speed, visual attention control, action coordination, etc.) has become a key indicator to measure their mission adaptability and execution efficiency. In the prior art, on the one hand, static simulation training or single-behavior feature collection methods are mostly used, which are difficult to effectively combine the task structure and multi-modal behavior data for ability modeling and training evaluation, and there are problems such as poor training scenario adaptability, coarse ability modeling granularity, and lack of personalization in training path formulation.

[0003] On the other hand, although mixed reality technology has been gradually applied to the field of flight training, current systems mostly focus on the immersive reproduction of operation skills, and have not yet formed a dynamic ability evaluation mechanism driven by behavior data, nor a system closed-loop design such as associative analysis of behavior characteristics and task structure, modeling of ability evolution trends, and personalized task adjustment, resulting in lack of pertinence in training feedback and difficulty in supporting the whole-process tracking and optimization of individual ability growth.

[0004] Therefore, there is an urgent need to propose a method and system for training psychomotor ability that integrates multi-modal behavior feature collection and in-depth analysis, and combines the dynamic adjustment of the mixed reality environment, so as to realize the personalization, intelligence, and visualization of the training process, and provide more scientific and efficient technical support for pilot ability improvement and mission adaptability training. Summary of the Invention

[0005] The present invention provides a method and system for training psychomotor ability based on mixed reality and behavior analysis, so as to solve the problem of how to realize the accurate modeling of pilots' psychomotor ability, the generation of personalized training paths, and the evaluation of growth trajectories by fusing and analyzing multi-modal behavior data (including action behavior data, eye movement path data, and operation reaction time data), and combining the dynamic adaptation mechanism of the mixed reality training environment.

[0006] To solve the above technical problems, In a first aspect, the present invention provides a method for training psychomotor ability based on mixed reality and behavior analysis, including: S100. Construct a multi-scene psychomotor training environment, generate a task structure diagram and environment configuration parameters, and push them to the mixed reality device for scene presentation; S200. Collect the action behavior data, eye movement path data, and operation reaction time data of the pilot during the training process, and construct multi-modal original behavior features; S300. Conduct fusion analysis on the multi-modal original behavior features, construct a set of psychomotor ability labels, and associate the task structure diagram to complete the ability-task matching modeling; S400. Based on the difference analysis results between the ability labels in the set of psychomotor ability labels and the preset ability standards, and the weight aggregation function, generate an ability difference map for task adjustment, and determine a personalized training path based on the difficulty adjustment function and the ability difference map; The weight aggregation function is: ; where represents the aggregated ability difference value of the i-th task node among multiple task nodes; is the difference weight of the -th ability dimension; represents the ability difference value of the -th ability dimension on the i-th task node; The difficulty adjustment function is: where represents the training difficulty adjustment coefficient corresponding to the i-th task node; S500. Based on the personalized training path and the ability difference map, adjust the task parameter configuration data, and push the adjusted task scenario configuration data to the mixed reality device. The adjusted task scenario configuration data is used to replace the current training task and execute a new round of training process; S600. Record and organize the set of psychomotor ability labels of the pilot in multiple training cycles, perform temporal modeling on all sets of psychomotor ability labels to generate an ability evolution trajectory, and generate a training process analysis report and an individual ability growth map based on the ability evolution trajectory.

[0007] Further, the task structure diagram at least includes: task process nodes, task logical relationships, and task content categories; the environment configuration parameters at least include: weather condition parameters, background change speed parameters, visual change range parameters, and limb operation frequency parameters; the S100, constructing a multi-scenario psychomotor training environment, generating a task structure diagram and environment configuration parameters, and pushing them to the mixed reality device for scenario presentation, includes: S110, obtaining a training task template and training target type information, selecting a corresponding training scenario type based on flight task requirements and setting a training level; S120, constructing the task process nodes, the task logical relationships, and the task content categories based on the training task structure composition information extracted from the training task template; S130, setting the weather condition parameters, the background change speed parameters, the visual change range parameters, and the limb operation frequency parameters required for the training scenario type based on the task process nodes, the task logical relationships, the task content categories, and the training level, generating a training task configuration file, and pushing it to the mixed reality device for scenario presentation.

[0008] Further, the action behavior data at least includes: push-pull operation behavior, hand click behavior, and posture adjustment interaction behavior; the S200, collecting the action behavior data, eye movement path data, and operation reaction time data of the pilot during the training process, and constructing multi-modal raw behavior characteristics, includes: S210, obtaining the push-pull operation behavior, the hand click behavior, and the posture adjustment interaction behavior of the pilot in the mixed reality task during the training process; S220, constructing line-of-sight trajectory information and time interval sequence based on the eye movement path data and the operation reaction time data of the pilot during the training process; S230, aligning and integrating the push-pull operation behavior, the hand click behavior, the posture adjustment interaction behavior, the line-of-sight trajectory information, and the time interval sequence to construct the multi-modal raw behavior characteristic data.

[0009] Further, the S300, performing fusion analysis on the multi-modal raw behavior characteristics, constructing a psychomotor ability label set, and associating it with the task structure diagram to complete the ability-task matching modeling, includes: S310, extracting key behavior index data from the multi-modal raw behavior characteristic data, and the key behavior index data at least includes: operation accuracy, reaction time, fixation reaction time, and gaze time; S320, performing fusion analysis on the operation accuracy, the reaction time, the fixation reaction time, and the gaze time to construct a psychomotor ability label set corresponding to the training task; S330, matching the psychomotor ability label set with the training process nodes in the task structure diagram to construct a mapping relationship between the ability labels and the task nodes to complete the ability-task matching modeling.

[0010] Further, the S400 generates an ability difference map for task adjustment based on the analysis result of the difference between the ability labels in the psychomotor ability label set and the preset ability standard, and the weight aggregation function, and determines a personalized training path based on the difficulty adjustment function and the ability difference map, including: S410 calculates the analysis result of the difference based on the ability labels in the psychomotor ability label set and the preset ability standard, and the analysis result of the difference is a set of difference numerical values between the label values; S420 generates the ability difference map according to the set of difference numerical values and the weight aggregation function; S430 generates the personalized training path based on the ability difference map and the training process structure, in combination with the difficulty adjustment function.

[0011] Further, the task parameter configuration data at least includes: the task type and the adaptation range of the ability label; the S500 adjusts the task parameter configuration data based on the personalized training path and the ability difference map, and pushes the adjusted task scenario configuration data to the mixed reality device, including: S510 extracts the task type and the adaptation range of the ability label based on the training task node parameters in the personalized training path; S520 adjusts the task type and the adaptation range of the ability label based on the ability difference map and the environment configuration parameters to generate an adjusted task type and an adjusted adaptation range of the ability label; S530 pushes the adjusted task type and the adjusted adaptation range of the ability label to the mixed reality device.

[0012] Further, the S600 records and organizes the psychomotor ability label sets of the pilot in multiple training cycles, performs time series modeling on all the psychomotor ability label sets to generate an ability evolution trajectory, and generates a training process analysis report and an individual ability growth map based on the ability evolution trajectory, including: S610 constructs a sequence of ability label changes arranged in chronological order based on the psychomotor ability label sets of the pilot in the multiple training cycles; S620 performs time series modeling on the sequence of ability label changes to extract the ability evolution trajectory during the training process; S630 generates the training process analysis report and the individual ability growth map based on the ability evolution trajectory.

[0013] Further, when performing time series modeling on the sequence of ability label changes in S620, a sliding window mechanism is adopted, the window length is multiple consecutive training cycles, and the evolution speed and fluctuation amplitude of each ability dimension are calculated through difference, slope, and curve fitting within the window to construct a multi-dimensional ability growth curve map, and the evolution trend is mapped to the corresponding nodes in the training task structure diagram to analyze the correlation between the ability dynamics and the task type, and the evolution trend is used to characterize the ability evolution trajectory.

[0014] Further, when adjusting the task parameter configuration data based on the personalized training path and the ability difference map in S500, if the i-th task node is marked as "strengthen", at least one of the background change speed parameter, the visual change range parameter, and the limb operation frequency parameter is increased; if the i-th task node is marked as "adjust", the target level environment parameter load is maintained; if the i-th task node is marked as "adapt", the original environment configuration parameters are retained.

[0015] In a second aspect, the present invention provides a psychomotor ability training system based on mixed reality and behavior analysis, including: A mixed reality task generation module, configured to construct a multi-scenario psychomotor training environment, generate a task structure diagram and environment configuration parameters, and push them to a mixed reality device for scenario presentation; A data collection module, configured to collect action behavior data, eye movement path data, and operation reaction time data of a pilot during training; A behavior analysis module, configured to construct the multi-modal original behavior features based on the action behavior data, the eye movement path data, and the operation reaction time data; and perform fusion analysis on the multi-modal original behavior features, construct a psychomotor ability label set, and associate the task structure diagram to complete the ability-task matching modeling; An ability reasoning and task planning module, configured to generate an ability difference map for task adjustment based on the difference analysis result between the ability labels in the psychomotor ability label set and the preset ability standards, and a weight aggregation function, and determine a personalized training path based on a difficulty adjustment function and the ability difference map; the weight aggregation function is: ; where represents the aggregated ability difference value of the i-th task node among multiple task nodes; is the difference weight of the -th ability dimension; represents the ability difference value of the -th ability dimension on the i-th task node; the difficulty adjustment function is: ; where represents the training difficulty adjustment coefficient corresponding to the i-th task node; A task scheduling module, configured to adjust the task parameter configuration data based on the personalized training path and the ability difference map, and push the adjusted task scenario configuration data to the mixed reality device, and the adjusted task scenario configuration data is used to replace the current training task and execute a new round of training process; The training and evaluation module is used to record and organize the set of psychomotor ability labels of pilots in multiple training cycles, perform time-series modeling on all sets of psychomotor ability labels, generate an ability evolution trajectory, and generate an analysis report on the training process and an individual ability growth map based on the ability evolution trajectory.

[0016] The following are its main beneficial effects: (1) Achieve the precise adaptation of training tasks to the ability status of pilots, and improve the personalization and targeting of training. The present invention integrates the mixed reality environment modeling technology and the multi-modal behavior data acquisition mechanism, constructs psychomotor ability labels based on key indicators such as action behavior data, eye movement path data, and operation reaction time data, and forms an ability-task mapping relationship by matching with the task structure diagram. It can automatically generate personalized training paths and task difficulty adjustment schemes according to the dynamic ability status of pilots, effectively avoiding problems such as fixed task intensity and lagging feedback in traditional training.

[0017] (2) Construct a mathematical modeling system integrating behavior data to achieve quantifiable evolution tracking of ability status.

[0018] The present invention introduces a high-dimensional feature fusion algorithm and an ability difference calculation function, and innovatively uses a weighted regression model and a time-series ability evolution function to model and dynamically analyze the differences of ability labels. Combining training task nodes and environmental configuration parameters, it promotes the formation of a closed-loop adjustment between task configuration and ability progress, and improves the accuracy of ability growth recognition and the adjustment sensitivity during the training process.

[0019] (3) Form an interpretable training effect evaluation map to achieve visualization and closed-loop optimization of training decisions The present invention constructs a sequence of changes in ability labels in multiple training cycles, extracts the ability growth trajectory based on time-series modeling, generates an ability growth map and an analysis report on the process, provides a visual basis for subsequent training task recommendation and strategy adjustment, and significantly improves the tracking transparency and optimization efficiency of training effects. Description of the Drawings

[0020] Figure 1 It is a schematic flow chart of a psychomotor ability training method based on mixed reality and behavior analysis provided by an embodiment of the present application; Figure 2 It is a schematic structural diagram of a psychomotor ability training system based on mixed reality and behavior analysis provided by an embodiment of the present application. Detailed Embodiments

[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0022] The following takes the psychological motor ability training system based on mixed reality and behavior analysis (abbreviation: system) as the execution subject, and elaborates in detail on the psychological motor ability training method provided by the embodiments of the present invention: Example 1: Refer to Figure 1 , which is a schematic flowchart of the psychological motor ability training method provided by the embodiments of the present invention. This method may at least include S100 - S600: S100. Construct a multi - scenario psychological motor training environment, generate a task structure diagram and environment configuration parameters, and push them to the mixed reality device for scene presentation.

[0023] S200. Collect the action behavior data, eye movement path data, and operation reaction time data of the pilot during the training process, and construct multi - modal original behavior features.

[0024] S300. Conduct fusion analysis on the multi - modal original behavior features, construct a set of psychological motor ability labels, and associate them with the task structure diagram to complete the ability - task matching modeling.

[0025] S400. Based on the difference analysis results between the ability labels in the set of psychological motor ability labels and the preset ability standards, as well as the weight aggregation function, generate an ability difference map for task adjustment, and determine a personalized training path based on the difficulty adjustment function and the ability difference map.

[0026] S500. Based on the personalized training path and the ability difference map, adjust the task parameter configuration data, and push the adjusted task scenario configuration data to the mixed reality device. The adjusted task scenario configuration data is used to replace the current training task and execute a new round of training process.

[0027] S600. Record and organize the set of psychological motor ability labels of the pilot in multiple training cycles, conduct temporal modeling on the set of psychological motor ability labels, generate an ability evolution trajectory, and generate an analysis report of the training process and an individual ability growth map based on the ability evolution trajectory.

[0028] Among them, the above - mentioned weight aggregation function is: ; It represents the aggregation ability difference value of the i-th task node among multiple task nodes; is the difference weight for the th ability dimension; It represents the ability difference value of the th ability dimension on the i-th task node; The above difficulty adjustment function is: It represents the training difficulty adjustment coefficient corresponding to the i-th task node.

[0029] In some embodiments, the task structure diagram at least includes: task flow nodes, task logical relationships, and task content categories; the environmental configuration parameters at least include: weather condition parameters, background change speed parameters, visual change range parameters, and limb operation frequency parameters; the above S100 at least includes S110 - S130: S110. Obtain the training task template and training target type information, and select the corresponding training scenario type based on the flight task requirements and set the training level.

[0030] In this step, the system first connects to the training task database to obtain the standardized training task templates corresponding to the pilot during the training phase. The number of these training task templates is at least one, and a training task template dataset can be constructed. Optionally, the training task template at least includes: task target type, task execution elements, task control logic, and task evaluation elements, etc. Then, the system can retrieve the required standardized training task templates from these four types of standardized training task templates: flight ability maintenance type, stress response type, multi-task switching type, and perception and interpretation type as needed.

[0031] Specifically, the system receives the training purpose configuration file input by the user. Optionally, the configuration file at least includes content such as the flight task phase (such as takeoff, stable flight, landing), the current ability evaluation score, and the recommended training dimension, etc. Then, the system automatically matches the target type of the above training task template according to the flight task phase in the configuration file, and extracts the standard task category associated with the training task template. Further, the system can select the scenario type suitable for the current training cycle. Optionally, the scenario type at least includes feature dimensions such as scenario abstraction level (highly abstract / low abstract), interaction complexity (single task / multi-task), and physical perception channel (vision-dominated / cross-sensory), etc.

[0032] Subsequently, the system sets the training level (abbreviation: training level) according to the pilot's training level, past training data, and task recommendation tags. The setting result of the training level will be used in the environmental parameter configuration of S130 as one of the input conditions for scene difficulty control.

[0033] This step outputs three items: the training task template dataset, the training target type identifier, and the set training level, which are passed as inputs to step S120 for constructing the task structure diagram.

[0034] S120. Based on the training task structure composition information extracted from the training task template, construct task flow nodes, task logical relationships, and task content categories.

[0035] Continuing with the training task template and training target type generated in S110, this step performs a structural analysis of the internal structure of the training task template, extracts task flow elements (i.e., task flow nodes, task logical relationships, and task content categories), and constructs a task structure diagram.

[0036] It should be noted that the above task structure diagram is a directed task execution diagram, where each node in the task structure diagram represents a task step, and each edge represents the front-back logical order between tasks.

[0037] In this section, first, the system analyzes the task phase information in the training task template, identifies and divides task flow nodes, such as typical node contents like "identify target identifier - perform simulation operation - complete visual confirmation - execute composite action - restore reference state". Each task node will be labeled with task function types (perception, decision-making, operation), associated task goals (such as visual accuracy, limb control), and the required user response time limit, etc.

[0038] Secondly, based on the time logic, causal trigger relationship, and task fusion requirements between nodes, the system configures the task path order in the task structure diagram to form a complete task flow diagram structure, and further labels task logical relationships, such as relationship patterns like "parallel execution", "serial switch", "conditional jump", etc.

[0039] Finally, the system classifies the task content of each node into types, generates task content category labels, and uses them as the basis for subsequent matching of ability labels. Task content categories cover operation tasks (such as hand movements, posture changes), reaction tasks (such as button confirmation, steering adjustment), and decision-making tasks (such as target judgment, information screening), etc.

[0040] The output results of this step include the task structure diagram, as well as the task flow node definition table, the task logical topology diagram, and the task content category label table, etc. Among them, the above task structure diagram will be directly used for the matching of the psychomotor ability label set and the training process nodes in the S300 module, and the task content category labels will be used in S400 for ability difference classification and attribution to form a task-ability multi-level label mapping relationship.

[0041] S130. Set the weather condition parameters, background change speed parameters, visual change range parameters, and limb operation frequency parameters required for the training scenario type based on the task process nodes, task logical relationships, task content categories, and training levels. Generate a training task configuration file and push it to the mixed reality device for scenario presentation.

[0042] Specifically, the system calls the environmental parameter model matching the training level according to the training level and extracts the control parameter indicators corresponding to that level. Based on the training objectives and ability types, this model defines the following four types of environmental parameters for each training level, namely the above-mentioned weather condition parameters, background change speed parameters, visual change range parameters, and limb operation frequency parameters. Specifically, The weather condition parameters include at least visual clarity, brightness contrast, and weather type (such as sunny, rainy, night, etc.); Background change speed parameter: Controls the background visual flow rate and is used to simulate different flight speeds or interference states during flight; Visual change range parameter: Used to adjust the range of the visual attention area during training, such as whether it is necessary to turn the head significantly and the edge attention ability; Limb operation frequency parameter: Sets the number of actions to be operated and the action switching frequency in the task, and defines the interaction density and response requirements.

[0043] The system maps each of the above four types of environmental parameters to the task path nodes in the task structure diagram one by one, and combines the task content category labels to numerically map the operation complexity, perception requirements, and response time limit of each node to complete the scene personalized configuration.

[0044] Subsequently, the system generates a training task configuration file. Optionally, this configuration file includes at least core elements such as the task structure diagram, environmental parameter set, training objective type identifier, and task content category table, and is packaged into a standard task data format and pushed to the mixed reality device. After receiving this configuration file, the mixed reality device generates a three-dimensional training scene according to the task definition and parameter settings in the configuration file to support the data collection of subsequent pilot training behaviors.

[0045] Through the organic connection of these three steps, S110, S120, and S130, the present invention realizes the complete process from the invocation of the training task template, the construction of the task structure diagram to the parametric generation of the mixed reality training scene. The entire process provides a unified and interpretable training task graph and environmental parameter basis for subsequent behavior collection, ability label extraction, path reasoning, and dynamic adjustment, thereby effectively supporting the closed-loop logic of "structure - parameter - behavior - label - feedback" in the entire psychomotor ability training system. The implementation of this module ensures the standardized output and dynamic adaptation ability of the system's personalized training tasks and is a key link connecting the upper and lower parts in the structure of the present invention.

[0046] In some embodiments, the action behavior data at least includes: push-pull operation behavior, hand click behavior, and posture adjustment interaction behavior; S200 at least includes S210 - S230: S210. Obtain the push-pull operation behavior, hand click behavior, and posture adjustment interaction behavior of the pilot during the training process in the mixed reality task.

[0047] In this step, during the mixed reality training process, the action behavior data acquisition module is enabled in real time to collect and analyze the operation data generated by the pilot when performing the training task.

[0048] Specifically, the acquisition of the action behavior data is based on the inertial measurement unit (IMU) and spatial positioning module carried by the mixed reality device, and in combination with the hand sensor, handle control device, or gesture recognition module, to obtain the interaction behavior of the pilot in the mixed reality task. Optionally, the at least key action types collected include: push-pull operation behavior, hand click behavior, and posture adjustment interaction behavior. Specifically, Push-pull operation behavior: Represents performing spatial direction operations such as forward / backward / lifting / pressing down in the mixed reality space; Hand click behavior: Represents an operation confirmation event triggered by a specific posture or device; Posture adjustment behavior: Represents the behavior of the pilot performing head rotation, perspective repositioning, or body posture change.

[0049] The action behavior data needs to be synchronously marked in combination with the task flow nodes preset in the training task structure diagram. For this purpose, the system calls the task node information in the training task configuration file generated in step S130 of S100 to perform task matching on the collected operation events, and records the training task number and trigger moment corresponding to the operation behavior, so as to realize the dynamic binding of the behavior data and the task phase.

[0050] In addition, during the acquisition process, the system will call the "limb operation frequency parameter" index in the environment configuration parameters set in S130 as a judgment reference to identify whether the current action frequency is within the task target setting range, and record the high-frequency operation or lag operation behavior as a subsequent ability analysis characteristic index.

[0051] The operation output of this step is a standardized structured action behavior data sequence, and each piece of data at least includes fields such as behavior type, behavior timestamp, task node number, operation frequency, and execution duration.

[0052] S220. Based on the eye movement path data and operation reaction time data of the pilot during the training process, construct the line-of-sight trajectory information and time interval sequence.

[0053] In this step, the system obtains the pilot's line-of-sight change behavior (i.e., eye movement path data) and operation reaction time data through an eye tracking device and an operation trigger recording module respectively.

[0054] First, the system obtains the two-dimensional / three-dimensional spatial coordinate information of the pilot's line of sight in real time through the eye tracking module built in or externally attached to the mixed reality device. Specifically, the above eye movement path data is recorded frame by frame, reflecting behavioral indicators such as the line-of-sight concentration area, saccade path, and fixation duration of the pilot during the execution of the training task.

[0055] To improve the task relevance of the eye movement path data, the system performs boundary screening on the collected line-of-sight trajectories based on the "visual change range parameter" generated by S130. Specifically, the system sets the spatial range of the target area in the training task and marks whether the pilot's line of sight moves within the target area, forming the basis for judging the "effective fixation interval" and "off-line-of-sight behavior". This visual parameter is one of the key indicators defined in the environmental configuration parameters, and its value comes from the flight task scenario setting level.

[0056] Second, the system collects operation reaction time data in parallel. The operation reaction time data refers to the time interval between the pilot receiving the mixed reality task prompt and completing the first operation behavior. This data is obtained by comparing the time difference between the task prompt frame trigger time of the mixed reality system and the action behavior data trigger time, accurately reflecting the pilot's response ability to sudden tasks or dynamic operations.

[0057] The output content of this step includes at least: the line-of-sight trajectory information sequence and the time interval sequence. Specifically, Line-of-sight trajectory information sequence: includes fields such as eye movement point coordinates, fixation duration, and fixation point density; Time interval sequence: includes the response time difference, task number, and operation time mark of each task trigger node.

[0058] The above output results will be used as the input items for the S230 behavior data fusion and serve as the key basis for generating ability labels such as "gaze reaction time" and "reaction time" in the S300 module.

[0059] S230 aligns and integrates the push-pull operation behavior, hand click behavior, posture adjustment interaction behavior, line-of-sight trajectory information, and time interval sequence to construct multi-modal raw behavior feature data.

[0060] Optionally, after S230, the method may further include: structurally storing the multi-modal raw behavior feature data.

[0061] This step is the core fusion step of the S200 module, aiming to align the task phases, synchronize the time axes, and structurally organize the data of the aforementioned multiple modalities to form a set of original behavioral feature data in a unified format.

[0062] Specifically, the system first aligns the action behavior data collected in S210, the eye movement path data and the operation reaction time data in S220 in terms of task nodes according to the training node information in the task structure diagram. For each training task node, the system constructs a "task behavior unit" and integrates all the operation behaviors, sight behaviors, and reaction time data of this task node into the same structure.

[0063] Secondly, the system synchronizes the time axes of the multi-modal data. Since the sampling frequencies of the action data and the eye movement data are different, the system uses a unified timestamp as the synchronization benchmark and performs interpolation and alignment operations on the sampling intervals to ensure that all behavioral data can be called based on the ternary relationship of "task node - time axis - data modality".

[0064] Then, the system executes the behavioral feature vector construction process. Based on the aligned dataset, the system generates original behavioral feature entries for each task behavior unit. Optionally, the above original behavioral feature entries at least include: operation behavior dimension, eye movement behavior dimension, reaction time dimension, etc. Specifically, The operation behavior dimension at least includes: operation type, operation amplitude, operation frequency, average action duration; The eye movement behavior dimension at least includes: average fixation time, saccade span, fixation distribution density, fixation point concentration area; The reaction time dimension at least includes: initial operation delay, average reaction time, extreme reaction interval ratio.

[0065] The above original features do not include any ability labels and are only behavioral performance indicators. In the S300 module, an ability label mapping model will be constructed through fusion analysis.

[0066] Finally, the system structurally stores the multi-modal original behavioral feature data and indexes it according to multi-dimensional labels such as pilot number, training task number, training time period, etc., for subsequent calls in training evaluation, ability modeling, and training adjustment path reasoning tasks.

[0067] Through the procedural operation of S210, S220, and S230, the system realizes the accurate collection and task-aligned fusion of the pilot's multimodal behaviors during training tasks, providing a solid data foundation for subsequent ability modeling and training adjustment. The constructed multimodal raw behavioral feature data not only covers the full-scale data of operational behaviors and visual response behaviors, but also structurally binds the behavioral performance with the task structure diagram, ensuring the mapability and traceability between behavioral features, ability labels, and task contents, thus effectively supporting the high-precision input and label semantic closed-loop in the ability modeling process of the S300 module.

[0068] In some embodiments, S300 at least includes S310 - S330: S310. Extract key behavioral indicator data from the multimodal raw behavioral feature data. The key behavioral indicator data at least includes: operation accuracy, reaction time, fixation reaction time, and gaze time.

[0069] This step aims to mathematically analyze the multimodal raw behavioral feature data and construct the basic indicator data on which the psychomotor ability labels depend. This process is carried out based on high-order behavioral functional modeling and temporal feature extraction strategies, mainly including temporal reconstruction, geometric mapping, and partial derivative mining of operational behaviors, reaction behaviors, and eye movement behaviors.

[0070] Specifically, assume that at time t, the collected k-th type of multimodal behavioral feature data is X_k(t), where X_k(t) respectively corresponds to: X_{k1}(t): Represents the operation accuracy behavior sequence; X_{k2}(t): Represents the operation reaction time data behavior sequence; X_{k3}(t): Represents the fixation reaction duration behavior sequence; X_{k4}(t): Represents the gaze time behavior sequence.

[0071] To further describe the dynamic change characteristics of behavioral features during the training process, the following functional transformation is introduced: ① High-order behavioral functional transformation function: Where: Y_{k}(t): Represents the dynamic response functional result of the k-th type of behavioral feature in the time domain; X_{k}(t): Represents the observed value of the k-th type of raw behavioral data at time t; : Represents the trajectory function after geometric mapping of , such as the transformation results of trajectory rotation, scaling, etc.; : Represents the task prompt function, which activates the behavioral response at a specific moment and represents the weight factor of the behavior during that period; : Represents the current time variable, which acts on time integration; : Represents the subscript of the behavior type, with a total of 4 types of behavior characteristics.

[0072] The system respectively performs the above transformations on to , extracts to their respective time-domain response characteristics, and aligns with the i-th training process node in the task structure diagram , and finally forms a quadruple of behavior indicators: Among them, represents the set of key behavior indicators extracted on the training process node , providing a data basis for ability fusion analysis and label generation; represents the dynamic response eigenvalue obtained after functional modeling of the operation accuracy behavior sequence on the training process node ; represents the dynamic response eigenvalue obtained after functional modeling of the reaction time behavior sequence on the training process node ; represents the dynamic response eigenvalue obtained after functional modeling of the gaze reaction time sequence on the training process node ; represents the dynamic response eigenvalue obtained after functional modeling of the fixation time sequence on the training process node .

[0073] S320. Conduct a fusion analysis on operation accuracy, reaction time, gaze reaction time, and fixation time, and construct a set of psychomotor ability labels corresponding to the training task.

[0074] This step is based on the quadruple of indicators extracted in S310 , that is , and uses a non-linear weighted fusion model to construct a set of psychomotor ability labels corresponding to the training task nodes.

[0075] ② Multimodal fusion ability calculation formula: ; Among them: : Represents the training process node The comprehensive psychomotor ability score of the pilot above; : Represents the Fusion weight of the i-th type of behavioral characteristics, satisfying , Exemplarily, it can be set to , Obtained by fitting according to the experience of historical training data; : Is a non-linear mapping function, and the Log-Sigmoid function is used as an example: The mapping can standardize the functional results to a unified scale, enhancing the comparability between different behavioral characteristics.

[0076] The system further extracts four sub-dimension ability labels based on the comprehensive score : : Represents the operation control ability, mainly derived from ; : Represents the reaction decision-making ability, mainly derived from ; : Represents the visual perception sensitivity, mainly derived from ; : Represents the gaze stability, mainly derived from .

[0077] Construct a psychomotor ability label set For: The label values of each ability label are obtained by weighted decomposition of the corresponding behavioral characteristics and the comprehensive score, and are used as the basis for ability label mapping.

[0078] S330. Match the psychomotor ability label set with the training process nodes in the task structure diagram to construct the mapping relationship between the ability label and the task node, so as to complete the ability-task matching modeling.

[0079] Among them, the training process node is the task node / task process node.

[0080] This step is based on the task structure diagram in S120 and the ability label set output in S320 , Construct the semantic mapping relationship between the training task node and the ability label, and form an ability-task comparison table that can be used for training scheduling and difference analysis.

[0081] ③ Ability label mapping function: Wherein: : represents a mapping function, which performs label matching according to the task process node number , the node task type, and the ability strength score; If the training process node is of the type "operation class", then the matching weight is preferentially used ; If the training process node is of the type "perception class" or "decision class", then it is respectively docked with or , .

[0082] The system combines the task flow logic in the task structure diagram and the semantic meaning of the ability labels to construct the following mapping table: Wherein, represents the task-ability label mapping table, which is used to describe the corresponding relationship between each training task node and the set of corresponding matched psychomotor ability labels ; represents the i-th training process node in the task structure diagram, which is the basic unit with a clear order and content category in the task structure.

[0083] It should be noted that the above task-ability label mapping table serves as the input index for the difference analysis and path generation in S400, supporting the subsequent ability gap identification and training configuration optimization.

[0084] The present invention proposes a behavior-ability modeling mechanism based on high-order functional analysis and multimodal non-linear fusion in the S310 analysis and path generation module, having the following remarkable technical effects: Fusing multi-source behavior features to form a standardized ability label structure, solving the problems of fuzzy ability indicators and unclear structure in traditional training; Introducing a non-linear weight modeling method to make the label expression have stronger interpretability and discrimination, significantly improving the pertinence of the training effect; Constructing a label-task node dynamic mapping mechanism to support the structural closed-loop of subsequent training path adjustment and ability difference analysis, and realizing the data-driven task adjustment logic closed-loop.

[0085] In some embodiments, S400 at least includes S410-S430: S410, calculating the difference analysis result based on the ability labels in the set of psychomotor ability labels and the preset ability standard, and the difference analysis result is a set of difference numerical values between the label values.

[0086] This step is based on the set of psychomotor ability labels constructed in S320 , in combination with the set of standard ability tags preset in the system , calculate the numerical differences between the abilities in each dimension.

[0087] Specifically, let the fusion tag value of the th ability dimension at the task node be , and its corresponding preset standard ability value be , then the difference metric between the two is expressed as: ① Difference value calculation formula: Where: : represents the ability difference value of the th ability dimension on the task node ; : the actual ability tag value calculated by the S320 module; : the standard ability tag value set by the system according to the training level, which can be specifically set as: 0.60 for junior level, 0.75 for intermediate level, and 0.90 for senior level. The value is derived from the standard ability evaluation model.

[0088] By performing the above calculations on the to four dimensions, the system can construct a complete ability difference vector: Difference set will be used as the core input index for constructing the ability difference map and optimizing the training path.

[0089] S420. Generate an ability difference map according to the difference value set and the weight aggregation function.

[0090] Optionally, after S420, the method may further include: classifying the ability gap between the task type and the task requirements.

[0091] This step performs weighted aggregation processing on the ability difference set output by S410 to generate the overall ability difference intensity index of the task node, which is used to support the training path scheduling and ability gap classification.

[0092] ② Weight aggregation function of the ability difference map: Where: : represents the i-th task node among multiple task nodes (i.e., the training process node Aggregation ability difference value of ( : Represents the difference weight of the th ability dimension, satisfying , for example, it can be set as , , , , and these difference weights are derived from the empirical weight configuration model; : The th label difference value from S410.

[0093] Subsequently, the system will Classify the ability gap levels according to the corresponding numerical magnitudes and categorize them into the following three intervals: Too low difference interval (e.g., 0.0–0.3): No adjustment necessary; Medium difference interval (e.g., 0.3–0.6): Moderate reinforcement; High difference interval (e.g., 0.6–1.0): Key adjustment required.

[0094] This clustering basis will be used as the priority basis for the next step of personalized path optimization.

[0095] S430. Based on the ability difference map and the training process structure, combined with the difficulty adjustment function, generate a personalized training path.

[0096] In this step, the system is based on the task node ability difference intensity obtained in S420 , and combined with the task node dependency relationship in the training process structure diagram in S120, optimize the configuration of the training path and dynamically allocate training difficulty parameters.

[0097] ③ Training path difficulty adjustment function: ; Among them, : Represents the training difficulty adjustment coefficient corresponding to the task node ; When is relatively large, the system assigns a higher training intensity and dynamically adjusts scene parameters such as hint frequency, task response time limit, interference intensity, etc. through the mixed reality device; At the same time, the system follows the node dependency relationship in the structure diagram and preferentially arranges task nodes with large ability gaps to improve the training pertinence.

[0098] Finally, the system outputs a table of the personalized training path: Among them, A table representing a personalized training path, which is used to describe the order of personalized training tasks generated based on the ability difference analysis and the task structure diagram and their corresponding difficulty adjustment parameters; the training path table will be used as the core input data structure for task generation and mixed reality device push in S500.

[0099] Represents the total number of task nodes, which is an integer greater than or equal to 2.

[0100] The present invention realizes differential analysis and personalized path recommendation based on behavior label fusion by constructing a linkage mechanism between a structured ability difference map and a training task process. Specifically: The aggregation modeling of multi-label differences is realized through the differential quantization formula ②, significantly improving the training accuracy and target matching degree; The proposed task difficulty adjustment function ③ can flexibly control the mixed reality training scenario, effectively supporting the dynamic adjustment of task intensity; A mapping path structure between training tasks and ability gaps is established, improving the system's adaptability to individual differences and solving the problems of homogeneous training and lack of pertinence in traditional systems.

[0101] In some embodiments, S500 at least includes S510 - S530: S510. Extract the task type and the applicable range of ability labels based on the training task node parameters in the personalized training path.

[0102] In this step, the system calls the table of the personalized training path output by the S430 module and reads the path node information one by one. The personalized training path is constructed with task nodes as the basic unit, and each node is associated with the corresponding ability label adjustment factor and training task attributes, including task objectives, task types, training priorities, and scenario constraints.

[0103] Specifically, the system first parses each training task node in the path structure, retrieves the matching task process node number in the corresponding training task structure diagram, and extracts the basic attribute parameters of the task node.

[0104] Optionally, the above basic attribute parameters at least include: Task node number; The task category to which it belongs (for example: visual perception category, hand-eye coordination category, spatial memory category, etc.); Training task objective (such as: completing a specific flight instruction simulation, judging the reaction of a specified index, etc.); Operation frequency and task duration (derived from the training level setting in S130); The set of ability labels corresponding to this node during the historical training process (supported by the ability label - task node mapping relationship generated by S330); For each task node, the system will match the subset of ability tags required for that node.

[0105] Furthermore, the system also needs to obtain the difference degree information of the corresponding tag dimension in the ability difference map output by the S400 module, and quantitatively mark the adaptation degree of the task node ability tags. This adaptation degree is divided by a preset ability tag interval, and combined with historical training results, the following adaptation mapping structure is established: If the ability tags required by the current task node are basically the same as the pilot's ability level, it is marked as "adapted"; If there is a medium deviation in a certain tag, it is marked as "adjust"; If there is a serious deviation in a certain tag, it is marked as "strengthen".

[0106] The system synchronously saves the above adaptation results with the path node for the differential adjustment of the next task parameter configuration data.

[0107] This step realizes the extraction of task node information highly correlated with the ability state from the training path structure, and constructs a three-layer structure of training task - ability tag - adaptation degree, providing an accurate reference for subsequent scenario generation.

[0108] S520. Based on the ability difference map and environmental configuration parameters, adjust the task type and the adaptation range of ability tags to generate an adjusted task type and an adjusted adaptation range of ability tags.

[0109] Based on the extraction of task node ability adaptation information in this step, the system further combines the environmental configuration parameters in the training tasks generated in S130 to differentially adjust the core parameters of each training task and form a final task scenario configuration data set.

[0110] Specifically, the system first retrieves the environmental parameter information in the previous training task configuration file, including: weather condition parameters, background change speed parameters, visual change range parameters, and limb operation frequency parameters.

[0111] The above four types of parameters have been associated with the training level and preliminarily configured in S130. In the current step, the system adaptively adjusts these environmental parameters according to the task type and ability adaptation degree of each training node, and the adjustment method depends on the tag difference degree classification in the ability difference map in S420: If the node is marked as "strengthen", at least one of the background change speed parameter, the visual prompt range, and the operation frequency is appropriately increased to increase the scenario difficulty; If the node is marked as "adjust", maintain the environmental parameter load at the target level. Specifically, maintain a medium environmental load; If the node is marked as "adapted", the original environmental configuration parameters are maintained to avoid interfering with normal ability migration.

[0112] The system will also perform dynamic fine-tuning by combining the training success rate information of the same type of tasks in the historical training cycles. For example, when the pilot performed poorly in the "space tracking task" during the previous training, the system automatically reduces the density of visual interference elements at that node or adjusts the interaction prompt rhythm.

[0113] During this process, the system constructs a three-dimensional parameter mapping structure of "task type - ability difference - environmental adjustment" to ensure the accuracy and adaptability of task scenario generation. Optionally, the finally formed updated task scenario configuration data includes at least the following fields: Scene ID; Task node number; Updated four types of environmental parameter values; Adaptation ability label and current status; Task execution time limit; Scene background and interaction logic script ID.

[0114] This configuration data structure will be used for the next step of mixed reality scene pushing and replacement.

[0115] S530. Push the adjusted task type and the adapted range of ability labels to the mixed reality device.

[0116] In this step, the system receives the updated task scenario configuration data generated by S520 and pushes it to the target device through the mixed reality training control interface.

[0117] Specifically, the system first completes the following operations through the task synchronization module: Detect the task status currently running on the mixed reality device; If it is in the task completion state or training pause state, allow loading of the new configuration; If the current task is still in progress, select the waiting-for-replacement or interrupt-execution-and-replace mode according to the system settings.

[0118] Then, the system performs structured parsing of the configuration data, including: Loading of scene background rendering resources; Update of the mapping of interaction logic control nodes; Binding of generated text for the semantic of ability label prompts; Injection of parameters into the visual interference and task response modules.

[0119] The above operations are all executed based on the standard interface of the mixed reality system SDK.

[0120] After the configuration is loaded, the system will push execution instructions to the mixed reality device, enter a new round of training task process, and initialize the scene operation log structure for subsequent collection of training process data (docking with the S600 module).

[0121] At the same time, the system will store the task configuration number, task type, ability label range, and adjustment information of this task in the training record database together to form "task-label-environment-response" structured data for training strategy analysis and ability growth curve modeling.

[0122] Through the implementation of this module, the system realizes an adaptive adjustment mechanism for task parameter configuration data based on the ability difference map, and combines the environmental configuration parameter data to update the mixed reality task scene in real time. Compared with the traditional training scheme with fixed difficulty and path, the dynamic scheduling mechanism of this module effectively improves the adaptability between the training task and the individual ability state, and improves the responsiveness and personalization level of the training system. This module realizes the full-process dynamic task control logic from "ability state recognition" to "environmental parameter update" and then to "training task scene push", breaks through the problem of rigid task configuration and difficult dynamic intervention in the existing technology, and enhances the practicality and intelligent level of the system in complex flight training scenarios.

[0123] In some embodiments, S600 at least includes S610-S630: S610, construct a chronological sequence of ability label changes based on the set of pilot psychomotor ability labels in multiple training cycles.

[0124] This step is executed after the task parameter adjustment and the dynamic push of the task scene (i.e., S530). After each training cycle ends, the system extracts the set of ability labels of the pilot after completing the current task process from the behavior analysis module. The set of ability labels is the training task ability characterization result constructed according to the fusion feature data via S320.

[0125] Specifically, the system first sets the training cycle index, numbers the set of ability labels of each training cycle in chronological order, and arranges them in the order of training execution as a sequence of ability label changes. The set of ability labels includes structured label items composed of multiple ability dimensions, and each label item has a task node association identifier, a corresponding index dimension, an evaluation value, and a timestamp.

[0126] Furthermore, the system uniformly encodes the ability label values of all pilots in each ability dimension during the training cycle to form a time series data matrix. The rows of this matrix represent the training cycle sequence, and the columns represent various ability label items, such as operation accuracy, reaction time, fixation response time, gaze duration, etc. Each cell value in the matrix corresponds to the ability label value of a specific ability dimension at a training node, marked with the corresponding timestamp for subsequent module calls.

[0127] For example, for the training cycle numbered the nth round, its corresponding set of ability labels is denoted as: Ability label sequence {A label sequence, the set of ability labels corresponding to the A label sequenceᵢ}, where i represents the number of ability dimensions involved in the current training task.

[0128] This sequence of ability label changes will directly serve as the input parameter for subsequent ability time series modeling (S620), and run through the input process of the ability evolution trajectory modeling.

[0129] S620: Perform time series modeling on the sequence of ability label changes to extract the ability evolution trajectory during the training process.

[0130] After obtaining the sequence of ability label changes arranged in chronological order, the system introduces a time series modeling method to identify the ability evolution trajectory based on the dynamic change trend of different ability label values of the pilot in multiple consecutive training cycles.

[0131] Optionally, when performing time series modeling on the sequence of ability label changes in S620, a sliding window mechanism is adopted. The window length is multiple consecutive training cycles. Inside the window, the evolution speed and fluctuation amplitude of each ability dimension are calculated through differences, slopes, and curve fitting. A multi-dimensional ability growth curve map is constructed, and the evolution trend is mapped to the corresponding nodes in the training task structure diagram to analyze the correlation between the ability dynamics and the task type. The evolution trend is used to characterize the ability evolution trajectory.

[0132] Specifically, the system adopts a long-term and short-term feature modeling strategy based on the sliding window mechanism. After segmenting the sequence of ability label changes in multiple training cycles into windows, the short-term ability fluctuation trend and long-term evolution direction are extracted, and the collaborative relationship between each ability dimension in the window is integrated. The sliding window length is set according to the training rhythm and the ability label sampling frequency, generally set as multiple consecutive (such as 3 to 5) training cycles, and window overlap is allowed.

[0133] Furthermore, during the modeling process, the system aggregates and calculates statistics such as the speed, amplitude, and frequency of changes in multiple capability dimensions. For indicators such as operation accuracy and gaze response time, the system calculates their evolution speed and fluctuation amplitude through difference, slope, curve fitting, etc., and then constructs a multidimensional curve map of the evolution of pilot capabilities. If the label value of a certain capability dimension maintains an upward trend for multiple consecutive rounds, the system identifies it as positive capability growth; if there are drastic fluctuations or downward trends in multiple consecutive training cycles, it is marked as a key fluctuation dimension and recorded for training strategy adjustment.

[0134] For example, for the change sequence of the ability label "reaction time": {R is the change of sequence:: sequence:: time", where each item is the label value under the corresponding training cycle, The system will construct a trend line for the sequence and perform differential approximation to capture its changing growth rate and fluctuation points. This trend line is used as part of the capability growth curve to reconstruct the capability trajectory.

[0135] Ultimately, the system maps the evolution curves of all capability dimensions uniformly into the pilot training task structure diagram, corresponding to specific task nodes, so as to analyze the corresponding capability growth dynamics with task types.

[0136] S630. Based on the capability evolution trajectory, generate a training process analysis report and an individual capability growth map for subsequent training strategy optimization.

[0137] After completing the capability evolution trajectory modeling, the system enters the training process analysis and capability growth map generation phase. This step is based on the capability evolution results output from S620, and conducts individual performance analysis and trend statistics in the task node dimension of the training structure diagram.

[0138] Specifically, the system constructs a structured training process analysis report, which includes the following key contents: Table of changes in capability label values ​​under different training cycles; Statistics of key capability dimensions with large capability growth; Capability degradation trend dimensions and their corresponding task types; Judgment of correlation between high volatility dimensions and task difficulty; Capability growth node mapping diagram (task flow chart + capability dimension label weight); Simulate and deduce personalized adjustment suggestions for the next round of training tasks.

[0139] Based on the generated training process analysis report, the system synchronously constructs a growth map of the pilot's individual capabilities. The map uses a graphical visualization method to present the growth trajectory of the pilot in each capability dimension in the form of a time-series coordinate, and each capability dimension corresponds to an evolution curve. The nodes of the map are composed of the training cycle number and the capability label, and the edge weights are weighted and synthesized by the capability growth rate and the training task complexity, which is convenient for intuitively displaying the capability evolution process.

[0140] For example, during the process of generating the map, the system maps the training cycle number to the horizontal axis coordinate, and uses the label values of each capability dimension as the vertical axis values, and presents them in the form of a line chart; and automatically adds analysis annotations at the places where the capabilities fluctuate violently to prompt whether the system needs to perform retraining on the capability dimension.

[0141] Finally, the system packages and outputs the training process analysis report and the capability growth map as the input to the training level setting logic and the training task template parameter optimization entry of the S100 module, realizes the ability feedback closed loop, and provides the original basis for the subsequent adaptive iteration of the training strategy.

[0142] By executing the steps described in this module, it is possible to realize the dynamic tracking and time-series analysis of the pilot's psychomotor ability in multiple training cycles, construct a continuous capability evolution trajectory, and output a capability growth map and a training process analysis report based on structured data, realizing the reverse drive of ability feedback to the training structure, and providing a quantitative basis for the personalized adjustment of the training process and the adaptive control of the training difficulty level.

[0143] The key innovation points of the present invention include: (1) Construct a capability-task mapping model based on the mixed reality task structure diagram and multi-modal behavior data.

[0144] Innovatively design the task structure diagram and the environmental configuration parameter construction mechanism, and combine the multi-modal original behavior data to establish a set of capability labels, and realize the automatic adaptation and dynamic adjustment of the training task through a one-to-one mapping of the label-task nodes.

[0145] (2) Integrate the capability difference map and the dynamic feedback adjustment algorithm to construct a personalized training path. The present invention designs a fusion type capability difference calculation and task adjustment mechanism, introduces a weighted regression model in dimensions such as operation accuracy and fixation duration, constructs a training path and adjusts the task configuration in combination with environmental parameters, and realizes the push of a dynamically updated mixed reality scene.

[0146] (3)A temporal ability growth modeling method based on the change sequence of ability labels in the set of psychomotor ability labels. The present invention constructs a temporally ordered change sequence of ability labels, and introduces an ability evolution trajectory function and a visible spectrum generation module, realizing the full-process tracking, modeling, and report output of training effects, and providing a personalized and quantifiable decision support mechanism for pilot training.

[0147] The following are its main beneficial effects: (1)Achieve the precise adaptation of training tasks to the ability status of pilots, and improve the personalization and targeting of training. The present invention integrates mixed reality environment modeling technology and multi-modal behavior data acquisition mechanism, constructs psychomotor ability labels based on key indicators such as action behavior data, eye movement path data, and operation reaction time data, and forms an ability-task mapping relationship by matching with the task structure diagram. According to the dynamic ability status of pilots, it can automatically generate personalized training paths and task difficulty adjustment schemes, effectively avoiding problems such as fixed task intensity and lagging feedback in traditional training.

[0148] (2)Construct a mathematical modeling system integrating behavior data to achieve quantifiable evolutionary tracking of ability status.

[0149] The present invention introduces a high-dimensional feature fusion algorithm and an ability difference calculation function, and innovatively uses a weighted regression model and a temporal ability evolution function to model ability labels and perform dynamic difference analysis. Combining training task nodes and environmental configuration parameters, it promotes the formation of a closed-loop adjustment between task configuration and ability progress, improving the accuracy of ability growth recognition and the adjustment sensitivity during training.

[0150] (3)Form an interpretable training effect evaluation spectrum to achieve visualization and closed-loop optimization of training decisions The present invention constructs a change sequence of ability labels in multiple training cycles, extracts the ability growth trajectory based on temporal modeling, generates an ability growth spectrum and a process analysis report, providing a visual basis for subsequent training task recommendation and strategy adjustment, and significantly improving the tracking transparency and optimization efficiency of training effects.

[0151] The following is a detailed description of the psychomotor ability training system based on mixed reality and behavior analysis: Embodiment 2: Figure 2 Show a structural schematic diagram of a psychomotor ability training system based on mixed reality and behavior analysis according to an embodiment of the present invention. As Figure 2 shown, the structure may include: A mixed reality task generation module 10, configured to construct a multi-scene psychomotor training environment, generate a task structure diagram and environmental configuration parameters, and push them to a mixed reality device for scene presentation.

[0152] The data acquisition module 20 is used to acquire the action behavior data, eye movement path data, and operation reaction time data of the pilot during training.

[0153] The behavior analysis module 30 is used to construct the multi-modal original behavior features based on the action behavior data, the eye movement path data, and the operation reaction time data; and perform fusion analysis on the multi-modal original behavior features, construct a set of psychomotor ability labels, and associate the task structure diagram to complete the ability-task matching modeling.

[0154] The ability reasoning and task planning module 40 is used to generate an ability difference map for task adjustment based on the analysis results of the differences between the ability labels in the set of psychomotor ability labels and the preset ability standards, and a weight aggregation function, and determine a personalized training path based on the difficulty adjustment function and the ability difference map; the weight aggregation function is: ; where represents the aggregated ability difference value of the i-th task node among multiple task nodes; is the difference weight of the th ability dimension; represents the ability difference value of the th ability dimension on the i-th task node; the difficulty adjustment function is: ; where represents the training difficulty adjustment coefficient corresponding to the i-th task node.

[0155] The task scheduling module 50 is used to adjust the task parameter configuration data based on the personalized training path and the ability difference map, and push the adjusted task scenario configuration data to the mixed reality device, and the adjusted task scenario configuration data is used to replace the current training task and execute a new round of training process.

[0156] The training evaluation module 60 is used to record and organize the set of psychomotor ability labels of the pilot in multiple training cycles, perform temporal modeling on the set of psychomotor ability labels, generate an ability evolution trajectory, and generate a training process analysis report and an individual ability growth map based on the ability evolution trajectory.

[0157] Optionally, the task structure diagram at least includes: task process nodes, task logical relationships, and task content categories; the environment configuration parameters at least include: weather condition parameters, background change speed parameters, visual change range parameters, and limb operation frequency parameters; the mixed reality task generation module 10 is specifically configured to obtain a training task template and training target type information, select a corresponding training scenario type based on the flight task requirements and set a training level; based on the training task structure composition information extracted from the training task template, construct the task process nodes, the task logical relationships, and the task content categories; based on the task process nodes, the task logical relationships, the task content categories, and the training level, set the weather condition parameters, the background change speed parameters, the visual change range parameters, and the limb operation frequency parameters required for the training scenario type, generate a training task configuration file, and push it to the mixed reality device for scenario presentation.

[0158] Optionally, the behavior analysis module 30 is specifically configured to, based on the push-pull operation behavior, the hand click behavior, and the posture adjustment interaction behavior of the pilot in the mixed reality task during the training obtained by the data acquisition module 20; based on the eye movement path data and the operation reaction time data of the pilot during the training obtained by the data acquisition module 20, construct line-of-sight trajectory information and a time interval sequence; align and integrate the push-pull operation behavior, the hand click behavior, the posture adjustment interaction behavior, the line-of-sight trajectory information, and the time interval sequence to construct the multi-modal raw behavior feature data.

[0159] Optionally, the behavior analysis module 30 is specifically configured to extract key behavior index data from the multi-modal raw behavior feature data, and the key behavior index data at least includes: operation accuracy, reaction time, fixation reaction time, and gaze time; perform fusion analysis on the operation accuracy, the reaction time, the fixation reaction time, and the gaze time to construct a set of psychomotor ability labels corresponding to the training task; match the set of psychomotor ability labels with the training process nodes in the task structure diagram to construct a mapping relationship between the ability labels and the task nodes, so as to complete the ability-task matching modeling.

[0160] Optionally, the ability reasoning and task planning module 40 is specifically configured to calculate a difference analysis result based on the ability labels in the set of psychomotor ability labels and the preset ability standard, and the difference analysis result is a set of difference numerical values between the label values; generate the ability difference map according to the set of difference numerical values and the weight aggregation function; generate the personalized training path based on the ability difference map and the training process structure, in combination with the difficulty adjustment function.

[0161] Optionally, the task parameter configuration data at least includes: the adaptation range between the task type and the ability label; a task scheduling module 50, specifically configured to extract the adaptation range between the task type and the ability label based on the training task node parameters in the personalized training path; adjust the adaptation range between the task type and the ability label based on the ability difference map and the environment configuration parameters to generate an adjusted task type and an adjusted ability label adaptation range; and push the adjusted task type and the adjusted ability label adaptation range to the mixed reality device.

[0162] Optionally, a training evaluation module 60 is specifically configured to construct an ability label change sequence arranged in chronological order based on the set of psychomotor ability labels of the pilot in the multiple training cycles; perform time series modeling on the ability label change sequence to extract the ability evolution trajectory during the training process; and generate the training process analysis report and the individual ability growth map based on the ability evolution trajectory.

[0163] Optionally, when performing time series modeling on the ability label change sequence, a sliding window mechanism is adopted, the window length is a continuous number of training cycles, and within the window, the evolution speed and fluctuation amplitude of each ability dimension are calculated through difference, slope, and curve fitting to construct a multi-dimensional ability growth curve map, and the evolution trend is mapped to the corresponding nodes in the training task structure diagram to analyze the correlation between the ability dynamics and the task type, and the evolution trend is used to represent the ability evolution trajectory.

[0164] Optionally, when adjusting the task parameter configuration data based on the personalized training path and the ability difference map, if the i-th task node is marked as "strengthen", at least one of the background change speed parameter, the visual change range parameter, and the limb operation frequency parameter is increased; if the i-th task node is marked as "adjust", the target level environment parameter load is maintained; if the i-th task node is marked as "adapt", the original environment configuration parameters are retained.

[0165] Beneficial effects of the embodiment: High precision in multi-modal behavior capture: The system can obtain multi-dimensional data such as action behavior data, eye movement path data, and operation reaction time data from the mixed reality device in real time, forming a high-precision behavior feature basis to provide data support for subsequent modeling.

[0166] Intelligent ability-task matching: Through the mapping relationship modeling between the ability label and the training task process node, precise adaptation between the training task and the individual ability is achieved.

[0167] Closed-loop training task scheduling: Combining the personalized training path and the ability difference map, the system can dynamically adjust the task parameters and the mixed reality scene, making the training content always adapt to the individual ability development.

[0168] Intelligent modeling of growth trends: By continuously modeling the sequence of changes in ability tags, the system can intuitively depict the pilot's ability growth process and provide intelligent support for subsequent training planning.

[0169] Closed-loop transmission of full-process parameters: Key data such as ability tags, task structure diagrams, ability difference maps, and environmental parameters are used to achieve closed-loop circulation among modules, forming a complete intelligent training system.

[0170] In summary, the system integrates mixed reality technology and intelligent analysis of behavioral data, breaks through the limitations of traditional flight training evaluation methods, and has broad practical application value and promotion prospects in improving training accuracy, self-adaptability, and evaluation intelligence.

[0171] Obviously, the embodiments described above are only a part of the embodiments of the present application, rather than all of them. The preferred embodiments of the present application are given in the drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosed content of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements on some of the technical features. Any equivalent structure directly or indirectly using the content of the specification and drawings of the present application in other related technical fields shall be within the scope of the patent protection of the present application by the same token.

Claims

1. A psychomotor ability training method based on mixed reality and behavior analysis, characterized in that: The following steps are involved: S100, constructing a multi-scenario psychomotor training environment, generating a task structure diagram and environment configuration parameters, and pushing them to a mixed reality device for scene presentation; S200, collecting the pilot's action behavior data, eye movement path data and operation reaction time data during the training process, and constructing multi-modal original behavior characteristics; S300, performing fusion analysis on the multimodal original behavior features, constructing a psychomotor ability label set, and associating the task structure diagram to complete ability-task matching modeling; S400, based on the difference analysis results between the ability labels in the psychomotor ability label set and the preset ability standards, and the weight aggregation function, generating an ability difference map for task adjustment, and determining a personalized training path based on the difficulty adjustment function and the ability difference map; The weight aggregation function is: ； in, Represents the aggregate capacity difference value of the i-th task node among multiple task nodes; For the The difference weights of the ability dimensions; Indicates the number of tasks on the i-th task node The ability difference value of each ability dimension; The difficulty adjustment function is: ； in, represents the training difficulty adjustment coefficient corresponding to the i-th task node; S500, based on the personalized training path and the capability difference map, adjusting the task parameter configuration data, and pushing the adjusted task scene configuration data to the mixed reality device, wherein the adjusted task scene configuration data is used to replace the current training task and execute a new round of training process; S600, recording and organizing the pilot's psychomotor ability label sets in multiple training cycles, performing time series modeling on all psychomotor ability label sets, generating an ability evolution trajectory, and generating a training process analysis report and an individual ability growth map based on the ability evolution trajectory.

2. The training method according to claim 1, characterized in that: The task structure diagram at least includes: task flow nodes, task logic relationships and task content categories; the environment configuration parameters at least include: weather condition parameters, background change speed parameters, visual change range parameters and limb operation frequency parameters; The S100, constructing a multi-scenario psychomotor training environment, generating a task structure diagram and environment configuration parameters, and pushing them to a mixed reality device for scene presentation, includes: S110, obtaining a training mission template and training target type information, selecting a corresponding training scenario type based on flight mission requirements, and setting a training level; S120, constructing the task flow nodes, the task logic relationships and the task content categories based on the training task structure composition information extracted from the training task template; S130, based on the task process nodes, the task logical relationships, the task content categories and the training levels, set the weather condition parameters, the background change speed parameters, the visual change range parameters and the limb operation frequency parameters required for the training scene type, generate a training task configuration file and push it to the mixed reality device for scene presentation.

3. The training method according to claim 1, characterized in that: The action behavior data at least includes: push-pull operation behavior, hand click behavior and posture adjustment interaction behavior; The step S200 collects the pilot's action behavior data, eye movement path data, and operation reaction time data during the training process to construct multi-modal original behavior features, including: S210, acquiring the push-pull operation behavior, the hand clicking behavior, and the posture adjustment interaction behavior of the pilot in the mixed reality task during the training process; S220, constructing sight track information and time interval sequence based on the eye movement path data and the operation reaction time data of the pilot during the training process; S230, aligning and integrating the push-pull operation behavior, the hand clicking behavior, the posture adjustment interaction behavior, the sight track information and the time interval sequence to construct the multimodal original behavior feature data.

4. The training method according to claim 2, characterized in that: S300, performing fusion analysis on the multi-modal original behavior features, constructing a psychomotor ability label set, and associating the task structure diagram to complete ability-task matching modeling, includes: S310, extracting key behavior indicator data from the multimodal original behavior feature data, wherein the key behavior indicator data at least includes: operation accuracy, reaction time, gaze reaction time and gaze time; S320, performing a fusion analysis on the operation accuracy, the reaction time, the gaze reaction time and the gaze time to construct a psychomotor ability label set corresponding to the training task; S330, matching the psychomotor ability label set with the training process nodes in the task structure diagram, and constructing a mapping relationship between the ability label and the task node to complete the ability-task matching modeling.

5. The training method according to claim 1, characterized in that: The step S400, based on the difference analysis results between the ability labels in the psychomotor ability label set and the preset ability standards, and the weight aggregation function, generates an ability difference map for task adjustment, and based on the difficulty adjustment function and the ability difference map, determines a personalized training path, including: S410, calculating a difference analysis result based on the ability labels in the psychomotor ability label set and the preset ability standard, wherein the difference analysis result is a difference value set between label values; S420, generating the capability difference map according to the difference value set and the weight aggregation function; S430: Based on the ability difference map and the training process structure, combined with the difficulty adjustment function, generate the personalized training path.

6. The training method according to claim 1, characterized in that: The task parameter configuration data includes at least: task type and capability tag adaptation range; The step S500, adjusting the task parameter configuration data based on the personalized training path and the capability difference map, and pushing the adjusted task scenario configuration data to the mixed reality device, includes: S510, extracting the task type and the capability tag adaptation range based on the training task node parameters in the personalized training path; S520: Based on the capability difference map and the environment configuration parameters, the task type and the capability tag adaptation range are adjusted to generate an adjusted task type and an adjusted capability tag adaptation range; S530: Push the adjusted task type and the adjusted capability tag adaptation range to the mixed reality device.

7. The training method according to claim 1, characterized in that: The S600 records and organizes the pilot's psychomotor ability tag sets in multiple training cycles, performs time series modeling on all psychomotor ability tag sets, generates ability evolution trajectories, and generates a training process analysis report and an individual ability growth map based on the ability evolution trajectory, including: S610, constructing a chronologically arranged ability label change sequence based on the pilot's psychomotor ability label set in the multiple training cycles; S620, performing time series modeling on the capability label change sequence to extract the capability evolution trajectory during the training process; S630: Generate the training process analysis report and the individual capability growth graph based on the capability evolution trajectory.

8. The training method according to claim 7, characterized in that: When performing time series modeling on the capability label change sequence in S620, a sliding window mechanism is adopted, and the window length is multiple consecutive training cycles. The evolution speed and fluctuation amplitude of each capability dimension are calculated within the window by difference, slope and curve fitting, and a multi-dimensional capability growth curve map is constructed. The evolution trend is mapped to the corresponding node in the training task structure diagram to analyze the correlation between capability dynamics and task types. The evolution trend is used to characterize the capability evolution trajectory.

9. The training method according to claim 1, characterized in that: When adjusting the task parameter configuration data based on the personalized training path and the ability difference map in S500, if the i-th task node is marked as "enhancement", at least one of the background change speed parameter, the visual change range parameter and the limb operation frequency parameter is increased; if the i-th task node is marked as "adjustment", the target level environmental parameter load is maintained; if the i-th task node is marked as "adaptation", the original environmental configuration parameters are retained.

10. A psychomotor ability training system based on mixed reality and behavior analysis, characterized in that: include: The mixed reality task generation module is used to build a multi-scenario psychomotor training environment, generate task structure diagrams and environment configuration parameters, and push them to the mixed reality device for scene presentation; Data collection module, used to collect pilots' action behavior data, eye movement path data and operation reaction time data during training; A behavior analysis module is used to construct multimodal original behavior features based on the action behavior data, the eye movement path data and the operation reaction time data; and to perform fusion analysis on the multimodal original behavior features, construct a psychomotor ability label set, and associate it with the task structure diagram to complete ability-task matching modeling; An ability reasoning and task planning module, for generating an ability difference map for task adjustment based on a difference analysis result between the ability labels in the psychomotor ability label set and a preset ability standard, and a weight aggregation function, and determining a personalized training path based on a difficulty adjustment function and the ability difference map; The weight aggregation function is: ;in, Represents the aggregate capacity difference value of the i-th task node among multiple task nodes; For the The difference weights of the ability dimensions; Indicates the number of tasks on the i-th task node The ability difference value of the ability dimension; the difficulty adjustment function is: ;in, represents the training difficulty adjustment coefficient corresponding to the i-th task node; A task scheduling module, used to adjust the task parameter configuration data based on the personalized training path and the capability difference map, and push the adjusted task scene configuration data to the mixed reality device, wherein the adjusted task scene configuration data is used to replace the current training task and execute a new round of training process; The training evaluation module is used to record and organize the pilots' psychomotor ability label sets in multiple training cycles, perform time series modeling on all psychomotor ability label sets, generate ability evolution trajectories, and generate training process analysis reports and individual ability growth maps based on the ability evolution trajectories.

Images