A learning path decision method and system based on multi-modal behavior sequence

Abstract

This invention discloses a learning path decision-making method and system based on multimodal behavior sequences, relating to the fields of educational informatization and intelligent teaching technology. The method includes: initializing the training task and scene tree; loading nodes, dependencies, branches, and evaluation rules; reading the initialization states of the basic data and learning results; collecting multimodal data in real time and uniformly labeling it; slicing, aligning, and extracting features to generate state vectors; constructing behavior vectors and scoring them, and determining node states; searching for branches to generate paths based on these vectors; outputting task resources and collecting feedback; updating behavior vectors and node states; and iterating until the task is completed. This invention solves the technical problem in existing technologies where the analysis, recommendation, and evaluation modules are fragmented and path generation depends on a fixed knowledge structure, making it impossible to generate personalized training paths based on learner behavior. It achieves the technical effect of improving the path targeting and global coherence in complex skill training scenarios.

Description

Technical Field

[0001] This invention relates to the field of educational informatization and intelligent teaching technology, specifically to a learning path decision-making method and system based on multimodal behavior sequences. Background Technology

[0002] With the development of educational informatization and intelligent teaching technologies, virtual simulation, augmented reality, virtual reality, and online training platforms are widely used in vocational education and skills training scenarios. Learners generate a large amount of learning behavior data during training, including course grades, test results, operation steps, click records, tool call records, incorrect operations, action trajectories, video images, and voice interactions. This data can reflect learners' knowledge mastery, operating habits, task completion abilities, and training weaknesses from different perspectives.

[0003] Existing learning analytics systems typically collect, analyze, and evaluate learner behavior, while personalized recommendation systems can recommend learning resources based on learner profiles or knowledge point mastery. However, in complex skills training scenarios, the learning process is often not simply about acquiring knowledge points, but rather a continuous training process composed of multiple task stages, operational scenarios, and step units. Current technologies largely remain at the level of state analysis, result evaluation, or resource recommendation, making it difficult to directly transform multimodal behavioral data into a basis for training path decisions, and also difficult to dynamically adjust subsequent training tasks based on learners' real-time performance. Especially in training projects such as electrical equipment installation, industrial robot operation, PLC debugging, and equipment maintenance, the learner's operational sequence, action trajectory, error types, retry behavior, and stage evaluation results all affect subsequent training arrangements. If a fixed training process or a knowledge point-based recommendation method is still used, it can easily lead to a mismatch between the training path and the learner's actual ability, resulting in problems such as repetitive training, untimely remediation, and unreasonable difficulty jumps. Summary of the Invention

[0004] This application provides a learning path decision-making method and system based on multimodal behavior sequences. The key is that it solves the technical problem in the prior art where the analysis, recommendation and evaluation modules are fragmented and path generation depends on a fixed knowledge structure, which makes it impossible to generate personalized training paths based on learner behavior. It achieves dynamic path decision-making based on multimodal behavior sequences driven by a scene tree structure, realizes the structured expression of the training process and the closed-loop optimization of behavior, path and feedback, and improves the path relevance and global coherence in complex skill training scenarios.

[0005] The first aspect of this application provides a learning path decision-making method based on multimodal behavior sequences, the method comprising: The training task and scene tree structure are initialized, and the hierarchical nodes, node dependencies, branching rules, and evaluation rules of the scene tree are loaded. Learner basic information and structured learning result data are read to initialize the learner state. Operation behavior data, spatial trajectory data, and perceptual interaction data are collected in real time, with unified timestamps added and marked according to the training steps. These data, together with the structured learning result data, form a multimodal behavior data stream. The multimodal behavior data stream is sliced, time-aligned, and feature-extracted. A user state vector is generated through a fusion model. Based on the user state vector, a behavior feature vector is constructed, and a comprehensive ability score is calculated to determine the completion status of the current node. Combining the completion status of the current node, path search and branch selection are performed in the scene tree to generate a personalized training path. The training task and auxiliary resources are output, and the execution results and process data are collected to generate feedback data. The behavior feature vector and scene tree node state are updated according to the feedback data. The updated results are returned to the node state judgment or path generation stage, forming a feedback loop and iterating until the training task is completed.

[0006] A second aspect of this application provides a learning path decision-making system based on multimodal behavior sequences, the system comprising: Initialization Module: Initializes the training task and scene tree structure, loads the hierarchical nodes, node dependencies, branching rules, and evaluation rules of the scene tree, and reads learner basic information and structured learning result data to initialize the learner state; Data Acquisition Module: Collects operational behavior data, spatial trajectory data, and perceptual interaction data in real time, adds a unified timestamp, and marks them according to the training steps, forming a multimodal behavior data stream together with the structured learning result data; Data Stream Processing Module: Slices, aligns, and extracts features from the multimodal behavior data stream, and generates a user state vector through a fusion model; Node State Judgment Module: Constructs a behavior feature vector based on the user state vector and calculates a comprehensive ability score to determine the completion status of the current node; Training Path Generation Module: Combines the current node completion status, performs path search and branch selection in the scene tree, generates a personalized training path, outputs the training task and auxiliary resources, and collects execution results and process data to generate feedback data; Task Iteration Module: Updates the behavior feature vector and scene tree node state according to the feedback data, returns the update result to the node state judgment or path generation stage, forms a feedback loop, and iterates until the training task is completed. One or more technical solutions provided in this application have at least the following technical effects or advantages: First, a corresponding scenario tree is established based on the training objectives, and the initial state is determined by combining learner basic information and existing learning results. Then, during the training process, multimodal behavioral data, including learner operations, trajectories, video, and audio, is continuously collected and organized into a behavioral data stream. Next, this data is aligned, analyzed, and fused to obtain state information reflecting the learner's current ability and performance. Based on this state, behavioral characteristics and ability scores are calculated to determine the learner's mastery at the current training node. Then, according to the node relationships and branching rules of the scenario tree, the next training content, remedial measures, or reinforcement paths are automatically selected. Finally, the learner's state and node states are continuously updated based on training feedback, and the training path is iteratively adjusted until the entire training task is completed, thereby achieving dynamic adjustment and personalized adaptation of the learning path. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a schematic diagram of a learning path decision-making method based on multimodal behavior sequences provided in an embodiment of this application.

[0009] Figure 2 This is a schematic diagram of a learning path decision system based on multimodal behavior sequences, provided as an embodiment of this application.

[0010] Figure labeling: Initialization module 11, data acquisition module 12, data stream processing module 13, node status judgment module 14, training path generation module 15, task iteration module 16. Detailed Implementation

[0011] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0012] Example 1, as Figure 1 As shown, this application provides a learning path decision-making method based on multimodal behavior sequences, wherein the method includes: Initialize the training task and scene tree structure, load the scene tree's hierarchical nodes, node dependencies, branching rules, and evaluation rules, and read learner basic information and structured learning result data to initialize the learner state.

[0013] In this embodiment, before starting the practical training, the system first establishes the training task based on the current course, professional direction, and training objectives. For example, it determines the practical training projects, training stages, assessment objectives, and available resources that the learner needs to complete. Then, it reads the scene tree structure corresponding to the training task from a pre-configured scene tree database. This scene tree includes at least a course layer, a practical training module layer, a task stage layer, an operation scene layer, a step unit layer, and an action and resource layer. Each node has a node number, node name, level, predecessor node, successor node, optional branches, completion conditions, scoring rules, and available resources. Afterward, the system loads the dependencies between nodes. For example, a wiring step can only be entered after terminal identification is completed, and a fault detection task can only be triggered after wiring confirmation is completed. Simultaneously, it loads branching rules and evaluation rules. For example, when the number of errors exceeds a threshold, it enters a remedial training node; when the comprehensive score reaches a set value, it enters the next main path node; and when the completion time is too long but the result is correct, it enters a reinforcement training node. Next, the system reads the learner's basic information and structured learning outcome data. The basic information includes the learner's ID, major, course stage, and training level; the structured learning outcome data includes pre-test scores, historical training scores, and unit task scores. The system standardizes this data, including Z-score-based normalization, One-Hot encoding, and missing data handling using default value imputation. It then generates the learner's initial state based on preset weights, determining their knowledge base, operational proficiency, weak skills, and recommended starting node. Finally, the learner's initial state is bound to the scene tree node states to determine the starting node, default main path, triggerable branch paths, and initial evaluation parameters for this training. This completes the initialization of the training task and scene tree structure, providing a foundation for subsequent multimodal behavior collection, node state judgment, and personalized path decision-making.

[0014] Furthermore, the scene tree is constructed from virtual assembly and debugging training scenarios in a virtual reality or augmented reality environment, including at least one of motor assembly and debugging, drone assembly and debugging, and PLC control system wiring and debugging; the scene tree adopts multi-level structured modeling, including at least a course layer, a training module layer, a task stage layer, an operation scene layer, a step unit layer, and an action and resource layer; wherein, the scene tree is used to define the main training path, reinforcement training branches, remedial training branches, and the triggering conditions of each branch.

[0015] Optionally, the system first establishes a scene tree root node with a specific training project as the object. This specific training project is a virtual assembly and adjustment training scene in a virtual reality or augmented reality environment. The virtual assembly and adjustment training scene can be at least one of motor assembly and adjustment training, drone assembly and adjustment training, and PLC control system wiring and debugging training, or it can be a combination of the above-mentioned multiple assembly and adjustment training scenes. Virtual assembly and debugging training scenarios present learners with training tasks such as assembly, wiring, debugging, testing, and troubleshooting through VR devices, AR devices, or virtual simulation training platforms. A scenario tree is used to structure the virtual assembly and debugging training process, breaking it down step-by-step according to the course syllabus and training objectives. The course layer defines the course name, application objectives, and applicable majors. For example, motor structure recognition training, electrical wiring training, and comprehensive maintenance assessment. When applied to motor assembly and debugging training, the course layer can correspond to the course objectives of motor structure recognition, electrical wiring, operation debugging, or comprehensive maintenance. When applied to UAV assembly and debugging training, the course layer can correspond to the course objectives of UAV structure assembly, flight control connection, power system debugging, or complete machine testing. When applied to PLC control system wiring and debugging training, the course layer can correspond to the course objectives of PLC wiring, program debugging, input / output testing, or control system integration debugging. In the practical training module layer, the course objectives are broken down into several functional modules, such as the structural identification module, disassembly and assembly module, wiring module, and testing module under the comprehensive maintenance assessment. Each module is configured with a module number, module objective, and entry conditions. For example, motor assembly and debugging training can be divided into structural identification module, disassembly and assembly module, wiring module, testing module, and operation verification module; UAV assembly and debugging training can be divided into frame assembly module, motor ESC installation module, flight control connection module, sensor calibration module, and trial operation testing module; PLC control system wiring and debugging training can be divided into hardware wiring module, program download module, input / output testing module, logic debugging module, and fault diagnosis module. Within the task phase layer, each module is further divided into preparation, execution, verification, and evaluation phases. For example, the wiring module includes a wiring preparation phase, a terminal identification phase, a line connection phase, and a connection verification phase. Sequential dependencies between phases are established. For UAV assembly and debugging scenarios, the flight control connection module can be divided into an interface identification phase, a cable connection phase, a sensor calibration phase, and a connection verification phase. For PLC control system wiring and debugging scenarios, the hardware wiring module can be divided into a power verification phase, an input wiring phase, an output wiring phase, and a power-on detection phase.In the operation scenario layer, specific training scenarios are configured for each stage, such as the U1 / V1 / W1 terminal identification scenario, the star connection scenario, and the thermal relay connection scenario. Scenario switching conditions and input / output data are defined. In UAV assembly and debugging, operation scenarios may include arm installation, ESC connection, flight controller fixing, and gyroscope calibration. In PLC control system wiring and debugging, operation scenarios may include input terminal identification, sensor connection, actuator connection, program download, and output response observation. In the step unit layer, each operation scenario is broken down into executable steps. For example, in the UAV motor ESC installation scenario, it can be broken down into steps such as selecting a screwdriver, installing the motor, connecting the ESC cable, fixing the wiring harness, and checking the rotation direction. In the PLC input / output test scenario, it can be broken down into steps such as selecting wires, connecting the input terminal, connecting the output terminal, downloading the test program, triggering the input signal, and observing the output status. In the action and resource layer, specific action requirements, auxiliary resources, error detection rules, and evaluation indicators are attached to each step unit. For example, in the step of connecting the first wire, actions such as correctly grabbing the wire, indicating the correct connection position, warning rules for incorrect wiring, and judgment conditions for the completion of this step can be attached. In the VR / AR virtual assembly and debugging environment, the action and resource layer can also attach actions such as virtual component grabbing, virtual tool usage, 3D assembly guidance model, AR overlay prompts, virtual wiring diagrams, component installation direction prompts, debugging parameter descriptions, and assembly and debugging error warning information.

[0016] After completing hierarchical modeling, the system establishes a structured attribute set for each node, including node ID, node type, parent-child relationship, predecessor dependent nodes, optional successor nodes, completion judgment conditions, scoring rules, and resource call identifiers. The system connects these nodes through directed relationships to form a main path structure. For example, a default training main path is formed according to the order of preparation → identification → execution → confirmation → verification. Simultaneously, branching rules are configured for key nodes in the scene tree to generate reinforcement training branches and remedial training branches. When a learner meets the correct condition at a node but the time taken exceeds a threshold or the number of repeated attempts exceeds a set value, the system triggers a reinforcement training branch, inserting the corresponding reinforcement training node after the original node, such as local repetitive training or process optimization training. When a key step error occurs, the score is below the threshold, or the prerequisite ability requirements are not met, the system triggers a remedial training branch, jumping the path to a preset remedial node, such as basic cognitive review or error correction training. When a learner performs exceptionally well and the score consistently exceeds the upper limit threshold, a jump branch can be triggered, skipping some intermediate nodes to enter a higher difficulty scenario. Furthermore, the system configures explicit trigger and regression conditions for each branch path. For example, after reinforcement training is completed, it returns to the original main path node; after remedial training achieves the target, it re-enters the main path; or, if multiple remedial attempts fail, it continues to extend to the next lower-level remedial branch. Through this approach, the scene tree not only possesses hierarchical structure expression capabilities but also executable path control logic. This allows it to dynamically select the main path, reinforcement path, or remedial path based on learner behavior during operation, achieving fine-grained scheduling and adaptive adjustment of the training process.

[0017] Real-time collection of operational behavior data, spatial trajectory data, and perceptual interaction data, with added unified timestamps and marked according to training steps, together with the structured learning result data to form a multimodal behavior data stream.

[0018] In one embodiment, after a learner enters the current training node and begins operation, the system initiates a data acquisition process, continuously acquiring multi-source data from the learner during the training process through the training platform, sensing devices, or interactive terminals. Specifically, the system collects the learner's operational behavior data, including operation step sequences, click behaviors, tool call records, etc., and also collects spatial trajectory data, including handle trajectory, head movement trajectory, hand spatial position, etc., as well as perceptual interaction data, including operation videos, key image frames, voice command data, etc. For the data from the above different sources, the system uniformly writes learner identifier, task identifier, scene node identifier, step unit identifier, and timestamp during collection, and categorizes and marks them according to the current training step, so that each behavior record can be mapped to a specific learner, specific training task, specific scene node, and specific operation step. Subsequently, the real-time collected operational behavior data, spatial trajectory data, and perceptual interaction data are correlated with the structured learning results data such as pre-test scores, historical training scores, unit task scores, and learning duration read before the start of training. The data is then organized into a continuous data stream according to the time sequence and training steps. The resulting multimodal behavioral data stream includes both the learner's existing learning foundation and their current operations, actions, videos, voice, and interactive performance during the training process. This data can serve as a unified input for subsequent time alignment, feature extraction, state recognition, and path decision-making, thereby achieving a comprehensive, continuous, and refined characterization of the learner's training process.

[0019] Furthermore, the structured learning outcome data includes pre-test scores, unit task scores, learning duration, historical training scores, phase evaluation results, and completion status of prerequisite courses; the operational behavior data includes operation step sequences, click behaviors, tool call records, task switching order, error operation records, undo operation records, and retry counts; the spatial trajectory data includes controller trajectory, head movement trajectory, hand spatial position, and posture changes in VR / AR virtual training; and the perceptual interaction data includes operation videos, key image frames, and voice command data.

[0020] Preferably, the system classifies and structures different types of data before and during the training. First, for structured learning outcome data, the system reads learners' historical data through the teaching platform or learning management system interface, including pre-test scores, unit task scores, cumulative learning time, historical training scores, phased evaluation results, and prerequisite course completion status. This data is then standardized to form a set of basic feature data that can be directly used for subsequent modeling. After learners enter the training operation phase, operational behavior data is collected in real time. During this process, the system records the learner's operational step sequence according to the training process, and simultaneously records the corresponding click behavior, tool call records, and task switching order for each step. Abnormal operations are also marked, such as the type of error, its location, and the number of times it occurred. The system also records the undo operation behavior and the corresponding number of retries. This operational behavior data is encoded in the form of an event stream, with step numbers and timestamps attached, to reflect the learner's operational logic and execution process. In immersive or interactive training environments, spatial trajectory data is further collected. This spatial trajectory data mainly comes from VR / AR virtual training environments. When learners perform training tasks through virtual reality or augmented reality devices, the system acquires corresponding trajectory data through the built-in sensors, controllers, head tracking devices, or hand recognition devices of the VR / AR devices. This includes controller trajectory, head movement trajectory, hand spatial position, and posture changes in VR / AR virtual training. The trajectory data is sampled and stored in time series. For example, in VR virtual training, the system collects the controller movement path and hand posture changes when learners operate virtual tools, grasp virtual parts, or connect virtual lines. In AR virtual training, the system collects the head perspective changes, hand spatial position, and posture changes when learners operate facing real devices or overlaid virtual models. The system can smooth the trajectory data, perform coordinate transformation, and extract key points as needed to facilitate subsequent analysis of learners' operation paths and the standardization of their actions. Simultaneously, perceptual interaction data is collected to supplement interactive information beyond behavior and trajectory. During this process, video recordings or key image frame extraction of the operation process are performed to reconstruct key operation scenarios. Learners' voice commands, questions, and interactions are also collected and converted into text information through speech recognition processing. Furthermore, the operation steps and time points corresponding to the video frames and voice data are marked to enhance the correlation between multimodal data. After completing the above data collection, structured learning result data, operational behavior data, spatial trajectory data, and perceptual interaction data undergo unified encoding processing to ensure consistent data format and time reference, and cross-modal data index relationships are established. Finally, all types of data are organized with the learner as the main subject and time series as the main thread, providing foundational support for subsequent data fusion, behavior modeling, and path decision-making, thereby achieving a multi-dimensional comprehensive representation of learners' learning foundation and real-time behavior.

[0021] The multimodal behavioral data stream is sliced, time-aligned, and feature-extracted, and a user state vector is generated through a fusion model.

[0022] In one embodiment, after obtaining the multimodal behavioral data stream, the data stream is first sliced ​​according to training steps, scene nodes, or preset time windows, dividing the continuous data into several behavioral segments. For each behavioral segment, the corresponding learner identifier, task identifier, node identifier, step number, and start and end time are retained. Subsequently, using a unified timestamp as a benchmark, different modal data are time-aligned to ensure that structured learning result data, operational behavior data, spatial trajectory data, and perceptual interaction data within the same time period or training step correspond to each other. After alignment, the system extracts various data features. For example, from structured learning result data, it extracts performance level, historical mastery, and learning duration features; from operational behavior data, it extracts operation sequence, click frequency, tool call count, error count, and retries count features; and from perceptual interaction data, it extracts video keyframe features, image recognition results, and voice command semantic features. The extracted features differ depending on the scenario. For example, in a motor wiring scenario, it extracts pre-test scores for wiring theory and the sequence of wiring steps; in a fault detection scenario, it extracts fault diagnosis question scores and the sequence of tool calls. Subsequently, the aforementioned features from different modalities are input into a fusion model, which performs joint calculations according to preset weights to generate a user state vector that represents the learner's current knowledge base, operational performance, behavioral norms, interaction status, and task completion ability. This process transforms scattered, heterogeneous, and multimodal data of varying frequencies into a unified state representation, providing a computational data foundation for subsequent behavioral feature construction, ability scoring, and learning path decision-making.

[0023] Furthermore, when processing the multimodal behavioral data, temporal alignment is completed in units of operation steps or time windows. The structured learning result features, operation behavior features, spatial trajectory features, and perceptual interaction features are input into the fusion model to obtain the user state vector.

[0024] Optionally, when processing multimodal behavioral data, the operation steps or preset time windows in the current training task are used as the basic processing unit to segment the data from different sources. For example, operation steps such as selecting tools, recognizing terminals, connecting wires, and submitting results can be treated as data segments. Alternatively, behavioral segments can be divided into time windows of 5 seconds, 10 seconds, or 30 seconds. Subsequently, based on a unified timestamp, the data generated within the same operation step or the same time window are time-aligned to ensure that structured learning result data, operational behavior data, spatial trajectory data, and perceptual interaction data can correspond to the same training stage. For spatial trajectory data with a high sampling frequency, resampling, coordinate system unification, trajectory smoothing, and key point extraction are performed first to ensure that the controller trajectory, head movement trajectory, hand spatial position, and posture changes collected in VR / AR virtual training maintain a correspondence with the operation steps, video frames, and voice commands on the time axis. After alignment, the system extracts structured learning result features F_result, operational behavior features F_behavior, spatial trajectory features F_trajectory, and perceptual interaction features F_perception. Structured learning result features reflect the learner's existing learning foundation; operational behavior features reflect the learner's current operational process; spatial trajectory features reflect the learner's action path, spatial position changes, and posture regularity in virtual reality or augmented reality training, such as trajectory length, movement speed, movement amplitude, hand stability, head viewpoint changes, and the degree of deviation between the actual trajectory and the standard operational trajectory; perceptual interaction features reflect the learner's action state and interaction performance. The system then inputs these four types of features into the fusion model, generating a user state vector as follows: User_State_Vector = Fusion(F_result, F_behavior, F_trajectory, F_perception), where Fusion represents the fusion calculation process, which can be implemented using weighted fusion, attention mechanisms, etc. Specifically, the input to the fusion model consists of structured learning result features F_result, operation behavior features F_behavior, spatial trajectory features F_trajectory, and perceptual interaction features F_perception, all completed after temporal alignment within the same operation step or time window. The fusion model first performs vectorization encoding and dimension mapping on different modal features to transform various features into a unified feature space. Then, it performs joint calculation on each modal feature according to preset weights or modal attention weights to obtain a unified user state vector User_State_Vector.During the fusion process, structured learning results features are used to provide learners' basic ability background, operational behavior features are used to provide information on the current step execution process, spatial trajectory features are used to provide information on the standardization of actions and trajectory deviations in VR / AR virtual assembly operations, and perceptual interaction features are used to provide interactive semantic information such as video, images, and voice. The fusion model complements and fuses the above information to output a state vector that can represent the learner's current knowledge base, operation process, spatial action state, interactive response, and task completion ability. This user state vector serves as the input for subsequent construction of behavioral feature vectors, comprehensive ability scoring, and path decision-making, thereby realizing the transformation of multimodal data from raw collection results to a unified decision state representation.

[0025] Based on the user state vector, construct a behavioral feature vector and calculate a comprehensive ability score to determine the completion status of the current node.

[0026] In one embodiment, after obtaining the user state vector, the system first extracts key behavioral indicators related to the evaluation of the current scene tree node from the vector. Specifically, based on the task type and evaluation rules of the current node, the system extracts features such as operation time, number of errors, number of repeated attempts, and path deviation from the user state vector, and assembles these features into a behavioral feature vector according to a unified format. Subsequently, the system performs weighted calculations on each behavioral feature according to the preset weight parameters and scoring rules of the current node to obtain the learner's comprehensive ability score for the current node. For example, for wiring-related nodes, the weights of the number of errors, wiring sequence, and path deviation can be increased; for fault detection-related nodes, the weights of tool call order, judgment accuracy, and key response status can be increased. The system compares the calculated comprehensive ability score with the completion threshold, remediation threshold, and reinforcement threshold of the current node, and, in conjunction with conditions such as whether there are errors in key steps, whether prerequisite dependencies are met, and whether the prescribed operations are completed, determines the completion status of the current node and marks it as completed, partially mastered, not mastered, or requiring remediation. Through the above process, the system can further convert the user state vector into node completion states that are evaluable, comparable, and usable for path decision-making, providing a basis for subsequent selection of the main path, reinforcement of training branches, or remedial training branches.

[0027] Furthermore, the behavioral feature vector includes operation time, number of errors, number of repeated attempts, degree of path deviation, step coherence, and key node response status. A comprehensive ability score is calculated based on the weighted calculation of each feature, and the node status is marked as completed, partially mastered, not mastered, or requiring remediation.

[0028] Optionally, after obtaining the user state vector, the corresponding behavioral indicators are first extracted based on the task type of the current scene tree node, and a behavioral feature vector is constructed. This behavioral feature vector includes at least the operation time t, number of errors e, number of repeated attempts r, path deviation p, step continuity c, and key node response k. Here, operation time refers to the time taken to complete the current node; number of errors refers to the number of errors that occur during the operation; number of repeated attempts refers to the number of times the same step is undone, redone, or repeated; path deviation refers to the degree of deviation between the actual operation path and the standard operation path; step continuity indicates whether the operation steps are executed consecutively in a reasonable order; and key node response indicates whether the key steps are completed correctly and the response is timely. All the above features are converted to a unified dimension through normalization and mapped to a preset value range. Subsequently, the system sets the weight parameters of each feature according to the evaluation rules of the current node, performs weighted calculation on the behavioral feature vector, and obtains the comprehensive ability score. For example, weights α, β, γ, δ, ε, and ζ can be set to correspond to t, e, r, p, c, and k, respectively. The comprehensive ability score is calculated by weighted summation, making the influence of key features on the scoring results adjustable under different task types. For time-related and error-related features, a reverse mapping method can be used, so that the longer the time taken and the more errors, the lower the score; for coherence and critical response-related features, a forward mapping method is used, so that the better the performance, the higher the score. After obtaining the comprehensive ability score, the system compares this score with multiple preset threshold ranges for the current node, and determines the node's status based on factors such as the presence of errors in key steps: when the score is higher than the completion threshold and all key steps are responded to correctly, the node is marked as completed; when the score is close to the completion threshold but there is excessive time consumption or repeated attempts, it is marked as partially mastered; when the score is lower than the completion threshold and there are obvious operational problems, it is marked as not mastered; when there are errors in key steps, consecutive errors, or the score is lower than the remediation threshold, it is marked as requiring remediation. Simultaneously, the system writes the determination results into the corresponding node status in the scene tree, using this as a basis for subsequent path branch selection and training adjustments, thereby achieving structured evaluation and decision support for the learner's current training performance.

[0029] Based on the current node completion status, perform path search and branch selection in the scene tree to generate a personalized training path, output training tasks and auxiliary resources, and collect execution results and process data to generate feedback data.

[0030] In one embodiment, after determining the completion status of the current node, the current node is used as the starting point for path search. The system reads the set of preceding dependent nodes, the set of successor nodes on the main path, the set of reinforcement training child nodes, the set of error correction training nodes, and the set of backtracking nodes from the scene tree. This is combined with the learner's behavioral feature vector and comprehensive ability score to select the next training path. When the current node status is "completed," the system prioritizes searching for the next training node that satisfies the dependency relationship along the main training path. When the node status is "partially mastered," reinforcement training nodes or additional prompts are inserted into the main path. When the node status is "not mastered," the system pauses on proceeding to subsequent nodes and selects a repetition training node or a prior review node corresponding to the current weakness. When the node status is "requiring remediation," the system jumps to a preset remedial training branch based on the error type, number of errors, and key step response. After the path is determined, a personalized training path for the current learner is generated, and the learner is output with corresponding training tasks, operation objectives, step prompts, demonstration videos, illustrations, error correction information, or remedial resources. After the learner completes the training task, the system continues to collect execution results and process data, such as task score, operation time, error type and number, and repeated operation records, and organizes the above data into feedback data. Through this process, the system can dynamically select the main path, reinforcement path, or remedial path in the scene tree according to the node status, so that the training task and auxiliary resources are matched with the learner's current mastery.

[0031] Furthermore, the path decision is executed starting from the current scene tree node, combined with the behavior feature vector, comprehensive ability score and node dependency relationship, including entering the next training scene according to the main path, jumping to the reinforcement training sub-node, inserting the error correction training node, skipping the intermediate node to enter the advanced training scene, and falling back to the previous node to retrain.

[0032] Optionally, after determining the current node's state, using that node as the starting point for path decision-making, the structural information associated with that node is first read from the scene tree. This includes the set of preceding dependent nodes, the set of main path successor nodes, the set of reinforcement training child nodes, the set of error correction training nodes, and the set of backtracking nodes. Simultaneously, the trigger condition parameters corresponding to each node are read, such as the scoring threshold, error count threshold, time consumption threshold, and path deviation threshold. Then, the current learner's behavioral feature vector and comprehensive ability score (Score) are used as input and matched against the aforementioned threshold conditions item by item. In specific execution, the main path determination is prioritized. If Score ≥ completion threshold and the key node response status (k) meets the requirements (e.g., all key steps are correct, and there are no errors exceeding the threshold), then nodes satisfying the dependency relationship are selected from the successor node set as the next training scene, and the path is updated to main path node → next training scene node. If the score is between the completion threshold and the reinforcement threshold, or if it meets conditions such as low error rate but high time consumption or a large number of repeated attempts, then a node corresponding to the current weak feature is selected from the reinforcement training sub-node set, and the path is adjusted to current node → reinforcement training sub-node → original main path node. Simultaneously, regression rules are set after the reinforcement training sub-node is completed. When a clear error is detected, such as a critical node failing to meet the requirements, an error correction mechanism is triggered. A node matching the error type is selected from the error correction training node set, such as a wiring error corresponding to a wiring error correction node, and this node is inserted into the current path, forming a path structure of current node → error correction training node → current node or subsequent node. Progress can only continue after the error correction node has passed. When multiple consecutive nodes meet the high score criteria, the learner is deemed to have the ability to advance to higher levels. In this case, higher-order nodes are selected from the successor node set, intermediate nodes are skipped, and a path of current node → advanced training scenario node is directly generated, recording the jump conditions for subsequent evaluation. When the score is determined to be below the remediation threshold or a serious error exists (such as a critical step error or unmet dependency conditions), the nearest capability compensation node or basic training node is selected from the set of preceding nodes as the fallback target. The path is adjusted to current node → preceding node, and remedial training sub-paths are added as needed. Simultaneously, corresponding path record data is generated for each path selection result, including node jump type, triggering conditions, and behavioral characteristic basis. Finally, based on the above decision results, the next training node and its resource set are determined, and the updated path is output as the current learner's personalized training path. Through this refinement process, path decision-making not only has clear judgment conditions and execution order but also enables refined branch selection and dynamic path scheduling based on behavioral characteristics.

[0033] Furthermore, the feedback data includes the current node score, operation time, error type and number, repeated operation records, teacher evaluation and system automatic evaluation results. The behavioral feature vector and node state are updated based on the feedback data to form a closed-loop iteration.

[0034] Optionally, after a learner completes the current training node, feedback data corresponding to that node is immediately collected, including the current node score, actual operation time, error type and number of errors, repeated operation records, undo records, teacher evaluation results, and system automatic evaluation results. This data is then bound to the current node number, learner number, task number, and timestamp. Subsequently, the system organizes and quantifies the feedback data. For example, error types are converted into error labels, teacher evaluations are converted into levels or scores, and the number of repeated operations, time, and node score are converted into computable features. These features are then merged and updated with the original behavioral feature vector to obtain a new behavioral feature vector. Afterward, the comprehensive ability score is recalculated based on the updated behavioral feature vector, and the status of the current node in the scene tree is updated synchronously. When the feedback data indicates that the learner's score is up to standard, the number of errors is low, and the operation process is stable, the node status is updated to "Completed"; when the score is close to standard but there is a long time commitment or repeated operations, it is updated to "Partially Mastered"; when the score is low or there are many errors, it is updated to "Not Mastered"; when a critical error occurs or the teacher evaluation indicates a need for intensive tutoring, it is updated to "Needs Remediation". After the update, the system re-inputs the new behavioral feature vectors and node states into the path decision-making process, re-determining whether the next step should proceed along the main path, enter reinforcement training, insert error correction nodes, or revert to a corrective course. This forms a closed-loop iterative process of behavior, decision-making, feedback, and re-decision-making, enabling the system to continuously adjust its ability judgments and path arrangements based on the learner's training results each time, thereby achieving dynamic optimization of the training path.

[0035] The behavior feature vector and scene tree node status are updated based on the feedback data. The update results are returned to the node status judgment or path generation stage, forming a feedback loop and iterating until the training task is completed.

[0036] In one embodiment, after obtaining feedback data, the system first normalizes and labels data such as the current node score, operation time, error type and number, repeated operation records, teacher evaluation, and system automatic evaluation results, and then merges and updates these data with the original behavioral feature vector to form a new behavioral feature vector. Specifically, the system corrects indicators such as operation time, number of errors, number of repeated attempts, path deviation, step coherence, and key node response based on the feedback data, and recalculates the comprehensive ability score. Simultaneously, the updated score is compared with the completion threshold, reinforcement threshold, and remediation threshold of the current scene tree node, and the node status is re-marked, for example, as completed, partially mastered, not mastered, or requiring remediation. After completing the behavioral feature vector and scene tree node status update, the system returns the updated results as new input data to the node status judgment stage or path generation stage. If the update result indicates that the current node status still needs to be re-determined, the system returns to the node status judgment stage to re-execute the score comparison and node status marking. If the update result clearly defines the current node status and meets the path selection conditions, the system returns to the path generation stage to directly perform subsequent path search and branch selection. Subsequently, based on the updated behavioral feature vector and node state, the system makes another ingress path decision, reselecting the next training node, reinforcement training node, error correction node, or remedial node in the scene tree, and outputting the corresponding training task and auxiliary resources. After the learner executes a new training node, the system continues to collect their behavioral data and feedback data, repeating the process of data update, state judgment, path decision, task execution, and feedback collection. This allows feedback data to continuously flow back between node state judgment and path generation until all target nodes in the scene tree are completed, or the learner reaches the preset training objectives, assessment standards, or training termination conditions. Through this iterative process, the system can continuously adjust the path based on the learner's performance in each round of training, transforming the training task from a fixed process into a dynamic adaptive training process.

[0037] In summary, the embodiments of this application have at least the following technical effects: First, the training task and scene tree structure are initialized, loading the scene tree's hierarchical nodes, node dependencies, branching rules, and evaluation rules. Learner basic information and structured learning result data are read to initialize the learner's state. Then, operational behavior data, spatial trajectory data, and perceptual interaction data are collected in real time, appended with unified timestamps, and marked according to the training steps, forming a multimodal behavior data stream together with the structured learning result data. Next, the multimodal behavior data stream is sliced, time-aligned, and feature-extracted, generating a user state vector through a fusion model. Then, a behavior feature vector is constructed based on the user state vector, and a comprehensive ability score is calculated to determine the completion status of the current node. Then, combining the current node completion status, path search and branch selection are performed in the scene tree to generate a personalized training path, outputting the training task and auxiliary resources, and collecting execution results and process data to generate feedback data. Finally, the behavior feature vector and scene tree node states are updated based on the feedback data, and the updated results are returned to the node state judgment or path generation stage, forming a feedback loop that iterates until the training task is completed. This invention addresses the technical problem in existing technologies where the analysis, recommendation, and evaluation modules are fragmented and path generation relies on a fixed knowledge structure, making it impossible to generate personalized training paths based on learner behavior. It achieves dynamic path decision-making based on a scene tree structure driven by multimodal behavior sequences, realizing structured expression of the training process and closed-loop optimization of behavior, path, and feedback, thereby improving the path relevance and global coherence in complex skill training scenarios.

[0038] Example 2, based on the same inventive concept as the learning path decision method based on multimodal behavior sequences in the foregoing examples, such as... Figure 2 As shown, this application provides a learning path decision system based on multimodal behavior sequences, wherein the system includes: Initialization Module 11: Initializes the training task and scene tree structure, loads the hierarchical nodes, node dependencies, branching rules, and evaluation rules of the scene tree, and reads learner basic information and structured learning result data to initialize the learner state; Data Acquisition Module 12: Collects operational behavior data, spatial trajectory data, and perceptual interaction data in real time, adds a unified timestamp, and marks them according to the training steps, forming a multimodal behavior data stream together with the structured learning result data; Data Stream Processing Module 13: Slices, aligns, and extracts features from the multimodal behavior data stream, and generates a user state vector through a fusion model; Nodes State Judgment Module 14: Constructs a behavioral feature vector based on the user state vector and calculates a comprehensive ability score to determine the completion status of the current node; Training Path Generation Module 15: Combines the completion status of the current node, performs path search and branch selection in the scene tree, generates a personalized training path, outputs training tasks and auxiliary resources, and collects execution results and process data to generate feedback data; Task Iteration Module 16: Updates the behavioral feature vector and scene tree node status according to the feedback data, returns the update results to the node status judgment or path generation stage, forms a feedback loop, and iterates until the training task is completed.

[0039] Furthermore, the initialization module 11 is used to perform the following methods: The scene tree is constructed from virtual assembly and debugging training scenarios in a virtual reality or augmented reality environment, including at least one of motor assembly and debugging, drone assembly and debugging, and PLC control system wiring and debugging; the scene tree adopts multi-level structured modeling, including at least a course layer, a training module layer, a task stage layer, an operation scene layer, a step unit layer, and an action and resource layer; wherein, the scene tree is used to define the main training path, reinforcement training branches, remedial training branches, and the triggering conditions of each branch.

[0040] Furthermore, the data acquisition module 12 is used to perform the following methods: The structured learning outcome data includes pre-test scores, unit task scores, learning duration, historical training scores, phase evaluation results, and completion status of prerequisite courses; the operational behavior data includes operation step sequences, click behaviors, tool call records, task switching order, error operation records, undo operation records, and retry counts; the spatial trajectory data includes controller trajectory, head movement trajectory, hand spatial position, and posture changes in VR / AR virtual training; and the perceptual interaction data includes operation videos, key image frames, and voice command data.

[0041] Furthermore, the data stream processing module 13 is used to perform the following method: When processing the multimodal behavioral data, temporal alignment is completed in units of operation steps or time windows. The structured learning result features, operation behavior features, spatial trajectory features, and perceptual interaction features are input into the fusion model to obtain the user state vector.

[0042] Furthermore, the node status determination module 14 is used to execute the following method: The behavioral feature vector includes operation time, number of errors, number of repeated attempts, degree of path deviation, step coherence, and response status of key nodes. A comprehensive ability score is calculated based on the weighted calculation of each feature, and the node status is marked as completed, partially mastered, not mastered, or needs remediation.

[0043] Furthermore, the training path generation module 15 is used to perform the following method: The path decision is executed starting from the current scene tree node, combined with the behavior feature vector, comprehensive ability score and node dependency relationship, including entering the next training scene according to the main path, jumping to the reinforcement training sub-node, inserting the error correction training node, skipping the intermediate node to enter the advanced training scene, and falling back to the previous node to retrain.

[0044] Furthermore, the training path generation module 15 is used to perform the following method: The feedback data includes the current node score, operation time, error type and number, repeated operation records, teacher evaluation and system automatic evaluation results. The behavioral feature vector and node status are updated based on the feedback data to form a closed loop iteration.

[0045] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A learning path decision-making method based on multimodal behavior sequences, characterized in that, The method includes: Initialize the training task and scene tree structure, load the scene tree's hierarchical nodes, node dependencies, branching rules, and evaluation rules, and read learner basic information and structured learning result data to initialize the learner state; Real-time collection of operational behavior data, spatial trajectory data, and perceptual interaction data, with added unified timestamps and marked according to training steps, together with the structured learning result data to form a multimodal behavior data stream; The multimodal behavior data stream is sliced, time-aligned, and feature-extracted, and a user state vector is generated through a fusion model. Based on the user state vector, construct a behavioral feature vector and calculate a comprehensive ability score to determine the completion status of the current node; Based on the current node completion status, perform path search and branch selection in the scene tree to generate a personalized training path, output training tasks and auxiliary resources, and collect execution results and process data to generate feedback data; The behavior feature vector and scene tree node status are updated based on the feedback data. The update results are returned to the node status judgment or path generation stage, forming a feedback loop and iterating until the training task is completed.

2. The learning path decision-making method based on multimodal behavior sequences according to claim 1, characterized in that, The structured learning outcome data includes pre-test scores, unit task scores, learning duration, historical training scores, phase evaluation results, and completion status of prerequisite courses. The operation behavior data includes operation step sequence, click behavior, tool call record, task switching order, erroneous operation record, undo operation record, and number of retries; The spatial trajectory data includes the controller trajectory, head movement trajectory, hand spatial position and posture changes in VR / AR virtual training; The perception and interaction data includes operation videos, key image frames, and voice command data.

3. The learning path decision-making method based on multimodal behavior sequences according to claim 2, characterized in that, The scene tree is constructed from virtual assembly and debugging training scenarios in a virtual reality or augmented reality environment, including at least one of motor assembly and debugging, drone assembly and debugging, and PLC control system wiring and debugging; The scene tree adopts a multi-level structured model, which includes at least a course layer, a training module layer, a task stage layer, an operation scene layer, a step unit layer, and an action and resource layer. The scenario tree is used to define the main training path, reinforcement training branches, remedial training branches, and the triggering conditions for each branch.

4. The learning path decision-making method based on multimodal behavior sequences according to claim 3, characterized in that, When processing the multimodal behavioral data, temporal alignment is completed in units of operation steps or time windows. The structured learning result features, operation behavior features, spatial trajectory features, and perceptual interaction features are input into the fusion model to obtain the user state vector.

5. The learning path decision-making method based on multimodal behavior sequences according to claim 4, characterized in that, The behavioral feature vector includes operation time, number of errors, number of repeated attempts, degree of path deviation, step coherence, and response status of key nodes. A comprehensive ability score is calculated based on the weighted calculation of each feature, and the node status is marked as completed, partially mastered, not mastered, or needs remediation.

6. The learning path decision-making method based on multimodal behavior sequences according to claim 5, characterized in that, The path decision is executed starting from the current scene tree node, combined with the behavior feature vector, comprehensive ability score and node dependency relationship, including entering the next training scene according to the main path, jumping to the reinforcement training sub-node, inserting the error correction training node, skipping the intermediate node to enter the advanced training scene, and falling back to the previous node to retrain.

7. The learning path decision-making method based on multimodal behavior sequences according to claim 1, characterized in that, The feedback data includes the current node score, operation time, error type and number, repeated operation records, teacher evaluation and system automatic evaluation results. The behavioral feature vector and node status are updated based on the feedback data to form a closed loop iteration.

8. A learning path decision-making system based on multimodal behavior sequences, characterized in that, The system is used to implement the learning path decision method based on multimodal behavior sequences according to any one of claims 1-7, the system comprising: Initialization module: Initializes the training task and scene tree structure, loads the hierarchical nodes, node dependencies, branching rules and evaluation rules of the scene tree, and reads the learner's basic information and structured learning result data to initialize the learner's state; Data acquisition module: collects operational behavior data, spatial trajectory data, and perceptual interaction data in real time, adds a unified timestamp and marks them according to the training steps, and forms a multimodal behavior data stream together with the structured learning result data; Data stream processing module: performs slicing, time alignment, and feature extraction on the multimodal behavior data stream, and generates user state vectors through a fusion model; Node status determination module: Constructs a behavioral feature vector based on the user status vector and calculates a comprehensive ability score to determine the completion status of the current node; Training path generation module: Based on the completion status of the current node, it performs path search and branch selection in the scene tree, generates a personalized training path, outputs training tasks and auxiliary resources, and collects execution results and process data to generate feedback data; Task Iteration Module: Updates the behavior feature vector and scene tree node status based on the feedback data, returns the update results to the node status judgment or path generation stage, forms a feedback loop and iterates until the training task is completed.

Images