Practical training evaluation method and system based on multi-modal data fusion

Abstract

The invention discloses a practical training evaluation method and system based on multi-modal data fusion. The method comprises the following steps: firstly, constructing a practical training task topology based on a knowledge graph, mapping elements such as knowledge points into nodes, and defining a relationship through directed edges; in response to task starting, dynamically deploying a multi-modal data fusion agent pre-loaded with a targeted model and a rule; in the operation process, the agent synchronously collects and processes original data from the machine vision and sensor network in real time, and a standardized feature flow is generated; then, on-line space-time correlation and reasoning are carried out on the multi-modal features based on a fusion strategy, and a task scene understanding model is dynamically constructed and maintained; after the task is completed, packaging a program code, a report and process abstract data exported by the model to form an enhanced practical training result package; and finally, multi-dimensional automatic comparison is carried out by calling the rule base and the case model, and an evaluation result is generated. According to the invention, intelligent perception and comprehensive evaluation of the whole practical training operation process are realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent education technology, specifically to a training and evaluation method and system based on multimodal data fusion. Background Technology

[0002] In modern vocational education and engineering training, practical training is a core component in cultivating students' practical operational abilities and problem-solving skills. However, traditional practical training assessment methods mainly rely on teachers' subjective observation, manually recorded checklists, and static scores of final results. This approach has significant limitations: First, teachers' attention cannot cover all students' continuous operational details, and assessments are prone to overlooking key operational errors or habitual mistakes, exhibiting strong subjectivity and poor consistency. Second, the assessment focus is concentrated on the final result, lacking detailed recording and analysis of the operational process, making it impossible to trace the root causes of errors and provide targeted feedback, thus lacking data support for teaching improvement.

[0003] With the development of information technology, digital training platforms have begun to be used, with some platforms integrating theoretical testing, resource management, and simple operation log recording. Some advanced platforms have attempted to introduce sensors or cameras to collect operational data. However, existing technical solutions are still in their early stages in processing and applying multimodal data. Most systems only achieve independent data collection and simple overlay presentation, such as storing video recordings and sensor readings separately and reviewing them independently. These discrete data lack deep spatiotemporal correlation and semantic fusion, making it impossible to automatically construct a unified understanding of the complete scenario of "student operation behavior - equipment status response - task goal achievement." As a result, assessment still requires teachers to spend a lot of time manually reviewing and analyzing multiple sources of information, resulting in low intelligence and limited efficiency improvement.

[0004] Furthermore, at the data analysis level, existing methods mostly employ fixed rule matching or simple threshold judgments, such as determining whether a sensor value exceeds a limit. This approach is rigid and poorly adaptable, unable to understand the logical sequences of complex operations, unable to identify non-quantitative, non-standard actions (such as incorrect tool usage posture), and even less able to infer the trainee's operational intentions and cognitive state from multimodal data. The assessment model is disconnected from the teaching content and practical training tasks; assessment rules require pre-coding and are difficult to adapt quickly to updates in teaching content or the introduction of new skills.

[0005] In summary, the core problems faced by existing practical training assessment technologies can be summarized as follows: First, the assessment dimensions are one-sided and the process data is fragmented. Relying on result evaluation and isolated single-modal data, they cannot objectively and comprehensively perceive and record the entire operational process from multiple dimensions. Second, the data analysis is superficial and lacks scenario understanding. They lack the ability to deeply integrate multimodal data and perform intelligent reasoning, and cannot automatically construct semantic operational scenario models from the data, resulting in a low level of assessment intelligence. Third, the teaching and assessment processes are disconnected. Assessment results are mainly based on scores or simple comments, failing to accurately drive adjustments to personalized learning paths and iterative optimization of teaching content and assessment rules themselves, thus failing to form a closed loop of "teaching-assessment-optimization."

[0006] Therefore, there is an urgent need for a training and evaluation method and system based on multimodal data fusion to solve the above-mentioned problems. Summary of the Invention

[0007] This application provides a training and evaluation method and system based on multimodal data fusion, which at least addresses the problems existing in the prior art.

[0008] A first aspect of this application provides a training evaluation method based on multimodal data fusion, comprising: S1: Construct a training task topology based on a knowledge graph. The knowledge graph maps knowledge points, skill points, learning resources and device interfaces to nodes, and defines the logical and data flow relationships between them through directed edges. S2: In response to the training task, dynamically deploy a multimodal data fusion agent adapted to the current task node. The fusion agent preloads targeted recognition models and fusion rules. S3: During the practical training, the raw data streams from machine vision devices and intelligent sensor networks are collected synchronously and processed in real time by the fusion agent to generate standardized feature streams with semantic labels. S4: The fusion agent, based on a preset fusion strategy, performs online spatiotemporal correlation and inference on standardized feature streams of different modalities, and dynamically constructs and maintains a task scenario understanding model that reflects the current operational context. S5: Upon completion of a task node, encapsulate the training program code, report, and process summary data derived from the task scenario understanding model to form an enhanced training outcome package and submit it. S6: On the training platform, a correlation analysis interface is provided, which synchronously presents the code, reports, and visual operation process reconstructed based on process summary data in the enhanced training result package; it calls the rule base and case model to perform multi-dimensional automatic comparison and generate evaluation results.

[0009] This application constructs a knowledge graph to uniformly organize teaching elements and introduces intelligent agents dynamically deployed at the edge. This enables synchronous perception of multimodal data, deep semantic fusion, and real-time scene understanding throughout the entire practical training process. It effectively overcomes the shortcomings of traditional assessments, such as single dimensions and fragmented data. It can automatically generate enhanced outcome packages that integrate process semantics and perform multidimensional intelligent comparisons based on these packages. It can also use assessment data to drive the adaptive optimization of the knowledge graph and assessment model, thereby closely linking teaching, assessment, and improvement. This forms a data-driven, self-evolving intelligent practical training assessment closed loop, significantly improving the objectivity, comprehensiveness, and teaching guidance value of the assessment.

[0010] In some embodiments of this application, step S1, which involves constructing a knowledge graph-based training task topology, specifically includes: constructing and storing the topology of the knowledge graph using a graph database, wherein node attributes include node type, resource identifier, access path of associated multimedia resources, multimodal data acquisition parameters required to complete the node, and a description of the expected standardized output; directed edge attributes include edge relationship type, content format and protocol specifications for data transmission between adjacent nodes, and condition thresholds for determining the completion status of the current node to activate subsequent nodes; the graph supports editing and logical verification through a visual interface, and can automatically generate or recommend personalized training paths containing different difficulty sequences and resource combinations based on preset teaching objectives.

[0011] This application, by employing a graph database and defining detailed node and edge attributes, achieves refined, structured, and computable management of the training task topology. This allows previously discrete knowledge points, skill points, and resources to be organically organized according to teaching logic, and clearly defines data flow and completion conditions. Visual editing and logical verification functions lower the barrier to constructing complex training paths, while the goal-based personalized path recommendation function enables the assessment system to flexibly adapt to the starting points and goals of different students, laying a solid foundation for accurate assessment tailored to individual needs.

[0012] In some embodiments of this application, the dynamic deployment of the multimodal data fusion agent in step S2 specifically includes: when a student starts a specific training task node, the system dynamically obtains and loads a lightweight model component set from the cloud model service center according to the preset multimodal data acquisition parameters in the node's attributes; the model component set includes at least a visual analysis model for recognizing specific hand movements and tool states in the current task scenario, a physical parameter analysis model for parsing the output signals of various sensors of the current training equipment, and a lightweight inference engine suitable for multi-source heterogeneous data fusion.

[0013] This application achieves dynamic, on-demand deployment of multimodal perception and analysis capabilities through cloud configuration and edge loading. This enables the system to accurately call upon the most relevant lightweight analysis models based on specific training task scenarios. This mechanism greatly improves the targeting and efficiency of edge-side data processing, providing a reliable guarantee for subsequent high-quality feature extraction and real-time fusion, and is key to realizing complex multimodal analysis in resource-constrained environments.

[0014] In some embodiments of this application, step S2 further includes: after the fusion agent is initialized, its built-in microservice kernel is responsible for scheduling and managing the visual analysis model, the physical parameter parsing model, and the lightweight inference engine; the kernel maintains a shared data cache based on a sliding time window locally for temporarily storing the intermediate feature data output in real time by each model; by constructing a standardized real-time data processing pipeline, the raw data access, feature extraction and standardization, and multimodal data alignment are executed sequentially, and finally a standardized feature stream with unified timestamps and semantic labels is output that can be directly used by subsequent steps.

[0015] This application, through its microservice-based kernel and shared cache design, ensures efficient collaboration and data consistency among different processing modules. The standardized real-time data processing pipeline enables end-to-end, low-latency conversion from raw heterogeneous data to standardized feature streams.

[0016] In some embodiments of this application, the raw data stream synchronously collected and processed by the fusion agent in step S3 further includes: a student physiological signal data stream collected by a wearable biosensor; the fusion agent has a built-in biosignal feature extraction module, which is used to calculate heart rate variability index, skin conductance response level or attention index in real time from the physiological signal data stream; the standardized feature stream not only includes operational behavior features and device status features, but also fuses physiological features as an independent modality.

[0017] This application extracts indicators such as focus and cognitive load in real time and integrates them with other modal features, enabling the system to more comprehensively understand the human-computer interaction process. This not only makes the evaluation results more interpretable but also provides adaptive human-computer interaction guidance based on real-time physiological states, achieving more humanized and intelligent training assistance and evaluation.

[0018] In some embodiments of this application, step S4, which involves dynamically building and maintaining a task scenario understanding model, specifically includes: the fusion agent receiving standardized feature streams from different data sources, dynamically calculating the contribution weights of visual features, physical features, and physiological features based on the target of the current task stage through a configurable attention weight allocation network; then, weighted fusion of the contribution weights, and online inference based on the weighted fusion feature sequence, outputting a joint estimate of the matching probability between the executed action and the standard process, the risk level of the operation behavior violating safety rules, and the abnormal index of the device system operating status.

[0019] This application achieves dynamic weighted fusion and deep semantic inference of multimodal features by introducing a configurable attention mechanism and temporal reasoning. It overcomes the limitations of simple weighting or rule matching, enabling the model to dynamically focus on key information based on task stages. Its output joint probability estimates (matching degree, risk level, anomaly index) provide a quantified, multi-faceted interpretation of the operational scenario, elevating raw sensor data and video streams to an understandable and judgmental semantic level, providing a direct and reliable basis for automated and refined evaluation decisions.

[0020] In some embodiments of this application, step S4 further includes: during the operation of the task scenario understanding model, the model continuously generates structured scenario understanding snapshots; when the matching probability inferred by the task scenario understanding model is lower than the safety threshold or the risk level exceeds the preset level, the task scenario understanding model immediately triggers a two-level response. The first level is to generate and push adaptive correction guidance information on the student operation interface, and the second level is to package and mark the high-risk event together with panoramic multimodal data fragments within a specific time window before and after its occurrence as key diagnostic events and embed them into the process summary data for subsequent review and in-depth analysis.

[0021] This application endows the scenario understanding model with the ability to intervene in real time and record in depth. A two-level response mechanism enables a process-oriented assessment: the first level provides immediate guidance to trainees, demonstrating the real-time and closed-loop nature of teaching intervention; the second level packages and marks high-risk events and their comprehensive contextual data, forming high-value diagnostic cases for in-depth analysis. This ensures both training safety and learning efficiency, while providing teachers with focused and comprehensive review materials, greatly improving the accuracy and efficiency of subsequent assessments and reviews, and achieving a combination of real-time correction and post-event in-depth analysis.

[0022] In some embodiments of this application, step S5, which forms an enhanced training outcome package, specifically includes: the enhanced training outcome package is encapsulated in a self-describing structured format, and its content includes at least a metadata block identifying the student's identity and task information, a block of the program source code file and executable file finally submitted by the student, a block of structured training report document written by the student, and a process summary data block derived from the task scenario understanding model; the process summary data block includes a list of key operation events organized by timeline, a summary of equipment status time series curves, and any key diagnostic events marked by the model and their associated multimodal data indexes, ensuring the integrity of the results and the traceability of the evaluation process.

[0023] This application ensures the integrity, self-containment, and traceability of the evaluated object through a self-describing structured encapsulation format. It organically integrates program code, text reports, and semantically rich process data (event lists, state curves, diagnostic events) into a single data package. This design allows evaluators to obtain all the information needed for the evaluation without navigating multiple systems or consulting scattered logs, and allows them to trace back to the original multimodal evidence through data indexing. This fundamentally solves the problem of fragmented evidence chains in process evaluation, guaranteeing the reliability and impartiality of the evaluation conclusions.

[0024] In some embodiments of this application, step S6 involves multi-dimensional automatic comparison to generate evaluation results, specifically including three analytical dimensions: Operational process standardization dimension, which involves dynamically time-normalizing and comparing the sequence of operational events in the process summary data with the standard operating procedures stored in the knowledge graph to calculate the step sequence fidelity and time efficiency score; Safety and quality compliance dimension, which involves matching the event and state data in the process summary data with the domain rule base to automatically identify violations and classify and quantify them; and Result technical quality dimension, which involves verifying the functional correctness of the program code through static analysis and dynamic testing, and using a case model library to evaluate the similarity of code quality and logical rationality. The scores from the three dimensions are combined with preset weights to generate a final evaluation report, which includes scores for each item, as well as a visual replay of key operational segments that led to deductions and improvement suggestions.

[0025] This application clearly defines three core dimensions of the assessment and their specific implementation algorithms, forming a comprehensive, multi-dimensional, and automated assessment framework. The process standardization dimension focuses on operational procedures, the compliance dimension focuses on safety and quality baselines, and the technical quality dimension focuses on the deliverables themselves. These three dimensions complement each other, covering the main aspects of skills assessment. Through automated methods such as dynamic time regulation, rule matching, and code analysis, the assessment achieves objectivity, standardization, and efficiency. The final structured report, coupled with visual playback and improvement suggestions, transforms the assessment results from simple scores into personalized learning guidance plans, directly promoting improved teaching effectiveness.

[0026] A second aspect of this application provides a training and evaluation system based on multimodal data fusion, comprising: a knowledge graph construction and management module for performing step S1; a fusion agent dynamic deployment module for performing step S2; a multimodal data synchronous acquisition and real-time processing module for performing step S3; a scene understanding model dynamic construction module for performing step S4; an enhanced result package generation and submission module for performing step S5; and an intelligent evaluation and correlation analysis module for performing step S6.

[0027] In the above embodiments, the system protects the hardware architecture and software module entities that implement the method. Through the collaborative work of various modules, efficient, reliable, and scalable physical support is provided for the implementation of the method. Attached Figure Description

[0028] Figure 1 A flowchart illustrating a training and evaluation method based on multimodal data fusion provided in an embodiment of this application; Figure 2 A schematic diagram of a training and evaluation system based on multimodal data fusion provided in an embodiment of this application; Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0030] To make the purpose, technical solution, and advantages of this application clearer, the following will be described in conjunction with the appendix. Figure 1-3 The following is an explanation using specific examples.

[0031] Please refer to Figure 1 , Figure 1 A training evaluation method based on multimodal data fusion, provided as an embodiment of this application, includes: S1: Construct a training task topology based on a knowledge graph. The knowledge graph maps knowledge points, skill points, learning resources and device interfaces to nodes, and defines the logical and data flow relationships between them through directed edges. S2: In response to the training task, dynamically deploy a multimodal data fusion agent adapted to the current task node. The fusion agent preloads targeted recognition models and fusion rules. S3: During the practical training, the raw data streams from machine vision devices and intelligent sensor networks are collected synchronously and processed in real time by the fusion agent to generate standardized feature streams with semantic labels. S4: The fusion agent, based on a preset fusion strategy, performs online spatiotemporal correlation and inference on standardized feature streams of different modalities, and dynamically constructs and maintains a task scenario understanding model that reflects the current operational context. S5: Upon completion of a task node, encapsulate the training program code, report, and process summary data derived from the task scenario understanding model to form an enhanced training outcome package and submit it. S6: On the training platform, a correlation analysis interface is provided, which synchronously presents the code, reports, and visual operation process reconstructed based on process summary data in the enhanced training result package; it calls the rule base and case model to perform multi-dimensional automatic comparison and generate evaluation results.

[0032] Understandably, a knowledge graph is a technology that represents knowledge using a graphical structure. Here, it's used to establish the topology of training tasks, abstracting discrete knowledge points, operational skill points, teaching materials, and real equipment interfaces into nodes in the graph, and using directed edges to clearly define their logical order, dependencies, and data transmission paths. The multimodal data fusion agent is a dynamically deployable software agent that can acquire and load appropriate recognition models and data processing rules from the cloud based on the current specific training task node. Machine vision devices and intelligent sensor networks are hardware facilities used to collect raw data; the former, such as cameras, captures the operator's actions and postures, while the latter, such as various sensors, monitors the equipment's current, voltage, displacement, and other physical parameters. The system integrates and processes these raw data streams in real time, transforming them into standardized feature streams with uniform format and specific meaning labels such as tightening, measuring, and wiring. The agent synchronizes and aligns these feature streams from different sources in time and space according to a strategy, and dynamically constructs a task scenario understanding model through inference. This model can interpret operational intentions, determine compliance, and monitor system status in real time. Process summary data is a condensed record of key operational events and states output by this model. Finally, the program code, training report, and process summary data are packaged into an enhanced training outcome package. During evaluation, the platform calls a pre-built rule base and excellent historical case models to conduct a multi-dimensional automatic comparative analysis of the outcome package, covering operational procedures, safety specifications, and technical achievements.

[0033] This application's embodiments achieve the systematic and personalized organization of practical training tasks through knowledge graphs, and utilize intelligent agents deployed at the edge to realize real-time semantic understanding and modeling of deeply integrated multimodal perception data throughout the entire operation process. This completely changes the traditional evaluation model that relies on result inspection and manual observation, enabling evaluation to be based on complete and objective process data, and expanding the evaluation dimensions from single results to the entire process behavior. Simultaneously, automated multi-dimensional comparison and the generation of structured evaluation reports greatly improve evaluation efficiency and consistency, and provide trainees with traceable, visualized, and accurate feedback. Ultimately, the entire method forms a complete data closed loop from teaching process data collection to intelligent evaluation, and then to driving teaching paths and model optimization, laying the core technological foundation for achieving adaptive and continuously improving intelligent practical training education.

[0034] In some embodiments of this application, step S1, which involves constructing a knowledge graph-based training task topology, specifically includes: constructing and storing the topology of the knowledge graph using a graph database, wherein node attributes include node type, resource identifier, access path of associated multimedia resources, multimodal data acquisition parameters required to complete the node, and a description of the expected standardized output; directed edge attributes include edge relationship type, content format and protocol specifications for data transmission between adjacent nodes, and condition thresholds for determining the completion status of the current node to activate subsequent nodes; the graph supports editing and logical verification through a visual interface, and can automatically generate or recommend personalized training paths containing different difficulty sequences and resource combinations based on preset teaching objectives.

[0035] As we can understand it, a graph database is a data management system specifically designed to store entities and the relationships between them. Here, it's used to persistently store the mesh structure of the entire knowledge graph. Node attributes describe the specific characteristics of each element in the graph. Node type indicates whether the node represents an abstract theoretical knowledge point, a specific operational skill point, a learning resource, or a callable real device interface. The resource identifier is the node's unique identity code within the system. The access path to the associated multimedia resource is a link pointing to the storage location of the accompanying instructional video, 3D simulation program, or electronic document. The multimodal data acquisition parameters required to complete the node are a set of instructions specifying which specific data acquisition devices and their operating modes need to be enabled to evaluate the task's execution process. For example, it might require activating a top camera to capture hand movements from a top-down perspective, while simultaneously acquiring current data from a sensor at a frequency of 100 times per second. The description of the expected standardized output clearly defines the conditions that a qualified output must meet to complete the task. Directed edge attributes define the connection rules between nodes; the edge relationship type describes the logic between two connected nodes, such as a dependency relationship that must be completed sequentially, or a parallel relationship that can be performed simultaneously. The content format and protocol specifications for data transmission between adjacent nodes define the data generated by a task node upon completion, such as a code snippet of a control program or a set of device parameters, and specify the data format and communication protocol used to transmit it to subsequent task nodes. The threshold condition used to determine the completion status of the current node to activate subsequent nodes is a quantifiable standard, such as requiring the current task's automatic score to be above 90 points, or a key operation to be correctly recognized by the system. The graph supports editing and logical verification through a visual interface, meaning that teaching administrators can intuitively design or adjust training task flows through graphical drag-and-drop, and the system backend will automatically check for logical errors such as circular dependencies in the flow. The ability to automatically generate or recommend personalized training paths containing different difficulty sequences and resource combinations based on preset teaching objectives means that the system can automatically plan the most suitable learning route for a student from a vast knowledge graph based on the student's initial ability assessment results and learning objectives. This may skip the basic modules already mastered and directly enter advanced challenges, or insert additional practice resources for their weak areas.

[0036] The specific implementation principle of this embodiment lies in the digital deconstruction and intelligent reorganization of the complex teaching system. It transforms the traditional linear course catalog into a network knowledge model rich in semantic associations and attribute constraints. For example, in intelligent manufacturing training, teachers can use "industrial robot circular trajectory programming" as a skill point node, linking trajectory planning demonstration videos and programming manuals to the node's attributes, and setting that evaluating this skill requires collecting actual robot trajectory data. The directed edges between nodes clearly indicate that students must first complete the two prerequisite nodes: "robot linear motion programming" and "coordinate system establishment."

[0037] This application achieves refined, structured, and computable management of training task topology by employing a graph database and defining detailed node and edge attributes. This allows previously discrete knowledge points, skill points, and resources to be organically organized according to teaching logic, and clearly defines data flow and completion conditions. Visual editing and logical verification functions lower the barrier to constructing complex training paths, while the goal-based personalized path recommendation function enables the assessment system to flexibly adapt to the starting points and goals of different students, laying a solid foundation for accurate assessment tailored to individual needs.

[0038] In some embodiments of this application, the dynamic deployment of the multimodal data fusion agent in step S2 specifically includes: when a student starts a specific training task node, the system dynamically obtains and loads a lightweight model component set from the cloud model service center according to the preset multimodal data acquisition parameters in the node's attributes; the model component set includes at least a visual analysis model for recognizing specific hand movements and tool states in the current task scenario, a physical parameter analysis model for parsing the output signals of various sensors of the current training equipment, and a lightweight inference engine suitable for multi-source heterogeneous data fusion.

[0039] Understandably, the cloud-based model service center is a centralized, remotely accessible server cluster responsible for storing, version management, and distributing various specialized algorithm models and data processing rules. The lightweight model component set refers to specially optimized software module packages that maintain core recognition accuracy while being smaller and having lower computational overhead, adapting to the limited local computing resources of student terminals or edge gateways. The visual analysis model is an artificial intelligence algorithm trained on a large amount of data, capable of recognizing visual elements and actions specific to the current task scenario from real-time video streams. For example, in electrical wiring training tasks, this model is configured to specifically recognize whether students are holding screwdrivers correctly and whether they are accurately inserting wires into terminal holes. The physical parameter analysis model is an algorithm used to process and understand raw signals from various physical sensors. For example, in hydraulic system debugging tasks, this model is responsible for analyzing the voltage signals from pressure sensors and converting them into pressure value readings with clear engineering significance, while also determining whether the reading is within a safe range. The lightweight inference engine is a core software module deployed locally, responsible for coordination and decision-making. It incorporates rules and strategies for multimodal data fusion, and can receive and synchronously process the output results from visual analysis models and physical parameter analysis models, perform time alignment and logical correlation on them, and finally make a comprehensive judgment.

[0040] The specific implementation process of this embodiment is as follows: When the trainee selects a specific task (such as "PLC control motor start / stop debugging") on the platform front end and clicks start, the system backend immediately parses the preset data acquisition requirements in the task node; subsequently, it sends a request to the cloud model service center to accurately obtain the visual model (recognizing button presses and indicator light status), physical analysis model (interpreting current and speed sensor data), and fusion inference engine, etc., tailored to this task. These components are packaged into a temporary fusion agent, which is quickly deployed and runs on the training computer or edge computing device next to the trainee's workstation.

[0041] This application achieves dynamic, on-demand deployment of multimodal perception and analysis capabilities through cloud configuration and edge loading. It solves the problems of cumbersome and inflexible traditional fixed data acquisition systems, enabling the system to accurately call upon the most relevant lightweight analysis models based on specific training task scenarios. This mechanism greatly improves the targeting and efficiency of edge-side data processing, providing a reliable guarantee for subsequent high-quality feature extraction and real-time fusion, and is key to realizing complex multimodal analysis in resource-constrained environments.

[0042] In some embodiments of this application, step S2 further includes: after the fusion agent is initialized, its built-in microservice kernel is responsible for scheduling and managing the visual analysis model, the physical parameter parsing model, and the lightweight inference engine; the kernel maintains a shared data cache based on a sliding time window locally for temporarily storing the intermediate feature data output in real time by each model; by constructing a standardized real-time data processing pipeline, the raw data access, feature extraction and standardization, and multimodal data alignment are executed sequentially, and finally a standardized feature stream with unified timestamps and semantic labels is output that can be directly used by subsequent steps.

[0043] Understandably, the fusion agent is a lightweight intelligent software entity deployed on student training terminals or edge computing nodes. Its built-in microservice kernel is a software architecture that breaks down core functions into independent, collaborative microservice components, specifically used for efficiently scheduling and managing visual analysis models, physical parameter parsing models, and a lightweight inference engine. The sliding time window shared data cache refers to a shared area in memory that stores recent data in chronological order. New data is continuously added, and old data is automatically removed after a set interval. This design ensures that different processing modules can access aligned intermediate data within the same time period. The standardized real-time data processing pipeline refers to a series of data processing stages connected in a fixed order. Starting with the input of the raw byte stream, it undergoes feature extraction to transform it into meaningful numerical vectors, then standardizes the data to make data of different dimensions comparable, and finally completes the precise alignment of multi-source data on timestamps, ultimately producing a feature data stream with unified semantic labels and a time benchmark.

[0044] The implementation principle of this embodiment is as follows: When a student starts a specific training task on the platform's student end, such as "Industrial Robot Trajectory Programming and Debugging," the system dynamically instantiates a fusion agent based on the node attributes of the task in the knowledge graph. The agent's microservice kernel is then activated, acting as a command center, simultaneously starting and coordinating three core services: a visual analysis service specifically for recognizing the student's hand gestures on the teaching pendant, a physical parameter service for parsing robot joint currents and servo states, and an inference service responsible for fusion judgment. The shared cache managed by the kernel continuously records intermediate results from the past few seconds, such as the "hand presses button A" event output by the visual service and the "joint 1 current surge" event output by the physical service. Subsequently, a pre-defined pipeline begins operation: it first receives raw video frames and current / voltage signals from the camera and sensor network; then, the visual model extracts the feature "button pressed" from the video, and the physical model extracts the feature "current value 15A" from the signals, converting them into a standard format; the pipeline's alignment module finds that the timestamps of these two features are extremely close, so it correlates them, adds a semantic label of "execution of start command," and attaches a unified timestamp, ultimately outputting a standardized feature record. This process continues, providing a real-time, clean, and preliminarily correlated data foundation for subsequent deep scene understanding.

[0045] The beneficial effects of this embodiment are a significant improvement in the real-time processing efficiency and collaborative capabilities of multimodal data at the edge. Through the design of a microservice-based kernel and shared cache, fine-grained scheduling of computing resources and low latency in data exchange are achieved, avoiding bottlenecks caused by inter-model communication or data format conversion. The standardized processing pipeline ensures a stable, reliable, and traceable transformation process from heterogeneous raw data to high-quality feature streams. This enables the fusion agent to function as a self-contained intelligent unit, independently completing complex real-time preprocessing and initial fusion of multi-source data in resource-constrained edge environments. This significantly reduces the computational and transmission pressure on cloud servers, providing crucial technical support for the real-time performance, scalability, and immediate response capabilities of the entire system in complex training scenarios.

[0046] In some embodiments of this application, the raw data stream synchronously collected and processed by the fusion agent in step S3 further includes: a student physiological signal data stream collected by a wearable biosensor; the fusion agent has a built-in biosignal feature extraction module, which is used to calculate heart rate variability index, skin conductance response level or attention index in real time from the physiological signal data stream; the standardized feature stream not only includes operational behavior features and device status features, but also fuses physiological features as an independent modality.

[0047] As we can understand, wearable biosensors refer to devices that can be worn on a user's body to collect physiological signals, such as smart bracelets or EEG headbands. The physiological signal data stream is the sequence of raw electrical or optical signals continuously generated by these sensors, reflecting the body's physiological state. The biosignal feature extraction module is a dedicated software component within the fusion agent. Its function is to calculate and output quantitative indicators with clear physiological or psychological significance from the raw, noisy physiological signal stream through algorithms. Examples include heart rate variability indicators reflecting stress and autonomic nervous system regulation, skin conductance response levels characterizing emotional excitement or tension, and attention indexes assessing the degree of concentration.

[0048] The implementation principle of this embodiment is to quantify and incorporate the trainees' internal cognition and psychological state into the evaluation system. Specifically, during the training process, in addition to operating the equipment, trainees also need to wear smart devices integrated with sensors. For example, during circuit board troubleshooting training, the smart bracelet worn by the student continuously collects their electrocardiogram (ECG) and skin conductance signals. The biosignal feature extraction module built into the fusion agent receives these raw signal streams in real time. It first filters and denoises the signals, and then uses specific algorithms, such as calculating the time variation of continuous heartbeat intervals to obtain heart rate variability, or measuring changes in skin conductivity to obtain skin conductance response values. These calculated indicators, along with the behavioral characteristics of the student repeatedly using a multimeter to measure a certain point as identified by the camera, and the equipment characteristics of abnormal voltage in a certain branch of the circuit obtained from the sensors, are sent together to the subsequent fusion process. The system correlates and fuses the physiological characteristics of decreased heart rate variability and increased skin conductance response with hesitant and repetitive operating behavior and the failure to locate the fault state, thereby inferring that the student may be affecting the troubleshooting efficiency due to tension or cognitive overload.

[0049] This application extracts indicators such as focus and cognitive load in real time and integrates them with other modal features, enabling the system to more comprehensively understand the human-computer interaction process. This not only makes the evaluation results more interpretable but also provides adaptive human-computer interaction guidance based on real-time physiological states, achieving more humanized and intelligent training assistance and evaluation.

[0050] In some embodiments of this application, step S4, which involves dynamically building and maintaining a task scenario understanding model, specifically includes: the fusion agent receiving standardized feature streams from different data sources, dynamically calculating the contribution weights of visual features, physical features, and physiological features based on the target of the current task stage through a configurable attention weight allocation network; then, weighted fusion of the contribution weights, and online inference based on the weighted fusion feature sequence, outputting a joint estimate of the matching probability between the executed action and the standard process, the risk level of the operation behavior violating safety rules, and the abnormal index of the device system operating status.

[0051] Understandably, a configurable attention weighting network is a simplified computational model that mimics human attention mechanisms. It dynamically and differentially focuses on features from different data sources based on the different objectives of the current task stage. For example, in the initial task configuration stage, it might focus more on the visual features of whether the student has read the safety procedures; while in the program execution stage, it focuses more on the physical features of the device's operating status. Online inference refers to the process by which the model performs real-time calculations and judgments during continuous data input. The final model output is a joint estimate of multiple key scenario attributes, specifically including the probability of matching the executed action with the standard procedure (a value between 0 and 1 representing the degree of compliance of the current operation step); the risk level of the operation behavior violating safety rules, used to quantitatively assess the safety of the operation; and an abnormal index of the device system's operating status, used to determine whether the device is in an unexpected fault or dangerous state.

[0052] The implementation principle of this embodiment is to simulate the observation, focusing, and comprehensive judgment thinking of a proctor. After receiving standardized feature streams from visual, physical, and physiological channels, the fusion agent does not treat all information equally. Its internal attention weight allocation network works first, dynamically adjusting the weights of each modality feature based on the current task stage goal defined in the knowledge graph. Taking the "Industrial Robot Precision Assembly" training as an example, when students begin the "assembly" task node, the multimodal fusion agent deployed on-site has already loaded a visual model (recognizing hand grasping and alignment actions), a physical model (analyzing six-axis force sensor data and visual positioning coordinates), and a rule engine specific to the scenario. The standardized feature stream received by the agent includes: visual features (such as "the robotic arm gripper is closed," "workpiece A is in the center of the field of view"), physical features (such as "axial pressure is 5N," "X-axis deviation is 0.1mm"), and optional physiological features (such as "student concentration index is 85"). At this time, the agent's built-in configurable attention weight allocation network begins to work. It dynamically adjusts the "discourse weight" of each modality feature according to the current stage goal of the assembly process. For example, in the initial grasping phase, the network assigns higher weights to "visual features" and "six-dimensional force sensor features" to focus on determining whether the grasping posture is correct and the force is appropriate, while the weight of physiological features may be lower at this stage. When entering the precise alignment phase, the network may automatically increase the weights of "high-precision visual positioning features" and "micro-force sensor features" to monitor the alignment process with millimeter-level precision. Simultaneously, if the student's physiological characteristics indicate "decreased concentration," the weight will be appropriately increased to assess the operational stability risk. This dynamic weight allocation allows the model to focus on different key signals at different stages, much like an experienced teacher. After dynamic weighting, all features are fused into a "context vector" that represents the current overall situation. A lightweight online inference model (such as an existing temporal convolutional network model) analyzes this vector sequence in real time. It does not independently judge each indicator but performs joint estimation: the output may include "the current alignment action matches the standard procedure with a probability of 92%", "due to instantaneous exceedance of axial pressure, the safety violation risk level has increased to medium", and "simultaneously, due to the continuous reduction in positioning deviation, the equipment status anomaly index is low". These three outputs are interconnected, collectively forming a quantified, multi-dimensional snapshot of the scene understanding. This model is dynamically built and maintained, meaning it is not a static checklist, but rather a memory and decision-maker that evolves in real time with the workflow. It continuously combines the latest inference snapshot with historical states to form a coherent understanding of task progress.For example, when the model detects that the "matching probability" drops due to a shaking motion, but the "risk level" does not increase and subsequent actions quickly correct the error, it can understand that this is a minor, recoverable mistake. Conversely, if the "risk level" and "anomaly index" rise simultaneously, even if the "matching probability" has not yet plummeted, it can provide an early warning of potential systemic failures. This deep, context-sensitive understanding is something that traditional systems based on threshold alarms or fixed rules cannot achieve, and it provides the core basis for generating accurate assessment reports and intervention recommendations.

[0053] This application achieves dynamic weighted fusion and deep semantic inference of multimodal features by introducing a configurable attention mechanism and temporal reasoning. It overcomes the limitations of simple weighting or rule matching, enabling the model to dynamically focus on key information based on task stages. Its output joint probability estimates (matching degree, risk level, anomaly index) provide a quantified, multi-faceted interpretation of the operational scenario, elevating raw sensor data and video streams to an understandable and judgmental semantic level, providing a direct and reliable basis for automated and refined evaluation decisions.

[0054] In some embodiments of this application, step S4 further includes: during the operation of the task scenario understanding model, the model continuously generates structured scenario understanding snapshots; when the matching probability inferred by the task scenario understanding model is lower than the safety threshold or the risk level exceeds the preset level, the task scenario understanding model immediately triggers a two-level response. The first level is to generate and push adaptive correction guidance information on the student operation interface, and the second level is to package and mark the high-risk event together with panoramic multimodal data fragments within a specific time window before and after its occurrence as key diagnostic events and embed them into the process summary data for subsequent review and in-depth analysis.

[0055] Understandably, the matching probability is a quantified value calculated by the model through inference, representing the degree of consistency between the student's current actual operation steps and the predefined standard operating procedures in the knowledge graph. The safety threshold and preset level are pre-set numerical limits used to determine whether the operation carries risk. When the matching probability falls below the safety threshold or the calculated risk level exceeds the preset level, the model immediately activates a two-level response mechanism. The first level of response generates and displays targeted text, graphic, or voice correction guidance information in real time on the software interface the student is using. The second level of response treats this high-risk event as an independent data unit. This data unit includes not only the event itself but also complete multimodal data fragments from a period before and after the event, such as all video frames, sensor reading sequences, and any possible physiological signal data within that time period. This data is packaged and marked as key diagnostic events and ultimately integrated into the process summary data for detailed review and analysis by teachers or the system.

[0056] The specific implementation principle of this embodiment lies in endowing the evaluation system with real-time process intervention and in-depth recording capabilities. The task scenario understanding model serves as the core, continuously monitoring the fused multimodal feature flow. For example, in an industrial robot teaching and programming training exercise, the standard procedure requires zero-point calibration before trajectory programming. The model visually identifies whether the student has performed the calibration action and uses sensors to confirm whether the robot has returned to the zero-point position signal. If the student skips calibration and starts programming directly, the probability of process matching calculated by the model will significantly decrease. Once it falls below a safety threshold, the system immediately executes a two-level response: First, a prompt box stating "Please perform zero-point calibration first" is prominently displayed on the student's programming interface, and subsequent operations may be paused; simultaneously, the system automatically saves panoramic data from 30 seconds before skipping calibration to 10 seconds after the prompt appears, including screen recordings (showing the student's operating interface), camera footage (capturing the student's operating behavior), robot controller status data streams, etc., packaging and marking it as a "missing critical safety step" event, and embedding it into the student's task process summary. This provides teachers with a complete and comprehensive chain of evidence for quickly locating problems and conducting safety awareness education during grading.

[0057] This application endows the scenario understanding model with the ability to intervene in real time and record in depth. A two-level response mechanism enables a process-oriented assessment: the first level provides immediate guidance to trainees, demonstrating the real-time and closed-loop nature of teaching intervention; the second level packages and marks high-risk events and their comprehensive contextual data, forming high-value diagnostic cases for in-depth analysis. This ensures both training safety and learning efficiency, while providing teachers with focused and comprehensive review materials, greatly improving the accuracy and efficiency of subsequent assessments and reviews, and achieving a combination of real-time correction and post-event in-depth analysis.

[0058] In some embodiments of this application, step S5, which forms an enhanced training outcome package, specifically includes: the enhanced training outcome package is encapsulated in a self-describing structured format, and its content includes at least a metadata block identifying the student's identity and task information, a block of the program source code file and executable file finally submitted by the student, a block of structured training report document written by the student, and a process summary data block derived from the task scenario understanding model; the process summary data block includes a list of key operation events organized by timeline, a summary of equipment status time series curves, and any key diagnostic events marked by the model and their associated multimodal data indexes, ensuring the integrity of the results and the traceability of the evaluation process.

[0059] Understandably, a self-describing structured format means that the data packet itself contains metadata describing its internal organization, allowing the receiver to parse its content without external documents. It typically contains several logical blocks: a metadata block recording basic identification information such as student ID, name, task name, and submission time; a program source code and executable file block storing the original code written by the student and the final generated program; a structured training report document block containing a summary document filled out by the student according to certain formatting requirements; and a process summary data block containing data automatically generated by the system that condenses the essence of the operation process. The list of key operation events organized chronologically resembles an operation log, the equipment status time-series curve summary reflects the changes in key equipment parameters during the training process in graphical form, and the key diagnostic events and their associated multimodal data indexes provide links or identifiers to original video and sensor data fragments, ensuring traceability from evaluation conclusions to original evidence.

[0060] The specific implementation principle of this embodiment is to construct a complete digital archive describing a training activity by defining a unified data encapsulation specification. During system implementation, when a student clicks to submit a task, the encapsulation module initiates a standardized workflow. For example, after completing a "PLC-based motor start-stop control" training exercise, the system automatically collects the following: the student's identity information and task ID (metadata), the ladder diagram program file written in the programming software (program block), the training report filled out online by the student including design ideas and debugging summaries (report block), and the process summary data of this training exercise generated by the fusion agent. This process summary may include: an event list (such as "15:30:21 Power on", "15:31:05 Download program to PLC", "15:31:20 Press start button"), a motor current curve, and a "safety specification ignored" diagnostic event index marked by the model when "the start button" was pressed because the emergency stop button was not checked first. All this data is packaged into a separate file or data package according to a predefined XML or JSON format and uploaded to the server. Once teachers receive this deliverable package, they can analyze its self-descriptive structure and clearly obtain all the assessment materials.

[0061] This application ensures the integrity, self-containment, and traceability of the evaluated object through a self-describing structured encapsulation format. It organically integrates program code, text reports, and semantically rich process data (event lists, state curves, diagnostic events) into a single data package. This design allows evaluators to obtain all the information needed for the evaluation without navigating multiple systems or consulting scattered logs, and allows them to trace back to the original multimodal evidence through data indexing. This fundamentally solves the problem of fragmented evidence chains in process evaluation, guaranteeing the reliability and impartiality of the evaluation conclusions.

[0062] In some embodiments of this application, step S6 involves multi-dimensional automatic comparison to generate evaluation results, specifically including three analytical dimensions: Operational process standardization dimension, which involves dynamically time-normalizing and comparing the sequence of operational events in the process summary data with the standard operating procedures stored in the knowledge graph to calculate the step sequence fidelity and time efficiency score; Safety and quality compliance dimension, which involves matching the event and state data in the process summary data with the domain rule base to automatically identify violations and classify and quantify them; and Result technical quality dimension, which involves verifying the functional correctness of the program code through static analysis and dynamic testing, and using a case model library to evaluate the similarity of code quality and logical rationality. The scores from the three dimensions are combined with preset weights to generate a final evaluation report, which includes scores for each item, as well as a visual replay of key operational segments that led to deductions and improvement suggestions.

[0063] Understandably, the operational process standardization dimension focuses on whether the student's actual operational actions are correct and efficient. Its core technology, dynamic time warping comparison, is an algorithm that can flexibly compare two sequences that may differ in length or speed on the timeline, such as a sequence of student operational events and a standard operating procedure sequence. Even if the student's operation has pauses or delays, it can accurately calculate the sequence fidelity and time efficiency. The safety and quality compliance dimension focuses on whether the operation and results comply with hard rules and quality standards. The domain rule base is a database storing all knowledge such as safe operating procedures and allowable ranges of process parameters. The system identifies violations by matching the event and state data in the process summary with the rule base entries. The outcome technical quality dimension evaluates the technical level of the student's final output, such as program code. Static analysis checks the syntax and structure without running the code, while dynamic testing verifies its functionality by running the code. The case model library contains typical excellent code examples and common error patterns for code quality similarity assessment. Finally, the system combines the quantitative scores from the three dimensions according to preset weights to generate an evaluation report that includes the total score, sub-scores, a link to a visual replay, and textual suggestions for improvement.

[0064] The specific implementation principle of this embodiment is to construct a three-dimensional, automated evaluation framework covering process, compliance, and outcome. After receiving the outcome package, the evaluation engine will launch three analysis pipelines in parallel. Taking the "CNC machine tool parts machining" training as an example, in the process standardization dimension, the system extracts the student's operation event sequence (such as "clamping the workpiece," "tool setting," "setting parameters," and "starting machining"), compares it with the standard SOP sequence for dynamic time warping, and finds that the student repeated "setting parameters" twice after "tool setting," and the total time exceeded the standard by 20%, thus calculating the score for this dimension. In the safety and quality compliance dimension, the system checks the spindle speed and feed rate data curves in the process summary, finds that they exceed the recommended values ​​for the material in the rule base, and identifies the visual event of not wearing protective glasses (from a critical diagnostic event), and deducts points accordingly. In the outcome technical quality dimension, the system performs static checks on the G-code written by the student and drives the simulation software to perform dynamic machining simulation to verify dimensional accuracy. At the same time, it compares it with efficient and safe code templates in the case library to evaluate its code quality. Finally, the system combines the three scores with a weight of 6:2:2 to generate a report. Users can directly click on the links in the report to replay the video footage of the "parameter setting repetition" and "speed over-limit" moments.

[0065] This application clearly defines three core dimensions of the assessment and their specific implementation algorithms, forming a comprehensive, multi-dimensional, and automated assessment framework. The process standardization dimension focuses on operational procedures, the compliance dimension focuses on safety and quality baselines, and the technical quality dimension focuses on the deliverables themselves. These three dimensions complement each other, covering the main aspects of skills assessment. Through automated methods such as dynamic time regulation, rule matching, and code analysis, the assessment achieves objectivity, standardization, and efficiency. The final structured report, coupled with visual playback and improvement suggestions, transforms the assessment results from simple scores into personalized learning guidance plans, directly promoting improved teaching effectiveness.

[0066] Please refer to Figure 2 A second aspect of this application provides a training and evaluation system based on multimodal data fusion, comprising: a knowledge graph construction and management module 21 for performing step S1; a fusion agent dynamic deployment module 22 for performing step S2; a multimodal data synchronous acquisition and real-time processing module 23 for performing step S3; a scene understanding model dynamic construction module 24 for performing step S4; an enhanced result package generation and submission module 25 for performing step S5; and an intelligent evaluation and association analysis module 26 for performing step S6.

[0067] See Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 3The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to execute the aforementioned training and evaluation method based on multimodal data fusion.

[0068] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0069] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0070] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0071] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation methods described in any embodiment of the training and evaluation method based on multimodal data fusion provided in the embodiments of this application, or they can execute the implementation methods of the electronic devices described in the embodiments of this application, which will not be repeated here.

[0072] In another embodiment of this application, an electronic device is provided. The electronic device stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the training and evaluation method based on multimodal data fusion described in the above embodiments. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in an electronic device, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0073] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0074] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0075] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0077] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0078] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0079] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training evaluation method based on multimodal data fusion, characterized in that, include: S1: Construct a training task topology based on a knowledge graph. The knowledge graph maps knowledge points, skill points, learning resources and device interfaces as nodes, and defines the logical and data flow relationships between them through directed edges. S2: In response to the training task, dynamically deploy a multimodal data fusion agent adapted to the current task node. The fusion agent preloads a targeted identification model and fusion rules. S3: During the practical training operation, the raw data stream from the machine vision device and the intelligent sensor network is collected and processed in real time through the fusion agent to generate a standardized feature stream with semantic tags. S4: The fusion agent, based on a preset fusion strategy, performs online spatiotemporal correlation and reasoning on the standardized feature streams of different modalities, and dynamically constructs and maintains a task scenario understanding model that reflects the current operation context; S5: When a task node is completed, encapsulate the training program code, report, and process summary data derived from the task scenario understanding model to form an enhanced training result package and submit it. S6: On the training platform, a correlation analysis interface is provided to synchronously present the code, reports, and visual operation process reconstructed based on the process summary data in the enhanced training result package; The system automatically compares the rule base and case models across multiple dimensions to generate evaluation results.

2. The training and evaluation method based on multimodal data fusion according to claim 1, characterized in that, The step S1, which involves constructing a knowledge graph-based training task topology, specifically includes: The topology of the knowledge graph is constructed and stored using a graph database. Node attributes include node type, resource identifier, access path to associated multimedia resources, multimodal data acquisition parameters required to complete the node, and a description of the expected standardized output. Directed edge attributes include edge relationship type, content format and protocol specifications for data transmission between adjacent nodes, and conditional thresholds for determining the completion status of the current node to activate subsequent nodes. The graph supports editing and logical verification through a visual interface and can automatically generate or recommend personalized training paths containing different difficulty sequences and resource combinations based on preset teaching objectives.

3. The training and evaluation method based on multimodal data fusion according to claim 1, characterized in that, The dynamic deployment of the multimodal data fusion agent in step S2 specifically includes: When a student initiates a specific training task node, the system dynamically acquires and loads a lightweight set of model components from the cloud model service center based on the preset multimodal data acquisition parameters in the node's attributes. The set of model components includes at least a visual analysis model for recognizing specific hand movements and tool states in the current task scenario, a physical parameter analysis model for analyzing the output signals of various sensors of the current training equipment, and a lightweight inference engine suitable for multi-source heterogeneous data fusion.

4. The training and evaluation method based on multimodal data fusion according to claim 3, characterized in that, Step S2 further includes: After the fusion agent is initialized, its built-in microservice kernel is responsible for scheduling and managing the visual analysis model, physical parameter parsing model, and lightweight inference engine. The kernel maintains a shared data cache based on a sliding time window locally to temporarily store the intermediate feature data output by each model in real time. By constructing a standardized real-time data processing pipeline, the raw data access, feature extraction and standardization, and multimodal data alignment are executed in sequence, and finally a standardized feature stream with unified timestamps and semantic labels is output for direct use in subsequent steps.

5. The training and evaluation method based on multimodal data fusion according to claim 1, characterized in that, The raw data stream synchronously collected and processed by the fusion agent in step S3 further includes: The student physiological signal data stream is collected through wearable biosensors; the fusion agent has a built-in biosignal feature extraction module, which is used to calculate heart rate variability, skin conductance response level or attention index in real time from the physiological signal data stream; the standardized feature stream not only includes operational behavior features and device status features, but also fuses physiological features as an independent modality.

6. The training and evaluation method based on multimodal data fusion according to claim 1, characterized in that, The dynamic construction and maintenance of the task scenario understanding model in step S4 specifically includes: The fusion agent receives standardized feature streams from different data sources and dynamically calculates the contribution weights of visual, physical, and physiological features based on the target of the current task stage through a configurable attention weight allocation network. Then, it performs weighted fusion of the contribution weights and performs online inference based on the weighted fused feature sequence to output a joint estimate of the matching probability of the executed action with the standard process, the risk level of the operation behavior violating safety rules, and the abnormal index of the device system operation status.

7. The training and evaluation method based on multimodal data fusion according to claim 6, characterized in that, Step S4 further includes: During operation, the task scenario understanding model continuously generates structured scenario understanding snapshots. When the matching probability inferred by the task scenario understanding model is lower than the safety threshold or the risk level exceeds the preset level, the task scenario understanding model immediately triggers a two-level response. The first level is to generate and push adaptive correction guidance information on the student's operation interface. The second level is to package and mark the high-risk event along with panoramic multimodal data fragments within a specific time window before and after its occurrence as key diagnostic events and embed them into the process summary data for subsequent review and in-depth analysis.

8. The training evaluation method based on multimodal data fusion according to claim 1, characterized in that, The enhanced training outcome package formed in step S5 specifically includes: The enhanced training outcome package is encapsulated in a self-describing structured format. Its content includes at least a metadata block identifying the student's identity and task information, a block of the student's final submitted program source code file and executable file, a block of the structured training report document written by the student, and a process summary data block derived from the task scenario understanding model. The process summary data block contains a list of key operation events organized chronologically, a summary of equipment status time-series curves, and any key diagnostic events marked by the model and their associated multimodal data indexes, ensuring the integrity of the outcome and the traceability of the evaluation process.

9. The training evaluation method based on multimodal data fusion according to claim 1, characterized in that, Step S6 involves multi-dimensional automatic comparison to generate evaluation results, specifically including three analytical dimensions: The standardization dimension of the operation process is calculated by dynamically time-normalizing and comparing the sequence of operation events in the process summary data with the standard operation procedures stored in the knowledge graph, and then calculating the step sequence fidelity and time efficiency score. In the safety and quality compliance dimension, by matching event and status data in process summary data with domain rule base, violations are automatically identified and classified and quantitatively scored. In terms of the technical quality of the results, the functional correctness of the program code is verified through static analysis and dynamic testing, and the similarity assessment of code quality and logical rationality is conducted using a case model library. The final evaluation report is generated by combining the scores from the three dimensions with preset weights. The evaluation report includes the scores for each item, as well as a visual replay of the key operational segments that led to the deduction of points and suggestions for improvement.

10. A training and evaluation system based on multimodal data fusion for implementing the method of any one of claims 1 to 9, characterized in that, include: The knowledge graph construction and management module is used to perform step S1; The integrated agent dynamic deployment module is used to execute step S2; A multimodal data synchronous acquisition and real-time processing module is used to perform step S3; The scene understanding model dynamic construction module is used to execute step S4; An enhanced deliverables generation and submission module is used to execute step S5; The intelligent evaluation and correlation analysis module is used to perform step S6.

Images