Skill practical operation training system and method based on AI glasses

Abstract

The invention discloses a skill practical operation training system and method based on AI glasses, and relates to the technical field of augmented reality and artificial intelligence. The system comprises teacher-side AI glasses, a background processing and analysis system, student-side AI glasses and a data and management platform. The teacher end AI glasses collect first visual angle demonstration data including vision, tracks and voice; the background processing and analysis system performs multi-modal feature fusion and time sequence alignment on the demonstration data, and constructs a skill operation knowledge graph comprising a standard action unit and allowable error range parameters; the student side AI glasses collect real-time operation data and upload the real-time operation data; the background processing and analysis system compares the real-time operation data with a standard parameter set by using a dynamic time warping algorithm, and generates a comparison result signal and a correction feedback instruction; and the student-side AI glasses display the virtual guide image according to the instruction. According to the invention, standardized motion modeling, real-time deviation detection and visual augmented reality guidance in the skill practical operation process are realized, and the problems of non-uniform traditional practical operation training standards and lack of real-time deviation correction are solved.

Description

Technical Field

[0001] This invention relates to the fields of augmented reality and artificial intelligence, specifically to a skills training system and method based on AI glasses. Background Technology

[0002] In fields such as industrial manufacturing, medical surgery, equipment maintenance, and vocational education, standardized practical skills training is fundamental to ensuring operational safety and production efficiency. Traditional skills training typically employs an apprenticeship-style on-site teaching model or relies on paper-based operation manuals and two-dimensional video tutorials for centralized instruction. With the development of wearable technology, some training scenarios are beginning to incorporate smart glasses devices. These glasses utilize their first-person perspective recording function to document the operational process or enable real-time guidance from experts to on-site personnel through remote video communication, aiming to enhance the informatization level and interactive experience of the training process.

[0003] However, in the existing skills training technology system, there is a significant digital bottleneck in the transformation process from expert demonstrations to standardized teaching content. Current practical course production mainly relies on manual post-editing of expert demonstration videos, combined with manually written documents to develop standard operating procedures (SOPs). This experience-based approach not only consumes a significant amount of time, leading to slow course updates, but more importantly, unstructured video recordings and subjective text descriptions are difficult for computer systems to automatically parse into quantifiable motion characteristic data and logically rigorous execution standards. This results in a lack of objectivity and consistency in practical training standards, making it difficult to support large-scale, high-precision automated skills assessment and reuse. Summary of the Invention

[0004] The first aspect of this invention provides a skills training system based on AI glasses, which includes AI glasses for teachers, a back-end processing and analysis system, AI glasses for students, and a data and management platform.

[0005] The AI ​​glasses for teachers are configured to collect first-person perspective demonstration data from teachers. This data includes visual image data acquired through the visual acquisition unit, operation trajectory data acquired through the inertial measurement unit, and voice command data acquired through the audio acquisition unit. The AI ​​glasses for teachers then send the visual image data, operation trajectory data, and voice command data to the backend processing and analysis system.

[0006] The backend processing and analysis system includes an expert demonstration analysis module, a skills operation knowledge graph storage module, a student model comparison module, and a personalized guidance strategy generation module. The expert demonstration analysis module receives first-person perspective demonstration data, performs feature extraction and fusion operations on the data to construct a skills operation knowledge graph, and stores this knowledge graph in the skills operation knowledge graph storage module. The skills operation knowledge graph includes the execution order of standard action units, key feature vectors, and allowable error range parameters.

[0007] In constructing the skill operation knowledge graph, the expert demonstration analysis module performs feature fusion and alignment operations. Specifically, the expert demonstration analysis module selects the video frame timestamps in the visual image data as the primary alignment index. Within the exposure period corresponding to each video frame timestamp, the expert demonstration analysis module extracts multiple quantized trajectory parameter points from the operation trajectory data and performs mean calculation to generate discrete trajectory feature vectors. Subsequently, the expert demonstration analysis module binds the discrete trajectory feature vectors with the corresponding visual features to form a visual-motion joint feature frame. Based on the timestamp interval information in the voice command data, the expert demonstration analysis module associates the structured text data with the visual-motion joint feature frame to generate a multimodal feature frame sequence. According to the kinematic segmentation points and semantic segmentation points in the multimodal feature frame sequence, the expert demonstration analysis module segments the multimodal feature frame sequence into multiple independent standard action units.

[0008] In addition, the expert demonstration analysis module also performs verification and graph generation operations. The module extracts the action classification probability vector from the multimodal feature frame sequence and uses the maximum probability value as the classification confidence score. When the classification confidence score is greater than or equal to the automatic pass threshold, the module performs statistical analysis on the quantized trajectory parameter points, calculating the arithmetic mean and standard deviation. Based on the arithmetic mean and standard deviation, the module constructs an allowable error range and writes this range into the node attribute fields of the skill operation knowledge graph.

[0009] The student-side AI glasses are configured to collect real-time operation data from student users and upload it to the backend processing and analysis system, while also receiving and displaying guidance information. The student model comparison module compares the real-time operation data with the standard parameter set in the skill operation knowledge graph, generating a comparison result signal.

[0010] In the real-time comparison logic, the student model comparison module extracts real-time feature vectors from real-time operation data. These real-time feature vectors include action type labels, hand spatial coordinates, tool operation angles, and real-time inertial trajectory data. The student model comparison module extracts the real-time inertial trajectory data and compares it with the standard trajectory template in the standard action unit, then uses a dynamic time warping algorithm to calculate the similarity distance between the real-time inertial trajectory data and the standard trajectory template. The student model comparison module compares the similarity distance with the trajectory deviation threshold in the allowable error range parameter. If the similarity distance is greater than the trajectory deviation threshold, the student model comparison module generates a comparison result signal containing a trajectory deviation code.

[0011] The personalized guidance strategy generation module generates correction feedback instructions based on the comparison result signal and sends these instructions to the student's AI glasses. When the comparison result signal contains a positional deviation code, the module constructs a 3D difference vector pointing from the student's hand's 3D coordinates to the standard target's 3D coordinates and generates a vector guidance instruction containing this vector. When the comparison result signal contains an angular deviation code, the module calculates the difference quaternion between the real-time tool posture quaternion and the standard tool posture quaternion and converts this difference quaternion into Euler angle deviation values. The student's AI glasses' near-eye display module renders a dynamic arrow model based on the vector guidance instructions or renders numerical visualization instructions based on the Euler angle deviation values.

[0012] To achieve accurate augmented reality overlay, the personalized guidance strategy generation module also performs coordinate transformation calculations. The module obtains the extrinsic and intrinsic parameter matrices of the camera on the student's AI glasses and uses the model-view-projection matrix to convert the preset placement coordinates corresponding to the standard action units into two-dimensional pixel coordinates on the screen plane of the student's AI glasses. The module then generates guidance rendering instructions containing these two-dimensional pixel coordinates to drive the near-eye display module to draw virtual tool images on the optical display plane.

[0013] The background processing and analysis system is also configured to perform intelligent evaluation operations. It aggregates real-time comparison log data during the practice process. This real-time comparison log data includes a deviation type identifier sequence and a deviation quantification value sequence. The system calculates the operation accuracy score by subtracting the total deduction value from a preset full-score benchmark. The total deduction value is calculated by weighting and summing the cumulative trigger count of each deviation code in the deviation type identifier sequence with the corresponding deduction value in a preset deduction weight table. Simultaneously, the system acquires the accelerometer data sequence uploaded by the student's AI glasses, calculates the variance of the accelerometer data sequence, and normalizes the variance value to a preset extreme jitter variance threshold to calculate the operation stability score.

[0014] The data and management platform is configured with a hybrid storage architecture consisting of a relational database management system and an unstructured object storage system. The platform receives abnormal video clip data streams uploaded by students' AI glasses, stores these streams in the unstructured object storage system, and generates Uniform Resource Locators (URLs). The platform then writes these URLs into the practice history log table in the relational database management system and establishes a mapping between the URLs and specific action nodes in the skill operation knowledge graph.

[0015] A second aspect of the present invention provides a skills training method based on AI glasses, which is applied to the aforementioned skills training system based on AI glasses, and includes the following steps: The teacher-side AI glasses collect first-person perspective demonstration data and send it to the backend processing and analysis system. The first-person perspective demonstration data includes visual image data, operation trajectory data, and voice command data.

[0016] The expert demonstration analysis module of the background processing and analysis system performs feature extraction and temporal fusion on the first-person perspective demonstration data to generate a multimodal feature frame sequence, and constructs a skill operation knowledge graph based on it.

[0017] The AI ​​glasses on the student's end collect real-time operation data and upload it to the back-end processing and analysis system.

[0018] The student model comparison module of the background processing and analysis system compares real-time operation data with the standard parameter set in the skill operation knowledge graph and generates a comparison result signal.

[0019] The personalized guidance strategy generation module of the background processing and analysis system generates correction feedback instructions based on the comparison result signals, and sends the correction feedback instructions to the student's AI glasses for display and prompts.

[0020] This invention provides a skills training system and method based on AI glasses. It has the following beneficial effects: 1. This invention uses an expert demonstration analysis module to perform multimodal feature fusion and temporal alignment on the first-person view video stream, operation trajectory data, and voice command data collected by the teacher's AI glasses. It also uses kinematic segmentation points to automatically segment standard action units, thereby constructing a skill operation knowledge graph containing key feature vectors and allowable error range parameters. This transforms the unstructured expert demonstration process into a standard operating procedure that can be understood and executed by a computer, avoiding the subjective bias and high time cost of manually writing practical teaching plans, and ensuring the objectivity and consistency of practical training standards.

[0021] 2. This invention uses a student model comparison module to calculate the similarity distance between real-time operation data and a standard parameter set using a dynamic time warping algorithm. Combined with a personalized guidance strategy generation module, it generates guidance rendering instructions containing two-dimensional pixel coordinates based on the comparison results. Virtual tool images or dynamic guidance arrows are then overlaid on the near-eye display module of the student's AI glasses. This achieves real-time deviation detection and augmented reality visualization correction of student user operations, solving the problem of erroneous actions becoming ingrained in traditional practical training due to the inability of instructors to provide one-on-one guidance, and improving the standardization of skill acquisition.

[0022] 3. This invention summarizes real-time comparison log data and acceleration sensor data sequences during the practice process through a background processing and analysis system. It calculates the operation accuracy score using a preset deduction weight table and obtains the operation stability score by calculating the acceleration variance and normalizing the data. At the same time, it combines the persistent storage and correlation analysis of abnormal video segments with the data and management platform to achieve multi-dimensional quantitative evaluation and traceable digital file management of the skill practice process. This overcomes the shortcomings of inconsistent scoring standards and difficulty in completely retaining process data in traditional manual assessment. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall architecture of the system of the present invention; Figure 2 A flowchart for constructing a knowledge graph of skill operations; Figure 3 An interactive sequence diagram for student practice and real-time guidance. Detailed Implementation

[0024] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Please see the appendix Figure 1 - Appendix Figure 3 This invention provides a skills training system based on AI glasses, including AI glasses for teachers, a back-end processing and analysis system, AI glasses for students, and a data and management platform.

[0026] The teacher-side AI glasses are wearable smart terminals with data acquisition and wireless communication capabilities. They are designed to be worn on the teacher's head. The glasses collect first-person perspective demonstration data from the teacher. This data includes visual image data captured by a camera, operational trajectory data captured by an inertial sensor, and voice command data captured by a microphone. The teacher-side AI glasses transmit this demonstration data to a backend processing and analysis system via a wireless communication network.

[0027] The backend processing and analysis system is a server or server cluster containing at least one processor and memory. It communicates with the teacher's AI glasses, the student's AI glasses, and the data and management platform. The system operates multiple functional modules for data parsing, comparison, and strategy generation.

[0028] The back-end processing and analysis system includes the following functional modules: expert demonstration analysis module, skills operation knowledge graph library, student model comparison module, and personalized guidance strategy generation module.

[0029] The expert demonstration analysis module communicates with the teacher's AI glasses. This module receives demonstration data and extracts features from it to identify the object of the operation, the type of action, and key voice information.

[0030] The skills operation knowledge graph library is connected to the expert demonstration analysis module. The skills operation knowledge graph library stores the structured knowledge graph generated by the expert demonstration analysis module. The structured knowledge graph includes standard operating procedure data, key action node data, allowable error range thresholds, and standard demonstration data fragments.

[0031] The student-end AI glasses are wearable smart terminals with environmental perception and augmented reality display capabilities. They are designed to be worn on the head of a student. The glasses collect real-time operational data during student use and upload it to a backend processing and analysis system. The glasses are also configured to receive feedback signals from the backend processing and analysis system and output guidance information through optical display components or audio components.

[0032] The student model comparison module is connected to both the skill operation knowledge graph library and the student's AI glasses. This module receives real-time operation data, maps it to the structured knowledge graph, and calculates the deviation between the real-time operation data and the standard operation procedure data.

[0033] The personalized guidance strategy generation module is connected to the student model comparison module. The personalized guidance strategy generation module generates control instructions based on the deviation values. These control instructions are sent to the student's AI glasses to trigger the display of visual guidance symbols, the playback of voice prompts, or error correction alarms.

[0034] The data and management platform is connected to the back-end processing and analysis system. The data and management platform is configured as a database to store expert demonstration model files, student practice log data, and evaluation report data, and provides a data retrieval interface.

[0035] The AI ​​glasses terminal provided in this embodiment of the invention mainly includes: a central processing module, a multimodal data acquisition module, a near-eye display module, a wireless communication module, and a power management module.

[0036] The central processing unit (CPU) includes an application processor (AP), a digital signal processor (DSP), and memory. The CPU establishes electrical connections with the multimodal data acquisition module, near-eye display module, wireless communication module, and power management module via an internal bus. The CPU is equipped with a unified clock source to send synchronous clock signals to all connected modules for timestamping the acquired multidimensional data.

[0037] The multimodal data acquisition module is used to acquire physical information data about the wearer and the environment. Specifically, the multimodal data acquisition module includes: a visual acquisition unit, an audio acquisition unit, an inertial measurement unit, and an eye-tracking unit.

[0038] The visual acquisition unit is embedded in the front-end frame of the AI ​​glasses terminal. The visual acquisition unit includes a visible light camera sensor and a depth camera sensor. The visible light camera sensor is configured to acquire RGB image data of the environment. The depth camera sensor is configured to acquire point cloud depth data of the environment. The central processing module performs spatial registration between the RGB image data and the point cloud depth data.

[0039] The audio acquisition unit includes multiple microphone arrays positioned at different locations on the temples of the AI ​​glasses. The audio acquisition unit is configured to acquire directional speech signals using beamforming technology. The audio acquisition unit converts the analog speech signal into a digital speech signal and transmits it to the central processing module for echo cancellation.

[0040] The inertial measurement unit (IMU) includes a microelectromechanical system (MEMS) triaxial gyroscope and a triaxial accelerometer. The IMU is configured to acquire angular velocity and acceleration data at a preset sampling frequency. The data output by the IMU is represented as a vector group. .in, Represents angular velocity. Represents acceleration. These represent the three axes of a three-dimensional coordinate system. The central processing module uses vector groups... Calculate the pose quaternion of the AI ​​glasses terminal.

[0041] The eye-tracking unit includes an infrared camera and an infrared LED. The eye-tracking unit is positioned facing the wearer's eyes. It is configured to capture the coordinates of the pupil center and corneal reflection points, and calculate the gaze point coordinates based on a preset eye geometry model.

[0042] The near-eye display module includes an optical display unit and a zoom drive unit. The optical display unit includes a microdisplay and a waveguide lens. The waveguide lens is configured to transmit light emitted from the microdisplay to the wearer's eye via total internal reflection.

[0043] The zoom drive unit is connected to the optical display unit. The zoom drive unit is used to resolve visual convergence-accommodation conflict (VAC). In this embodiment, the zoom drive unit employs a displacement mechanism driven by a liquid lens or a voice coil motor. The zoom drive unit receives a focal length control signal from the central processing module and adjusts the virtual image distance of the virtual image.

[0044] Virtual image distance Satisfies the imaging formula: .in, The combined focal length of the optical system. The object distance is from the microdisplay to the optical center. The zoom drive unit changes the combined focal length. or object distance To adjust the virtual image distance , making the virtual image distance It is consistent with the gaze depth detected by the eye-tracking unit.

[0045] The wireless communication module includes radio frequency transceiver circuitry and an antenna. It supports Wi-Fi 6 or 5G communication protocols. The wireless communication module is used to upload video streams, audio streams, and sensor data streams packaged by the central processing module to the server, and to receive AR rendering commands from the server.

[0046] The power management module includes a lithium polymer battery and a power management integrated circuit (PMIC). The power management module is configured to regulate and shunt the input voltage, providing the required DC operating voltage for the aforementioned modules.

[0047] The backend processing and analysis system provided in this embodiment of the invention is deployed on the server side. The backend processing and analysis system includes: an expert demonstration analysis module, a skill operation knowledge graph storage module, a student model comparison module, and a personalized guidance strategy generation module.

[0048] The expert demonstration analysis module is configured to receive demonstration data from the teacher's AI glasses. The module includes computer vision algorithms, speech recognition algorithms, and motion trajectory calculation algorithms.

[0049] The expert demonstration analysis module runs a computer vision algorithm program to decompose the video stream data into a sequence of image frames. The module identifies tool features, workpiece features, and hand joint coordinates within the image frame sequence. Based on the start and end features of the action, the module segments the continuous image frame sequence into independent action segments.

[0050] The expert demonstration analysis module runs a speech recognition algorithm program, converting the speech data stream into text data. From the text data, the expert demonstration analysis module extracts verb and noun combinations to generate action description tags.

[0051] The expert demonstration analysis module runs the motion trajectory calculation algorithm program, filters and calculates the attitude of the inertial sensor data, and generates the motion trajectory curve of the hand in three-dimensional space.

[0052] The expert demonstration analysis module uses timestamp indexes to associate action segments, action description tags, and motion trajectory curves to generate a multimodal feature vector sequence.

[0053] The skills operation knowledge graph storage module is connected to the expert demonstration analysis module. The skills operation knowledge graph storage module is configured to build and store standard operating procedure graphs.

[0054] The skill operation knowledge graph storage module transforms multimodal feature vector sequences into directed graph structures. .in, Represents key action nodes. This represents the flow order relationship between action nodes.

[0055] The skill operation knowledge graph storage module stores the standard parameter set for each key action node. The standard parameter set includes: standard operation duration. Standard hand spatial coordinate sequence Standard tool operating angle and allowable error threshold .

[0056] The skills operation knowledge graph storage module establishes an index mapping relationship between standard parameter sets and correct demonstration video clips.

[0057] The student model comparison module is connected to both the skill operation knowledge graph storage module and the student-side AI glasses. The student model comparison module is configured to execute sequence matching logic.

[0058] The student model comparison module receives real-time operation data streams uploaded by students' AI glasses. The module uses Dynamic Time Warping (DTW) to calculate the similarity distance between the real-time operation data streams and the standard parameter set. .

[0059] Similarity distance The calculation formula is: in, For inclusion A sequence of real-time student actions in each frame. For inclusion A sequence of expert standard actions for each frame. To standardize the path, , of which Path elements Indicates the first action in the student's real-time action sequence. Frame and expert standard action sequence Frame alignment. The Euclidean distance is the distance between two feature points.

[0060] The student model comparison module will use similarity distance With the allowable error threshold Perform a numerical comparison. If the similarity distance... Greater than the allowable error threshold The student model comparison module determined that there was a deviation in the current operation.

[0061] The student model comparison module outputs deviation type identifiers and deviation quantification values. Deviation type identifiers include: sequence error identifier, position deviation identifier, angle deviation identifier, and timeout identifier.

[0062] The personalized guidance strategy generation module is connected to the student model comparison module. The personalized guidance strategy generation module stores a preset guidance strategy library.

[0063] The personalized guidance strategy generation module retrieves the corresponding feedback instructions from the guidance strategy library based on the deviation type identifier and the deviation quantification value.

[0064] When the student model comparison module does not output a deviation type identifier, the personalized guidance strategy generation module generates a confirmation command. The confirmation command is used to drive the student's AI glasses to display a symbol or play a confirmation sound effect.

[0065] When the student model comparison module outputs a positional deviation indicator or an angle deviation indicator, the personalized guidance strategy generation module generates a correction instruction. This correction instruction includes AR-rendered coordinate data and speech-synthesized text data. The AR-rendered coordinate data drives the student's AI glasses to draw virtual guide arrows or virtual tool outlines on the display plane. The speech-synthesized text data provides prompts for directional adjustments.

[0066] When the deviation quantification value output by the student model comparison module exceeds the preset safety warning value, the personalized guidance strategy generation module generates an interrupt command. The interrupt command is used to drive the student's AI glasses to play an alarm signal and retrieve the corresponding correct demonstration video clip from the skill operation knowledge graph storage module for overlay playback.

[0067] The expert demonstration data collection process provided in this embodiment of the invention is executed by the teacher's AI glasses. The expert demonstration data collection content includes: In response to the data acquisition start command, the central processing module of the teacher-side AI glasses sends synchronization trigger signals to the visual acquisition unit, audio acquisition unit, and inertial measurement unit. The central processing module locks its internal hardware clock source and uses the current system time as the starting time reference.

[0068] The main camera sensor in the visual acquisition unit acquires first-person perspective video stream data at a preset frame rate. The first-person perspective video stream data includes images of the operated device, the operating tools, and the teacher's hand within the operating area.

[0069] The depth camera sensor in the vision acquisition unit collects depth point cloud data of the operating area. The depth point cloud data includes the relative spatial distance between the operating tool and the operated device, as well as the three-dimensional spatial coordinates of the teacher user's hand joints.

[0070] The audio acquisition unit collects ambient acoustic signals through a microphone array. The unit then performs beamforming processing on the ambient acoustic signals to separate voice narration data and operational feedback audio data. The voice narration data includes the waveform of the teacher's spoken commands. The operational feedback audio data includes the sound waveform of physical impacts generated when tools come into contact with equipment.

[0071] The inertial measurement unit (IMU) acquires motion attitude data at a higher sampling frequency than the visual acquisition unit. The motion attitude data consists of three-axis acceleration vectors and three-axis angular velocity vectors. This data characterizes the pitch angle, yaw angle, and displacement acceleration of the teacher's AI glasses in the world coordinate system.

[0072] The central processing module establishes a data buffer queue. It receives first-view video stream data and depth point cloud data from the vision acquisition unit, voice narration data and operation feedback audio data from the audio acquisition unit, and motion attitude data from the inertial measurement unit.

[0073] The central processing module, based on a hardware clock source, assigns a unique generation timestamp to each frame of first-view video stream data, each frame of depth point cloud data, each audio sampling packet, and each motion posture data point.

[0074] The central processing module encapsulates first-person view video stream data, depth point cloud data, voice narration data, operation feedback sound data, and motion posture data into a multimodal demonstration data package based on the generated timestamp.

[0075] The wireless communication module is connected to the central processing module. The wireless communication module sends multimodal demonstration data packets to the back-end processing and analysis system. The expert demonstration analysis module in the back-end processing and analysis system parses the multimodal demonstration data packets to construct a skills operation knowledge graph.

[0076] The multimodal data processing procedure provided in this embodiment of the invention is executed by the expert demonstration analysis module in the background processing and analysis system. The multimodal data processing content includes: The expert demonstration analysis module receives multimodal demonstration data packets from the teacher's AI glasses. The module then initiates parallel processing threads to perform feature extraction operations on the visual image data, inertial trajectory data, and audio data.

[0077] In the visual image data processing workflow, the expert demonstration analysis module decodes the first-person view video stream data frame by frame. The module runs an object detection algorithm to locate the objects and tools being manipulated within the image frames. Finally, the module outputs the pixel region bounding box coordinates and category confidence scores for the objects and tools being manipulated.

[0078] The expert demonstration analysis module combines the depth point cloud data corresponding to the first-person view video stream data to convert the pixel region bounding box coordinates into three-dimensional spatial coordinates. The module also calculates the Euclidean distance and relative angle between the center point of the manipulation tool and the center point of the manipulated object.

[0079] The expert demonstration analysis module runs a posture estimation algorithm to extract the joint coordinate data of the teacher user's hands. Based on the temporal variation characteristics of the joint coordinate data, the module identifies the type of hand movement. Hand movement types include: grasping, twisting, pressing, and placing. The module generates a visual semantic sequence containing object labels, tool labels, relative position data, and movement type labels.

[0080] In the inertial trajectory data processing flow, the expert demonstration analysis module performs low-pass filtering on the triaxial acceleration and triaxial angular velocity data to remove high-frequency noise interference. The expert demonstration analysis module then performs attitude calculation on the filtered triaxial acceleration and angular velocity data.

[0081] The expert demonstration analysis module calculates the spatial attitude angles of the teacher's AI glasses using quaternion operations. Within a preset time window, the module integrates the triaxial acceleration data to calculate the displacement and velocity vectors of the hand in three-dimensional space. The module then generates a quantized trajectory parameter set containing the spatial attitude angles, displacement vectors, and velocity vectors.

[0082] In the voice audio data processing flow, the expert demonstration analysis module performs automatic speech recognition processing on the voice explanation data, converting the audio signal into a text character sequence. The expert demonstration analysis module then performs word segmentation and entity extraction operations on the text character sequence.

[0083] The expert demonstration analysis module extracts verbs from the text character sequence as operation instruction entities. It also extracts nouns as operation object entities. Furthermore, it extracts quantifiers and modifiers as operation parameter entities. Finally, it identifies the logical order of operation steps based on connecting words. The module generates structured text data containing operation instruction entities, operation object entities, operation parameter entities, and the logical order.

[0084] The expert demonstration analysis module writes the generated visual semantic sequence, quantized trajectory parameter set, and structured text data into a memory buffer as input data for subsequent feature fusion steps.

[0085] The feature fusion and alignment process provided in this embodiment of the invention is executed by the expert demonstration analysis module in the background processing and analysis system. The feature fusion and alignment process includes: The expert demonstration analysis module reads the visual semantic sequence, quantized trajectory parameter set, and structured text data from the memory buffer. The module selects the video frame timestamps from the visual semantic sequence as the primary alignment index.

[0086] The expert demonstration analysis module performs frequency alignment. Since the sampling frequency of the quantized trajectory parameter set is higher than that of the visual semantic sequence, the expert demonstration analysis module extracts multiple quantized trajectory parameter points within the exposure period corresponding to each video frame timestamp. The module then calculates the mean of these extracted quantized trajectory parameter points, generating discrete trajectory feature vectors that correspond one-to-one with each video frame timestamp.

[0087] The expert demonstration analysis module performs a spatial-semantic mapping operation. It concatenates discrete trajectory feature vectors into the feature channels of the visual semantic sequence, forming a joint visual-motion data frame.

[0088] The expert demonstration analysis module reads the timestamp interval information from the structured text data. It then maps the operation instruction entities and operation object entities from the structured text data to multiple vision-motion joint data frames covered by the timestamp intervals.

[0089] The expert demonstration analysis module performs semantic consistency verification. It calculates the text similarity between the operational object entities in the structured text data and the object classification labels in the visual semantic sequence. If the text similarity value is greater than a preset matching threshold, the expert demonstration analysis module writes the structured text data into the metadata field of the visual-motion joint data frame, generating a multimodal feature frame sequence.

[0090] The expert demonstration analysis module performs action boundary segmentation on the multimodal feature frame sequence. It iterates through the velocity vector values ​​in the sequence. The module identifies consecutive frame intervals where the velocity vector values ​​are below a rest threshold and marks the center point of each interval as a kinematic segmentation point.

[0091] The expert demonstration analysis module iterates through the logical connector entities in the multimodal feature frame sequence. The module marks time points containing logical connectors such as "start," "then," and "finally" as semantic segmentation points.

[0092] The expert demonstration analysis module integrates kinematic segmentation points and semantic segmentation points to determine the start and end frame indices of the standard action units. Based on these indices, the module segments the multimodal feature frame sequence into multiple independent standard action units.

[0093] The expert demonstration and analysis module assigns a unique sequence number to each standard action unit. It then encapsulates these standard action units as action node objects within a skill operation knowledge graph. Finally, the module constructs logical jump relationships between these action node objects in ascending order of their sequence numbers.

[0094] The verification and atlas generation process provided in this embodiment of the invention is executed by a background processing and analysis system. The verification and atlas generation process includes: the expert demonstration analysis module receiving multimodal action sequence data after action segmentation processing.

[0095] The expert demonstration analysis module extracts the action classification probability vector from the multimodal action sequence data. The module then selects the highest probability value from this vector as the classification confidence score.

[0096] The expert demonstration analysis module compares the classification confidence score with a preset automatic pass threshold.

[0097] When the classification confidence score is greater than or equal to the automatic pass threshold, the expert demonstration analysis module executes the automated parameter extraction logic. The expert demonstration analysis module determines the action type label in the multimodal action sequence data as the standard action name.

[0098] The expert demonstration analysis module performs statistical analysis on the quantized trajectory parameter set in multimodal motion sequence data. The module calculates the arithmetic mean and standard deviation of the trajectory parameter points.

[0099] The expert demonstration analysis module constructs an allowable error range based on the arithmetic mean and standard deviation. The upper limit of the allowable error range is the arithmetic mean plus a preset multiple of the standard deviation, and the lower limit is the arithmetic mean minus a preset multiple of the standard deviation.

[0100] When the classification confidence score is lower than the automatic pass threshold, the expert demonstration analysis module generates a manual verification request instruction. The expert demonstration analysis module then sends the multimodal action sequence data and the manual verification request instruction to the data and management platform.

[0101] The data and management platform parses multimodal action sequence data. It then generates display rendering data that includes video playback footage and predicted label text.

[0102] The data and management platform detects operation signals from input devices. These operation signals correspond to modifications to the action start frame index, action end frame index, or action type label.

[0103] The data and management platform generates a correction parameter package based on the operation signal and returns the correction parameter package to the expert demonstration analysis module.

[0104] The expert demonstration analysis module updates the multimodal action sequence data based on the correction parameter package.

[0105] The expert demonstration analysis module constructs a skill operation knowledge graph based on multimodal action sequence data that has undergone automated parameter extraction or manual correction.

[0106] The expert demonstration analysis module instantiates a directed graph data structure in memory. The expert demonstration analysis module encapsulates each multimodal action sequence data into a graph node object. .

[0107] The expert demonstration analysis module writes the standard action name, standard demonstration video clip index, key action feature vector, common error type set, and allowable error range into the graph node object. The attribute fields.

[0108] The expert demonstration analysis module creates connection graph node objects based on the chronological order of timestamps in the multimodal action sequence data. Directed edge objects .

[0109] The expert demonstration analysis module is used in directed edge objects. Configure state transition conditions. State transition conditions include the completion flag of the preceding node and the passing flag of the environmental detection.

[0110] The expert demonstration analysis module demonstrates the application of directed graph data structures. Execute the integrity verification algorithm.

[0111] The expert demonstration analysis module runs a depth-first search (DFS) algorithm to traverse the directed graph data structure. The expert demonstration analysis module detects the existence of unreachable isolated node objects. It also detects the existence of closed-loop paths pointing to themselves.

[0112] After the integrity verification is passed, the expert demonstration analysis module will have a directed graph data structure. Serialized into a binary map file.

[0113] The expert demonstration analysis module stores the binary graph file into the skill operation knowledge graph storage module.

[0114] The student data collection and uploading process provided in this embodiment of the invention is executed by the student-side AI glasses. The student data collection and uploading content includes: The central processing module (CPU) of the student's AI glasses receives the practice start command. The CPU sends a power-on signal to the multimodal data acquisition module. The CPU initiates a connection request to the backend processing and analysis system via the wireless communication module. A bidirectional long-term data transmission connection is established between the CPU and the backend processing and analysis system.

[0115] The visual acquisition unit acquires first-person perspective video stream data at a preset sampling frequency. This first-person perspective video stream data includes images of the student user's hand, the operating tool, and the device being operated. The visual acquisition unit also acquires depth point cloud data synchronized with the first-person perspective video stream data. This depth point cloud data includes distance values ​​for each pixel.

[0116] The inertial measurement unit (IMU) collects real-time attitude data from the student's AI glasses. This real-time attitude data includes triaxial acceleration and triaxial angular velocity values. The data is used to calculate the Euler angles and displacement components of the student's AI glasses in the world coordinate system.

[0117] The audio acquisition unit activates the audio acquisition circuit. The audio acquisition unit acquires the student's voice signal and the environmental feedback sound signal. The audio acquisition unit performs analog-to-digital conversion on the voice signal and the environmental feedback sound signal to generate a digital audio data stream.

[0118] The central processing module reads the internal hardware clock counter. The central processing module assigns system acquisition timestamps to each frame of first-person view video stream data, each frame of depth point cloud data, each real-time attitude data sampling point, and each frame of digital audio data stream.

[0119] The central processing module performs video encoding and compression on the first-person view video stream data and depth point cloud data. The central processing module constructs a real-time practical data packet. The central processing module writes the compressed video data, real-time attitude data, and digital audio data stream into the payload section of the real-time practical data packet. The central processing module writes the student user identification code (User_ID) and course task identification code (Task_ID) into the header fields of the real-time practical data packet.

[0120] The wireless communication module uses Real-Time Transport Protocol (RTP) or WebRTC to send real-time data packets. The module monitors the network round-trip time (RTT) and packet loss rate of the data transmission channel in real time.

[0121] When the network round-trip time or packet loss rate exceeds a preset threshold, the wireless communication module sends a congestion control signal to the central processing module. The central processing module then reduces the encoding bitrate of the first-person view video stream data or decreases the size of the transmission buffer based on the congestion control signal. The wireless communication module then uploads the real-time practical data packets to the student model comparison module in the background processing and analysis system.

[0122] The real-time comparison logic provided in this embodiment of the invention is executed by the student model comparison module in the background processing and analysis system. The real-time comparison logic includes: The student model comparison module initializes the task status pointer in memory. The module then locks the task status pointer to the first standard action node in the skill operation knowledge graph storage module.

[0123] The student model comparison module receives real-time practice data packets from the student's AI glasses. From these packets, the module extracts real-time feature vectors containing action type labels, hand spatial coordinates, tool operation angles, and real-time inertial trajectory data.

[0124] The student model comparison module performs a step sequence consistency check. It reads the currently locked standard action node from the task status pointer. The module then obtains the preset action type identifier for the standard action node. Finally, it matches the action type label in the real-time feature vector with the preset action type identifier.

[0125] If the action type label does not match the preset action type identifier, the student model comparison module retrieves subsequent nodes in the skill operation knowledge graph storage module that have a direct connection relationship with the current standard action node. If the action type label matches the preset action type identifier of a subsequent node, the student model comparison module determines that a step jump has occurred and generates a sequence error prompt signal. If the action type label does not match any of the preset action type identifiers of subsequent nodes, the student model comparison module determines that non-standard action interference has occurred and ignores the current frame data.

[0126] If the action type label matches the preset action type identifier, the student model comparison module performs an action quality accuracy check.

[0127] The student model comparison module calculates the Dynamic Time Warping (DTW) similarity distance between the real-time inertial trajectory data in the real-time feature vector and the standard trajectory template in the standard action node. The student model comparison module then compares the DTW similarity distance with the allowable trajectory error threshold defined by the standard action node. If the DTW similarity distance is greater than the allowable trajectory error threshold, the student model comparison module generates a trajectory deviation signal.

[0128] The student model comparison module calculates the Euclidean distance between the hand spatial coordinates in the real-time feature vector and the standard spatial coordinate interval in the standard action node. If the Euclidean distance exceeds the preset spatial tolerance range, the student model comparison module generates a positional deviation signal.

[0129] The student model comparison module calculates the cosine of the angle between the tool operation angle in the real-time feature vector and the standard angle normal vector in the standard action node. If the cosine of the angle is lower than a preset angle similarity threshold, the student model comparison module generates an angle deviation signal.

[0130] The student model comparison module performs timeliness checks on actions. It records the cumulative execution time of the current standard action node. The module compares the cumulative execution time with the standard time threshold defined for the standard action node. If the cumulative execution time exceeds the standard time threshold, the module generates a timeout alarm signal.

[0131] The student model comparison module performs action completion assessment. It checks whether the real-time feature vectors meet the termination conditions of the standard action node. Termination conditions include: the hand's spatial coordinates entering the standard termination region or the cumulative execution time reaching the minimum necessary duration.

[0132] If the student model comparison module detects the generation of trajectory deviation signal, position deviation signal, angle deviation signal or timeout alarm signal, the student model comparison module will send the above signals to the personalized guidance strategy generation module and keep the task status pointer unchanged.

[0133] If the student model comparison module does not detect any deviation signal and detects that the termination condition is met, the student model comparison module generates a step pass signal and updates the task status pointer to point to the next standard action node in the skill operation knowledge graph storage module.

[0134] The personalized guidance strategy generation process provided in this embodiment of the invention is executed by the personalized guidance strategy generation module in the background processing and analysis system. The personalized guidance strategy generation content includes: The personalized guidance strategy generation module constructs a policy mapping lookup table in memory. This table stores the index relationships between deviation type identifiers and augmented reality (AR) rendering templates. The personalized guidance strategy generation module receives comparison result data packets from the student model comparison module.

[0135] When the comparison result data packet contains a step-through signal, the personalized guidance strategy generation module queries the skill operation knowledge graph storage module. The personalized guidance strategy generation module obtains the standard tool's 3D model data corresponding to the next standard action node and the standard tool's preset placement coordinates in the world coordinate system.

[0136] The personalized guidance strategy generation module performs coordinate transformation calculations. It obtains the current extrinsic and intrinsic parameter matrices of the camera on the student's AI glasses. Using the model-view-projection matrix, it converts the preset placement coordinates into two-dimensional pixel coordinates on the screen plane of the student's AI glasses. The module then generates guided rendering instructions. These instructions include standard tool 3D model data and two-dimensional pixel coordinates. The near-eye display module of the student's AI glasses executes rendering according to the guided rendering instructions, drawing a semi-transparent virtual tool image on the optical display plane.

[0137] When the comparison result data packet contains positional deviation codes, the personalized guidance strategy generation module extracts the three-dimensional coordinates of the student's hand and the standard target three-dimensional coordinates of the standard action node from the real-time feature vector. The personalized guidance strategy generation module constructs a three-dimensional difference vector pointing from the three-dimensional coordinates of the student's hand to the three-dimensional coordinates of the standard target.

[0138] The personalized guidance strategy generation module generates vector guidance instructions. These instructions include the magnitude, direction cosine, and color attribute values ​​of a 3D difference vector. Based on these instructions, the near-eye display module renders a dynamic arrow model on the spatial line segment connecting the student's hand's 3D coordinates to the standard target's 3D coordinates.

[0139] When the comparison result data packet contains trajectory deviation codes, the personalized guidance strategy generation module reads the standard trajectory point sequence from the skill operation knowledge graph storage module. Based on the standard trajectory point sequence, the personalized guidance strategy generation module constructs a three-dimensional strip mesh model (TriangleStrip).

[0140] The personalized guidance strategy generation module generates path coverage instructions. These instructions drive the near-eye display module to overlay a 3D strip mesh model onto the real-world image. The 3D strip mesh model is marked with preset highlight textures to visualize the standard operation path.

[0141] When the comparison result data packet contains angle deviation codes, the personalized guidance strategy generation module obtains the real-time tool attitude quaternion and the standard tool attitude quaternion. The personalized guidance strategy generation module calculates the difference quaternion between the real-time tool attitude quaternion and the standard tool attitude quaternion. The personalized guidance strategy generation module then converts the difference quaternion into Euler angle deviation values.

[0142] The personalized guidance strategy generation module generates numerical visualization instructions. These instructions include Euler angle deviation values ​​and rotation direction indicator icons. The near-eye display module renders a text layer containing Euler angle deviation values ​​at the edge of the screen.

[0143] When the comparison result data packet contains a timeout alarm signal or consecutive error codes, the personalized guidance strategy generation module generates a video stream push instruction. The video stream push instruction includes an expert correct demonstration video frame sequence associated with the current standard action node and picture-in-picture (PiP) layout parameters. The near-eye display module creates a video overlay in the off-center area of ​​the display field of view. The near-eye display module then plays the expert correct demonstration video frame sequence within the video overlay.

[0144] The personalized guidance strategy generation module performs adaptive rendering parameter adjustments. It continuously monitors and compares the deviation quantization values ​​in the result data packet. Finally, it establishes a linear mapping function between the deviation quantization values ​​and the rendering opacity channel.

[0145] As the deviation quantization value decreases over time, the personalized guidance strategy generation module reduces the value of the rendering transparency channel based on a linear mapping function. The near-eye display module then uses the reduced rendering transparency channel value to dim the display brightness of virtual tool images, dynamic arrow models, or 3D strip mesh models.

[0146] The intelligent assessment process provided in this embodiment of the invention is executed by a background processing and analysis system, and interacts with the data and management platform and the student's AI glasses. The intelligent assessment content includes: the student model comparison module detecting that the task status pointer points to the end node in the skill operation knowledge graph storage module.

[0147] The student model comparison module generates a task termination signal. The student model comparison module packages the process log data during the practice process. The process log data includes the action node sequence index, the actual execution time of each action node, the trajectory deviation code set, the position deviation code set, the angle deviation code set, and the sequence error code set.

[0148] The background processing and analysis system receives process log data and initiates multidimensional scoring calculation logic.

[0149] The background processing and analysis system performs an accuracy score calculation. It then iterates through the process log data, collecting the sets of trajectory deviation codes, position deviation codes, angle deviation codes, and sequence error codes. Finally, the system calculates the cumulative number of triggers for each type of deviation code.

[0150] The background processing and analysis system reads a pre-defined deduction weight table from the database. The deduction weight table defines the deduction value corresponding to each deviation code.

[0151] The background processing and analysis system utilizes formulas Calculate the operational accuracy score. Among them, This is the benchmark value for a perfect score. For the first The cumulative number of times the class deviation code is triggered. For the first The deduction value for the class deviation code.

[0152] The background processing and analysis system performs an operation efficiency score calculation. It accumulates the actual execution time of all action nodes in the process log data to obtain the total execution time.

[0153] The background processing and analysis system reads the standard time threshold for the task from the skill operation knowledge graph storage module. The system then calculates the time difference between the total time consumed and the standard time threshold.

[0154] If the total time consumed is less than or equal to the standard working time threshold, the background processing and analysis system sets the operation efficiency score to full. If the total time consumed is greater than the standard working time threshold, the background processing and analysis system linearly deducts the full score according to a preset time decay coefficient to obtain the operation efficiency score.

[0155] The background processing and analysis system performs operational stability scoring calculations. It also acquires accelerometer data sequences uploaded by students using their AI glasses during the practice session.

[0156] The background processing and analysis system calculates the variance of the accelerometer data sequence. The system also reads a preset limit for the variance of the jitter.

[0157] The background processing and analysis system utilizes formulas Calculate the operational stability score. Among them, This represents the variance of the accelerometer data sequence. This is the threshold for the extreme jitter variance.

[0158] The background processing and analysis system generates improvement suggestion data. The system also extracts the deviation type codes with the highest trigger frequency from the process log data.

[0159] The background processing and analysis system retrieves the corresponding error cause description text and targeted reinforcement training plan text from the skill operation knowledge graph storage module based on the deviation type code.

[0160] The background processing and analysis system encapsulates the operation accuracy score, operation efficiency score, operation stability score, error cause description text, and targeted reinforcement training plan text into a structured evaluation report data package.

[0161] The backend processing and analysis system sends the structured assessment report data package to the data and management platform. The data and management platform parses the structured assessment report data package. The data and management platform then stores the parsed data in the student user's personal electronic record database.

[0162] The backend processing and analysis system sends the structured evaluation report data package to the student's AI glasses. The near-eye display module of the student's AI glasses renders a multi-dimensional radar chart based on the operation accuracy score, operation efficiency score, and operation stability score.

[0163] The near-eye display module renders error cause description text and targeted reinforcement training scheme text in the lower area of ​​the multi-dimensional radar chart.

[0164] The data storage process provided in this embodiment of the invention is executed by a data and management platform. The data storage content includes: The data and management platform is deployed with a hybrid storage architecture. This architecture consists of a relational database management system (RDBMS) and an unstructured object storage system (ObjectStorage). The data and management platform establishes a data synchronization channel with the back-end processing and analysis system through a data communication interface.

[0165] The data and management platform receives serialized graph files from the skills operation knowledge graph storage module. The platform writes the serialized graph files as binary large objects (BLOBs) to the unstructured object storage system. The platform then generates a storage address index pointing to the serialized graph files.

[0166] The data and management platform creates a graph metadata table within a relational database management system. The platform writes the version number, creation timestamp, applicable device model identifier, associated expert identifier, and storage address index of the skill operation knowledge graph into the graph metadata table.

[0167] The data and management platform receives structured assessment report data packets from the back-end processing and analysis system. The platform parses these packets and extracts the student user identification code. Using this student user identification code as the primary key, the platform queries the student's personal electronic record in the relational database management system.

[0168] The data and management platform updates the operational accuracy, operational efficiency, and operational stability scores from the structured assessment report data package into the competency dimension fields of the individual's electronic profile. The platform also inserts a new record into the practice history log table. This new record includes the date, time, and total time spent on the current practice session.

[0169] The data and management platform receives abnormal video clip data streams uploaded by students' AI glasses. The platform then stores these abnormal video clip data streams in an unstructured object storage system. Finally, the platform retrieves the Uniform Resource Locator (URL) of the abnormal video clip data streams from the unstructured object storage system.

[0170] The data and management platform writes the Uniform Resource Locator (URL) into the error association field of the exercise history log table. The platform then establishes a mapping between the error association field and specific action nodes in the graph metadata table.

[0171] The data and management platform runs offline aggregation and analysis tasks. It performs statistical calculations on a large amount of historical practice data from a relational database management system. The platform calculates the average pass rate, average time taken, and distribution of frequently missed actions for specific course tasks. The platform persistently stores the statistical calculation results as statistical report views.

[0172] The data and management platform provides parameter optimization query services. When the backend processing and analysis system initiates a threshold calibration request, the data and management platform queries the normal distribution statistical parameters of historical practice data. The data and management platform returns the normal distribution statistical parameters to the backend processing and analysis system. The backend processing and analysis system uses the normal distribution statistical parameters to correct the allowable error range threshold of the standard action nodes.

[0173] This invention provides a skills training method based on AI glasses, which is applied to the aforementioned skills training system based on AI glasses. The skills training method based on AI glasses includes the following steps: Step S1: Expert Demonstration of Data Collection and Uploading The teacher-side AI glasses receive the data acquisition start command. The central processing module of the teacher-side AI glasses sends synchronization trigger signals to the visual acquisition unit, audio acquisition unit, and inertial measurement unit.

[0174] The AI ​​glasses for teachers use a vision acquisition unit to collect first-person perspective video stream data and depth point cloud data. The first-person perspective video stream data and depth point cloud data include images of the object being manipulated, images of the tools being manipulated, and images of the teacher's hand.

[0175] The AI ​​glasses for teachers use an inertial measurement unit to collect head and body movement posture data. The AI ​​glasses also use an audio acquisition unit to collect voice narration data and environmental operation sound data.

[0176] The central processing module of the teacher-side AI glasses performs timestamp alignment on the first-person perspective video stream data, depth point cloud data, motion posture data, voice narration data, and environmental operation sound data. The central processing module then encapsulates the aligned multimodal data into a sample data package. The teacher-side AI glasses send this sample data package to the backend processing and analysis system via a wireless communication module.

[0177] Step S2: Construction of Skill Operation Knowledge Graph The expert demonstration analysis module of the background processing and analysis system receives the demonstration data packet. The expert demonstration analysis module performs feature extraction calculations on the demonstration data packet to generate a visual semantic sequence, a quantized trajectory parameter set, and structured text data.

[0178] The expert demonstration analysis module uses the frame timestamps of the visual semantic sequence as a benchmark, and integrates the visual semantic sequence, the quantized trajectory parameter set, and structured text data to generate a multimodal feature frame sequence.

[0179] The expert demonstration analysis module divides the multimodal feature frame sequence into multiple independent standard action units based on the kinematic extrema and semantic logic nodes in the sequence. The module then calculates the allowable error range parameter for each standard action unit.

[0180] The expert demonstration and analysis module constructs a skill operation knowledge graph. This knowledge graph includes the execution sequence logic of each standard action unit, key feature vectors, indexes of standard demonstration video clips, and allowable error range parameters. The expert demonstration and analysis module then stores this knowledge graph in the skill operation knowledge graph storage module.

[0181] Step S3: Real-time guidance for student practice The student's AI glasses receive the exercise start command. The student's AI glasses collect real-time first-person view video stream data, real-time depth data, and real-time pose data from the student user. The student's AI glasses package this real-time first-person view video stream data, real-time depth data, and real-time pose data into a real-time practice data package. The student's AI glasses then upload this real-time practice data package to the backend processing and analysis system.

[0182] The student model comparison module of the backend processing and analysis system receives real-time practical data packets. It then reads the standard parameter set of the current standard action node from the skill operation knowledge graph storage module. Finally, the module calculates the similarity distance between the real-time practical data packet and the standard parameter set.

[0183] The student model comparison module compares the similarity distance value with the allowable error range parameter. If the similarity distance value exceeds the allowable error range parameter, the student model comparison module generates a deviation signal.

[0184] The personalized guidance strategy generation module of the backend processing and analysis system receives deviation signals. Based on these signals, the module generates corrective feedback instructions. These instructions include augmented reality (AR) rendered coordinates and voice prompts.

[0185] The student-side AI glasses receive correction feedback commands. The near-eye display module of the student-side AI glasses overlays virtual guide arrows or virtual tool outlines based on AR rendering coordinates. The audio component of the student-side AI glasses plays voice prompts.

[0186] Step S4: Intelligent Assessment and Data Archiving When the student model comparison module detects that all standard action units in the skill operation knowledge graph have been executed, the student model comparison module generates a task termination signal.

[0187] The background processing and analysis system summarizes the deviation records during the practice process. Based on this data, the system calculates scores for operational accuracy, efficiency, and stability. Finally, the system retrieves corresponding improvement suggestions.

[0188] The backend processing and analysis system generates a structured evaluation report. This report is then sent to the student's AI glasses. The student's AI glasses display the operation accuracy score, operation efficiency score, operation stability score, and improvement suggestions on the near-eye display module.

[0189] The back-end processing and analysis system sends the structured assessment report file to the data and management platform. The data and management platform then stores the structured assessment report file in the student user's historical record database.

[0190] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A skills training system based on AI glasses, characterized in that: include: AI glasses for teachers, back-end processing and analysis system, AI glasses for students, and data and management platform; The AI ​​glasses for teachers are configured to collect first-person perspective demonstration data from teachers. The demonstration data includes visual image data collected by the visual acquisition unit, operation trajectory data collected by the inertial measurement unit, and voice command data collected by the audio acquisition unit, and are sent to the background processing and analysis system. The background processing and analysis system includes: an expert demonstration analysis module, a skills operation knowledge graph storage module, a student model comparison module, and a personalized guidance strategy generation module. The expert demonstration analysis module is used to receive the demonstration data, extract and fuse its features, construct a skill operation knowledge graph, and store it in the skill operation knowledge graph storage module; the skill operation knowledge graph includes the execution order of standard action units, key feature vectors, and allowable error range parameters; The student-side AI glasses are configured to collect real-time operation data of student users and upload it to the back-end processing and analysis system, as well as receive and display guidance information; The student model comparison module is used to compare the real-time operation data with the standard parameter set in the skill operation knowledge graph and generate a comparison result signal. The personalized guidance strategy generation module is used to generate a correction feedback instruction based on the comparison result signal, and send the correction feedback instruction to the student's AI glasses.

2. The skills training system based on AI glasses according to claim 1, characterized in that, The expert demonstration analysis module is configured to perform the following feature fusion and alignment operations: selecting the video frame timestamp in the visual image data as the primary alignment index; within the exposure period corresponding to each video frame timestamp, extracting multiple quantized trajectory parameter points from the operation trajectory data and performing mean calculation to generate a discrete trajectory feature vector; binding the discrete trajectory feature vector with the corresponding visual features to form a visual-motion joint feature frame; based on the timestamp interval information in the voice command data, associating the structured text data with the visual-motion joint feature frame to generate a multimodal feature frame sequence; and cutting the multimodal feature frame sequence into multiple independent standard action units according to the kinematic segmentation points and semantic segmentation points in the multimodal feature frame sequence.

3. The skills training system based on AI glasses according to claim 2, characterized in that, The expert demonstration analysis module is also configured to perform the following verification and graph generation operations: extract the action classification probability vector of the multimodal feature frame sequence and use the maximum probability value as the classification confidence value; when the classification confidence value is greater than or equal to the automatic pass threshold, perform statistical analysis on the quantized trajectory parameter points and calculate the arithmetic mean and standard deviation values; construct an allowable error range based on the arithmetic mean and standard deviation values, and write the allowable error range into the node attribute field of the skill operation knowledge graph.

4. The skills training system based on AI glasses according to claim 1, characterized in that, The student model comparison module is configured to execute the following real-time comparison logic: extract real-time feature vectors from the real-time operation data, the real-time feature vectors including action type labels, hand spatial coordinates, tool operation angles, and real-time inertial trajectory data; extract the real-time inertial trajectory data and the standard trajectory template in the standard action unit; calculate the similarity distance between the real-time inertial trajectory data and the standard trajectory template using the Dynamic Time Warping (DTW) algorithm; compare the similarity distance with the trajectory deviation threshold in the allowable error range parameter; if the similarity distance is greater than the trajectory deviation threshold, generate the comparison result signal containing the trajectory deviation code.

5. The skills training system based on AI glasses according to claim 4, characterized in that, The personalized guidance strategy generation module is configured as follows: when the comparison result signal contains a position deviation code, a three-dimensional difference vector is constructed pointing from the three-dimensional coordinates of the student's hand to the three-dimensional coordinates of the standard target, and a vector guidance instruction containing the three-dimensional difference vector is generated; when the comparison result signal contains an angle deviation code, the difference quaternion between the real-time tool posture quaternion and the standard tool posture quaternion is calculated and converted into an Euler angle deviation value; the near-eye display module of the student's AI glasses renders a dynamic arrow model according to the vector guidance instruction, or renders a numerical visualization instruction according to the Euler angle deviation value.

6. The skills training system based on AI glasses according to claim 5, characterized in that, The personalized guidance strategy generation module is also configured to perform coordinate transformation operations: obtain the camera extrinsic and intrinsic parameter matrices of the student AI glasses; use the Model-View-Projection Matrix (MVPMatrix) to convert the preset placement coordinates corresponding to the standard action unit into two-dimensional pixel coordinates on the screen plane of the student AI glasses; generate a guided rendering instruction containing the two-dimensional pixel coordinates to drive the near-eye display module to draw virtual tool images on the optical display plane.

7. The skills training system based on AI glasses according to claim 1, characterized in that, The background processing and analysis system is also configured to perform the following intelligent evaluation operations: summarize real-time comparison log data during the practice process, the real-time comparison log data including deviation type identifier sequence and deviation quantification value sequence; calculate the operation accuracy score by subtracting the total deduction value from the preset full score benchmark value, wherein the total deduction value is obtained by weighted summation of the cumulative trigger count of each type of deviation code in the deviation type identifier sequence and the corresponding deduction value in the preset deduction weight table; obtain the acceleration sensor data sequence uploaded by the student's AI glasses, calculate its variance value, and normalize the variance value with the preset limit jitter variance threshold to calculate the operation stability score.

8. The skills training system based on AI glasses according to claim 1, characterized in that, The data and management platform is configured as follows: deploying a hybrid storage architecture consisting of a relational database management system and an unstructured object storage system; receiving abnormal video segment data streams uploaded by the student's AI glasses, storing them in the unstructured object storage system, and generating a Uniform Resource Locator (URL); writing the URL into the practice history log table in the relational database management system, and establishing a mapping relationship between the URL and specific action nodes in the skill operation knowledge graph.

9. The skills training system based on AI glasses according to claim 1, characterized in that, Both the teacher-side AI glasses and the student-side AI glasses include: a near-eye display module, including an optical display unit and a zoom drive unit; the zoom drive unit is configured to receive a focal length control signal and adjust the virtual image distance of the virtual image by changing the combined focal length or object distance of the optical system, so that the virtual image distance is consistent with the gaze depth detected by the eye-tracking unit.

10. A skills training method based on AI glasses, applied to a skills training system based on AI glasses as described in any one of claims 1 to 9, the method comprising the following steps: The teacher-side AI glasses collect first-person perspective demonstration data and send it to the backend processing and analysis system. The demonstration data includes visual image data, operation trajectory data, and voice command data. The expert demonstration analysis module of the backend processing and analysis system extracts features and performs temporal fusion on the demonstration data to generate a multimodal feature frame sequence and constructs a skill operation knowledge graph based on it. The student-side AI glasses collect real-time operation data and upload it to the backend processing and analysis system. The student model comparison module of the backend processing and analysis system compares the real-time operation data with the standard parameter set in the skill operation knowledge graph and generates a comparison result signal. The personalized guidance strategy generation module of the backend processing and analysis system generates correction feedback instructions based on the comparison result signal and sends them to the student-side AI glasses for display and prompting.

Images