Gesture interactive virtual work order integration method and device based on AR glasses

Abstract

The invention provides a gesture interactive virtual work order integration method and device based on AR glasses, and relates to the field of data processing. The method comprises the following steps: acquiring RGB image data and multi-modal sensor data which are sent by AR glasses and aim at the hand of a user; according to the RGB image data, performing skeleton tracking on a preset key part of the hand of the user, and determining a first gesture action; according to the multi-modal sensor data, calculating to obtain a gesture speed, a gesture acceleration and a gesture movement track; fusing the first gesture action through the gesture speed, the gesture acceleration and the gesture movement track to obtain a second gesture action; the second gesture action is mapped to a preset virtual work order, a dynamic integration instruction is obtained, and the preset virtual work order is generated on AR glasses in advance; and controlling the AR glasses to execute the dynamic integration instruction, and generating a target virtual work order. By implementing the technical scheme provided by the invention, the inspection efficiency can be improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a gesture interactive virtual work order integration method and device based on AR glasses. Background Art

[0002] In scenarios such as industrial inspection and equipment maintenance, virtual work orders have become important task management tools, widely used for recording, tracking, and executing various operation processes.

[0003] Currently, the existing virtual work order interaction methods mainly rely on touch screens, mice, keyboards, or other external input devices. Therefore, inspection personnel often need to carry tools and wear protective equipment, and even work in harsh environments such as high temperature, high humidity, and high noise. Using traditional input devices may bring many inconveniences. For example, it is difficult to accurately click on the touch screen when wearing gloves, and the mouse operation of the work order may become unstable in poor lighting or limited visibility. This interaction mode seriously reduces the inspection efficiency.

[0004] Therefore, there is an urgent need for a gesture interactive virtual work order integration method and device based on AR glasses. Summary of the Invention

[0005] This application provides a gesture interactive virtual work order integration method and device based on AR glasses, which is convenient for improving the inspection efficiency.

[0006] In the first aspect of this application, a gesture interactive virtual work order integration method based on AR glasses is provided. The method includes: obtaining RGB image data and multi-modal sensor data of the user's hand sent by the AR glasses; performing skeleton tracking on preset key parts of the user's hand according to the RGB image data to determine the first gesture action; calculating the gesture speed, gesture acceleration, and gesture movement trajectory according to the multi-modal sensor data; fusing the first gesture action through the gesture speed, the gesture acceleration, and the gesture movement trajectory to obtain a second gesture action; mapping the second gesture action to a preset virtual work order to obtain a dynamic integration instruction, where the preset virtual work order is pre-generated on the AR glasses; controlling the AR glasses to execute the dynamic integration instruction to generate a target virtual work order.

[0007] By adopting the above technical solutions, by combining RGB images and multi-modal sensor data, not only relying on visual information but also using kinematic features, the misjudgment rate of single visual recognition is reduced, and the accuracy of gesture recognition is improved. By using the skeleton tracking technology, the key parts of the hand can be accurately located, ensuring the robustness of gesture recognition and enabling stable operation even in complex environments. By adopting the method of dynamic gesture fusion, the system can understand continuous gestures, not limited to static gestures, thus reducing the stiffness during the interaction process and making gesture control more intuitive and in line with the natural movement habits of users. Combining gesture trajectory analysis makes the expression of complex gestures smoother, reduces misoperations, and improves the user experience. By fusing gesture speed, acceleration, and trajectory information, the system can distinguish similar but different-intention gestures, improving the refinement of the interaction. Combining motion pattern analysis, the system can automatically judge the execution strength and direction of gestures, thereby more accurately matching the corresponding interaction commands and improving the intelligent level of human-computer interaction. This solution does not rely on traditional input devices such as touchscreens and mice, and completely completes virtual work order operations through gesture interaction, reducing the burden on physical interaction devices and enabling users to operate efficiently without contact. It is applicable to scenarios that require frequent operations, reduces operation interruptions caused by device switching, and improves the continuity of task execution. Combining multi-modal data fusion enables gesture recognition to maintain a high accuracy under different environmental conditions and has stronger adaptability. In high-intensity and complex environments such as industrial sites, it reduces the time for users to switch tasks during virtual work order operations and improves the overall operation efficiency.

[0008] Optionally, the obtaining of the RGB image data and multi-modal sensor data of the user's hand sent by the AR glasses specifically includes: receiving the original image data sent by the binocular camera located on the AR glasses; receiving the original sensor data sent by the IMU, optical flow sensor, and microelectromyogram sensor located on the AR glasses; and performing data processing on the original image data and the original sensor data to obtain the RGB image data and the multi-modal sensor data, where the data processing includes denoising, filtering, feature classification, and normalization processing.

[0009] By adopting the above technical solution, through denoising, filtering, feature classification, and normalization processing of the original image data of the binocular camera and various sensor data, the interference of environmental noise is effectively reduced, ensuring the stability and consistency of the input data, thereby improving the accuracy of gesture recognition. Data preprocessing can eliminate the systematic errors brought by the camera and sensors, enhance the adaptability of the model to complex scenarios, and reduce the occurrence of misidentifications. Combining multi-modal data of RGB images, IMUs, optical flow sensors, and microelectromyogram sensors enables the system to comprehensively analyze the visual features, motion states, and physiological signals of gestures, improving the ability to understand complex gestures. Not relying on a single data source enhances the stability of the system under different lighting conditions, environmental interferences, and operating conditions, ensuring the continuity and reliability of gesture interaction. Combining IMU and optical flow sensor data can obtain information on the motion trajectory, speed, and acceleration of gestures, enabling the system to perceive the dynamic features of gestures, not limited to static gesture recognition, and thus improving the naturalness and flexibility of interaction. Through the data input of the microelectromyogram sensor, the muscle signals of the user can be further captured, enhancing the perception of subtle gestures and force changes, and improving the depth of understanding of the user's intentions by the system. Normalization processing can unify the gesture features of different users, reduce the impact of individual differences on the recognition accuracy, and make the system more versatile, suitable for different populations and usage habits. Through feature classification and normalization, the data under different devices and different environments can be standardized, ensuring stability and consistency across devices and environments. In the preprocessing stage, through data denoising, feature extraction, and classification, redundant information can be reduced, the computational efficiency of the algorithm can be improved, and the fast response ability of real-time gesture recognition can be ensured. Normalization processing reduces the impact of data dimensions, enabling the system to match gesture patterns more quickly, improving the fluency of interaction and the operating efficiency of the system.

[0010] Optionally, the determining the first gesture action by performing skeleton tracking on a preset key part of the user's hand according to the RGB image data specifically includes: using an image segmentation network to perform pixel-level segmentation on the RGB image to extract the contour data of the user's hand; constructing a 3D skeleton model of the user's hand by using binocular disparity calculation according to the RGB image data; using a convolutional neural network to perform key point recognition on the 3D skeleton model to obtain a set of key points; and matching the preset key part according to the set of key points and the contour data.

[0011] By adopting the above technical solutions, pixel-level segmentation is performed using an image segmentation network, which can effectively remove background interference, ensure the accurate extraction of the hand region, and make subsequent skeleton tracking more stable. With high-precision hand contour data, the adaptability of the system in complex environments is enhanced, and the influence of factors such as light changes and occlusions on gesture recognition is reduced. By calculating binocular disparity to construct a 3D skeleton model, compared with the method that only relies on 2D image recognition, the ability to perceive depth information in gesture recognition can be effectively improved, enabling the system to recognize gesture actions at different distances and angles. This solution can make up for the deficiency of monocular vision in lacking depth information and improve the recognition stability of the system at different spatial positions. Using a convolutional neural network to identify key points of the 3D skeleton model can effectively extract key parts such as finger joints and the center of the palm, improving the accuracy and robustness of recognition. Combining the feature extraction ability of deep learning, the system can adapt to the hand shape differences of different users and enhance the generalization ability. By matching the key point set with the hand contour data, the key parts identified can be made more stable, avoiding recognition deviations caused by errors in a single feature. Adopting multi-dimensional feature data fusion enables the system to accurately track the hand structure during dynamic gestures, ensuring the continuity and stability of interaction. This method can adapt to gesture recognition under different lighting, backgrounds, and occlusions, reducing the impact of environmental changes on the interaction experience. Combining the 3D skeleton model and the deep learning model enables the system to maintain a high recognition accuracy under complex gestures. By optimizing the key point extraction process using deep learning and computer vision methods, the consumption of computing resources can be reduced, and the system response speed can be improved. Adopting an end-to-end feature matching method reduces the cumbersome steps in traditional gesture recognition methods, improves the overall computing efficiency, and ensures the smoothness of real-time interaction.

[0012] Optionally, the skeleton tracking of the preset key parts of the user's hand according to the RGB image data to determine the first gesture action specifically further includes: determining a first key point and a second key point, where the first key point and the second key point are any two key points among the multiple key points included in the key point set; according to the 3D skeleton model, determining a first coordinate corresponding to the first key point and determining a second coordinate corresponding to the second key point, where both the first coordinate and the second coordinate are 3D coordinates; using temporally consecutive image frames, and combining Kalman filtering and / or recursive least squares method to smooth the first coordinate and the second coordinate to obtain the first gesture action.

[0013] By adopting the above technical solutions, by selecting the 3D coordinates of the first key point and the second key point, the spatial movement trajectory of the gesture can be described more accurately, avoiding recognition deviation caused by single-point data error. Combining with the 3D skeleton model ensures more stable key point positioning, enabling the system to capture the actual gesture actions of the user more accurately and reducing misjudgment. Using temporally consecutive image frames ensures that the system can analyze the gesture trajectory based on historical information, making the recognition of dynamic gestures more stable and smooth, and avoiding the problem of single-frame misrecognition. Through the processing of time-series data, the adaptability of the system to rapidly changing gestures is improved, making the interaction more natural. Using Kalman filtering and / or recursive least squares method to smooth the coordinates effectively reduces data fluctuations caused by sensor jitter, light changes or background complexity, improving the stability of gesture recognition. These filtering algorithms can eliminate sudden interferences, making the gesture actions more coherent, reducing jitter and misrecognition, and improving the user interaction experience. By calculating the 3D coordinates of multiple key points, the overall movement trend of the hand can be better understood, rather than just the movement of a single finger or palm, thereby enhancing the recognition ability for complex gestures (such as gesture rotation, continuous gesture conversion, etc.). The relative movement information of the key points can be used to distinguish different types of gestures, improving the gesture classification ability of the system and making gesture operations more diverse. Kalman filtering is an efficient dynamic system estimation method that can reduce the computational amount while ensuring high precision, enabling the system to process gesture data in real time with low latency. The recursive least squares method can perform adaptive optimization in the continuously updated gesture data stream, improving the prediction accuracy and reducing the system response time, enabling the user to obtain a smoother interaction experience.

[0014] Optionally, the step of fusing the first gesture action through the gesture speed, the gesture acceleration, and the gesture movement trajectory to obtain a second gesture action specifically includes: comparing the gesture speed with a preset gesture speed threshold, and comparing the gesture acceleration with a preset gesture acceleration threshold; if it is determined that the gesture speed is greater than or equal to the preset gesture speed threshold and the gesture acceleration is greater than or equal to the preset gesture acceleration threshold, then determining that the action category is a fast action, where the action category includes the fast action and the slow action; based on the action category being the fast action, analyzing the gesture movement trajectory to generate gesture movement dynamic features; adding the gesture movement dynamic features to the first gesture action for dynamic fusion to obtain the second gesture action.

[0015] By adopting the above technical solution, through real-time calculation and threshold judgment of gesture speed and acceleration, the system can distinguish different action types, thereby more accurately identifying the user's intention. This dynamic adaptation mechanism enables the system to adjust according to different users' operation habits, improving the adaptability and robustness of recognition. Through the analysis of the gesture movement trajectory, the system can more subtly understand the user's operation method, not only relying on the static gesture form, but combining dynamic movement features to improve the accuracy of gesture interaction. This method can effectively reduce misjudgment, enabling the system to accurately distinguish the user's true operation intention in complex gesture sequences and improving the flexibility of gesture recognition. Since fast actions are often related to emergency operations or efficient interactions, this solution ensures that fast actions can be quickly recognized and processed by presetting speed and acceleration thresholds, reducing system response latency and improving interaction efficiency. This optimization strategy allows users to operate more naturally with fast gestures without having to deliberately slow down their actions to adapt to the system's recognition ability. By extracting the dynamic features of gesture movement, the user's operation method can be more comprehensively described, thereby enhancing the accuracy of gesture fusion. This method can ensure a high recognition accuracy for dynamic gestures during changes in different speeds and directions, avoiding action loss or misclassification caused by simply based on static key point recognition. Combining speed, acceleration, and movement trajectory for fusion enables the system to filter out invalid or misrecognized gesture actions, reducing false triggers caused by environmental interference, users' unintentional actions, and other factors. This multi-dimensional fusion method effectively improves the stability of the system in complex scenarios, reduces the misrecognition rate, and improves the user experience.

[0016] Optionally, mapping the second gesture action to a preset virtual work order to obtain a dynamic integration instruction specifically includes: determining a preset mapping rule according to the second gesture action, where the preset mapping rule includes the correspondence between the second gesture action and the preset virtual work order, and the preset virtual work order includes an inspection task, operation options, and a work order structure; matching the preset mapping rule with the preset virtual work order and generating the dynamic integration instruction.

[0017] By adopting the above technical solution, through the preset mapping rules, it is ensured that the gesture actions can establish an accurate corresponding relationship with the specific operations in the virtual work order, thereby avoiding recognition ambiguity or mis-triggering and improving the interaction accuracy of the system. This precise matching mechanism enables the user's gestures to directly drive the work order operations without additional confirmation steps, improving the operation efficiency. Since the mapping rules are preset and configurable, the mapping logic can be adjusted for different scenarios and different work order tasks to meet different application requirements, making the system more flexible and adaptable. This solution supports dynamic updates of the gesture mapping relationship, enabling the system to evolve with the changes in user requirements and enhancing the long-term usability. By adopting the method of dynamic integration instructions, the gesture operations can directly trigger the inspection tasks, operation options or adjustments to the work order structure, enabling the user to complete the tasks in a more natural interaction environment and improving the user experience. By eliminating the traditional manual input or complex menu operations, the operation time of the user on the interface is reduced, making the interaction more intuitive and efficient. Combining gesture recognition, mapping rules and work order structure enables the system to autonomously identify and execute the corresponding operations, reducing the user's learning cost and the probability of misoperations, and making the whole process more intelligent. This solution can be further combined with AI to optimize the mapping rules, making the gesture interaction more in line with user habits and achieving intelligent adaptation.

[0018] Optionally, controlling the AR glasses to execute the dynamic integration instruction to generate a target virtual work order specifically includes: generating an operation animation and an operation result for the preset virtual work order according to the dynamic integration instruction; sending the operation animation and the operation result to the AR glasses and controlling the AR glasses to display the operation animation and the operation result to obtain the target virtual work order.

[0019] By adopting the above technical solutions, through the real-time generation of operation animations and operation results, users can intuitively see the feedback of gesture operations, improving the visibility and understanding of interactions. This method effectively reduces misoperations caused by users' uncertainty about the execution of gestures, enhancing the intuitiveness and reliability of interactions. By directly controlling the AR glasses using dynamic integrated instructions, real-time operations and feedback are achieved, reducing interaction latency and making the execution of virtual work orders smoother. This solution ensures seamless connection between gesture operations and work order execution, improving the system's response speed and enabling users to complete tasks efficiently. By presenting operation animations in the AR glasses, users can obtain immersive task guidance without switching interfaces or relying on external display devices, thereby enhancing focus and task completion rate. This method reduces the need for users to input additional information, making the workflow smoother and reducing cognitive burden. By presenting the operation results in real time on the AR glasses, users can promptly confirm the status of work order execution, ensuring that tasks are completed correctly and reducing rework caused by misunderstandings or operation errors. This solution improves the accuracy of task execution, making the entire process more reliable and efficient.

[0020] In the second aspect of the present application, a gesture interactive virtual work order integration device based on AR glasses is provided. The gesture interactive virtual work order integration device includes an acquisition module and a processing module. Among them, the acquisition module is used to acquire RGB image data and multi-modal sensor data of the user's hand sent by the AR glasses; the processing module is used to perform skeleton tracking on preset key parts of the user's hand according to the RGB image data to determine the first gesture action; the processing module is further used to calculate the gesture speed, gesture acceleration, and gesture movement trajectory according to the multi-modal sensor data; the processing module is further used to fuse the first gesture action through the gesture speed, the gesture acceleration, and the gesture movement trajectory to obtain a second gesture action; the processing module is further used to map the second gesture action to a preset virtual work order to obtain a dynamic integrated instruction, where the preset virtual work order is pre-generated on the AR glasses; the processing module is further used to control the AR glasses to execute the dynamic integrated instruction to generate a target virtual work order.

[0021] In the third aspect of the present application, an electronic device is provided. The electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, and both the user interface and the network interface are used to communicate with other devices. The processor is used to execute the instructions stored in the memory to enable the electronic device to execute the method described above.

[0022] In the fourth aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions that, when executed, execute the method described above.

[0023] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages: By combining RGB images and multi-modal sensor data, not only relying on visual information but also utilizing kinematic features, the misjudgment rate of single visual recognition is reduced, and the accuracy of gesture recognition is improved. By adopting the skeleton tracking technology, the key parts of the hand can be accurately located, ensuring the robustness of gesture recognition and enabling stable operation even in complex environments. By adopting the method of dynamic gesture fusion, the system can understand continuous gestures, not limited to static gestures only, thereby reducing the stiffness during the interaction process and making gesture control more intuitive and in line with the natural movement habits of users. Combining gesture trajectory analysis makes the expression of complex gestures smoother, reduces misoperations, and improves the user experience. By fusing gesture speed, acceleration, and trajectory information, the system can distinguish similar but different-intention gestures, improving the refinement degree of interaction. Combining motion pattern analysis, the system can automatically judge the execution strength and direction of gestures, thereby more accurately matching corresponding interaction instructions and improving the intelligent level of human-computer interaction. This solution does not rely on traditional input devices such as touch screens and mice, and completely completes virtual work order operations through gesture interaction, reducing the burden on physical interaction devices and enabling users to operate efficiently without contact. It is applicable to scenarios that require frequent operations, reduces operation interruptions caused by device switching, and improves the continuity of task execution. Combining multi-modal data fusion enables gesture recognition to maintain high accuracy under different environmental conditions and has stronger adaptability. In high-intensity and complex environments such as industrial sites, it reduces the time for task switching during virtual work order operations by users and improves the overall operation efficiency. Description of the Drawings

[0024] Figure 1 It is a schematic flowchart of a gesture interactive virtual work order integration method based on AR glasses provided by an embodiment of this application; Figure 2 It is another schematic flowchart of a gesture interactive virtual work order integration method based on AR glasses provided by an embodiment of this application; Figure 3 It is a schematic module diagram of a gesture interactive virtual work order integration device based on AR glasses provided by an embodiment of this application; Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of this application.

[0025] Description of the reference numerals: 31, acquisition module; 32, processing module; 41, processor; 42, communication bus; 43, user interface; 44, network interface; 45, memory. Detailed Embodiments

[0026] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments.

[0027] In the description of the embodiments of this application, words such as "for example" or "for illustration" are used to give examples, illustrations or explanations. Any embodiment or design solution described as "for example" or "for illustration" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "for example" or "for illustration" aims to present relevant concepts in a specific way.

[0028] In the description of the embodiments of this application, the meaning of the term "a plurality of" refers to two or more. For example, a plurality of systems refers to two or more systems, and a plurality of screen terminals refers to two or more screen terminals. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the technical features indicated. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. The terms "include", "comprise", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0029] In scenarios such as industrial inspection and equipment maintenance, virtual work orders have become key task management tools and are widely used in the recording, tracking and execution of operation processes. However, the current virtual work order interaction methods mainly rely on external input devices such as touch screens, mice, and keyboards, making it inconvenient for inspection personnel in actual operations. Especially when they need to carry tools, wear protective equipment, or are in complex environments such as high temperature, high humidity, and high noise, the traditional interaction methods often cannot meet the requirements of efficient operation. For example, wearing gloves may cause inaccurate clicks on the touch screen, and when the light is insufficient or the line of sight is restricted, the stability of mouse operations decreases, thus affecting the smooth execution of inspection tasks and reducing the overall work efficiency.

[0030] To solve the above technical problems, this application provides a gesture interactive virtual work order integration method based on AR glasses. Refer to Figure 1 , Figure 1 is a schematic flow chart of a gesture interactive virtual work order integration method based on AR glasses provided in the embodiments of this application. This method is applied to a server and includes steps S110 to S150. The above steps are as follows: S110. Obtain RGB image data and multi-modal sensor data of the user's hand sent by the AR glasses.

[0031] Specifically, the binocular cameras equipped on the AR glasses can capture the color images of the user's hand and calculate the depth information of the hand through binocular disparity. After receiving these original images, the server will perform preprocessing (such as denoising, filtering, white balance correction, etc.) for subsequent gesture recognition and 3D modeling. The multi-modal sensor data includes the attitude information (acceleration, angular velocity) of the hand provided by the IMU (Inertial Measurement Unit), which is used to infer the motion direction, speed, and dynamic changes of the gesture. The optical flow sensor infers the movement of the hand relative to the environment by detecting the motion patterns of pixel points, assisting in calculating the gesture trajectory. The micro electromyogram sensor (EMG) detects the muscle signals of the hand and judges the micro-motions such as the grasping and stretching of the fingers, further improving the accuracy of gesture recognition. After obtaining these data, the server will perform fusion calculations, combining the RGB images with the multi-modal sensor data to enhance the accuracy, real-time performance, and robustness of gesture recognition.

[0032] For example, assume the scenario is a factory equipment inspection scenario. In a certain intelligent factory, the inspector wears AR glasses to perform the equipment inspection task, and the AR glasses need to recognize the inspector's gestures to achieve contactless operation. The inspector extends his palm and is about to open the equipment inspection work order through gesture interaction. The binocular cameras of the AR glasses capture the hand image and send it to the server. The IMU measures the angular velocity and acceleration of the palm lifting upward to judge the user's intention. The micro electromyogram sensor detects the micro-contraction of the palm muscles and infers that the user may be about to make a click gesture. The server receives this data and combines the gesture contour in the RGB image to judge whether the inspector has performed the gesture of "unfolding the work order". After the server combines visual recognition and sensor information and judges that the gesture action is correct, it sends an instruction to the AR glasses to unfold the equipment inspection work order on the AR interface. After the inspector confirms the work order, he can continue to use gestures to turn pages or check options, and the entire process does not require touching the screen, improving the operation convenience.

[0033] In a possible implementation manner, obtaining the RGB image data and multi-modal sensor data of the user's hand sent by the AR glasses specifically includes: receiving the original image data sent by the binocular cameras located on the AR glasses; receiving the original sensor data sent by the IMU, optical flow sensor, and micro electromyogram sensor located on the AR glasses; performing data processing on the original image data and the original sensor data to obtain the RGB image data and multi-modal sensor data, and the data processing includes denoising, filtering, feature classification, and normalization processing.

[0034] Specifically, the binocular camera (RGB image data) provides a high-resolution color image of the user's hand and calculates the 3D depth information of the hand using binocular disparity. The original image data is usually transmitted in RGB format and may contain interference information such as illumination changes and noise. The AR glasses send the image data to the server or edge computing device through a wireless network or a wired interface (such as USB, Wi-Fi, 5G, etc.). The IMU (Inertial Measurement Unit) is used to detect the acceleration, angular velocity, and direction changes of the hand to assist in analyzing gesture dynamics. The three-axis acceleration (X, Y, Z) and three-axis gyroscope (angular velocity) data can be collected dozens to hundreds of times per second. The optical flow sensor calculates the displacement of the hand relative to the environment by tracking the movement of pixel points to help estimate the gesture trajectory. The optical flow vector information records the relative movement of the hand between consecutive frames. The micro electromyogram sensor (EMG) detects the muscle activity of the hand and is used to identify micro-motions (such as finger bending, clenching, and opening). The electromyogram signal, usually in mV, represents the intensity of the electrical activity during muscle contraction.

[0035] The original data usually contains noise and interference information. Therefore, the following processing is required to improve the accuracy of subsequent calculations: Denoising is used to eliminate the illumination noise and motion blur in the image, as well as the high-frequency interference signals in the IMU and EMG data. Gaussian filtering and median filtering are used for the image data to remove optical noise. Low-pass filtering is used for the IMU / EMG data to remove high-frequency jitter signals. Filtering is used to smooth the data, improve stability, and avoid misjudgment caused by short-term jitter. Kalman filtering or mean filtering is used for the IMU data to reduce the error in attitude calculation. Bilateral filtering is used for the optical flow data to enhance the boundary information of moving objects. Feature classification is used to extract key information, such as the position, speed, acceleration, and muscle activity pattern of the hand, and classify it for gesture recognition. Deep learning models (such as CNN, ResNet, etc.) are used for the RGB image to extract features, such as finger skeletons and joint angles. The IMU data is based on time series analysis to extract the motion trend and rhythm. Pattern recognition algorithms are used for the EMG data to distinguish different gesture muscle activity patterns. Normalization makes the data units and scales of different sensors consistent to avoid calculation deviations caused by different numerical ranges. The IMU / EMG data is converted to the range of 0-1 or -1 to 1 for multi-modal fusion. The RGB image uses a normalization method (such as zero-mean normalization) to adjust the pixel intensity distribution.

[0036] S120. Based on the RGB image data, perform skeleton tracking on the preset key parts of the user's hand to determine the first gesture action.

[0037] Specifically, the server first receives RGB image data from the AR glasses. These data are captured by a binocular camera (or a monocular camera combined with depth estimation) and contain visible light image information of the user's hand. The binocular camera can provide more accurate depth information to help construct a 3D hand skeleton. The monocular camera can infer 3D information through depth estimation algorithms (such as CNN-based monocular depth estimation). The server uses deep learning models (such as OpenPose, MediaPipeHands, HandPose, etc.) to identify the joint points of the hand and establish a skeleton model of the hand. Specifically, it includes the following steps: The server first performs pixel-level segmentation on the RGB image to extract the hand from the background, reducing unnecessary interference information. A semantic segmentation network (such as DeepLabV3+) or a hand detection model (such as YOLO, SSD) is used for hand detection. The server constructs a hand skeleton based on the RGB image and calculates the 3D coordinates of the key points of the hand (such as finger joints, palm center, etc.). If a binocular camera is used, the disparity between the two viewpoints can be calculated to infer the depth information. The depth information of the hand is predicted based on a trained neural network (such as DepthNet). The server uses a convolutional neural network (CNN) or a Transformer-based model to detect 21 key points of the hand (including multiple joint points of the thumb, index finger, middle finger, ring finger, and little finger). By analyzing the positional relationships of these key points, information such as the bending degree of the fingers and the palm direction is further determined.

[0038] In a possible implementation manner, according to the RGB image data, skeleton tracking is performed on the preset key parts of the user's hand to determine the first gesture action, specifically including: using an image segmentation network to perform pixel-level segmentation on the RGB image to extract the contour data of the user's hand; according to the RGB image data, using binocular disparity calculation to construct a 3D skeleton model of the user's hand; using a convolutional neural network to perform key point recognition on the 3D skeleton model to obtain a set of key points; and matching the preset key parts according to the set of key points and the contour data.

[0039] Specifically, use deep learning-based image segmentation networks (such as DeepLabV3+, Mask R-CNN, U-Net) to perform pixel-level segmentation on RGB images and extract the hand region. Generate a binary mask, where the hand region is 1 and other regions are 0, thus obtaining clear hand contour data. In the factory inspection scenario, the inspector wears AR glasses, and the system needs to recognize his gesture operations. When the background is complex (such as there are many devices and workpieces), image segmentation can accurately extract the hand region and avoid misidentifying other objects in the background, such as tools or machine parts. Use binocular cameras to obtain the depth information of the hand to generate an accurate 3D skeleton. The server receives the binocular camera data from the AR glasses and obtains two RGB images, the left and the right. Calculate the disparity of the hand key points in the left and right images, and calculate the depth information (Z-axis coordinate) of each pixel point through triangulation. Convert the 2D hand image into 3D point cloud data, and use a skeleton regression model (such as the MANO model, HandPose model) to generate a complete 3D hand skeleton. In virtual work order operations, if the inspector makes a "point and click with finger" action, the system needs to recognize the depth information of the finger to determine whether he is pointing at a nearby virtual button or a distant menu option. Without 3D information, the system may misjudge the target of the gesture. On the 3D skeleton model, accurately locate the hand key points (such as the palm center, fingertips, knuckles, etc.). Use a pre-trained convolutional neural network (CNN) or Transformer-based model (such as MediaPipe Hands, OpenPose, HandPoseNet) to extract features from the 3D skeleton. Identify 21 hand key points (including the fingertips, knuckles, palm center of the five fingers, etc.). Calculate the 3D coordinates (X, Y, Z) of each key point. By analyzing the geometric relationship between the key point set and the hand contour data, judge the accurate position of the key part. If the key point falls within the contour range, confirm that the point is valid. If the key point deviates from the contour, use a smoothing filter or optimization algorithm (such as Kalman filter, RANSAC) for correction.

[0040] S130. Calculate the gesture speed, gesture acceleration, and gesture movement trajectory based on the multi-modal sensor data.

[0041] Specifically, multi-modal sensors refer to IMU (Inertial Measurement Unit), optical flow sensors, micro electromyography (EMG) sensors, etc., which provide multi-dimensional information on hand movements. The IMU (Inertial Measurement Unit) consists of an accelerometer and a gyroscope, which measure linear acceleration and angular velocity respectively. Through the angular velocity of the gyroscope, the rotation angle of the hand in 3D space can be deduced. Through the data of the accelerometer, the motion state of the hand can be estimated. The optical flow sensor tracks the motion changes of the hand in the field of view and measures the relative displacement, which helps to optimize the stability of trajectory calculation. The micro electromyography (EMG) sensor monitors the electrical activity of the hand muscles, judges the hand force, and assists in distinguishing different gesture types. The server calculates gesture speed, gesture acceleration, and gesture trajectory through multi-modal sensor data, which can not only distinguish static gestures and dynamic gestures, but also accurately analyze the user's intention, greatly improving the naturalness and reliability of AR interaction. This solution is widely applicable to fields such as industrial inspection, AR remote control, and smart home interaction, providing strong technical support for contactless operation.

[0042] In a possible implementation manner, according to the RGB image data, skeleton tracking is performed on the preset key parts of the user's hand to determine the first gesture action. Specifically, it further includes: determining a first key point and a second key point, where the first key point and the second key point are any two key points among the multiple key points included in the key point set; according to the 3D skeleton model, determining the first coordinate corresponding to the first key point and determining the second coordinate corresponding to the second key point, where both the first coordinate and the second coordinate are 3D coordinates; using temporally consecutive image frames, and smoothing the first coordinate and the second coordinate by combining Kalman filtering and / or recursive least squares method to obtain the first gesture action.

[0043] Specifically, the 3D skeleton model of the hand contains multiple key points (such as fingertips, knuckles, wrists, etc.). The first key point and the second key point are any two of these key points. For example, the first key point may be the fingertip of the index finger. The second key point may be the fingertip of the thumb. The purpose of selecting these key points is to calculate the gesture form through their coordinates, such as whether the fingers are pinched or bent. Since the binocular cameras of the AR glasses can sense depth information, the 3D coordinates of the hand key points can be calculated. The coordinates of each key point include X (left-right), Y (up-down), and Z (front-back depth) information, which helps to construct a complete 3D hand skeleton. Through the coordinate relationships of multiple key points, the overall action of the hand can be determined, such as "palm open" or "finger bent". For example, when the user makes a "waving" gesture, the system can track: the change in the Z coordinate of the wrist key point (judging whether it is approaching / away from the camera). The change in the X coordinate of the finger key point (judging whether the hand is swinging horizontally). Combining these pieces of information, the system can identify the waving gesture.

[0044] In actual use, due to noise in camera shooting and data transmission, the gesture trajectory may jitter or have errors. To improve accuracy, signal processing algorithms are usually used for smoothing. Kalman filtering uses the data of the current frame and historical frames to predict the key point positions of the next frame and correct errors. The recursive least squares method continuously adjusts the key point coordinates to make the gesture trajectory smoother and reduce jitter.

[0045] S140. Fuse the first gesture action based on the gesture speed, gesture acceleration, and gesture movement trajectory to obtain a second gesture action.

[0046] Specifically, gesture speed refers to the moving rate of the hand key points over time, that is, the distance the hand moves per unit time. Calculate its speed through the change in hand position at multiple time points. For example: If the hand moves from coordinates (0, 0, 0) to (0.05, 0.02, 0) in 0.1 seconds, the speed can be expressed as (0.5, 0.2, 0) m / s. Function: It can be used to distinguish "fast actions" and "slow actions". For example, "quickly waving the hand" and "slowly swinging" may correspond to different instructions. Example, the user quickly waves the hand → may indicate a "cancel" or "exit" operation. The user slowly moves the palm → may indicate a "page turning" or "parameter adjustment" operation. Gesture acceleration is the rate of change of the hand movement speed, that is, the "urgency" of the gesture. Calculation method: Calculate the acceleration through continuous speed data. For example: If the speed increases from (0.5, 0.2, 0) m / s to (1.0, 0.4, 0) m / s in 0.2 seconds, the acceleration is (2.5, 1.0, 0) m / s². Function: It can be used to detect "sudden" or "slow" actions. For example: High acceleration → may indicate a "pat" or "quick confirmation" operation. Low acceleration → may indicate a "stable slide" or "continuous press" operation. Example, the user quickly raises the palm (high acceleration) → may indicate "opening the menu" or "activating a certain function". The user slowly pushes the palm (low acceleration) → may indicate "adjusting the volume" or "precision positioning".

[0047] The gesture movement trajectory is the complete path of the hand movement, which describes the change trend of the gesture in the time dimension. Combine the hand positions at multiple time points to draw the movement path. For example: If the gesture forms a circular trajectory from point A → B → C → D, it may be a "rotation" gesture. If the trajectory is a straight line upwards, it may be a "slide" gesture. Function: Used to distinguish different types of gestures. For example: A "Z" - shaped trajectory → may indicate an "undo" operation. A "circular trajectory" → may indicate a "rotation" or "scaling" operation. Example, the user draws a circle (circular trajectory) → may be used to adjust parameters (such as rotating a model). The user's palm swipes from left to right (linear trajectory) → may be used for "page turning" or "switching modes".

[0048] In a possible implementation, the first gesture action is fused through gesture speed, gesture acceleration, and gesture movement trajectory to obtain a second gesture action, which specifically includes: comparing the gesture speed with a preset gesture speed threshold, and comparing the gesture acceleration with a preset gesture acceleration threshold; if it is determined that the gesture speed is greater than or equal to the preset gesture speed threshold and the gesture acceleration is greater than or equal to the preset gesture acceleration threshold, then the action category is determined as a fast action, and the action categories include fast actions and slow actions; based on the action category being a fast action, the gesture movement trajectory is analyzed to generate gesture movement dynamic features; the gesture movement dynamic features are added to the first gesture action for dynamic fusion to obtain the second gesture action.

[0049] Specifically, the server first compares the calculated gesture speed with the preset speed threshold. At the same time, the gesture acceleration is compared with the preset acceleration threshold. Based on the combination of the two, the "action category" of the gesture is determined: fast action or slow action. Example, the user quickly swings their hand (high speed + high acceleration) → classified as a "fast action", such as "return to the main interface". The user slowly moves their finger (low speed + low acceleration) → classified as a "slow action", such as "fine drag". If the gesture is classified as a "fast action", the server will further analyze its movement trajectory. The trajectory shape determines the final gesture category, for example: linear movement → may be "swipe to flip the page". Z-shaped movement → may be an "undo" operation. Drawing a circle → may be "rotate" or "adjust". Example, the user quickly waves from left to right (linear trajectory) → may be recognized as "next page". The user quickly draws a "Z" → may trigger an "undo operation". The server generates dynamic features of the gesture based on the analysis results, including: movement direction, trajectory shape, action duration. Example, the user quickly draws a semi-circle → the server detects "clockwise trajectory + fast speed", mapped to the "rotate and adjust" function. The user slowly draws a circle → the server detects "counterclockwise trajectory + slow speed", mapped to the "fine-tune rotation angle" function. The server fuses the first gesture action with the dynamic features to generate the final second gesture action. This gesture action is more accurate than the original recognition result and can avoid misrecognition. Example, the user quickly waves + linear trajectory → "switch interface"; the user slowly drags + smooth trajectory → "slide and adjust"; the user suddenly lifts up + short trajectory → "activate function".

[0050] S150. Map the second gesture action to a preset virtual work order to obtain a dynamic integration instruction, where the preset virtual work order is pre-generated on the AR glasses.

[0051] Specifically, the second gesture action is an optimized gesture after fusion, which combines gesture speed, acceleration, and trajectory features and can more accurately represent the user's intention. The virtual work order is a task template created and stored in advance on the AR glasses for scenarios such as industrial inspection and equipment maintenance. The dynamic integration instruction is an instruction generated by the server for manipulating the virtual work order. It is triggered by the user's gesture and mapped to specific task operations. The server is responsible for parsing the user's gesture action, mapping it to the preset virtual work order on the AR glasses, and then generating a dynamic integration instruction to achieve an efficient and intuitive interaction method. This process improves the operation efficiency of tasks such as industrial inspection and equipment maintenance, enabling users to control the work order content more naturally and getting rid of the constraints of traditional input devices.

[0052] In a possible implementation, mapping the second gesture action to a preset virtual work order to obtain a dynamic integration instruction specifically includes: determining a preset mapping rule according to the second gesture action. The preset mapping rule includes the correspondence between the second gesture action and the preset virtual work order. The preset virtual work order includes inspection tasks, operation options, and work order structures; matching the preset mapping rule with the preset virtual work order and generating a dynamic integration instruction.

[0053] Specifically, the server will query the preset mapping rules in the current AR glasses to find tasks related to the gesture. Determine specific operation items according to the specific position of the gesture action. Extract corresponding work order instructions such as "confirm task", "enter details", "return to the previous level", etc. Operate the virtual work order directly through the gesture, without traditional mouse / touch screen input, improving work fluency. Suitable for situations such as wearing gloves and insufficient light, reducing the dependence on input devices. Through the preset mapping rules, reduce misrecognition and improve operation accuracy. The intuitive and natural gesture interaction method makes the operation more ergonomic and improves user satisfaction. The server is responsible for parsing the user's gesture action, mapping it to the preset virtual work order, and then generating a dynamic integration instruction to achieve an efficient and intuitive interaction method. This process improves the operation efficiency of tasks such as industrial inspection and equipment maintenance, enabling users to control the work order content more naturally and getting rid of the constraints of traditional input devices.

[0054] S160. Control the AR glasses to execute the dynamic integration instruction to generate a target virtual work order.

[0055] Specifically, the server is responsible for parsing the user's gesture intentions, converting them into work order operation instructions, and then sending the instructions to the AR glasses. Ensure that the AR glasses can accurately execute the user's gesture operations according to the preset rules. The server needs to communicate in real time to ensure low latency and high accuracy in instruction transmission. Make the AR glasses become the "intelligent execution end" to ensure the synchronization of user gestures and work order operations. The dynamic integration instruction is an operation instruction generated by the server according to the preset rules of user gestures and virtual work orders. For example: "Page turning" instruction: Switch to the next inspection task. "Confirmation" instruction: Submit the inspection result. "Zoom" instruction: Zoom in on the device drawing. Bridge the user's intentions and work order operations so that gestures can directly drive the work order process. The target virtual work order is the final work order interface after user gesture interaction, server parsing, and AR glasses execution. It is not a static template but a real-time updated task interface that can dynamically display the operation results. The target virtual work order contains feedback information on user operations, such as: Operation logs (e.g., a certain task has been completed). Dynamic data (e.g., device status update). Interaction animations (e.g., button color change, interface switching). Finally, present the work order status after user interaction to ensure the real-time and visualization of task management. The server remotely controls the AR glasses, reduces manual input by the user, and improves the task execution speed. Get rid of the mouse, keyboard, and touch screen, and achieve full-process control through gestures. Through dynamic instructions and visual feedback, make the work order interaction smoother and more intuitive. The server calculates and optimizes gesture actions to ensure accurate recognition and low error operation rate.

[0056] In a possible implementation manner, controlling the AR glasses to execute the dynamic integration instruction to generate the target virtual work order specifically includes steps S210 to S230, and the above steps are as follows: S210. Generate an operation animation and an operation result for the preset virtual work order according to the dynamic integration instruction; S220. Send the operation animation and the operation result to the AR glasses and control the AR glasses to display the operation animation and the operation result to obtain the target virtual work order.

[0057] Specifically, it is generated by the server according to factors such as user gestures and device status and is used to control how the AR glasses execute tasks. Convert the user's interaction actions (such as gestures, voices) into instructions executable by the AR glasses. Standardized task templates defined in advance in the system, such as equipment inspection, maintenance operations, etc. Structured data includes: Inspection task list (e.g., "Check the device temperature"). Operation options (e.g., "Confirm passed" or "Report abnormality"). Task status (e.g., "Not completed" → "Completed"). Visually display the dynamic effect of task status change to enhance the interaction experience. The operation result is the final task feedback after the user executes the operation to ensure the traceability of the operation. It includes data records (e.g., "Inspection completed, temperature normal"); status update (e.g., "Task has been confirmed"); task flow (e.g., "Enter the next operation").

[0058] The present application also provides a gesture interactive virtual work order integration device based on an AR glasses. Refer to Figure 3 , Figure 3 which is a schematic diagram of the modules of a gesture interactive virtual work order integration device based on an AR glasses provided in an embodiment of the present application. The device is a server, and the server includes an acquisition module 31 and a processing module 32. Among them, the acquisition module 31 acquires RGB image data and multi-modal sensor data of the user's hand sent by the AR glasses; the processing module 32 performs skeleton tracking on preset key parts of the user's hand according to the RGB image data to determine the first gesture action; the processing module 32 calculates the gesture speed, gesture acceleration, and gesture movement trajectory according to the multi-modal sensor data; the processing module 32 fuses the first gesture action through the gesture speed, gesture acceleration, and gesture movement trajectory to obtain the second gesture action; the processing module 32 maps the second gesture action to a preset virtual work order to obtain a dynamic integration instruction, and the preset virtual work order is generated on the AR glasses in advance; the processing module 32 controls the AR glasses to execute the dynamic integration instruction to generate a target virtual work order.

[0059] In a possible implementation manner, the acquisition module 31 acquires RGB image data and multi-modal sensor data of the user's hand sent by the AR glasses, specifically including: the acquisition module 31 receives the original image data sent by the binocular camera located on the AR glasses; the acquisition module 31 receives the original sensor data sent by the IMU, optical flow sensor, and microelectromyogram sensor located on the AR glasses; the processing module 32 performs data processing on the original image data and the original sensor data to obtain RGB image data and multi-modal sensor data, and the data processing includes denoising, filtering, feature classification, and normalization processing.

[0060] In a possible implementation manner, the processing module 32 performs skeleton tracking on preset key parts of the user's hand according to the RGB image data to determine the first gesture action, specifically including: the processing module 32 uses an image segmentation network to perform pixel-level segmentation on the RGB image to extract the contour data of the user's hand; the processing module 32 constructs a 3D skeleton model of the user's hand according to the RGB image data by using binocular disparity calculation; the processing module 32 uses a convolutional neural network to perform key point recognition on the 3D skeleton model to obtain a key point set; the processing module 32 matches the preset key parts according to the key point set and the contour data.

[0061] In a possible implementation, the processing module 32 performs skeleton tracking on preset key parts of the user's hand according to the RGB image data to determine the first gesture action. Specifically, it further includes: the processing module 32 determines a first key point and a second key point, where the first key point and the second key point are any two key points among the multiple key points included in the key point set; the processing module 32 determines the first coordinate corresponding to the first key point and the second coordinate corresponding to the second key point according to the 3D skeleton model, and both the first coordinate and the second coordinate are 3D coordinates; the processing module 32 uses temporally continuous image frames and combines Kalman filtering and / or recursive least squares method to smooth the first coordinate and the second coordinate to obtain the first gesture action.

[0062] In a possible implementation, the processing module 32 fuses the first gesture action through gesture speed, gesture acceleration, and gesture movement trajectory to obtain the second gesture action. Specifically, it includes: the processing module 32 compares the gesture speed with a preset gesture speed threshold and compares the gesture acceleration with a preset gesture acceleration threshold; if the processing module 32 determines that the gesture speed is greater than or equal to the preset gesture speed threshold and the gesture acceleration is greater than or equal to the preset gesture acceleration threshold, it determines that the action category is a fast action, and the action category includes fast actions and slow actions; based on the action category being a fast action, the processing module 32 analyzes the gesture movement trajectory to generate gesture movement dynamic features; the processing module 32 adds the gesture movement dynamic features to the first gesture action for dynamic fusion to obtain the second gesture action.

[0063] In a possible implementation, the processing module 32 maps the second gesture action to a preset virtual work order to obtain a dynamic integration instruction. Specifically, it includes: the processing module 32 determines a preset mapping rule according to the second gesture action, and the preset mapping rule includes the correspondence between the second gesture action and the preset virtual work order. The preset virtual work order includes inspection tasks, operation options, and work order structures; the processing module 32 matches the preset mapping rule with the preset virtual work order and generates a dynamic integration instruction.

[0064] In a possible implementation, the processing module 32 controls the AR glasses to execute the dynamic integration instruction to generate a target virtual work order. Specifically, it includes: the processing module 32 generates an operation animation and an operation result for the preset virtual work order according to the dynamic integration instruction; the processing module 32 issues the operation animation and the operation result to the AR glasses and controls the AR glasses to display the operation animation and the operation result to obtain the target virtual work order.

[0065] It should be noted that when the device provided in the above embodiment realizes its functions, only the division of the above-mentioned functional modules is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept. For the specific implementation process, please refer to the method embodiment, which will not be elaborated here.

[0066] This application also provides an electronic device. Referring to Figure 4 , Figure 4 which is a schematic structural diagram of an electronic device provided by an embodiment of this application. The electronic device may include: at least one processor 41, at least one network interface 44, a user interface 43, a memory 45, and at least one communication bus 42.

[0067] Among them, the communication bus 42 is used to realize the connection and communication between these components.

[0068] Among them, the user interface 43 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 43 may further include a standard wired interface and a wireless interface.

[0069] Among them, the network interface 44 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface).

[0070] Among them, the processor 41 may include one or more processing cores. The processor 41 connects various parts within the entire server through various interfaces and lines, and executes various functions of the server and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 45, and by calling data stored in the memory 45. Optionally, the processor 41 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 41 may integrate one or a combination of several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 41 and may be implemented separately by a chip.

[0071] Among them, the memory 45 may include a Random Access Memory (RAM) or a Read-Only Memory. Optionally, the memory 45 includes a non-transitory computer-readable storage medium. The memory 45 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 45 may include a program storage area and a data storage area. Among them, the program storage area can store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area can store the data involved in the above-mentioned method embodiments. Optionally, the memory 45 can also be at least one storage device located far from the aforementioned processor 41. As Figure 4 shown, in the memory 45 as a computer storage medium, there may be included an operating system, a network communication module, a user interface module, and an application program for a gesture interactive virtual work order integration method based on AR glasses.

[0072] In Figure 4 the electronic device shown, the user interface 43 is mainly used to provide an input interface for the user and obtain the data input by the user; while the processor 41 can be used to call the application program stored in the memory 45 for a gesture interactive virtual work order integration method based on AR glasses. When executed by one or more processors, the electronic device executes the method as described in one or more of the above embodiments.

[0073] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be in other sequences or performed simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0074] This application also provides a computer-readable storage medium, and the computer-readable storage medium stores instructions. When executed by one or more processors, the electronic device executes the method as described in one or more of the above embodiments.

[0075] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0076] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection between each other can be through some service interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical or other form.

[0077] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0078] In addition, in each embodiment of the present application, the functional units can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. And the aforementioned memory includes: various media such as USB flash drives, mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0080] The above are only exemplary embodiments of the present disclosure and should not be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure still fall within the scope covered by the present disclosure. Those skilled in the art will easily think of other implementation schemes of the present disclosure after considering the specification and the disclosed practice truth. The present application aims to cover any variations, uses, or adaptive changes of the present disclosure, and these variations, uses, or adaptive changes follow the general principles of the present disclosure and include the common general knowledge or conventional technical means in the technical field not recorded in the present disclosure. The specification and the embodiments are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A gesture interactive virtual work order integration method based on AR glasses, characterized in that, The method includes: Obtaining RGB image data and multi-modal sensor data for the user's hand sent by the AR glasses; Performing skeleton tracking on preset key parts of the user's hand according to the RGB image data to determine a first gesture action; Calculating a gesture speed, a gesture acceleration, and a gesture movement trajectory according to the multi-modal sensor data; Fusing the first gesture action through the gesture speed, the gesture acceleration, and the gesture movement trajectory to obtain a second gesture action; Mapping the second gesture action to a preset virtual work order to obtain a dynamic integration instruction, where the preset virtual work order is pre-generated on the AR glasses; Controlling the AR glasses to execute the dynamic integration instruction to generate a target virtual work order.

2. The gesture interactive virtual work order integration method based on an AR glasses according to claim 1, wherein The obtaining of the RGB image data and the multi-modal sensor data for the user's hand sent by the AR glasses specifically includes: Receiving original image data sent by a binocular camera located on the AR glasses; Receiving original sensor data sent by an IMU, an optical flow sensor, and a microelectromyogram sensor located on the AR glasses; Performing data processing on the original image data and the original sensor data to obtain the RGB image data and the multi-modal sensor data, where the data processing includes denoising, filtering, feature classification, and normalization processing.

3. The gesture interactive virtual work order integration method based on an AR glasses according to claim 1, wherein The performing of skeleton tracking on preset key parts of the user's hand according to the RGB image data to determine a first gesture action specifically includes: Using an image segmentation network to perform pixel-level segmentation on the RGB image to extract contour data of the user's hand; Constructing a 3D skeleton model of the user's hand according to the RGB image data by using binocular disparity calculation; Performing key point recognition on the 3D skeleton model by using a convolutional neural network to obtain a key point set; Matching to obtain the preset key parts according to the key point set and the contour data.

4. The gesture interactive virtual work order integration method based on an AR glasses according to claim 3, characterized in that The performing of skeleton tracking on preset key parts of the user's hand according to the RGB image data to determine a first gesture action specifically further includes: Determining a first key point and a second key point, where the first key point and the second key point are any two key points among the multiple key points included in the key point set; Determining a first coordinate corresponding to the first key point and a second coordinate corresponding to the second key point according to the 3D skeleton model, where both the first coordinate and the second coordinate are 3D coordinates; Using temporally consecutive image frames and combining Kalman filtering and / or recursive least squares method to perform smoothing processing on the first coordinate and the second coordinate to obtain the first gesture action.

5. The gesture interactive virtual work order integration method based on an AR glasses according to claim 1, wherein The fusing of the first gesture action through the gesture speed, the gesture acceleration, and the gesture movement trajectory to obtain a second gesture action specifically includes: Comparing the gesture speed with a preset gesture speed threshold, and comparing the gesture acceleration with a preset gesture acceleration threshold; If it is determined that the gesture speed is greater than or equal to the preset gesture speed threshold and the gesture acceleration is greater than or equal to the preset gesture acceleration threshold, it is determined that the action category is a fast action, and the action category includes the fast action and the slow action; Based on the action category being the fast action, analyze the gesture movement trajectory to generate gesture movement dynamic features; Add the gesture movement dynamic features to the first gesture action for dynamic fusion to obtain the second gesture action.

6. The gesture interactive virtual work order integration method based on an AR glasses according to claim 1, wherein The mapping of the second gesture action to a preset virtual work order to obtain a dynamic integration instruction specifically includes: According to the second gesture action, determine a preset mapping rule, where the preset mapping rule includes the correspondence between the second gesture action and the preset virtual work order, and the preset virtual work order includes an inspection task, operation options, and a work order structure; Match the preset mapping rule with the preset virtual work order and generate the dynamic integration instruction.

7. The gesture interactive virtual work order integration method based on an AR glasses according to claim 1, wherein The controlling the AR glasses to execute the dynamic integration instruction to generate a target virtual work order specifically includes: According to the dynamic integration instruction, generate an operation animation and an operation result for the preset virtual work order; Send the operation animation and the operation result to the AR glasses and control the AR glasses to display the operation animation and the operation result to obtain the target virtual work order.

8. A gesture interactive virtual work order integration device based on an AR glasses, characterized in that, The gesture interactive virtual work order integration device includes an acquisition module (31) and a processing module (32), where, The acquisition module (31) is used to acquire RGB image data and multi-modal sensor data for the user's hand sent by the AR glasses; The processing module (32) is used to perform skeleton tracking on preset key parts of the user's hand according to the RGB image data to determine the first gesture action; The processing module (32) is further used to calculate the gesture speed, gesture acceleration, and gesture movement trajectory according to the multi-modal sensor data; The processing module (32) is further used to fuse the first gesture action through the gesture speed, the gesture acceleration, and the gesture movement trajectory to obtain the second gesture action; The processing module (32) is further used to map the second gesture action to a preset virtual work order to obtain a dynamic integration instruction, and the preset virtual work order is pre-generated on the AR glasses; The processing module (32) is further used to control the AR glasses to execute the dynamic integration instruction to generate a target virtual work order.

9. An electronic device, characterized in that, The electronic device includes a processor (41), a memory (45), a user interface (43), and a network interface (44). The memory (45) is used to store instructions. Both the user interface (43) and the network interface (44) are used to communicate with other devices. The processor (41) is used to execute the instructions stored in the memory (45) so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method according to any one of claims 1 to 7.

Images