Gesture recognition algorithm and system based on color attention in AR operation environment

Abstract

The invention discloses an improved gesture recognition method suitable for an operation environment and an operation tool transmission system. The method comprises the following implementation steps: capturing an image by using a first visual angle of AR glasses, projecting a material required by a current operation into the AR glasses for virtual imaging to assist the operation, transmitting the image to an ROS system through TCP / IP and RosConnected, preprocessing the image, performing gesture recognition by using a RegNetCA network, entering a judgment gate after a gesture recognition result is obtained, and completing the operation according to the judgment gate. The robot arm is guided to grab and transfer corresponding objects, d435i on the kinova robot arm is used for object recognition, Yolov8 is used for recognizing the objects and returning three-dimensional coordinates of the objects, coordinates of the objects under a camera coordinate system are converted into overall coordinates through coordinate conversion, and after the coordinates are obtained, the robot arm reaches a target point position to grab the objects and transfer the objects to a designated position. According to the invention, a complete non-contact object transmission system is formed, and the accuracy of gesture recognition is improved through a color-fused gesture recognition algorithm.

Description

Technical Field

[0001] This application relates to the fields of AR, machine learning and human-computer interaction, and a gesture recognition algorithm and system based on color attention in an AR surgical environment. Background Art

[0002] In modern medical practice, it is crucial to ensure a sterile environment and prevent and control infections during surgery. Traditional surgical instrument delivery methods often rely on medical staff to perform them directly, which increases the risk of infection in a sterile environment or when facing a high risk of infectious diseases. In this regard, the combination of augmented reality technology and robotic arm systems has shown great potential. AR technology can display case views, anatomical structures, and preoperative planning information in real time during surgery, giving surgeons clearer visual references when performing complex operations. This real-time information presentation not only enhances the doctor's understanding of the surgical area, but also reduces information omissions and misoperations that may occur during traditional surgery. At the same time, the robotic arm system can significantly reduce the risk of infection that may be introduced during the delivery of traditional surgical instruments by providing high-precision operations and stable instrument delivery capabilities.

[0003] In the field of gesture recognition, especially egocentric (first-person perspective) gesture recognition, there is still a lack of high-quality gesture datasets that are adapted to specific scenarios. Existing datasets such as EgoGesture provide a foundation, but they are still difficult to fully meet the needs of specific applications in terms of scale, variability, and realism. In particular, in surgical environments, the coverage and applicability of standard gesture recognition datasets are low. Although the hand detection method proposed by Yang and Zhu improves the accuracy of detection, it has limited effect under complex lighting conditions or dirty gloves. In addition, existing datasets usually ignore the particularity of the hand state of medical staff during surgery, such as gestures with gloves, blood stains, or other possible interference. These limitations hinder the development of gesture recognition technology in surgical scenarios. Therefore, datasets customized for surgical scenarios have become the key to improving the accuracy of gesture recognition.

[0004] Deep learning is widely used in gesture recognition, mainly through convolutional neural networks (CNN), long short-term memory networks (LSTM) and Transformer based on attention mechanism to improve recognition accuracy. The 3D CNN method proposed by Ge et al. significantly improves the recognition ability of gesture structure by utilizing the three-dimensional information of depth images. However, CNN still has the problem of reduced accuracy when processing gestures in dynamic and complex backgrounds. The LSTM method is good at processing sequence data, but its recognition effect is not ideal under the complex environmental changes in surgical scenes. In contrast, Transformer has stronger parallel processing capabilities and can capture dependencies in sequences, but the computational complexity is high. With the continuous optimization of network design architecture, the proposal of RegNet provides new possibilities for improving the accuracy of gesture recognition in surgical scenes while maintaining high efficiency. However, current deep learning models still face challenges in dealing with complex light and gesture changes in surgical environments.

[0005] In modern surgical procedures, the use of robotic arms has become an important way to reduce direct human contact and control of surgical instruments. Guoguang Du et al. emphasized the key role of object positioning and posture estimation in robotic arm grasping. However, existing robotic arm control systems still have certain limitations when dealing with high dynamics and complex operations. In order to achieve accurate surgical instrument delivery in a surgical environment, the robotic arm not only needs to have high-precision object recognition and grasping capabilities, but also needs to be combined with real-time gesture recognition and coordinate conversion mechanisms. Current research has made progress in autonomous motion control and precise grasping of robotic arms, but there is still room for further improvement when combined with complex surgical environments. The present invention improves the real-time grasping control of the robotic arm by combining AR technology with gesture recognition, significantly improving the accuracy and reliability of the operation.

[0006] In the field of human-computer interaction, the combination of augmented reality (AR) technology and robotic arm systems has shown great potential. The combination of AR and tactile feedback devices proposed by Yan et al. significantly improved the real-time performance of human-robot collaboration. Although Wang et al.'s AR and gesture recognition system has improved the intuitiveness and accuracy of operations, its application in high-risk environments such as surgery is still immature. In surgical environments, real-time and high-precision gesture recognition and sterile instrument delivery systems remain important directions for technological development. Although YOLOv8 performs well in object detection, there are still challenges in accuracy and safety when applying it to precise surgical instrument delivery. Existing systems lack support for real-time feedback and precise control in complex surgical scenarios, which has become a bottleneck for further development. Summary of the invention

[0007] The purpose of this invention is to propose a contactless surgical instrument delivery method through AR and a robotic arm for the operating room surgical environment. The AR-based surgical environment can assist the surgical system to provide support information for doctors' operations. Aiming at the human-computer interaction needs in AR gestures, an AR gesture dataset SHGD is constructed to provide data support for AR human-computer interaction based on gestures. At the same time, this paper provides a RegNetCA algorithm to provide algorithmic support for gesture interaction, simulates an AR surgical environment, and creates a robotic arm-assisted surgical delivery system to assist doctors in delivering surgical instruments during surgery.

[0008] The idea of ​​implementing the present invention is: first, the input information is captured by an AR head display, the information is preprocessed, and it is determined whether the input data is empty. If it is empty, it is not executed. If it is not empty, gesture recognition is performed, and the gesture is predicted by combining the gesture recognition model trained by the data set and the improved RegNetCA network. The color features of the input image are captured by the color histogram module. The network structure based on RegNet extracts high-level features through multiple convolution and residual modules. The color attention module uses color information to enhance attention to specific areas, and classification is used to obtain recognition results. In order to avoid execution errors and consider time costs, if the predicted results are equivalent within ten times, the judgment transfer instruction is entered. The video refresh rate here can reach 10-15. The judgment gate is used to determine which object the robotic arm should grab, and Yolov8 is used to perform object recognition to obtain the object position. The three-dimensional coordinates of the object are obtained by aligning the depth camera with the RGB camera. The coordinate system of the object under the current camera is converted into the overall coordinate system through the coordinate system conversion, and the position is sent to the robotic arm. The robotic arm plans the best running trajectory through path planning, and grabs and transfers the object to the specified position.

[0009] The specific steps of the present invention are as follows: Step 1: Collect gesture datasets and annotate the gestures and datasets.

[0010] The collection of gesture images is divided into two scenarios: one is wearing gloves with fake blood, and the other is wearing gloves without fake blood. The collector needs to wear AR glasses to collect first-person images; Extract frames from camera video data and classify gesture images; The image dataset is divided into 70% training set, 10% validation set and 20% test set.

[0011] Step 2: Preprocess the image: This includes image cropping and normalization, converting the image to HSV color space, and then creating a mask of the glove color; The mask is processed by morphological operation, and the contour closest to the center is found by contour extraction method, and finally the contour area is extracted as the hand area; Step 3: Project the materials needed for the current surgery into the AR eye for virtual imaging-assisted surgery: For the patient's surgical data, the system can use 3D modeling technology to project the detailed appearance, size, location and other information of the patient's lesion into the doctor's AR field of view in the form of a three-dimensional model, helping the doctor to accurately locate and use it in actual surgical operations. Step 4: Use RosConnected to communicate between the AR system and the Ros system: The surgeon issues commands through gesture input, and the system transmits the commands to the ROS system in real time through the RosConnected protocol; Step 5: Use the judgment gate to determine whether the gesture data is empty: Determine whether there is a need for gesture control in the current AR field of view; If yes, gesture recognition is performed; Step 6: Determine the gesture type through the RegNetCA network: The RegNetCA network consists of the RegNet base network and the color attention module; As the backbone of the model, RegNet extracts spatial features from images through a series of convolution, pooling, and nonlinear activation operations. In the model design, the width and depth of the convolution layer are defined so that it can process complex images efficiently. The color attention module generates an attention weight vector based on the color histogram of the input image through a fully connected layer. This weight vector is fused with the spatial features extracted by RegNet. In this way, the network can adjust the weights of each channel according to the color features and enhance its sensitivity to color information. This plays an important role in complex color interference in surgical scenes; By training the network, the optimal parameters are obtained and saved locally for calling for gesture recognition; Step 7: Enter the judgment gate according to the gesture recognition result: Determine the object content recognized by the robot arm camera based on the gesture recognition result; If the gesture recognition results are the same in ten video frames, the robot arm grabbing command is executed; Step 8: Perform object recognition through Yolov8 to obtain the object coordinates in the global coordinate system: Identify the object to be grasped based on the gesture recognition content; Calibrate the collected image of the object, put it into Yolov8 model training, and obtain the recognition model of the custom object Use the trained object recognition model to identify the object and obtain the object coordinates. Use depth-RGB alignment to obtain the three-dimensional coordinates of the object. Through coordinate transformation, the object coordinates in the camera coordinate system are transformed into the object coordinates in the global coordinate system; Step 9: The robot performs path planning based on the converted coordinates, grabs the object, and delivers the object to the specified location: Through the path planning algorithm, the optimal path from the current position of the robot arm to the grasping position of the target object is planned; Perform grasping tasks and deliver objects safely to the intended location; BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a system flow chart in AR robot interaction.

[0013] Figure 2 This is a diagram of the RegNetCA network structure. DETAILED DESCRIPTION

[0014] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0015] See attached Figure 1 , the steps of the present invention are further described in detail.

[0016] Step 1: Collect gesture datasets and annotate the gestures and datasets.

[0017] In order to adapt to the surgical environment, we simulated a real surgical environment. The subjects need to wear Hololens2 glasses, surgical gown sleeves and surgical gloves for collection. We collected an image dataset including multiple gestures. The dataset consists of seven different gestures of twenty people, totaling about 2,000 photos. The experimenter needs to wear surgical nitrile gloves to collect pictures under light. We also used props blood to smear on the gloves to simulate the real scene to ensure the diversity and representativeness of the dataset. The collection of gesture pictures is divided into two scenes, one is wearing gloves with fake blood data, and the other is wearing gloves without fake blood data, which are collected in different scenes; the collector needs to wear AR glasses for first-person image collection; extract frames from the camera video data and classify the gesture pictures; divide the image dataset into 70% training set, 10% validation set and 20% test set.

[0018] Step 2: Preprocess the image: This includes image cropping and normalization, converting the image to the HSV color space, and then creating a mask of the glove color; processing the mask through morphological operations, using the contour extraction method to find the contour closest to the center, and finally extracting the contour area as the hand area; Step 3: Project the materials needed for the current surgery into the AR eye for virtual imaging-assisted surgery: For the patient's surgical data, the system can use 3D modeling technology to project the detailed appearance, size, location and other information of the patient's lesion into the doctor's AR field of view in the form of a three-dimensional model, helping the doctor to accurately locate and use it in actual surgical operations. Step 4: Use RosConnected to communicate between the AR system and the Ros system: The surgeon issues commands through gesture input, and the system transmits the commands to the ROS system in real time through the RosConnected protocol; AR visual information is sent from HoloLens 2 to the topic in the ROS system.

[0019] The application running on HoloLens 2 converts the captured visual data and gesture commands into a standardized message format and sends this data to the ROS node in the Ubuntu system through the TCP / IP protocol. The ROSconnected framework is responsible for establishing a communication link in the ROS environment and publishing these messages to the corresponding topics. In the Ubuntu system, the ROS node is responsible for receiving and processing the data sent from HoloLens 2. The ROS node processes the received visual information and control commands into control instructions for the robotic arm through recognition. These instructions are then sent to the Kinova robotic arm to instruct it to perform the corresponding actions. In this way, AR visual information and ROS instructions can be seamlessly integrated and synchronized. Step 5: Use the judgment gate to determine whether the gesture data is empty: Determine whether there is a need for gesture control in the current AR field of view; if so, perform gesture recognition; determine the gesture type through the RegNetCA network: the RegNetCA network consists of the RegNet basic network and the color attention module; as the backbone of the model, RegNet extracts spatial features in the image through a series of convolution, pooling, and nonlinear activation operations. In the model design, the width and depth of the convolution layer are defined so that it can process complex images efficiently. The color attention module generates an attention weight vector based on the color histogram of the input image through the fully connected layer, and this weight vector is fused with the spatial features extracted by RegNet. In this way, the network can adjust the weights of each channel according to the color features and enhance the sensitivity to color information. This plays an important role in the complex color interference in surgical scenes; by training the network, its optimal parameters are obtained and saved locally for calling for gesture recognition; Step 6: Enter the judgment gate according to the gesture recognition result: determine the object content recognized by the robot arm camera according to the gesture recognition result; if the gesture recognition results are the same ten times in the video frame, execute the robot arm grabbing instruction; Step 7: Use Yolov8 to identify the object and obtain the object coordinates in the global coordinate system: The YOLOv8 model is trained for target detection. YOLOv8 is an advanced deep learning model that can achieve efficient and accurate object detection. The labeled image dataset is input into the YOLOv8 model, and hyperparameters are adjusted and trained to optimize the performance of the model. During the training process, we set the learning rate lr to 0.01, batch size batch=32, training rounds epochs=100, fixed image input size. Each image input to the model will be resized to 640x640 pixels and cross-validation is used to evaluate the accuracy and robustness of the model. Through testing on the validation set, the model achieved the expected detection accuracy. The trained model was used to identify the object to be grasped based on the gesture recognition content. The object coordinates were obtained and the three-dimensional coordinates of the object were obtained by depth-RGB alignment. The object coordinates in the camera coordinate system were converted to the object coordinates in the global coordinate system through coordinate conversion. The robot arm performed path planning based on the converted coordinates, grasped the object, and delivered the object to the specified location. The path planning algorithm was used to plan the optimal path from the current position of the robot arm to the grasping position of the target object. The grasping task was performed and the object was safely delivered to the predetermined location. The complete RegNetCA network model architecture is as follows Figure 2 As shown in the figure, it includes the color attention module and the RegNet network infrastructure. The input is the gesture image after data preprocessing, and the output is the final result of gesture recognition. The gesture recognition through AR glasses finally achieves the effect of human-computer interaction.

[0020] The software environment of the present invention is selected on a server with Ubuntu 20.04.4 system, a GeForceRTX3090 Ti GPU, and 24G video memory. Python 3.9 and pytroch-gpu 1.12.0 are used.

[0021] The various embodiments in the specification of this application, if there are the same or similar parts, can refer to each other, especially for the system embodiment, because its method is basically similar to the method embodiment, so the description is relatively brief, and the relevant content can be found in the description of the method embodiment. Of course, the technical solutions in the specification of this application are not limited to the above examples, and any technical features not described in detail can be implemented based on the prior art, which will not be repeated here. It should be noted that the above embodiments and drawings are only used to illustrate the technical solutions of this application, and the purpose is to help understand, rather than to limit this application. For ordinary technicians in this field, without departing from the spirit and substantive scope of this application, all modifications, improvements, additions or substitutions made should be deemed to be within the scope of protection of the claims of this application. These modifications and changes, if in accordance with the purpose of this application, should be within the scope of protection of the claims.

Claims

1. A gesture recognition method and system in AR robot interaction, characterized in that: The system comprises: Capturing surgical scene images from the first person perspective using augmented reality (AR) glasses; Virtually project the materials needed for the current surgery onto AR glasses to assist the surgery; Transmit the captured images to the ROS system via TCP / IP and ROSConnected protocol; Preprocess the image and use the RegNetCA network for gesture recognition; Based on the gesture recognition results, enter the judgment gate and control the robotic arm to grab the corresponding object; Object recognition using the D435i camera on the Kinova robotic arm; Return the 3D coordinates of the object based on the Yolov8 model, and convert the coordinates of the object in the camera coordinate system into the global coordinate system; The robot arm plans the path based on the converted coordinates, grasps the object, and delivers the object to the specified location.

2. The system according to claim 1, characterized in that The AR glasses display surgical materials in real time through virtual imaging to assist surgical operations, including: The AR glasses’ camera captures real-time images of the surgical environment; According to the surgical process or the doctor's needs, the required surgical material information is superimposed on the display interface of the AR glasses through virtual imaging.

3. The system according to claim 2, characterized in that The steps of transmitting images via TCP / IP and RosConnected include image compression to reduce network latency, including: Before image transmission, an image compression algorithm is used to compress the acquired image to reduce the size of the image file; Under the communication framework of ROSConnected, network optimization technology is used to further reduce transmission delays and ensure the real-time nature of image data.

4. The system according to claim 3, characterized in that The image is preprocessed, wherein the RegNetCA network is customized and improved to improve the accuracy of gesture recognition in surgical scenarios, including: By creating a mask of the glove color, performing morphological processing, contour extraction (extracting the contour closest to the middle), and finally obtaining the processed image; By separating the color channels of the input image and calculating the color histogram, the distribution of different colors in each channel is quantified; The feature weights of each color channel are learned through a two-layer fully connected neural network, and the attention weights of each channel are output, and the feature maps are dynamically adjusted using the weights.

5. The method according to claim 4, characterized in that The robotic arm controls the judgment gate based on the gesture recognition result, specifically including judging the confidence of the recognized gesture through the threshold, thereby deciding whether to perform the corresponding robotic arm action, including: The gesture recognition result is obtained through recognition. Only when the same gesture is recognized ten times will it enter the judgment gate to determine which object to grab, and then perform the corresponding robotic arm action.

6. The method according to claim 5, characterized in that The object recognition using the D435i camera on the Kinova robot arm includes: Use Yolov8 to train custom objects, identify objects, fix them to corresponding objects, and obtain the two-dimensional coordinates of the objects.

7. The method according to claim 6, characterized in that Returns the 3D coordinates of the object based on the Yolov8 model, and converts the coordinates of the object in the camera coordinate system to the global coordinate system, including: By retrieving the depth image of the object and aligning it with the RGB, the three-dimensional point coordinates of the identified object are obtained; the transformation matrix between the camera and the robotic arm is obtained through the calibration process, and the coordinate conversion algorithm is applied to convert the object coordinates under the camera into the coordinates under the robotic arm coordinate system.

8. The method according to claim 7, characterized in that The robot arm performs path planning based on the converted coordinates, grabs the object, and delivers the object to the specified location, including: The robot arm motion path is preset through the path planning algorithm to ensure the accuracy and speed of the robot arm motion; Fix the doctor's location, and when the doctor specifies the corresponding gesture, the corresponding item will be delivered to the corresponding point.

9. The method according to claim 8, characterized in that A complete contactless object transfer system is formed, including: Collect gesture data sets in a simulated surgical environment and use a color-integrated gesture recognition algorithm to train a gesture recognition model; Wear AR to present surgical assistance information and input gestures from the first person perspective; Determine the gesture detection result. If the gesture results recognized in the video are consistent ten times, the obtained result will be input into the target recognition; Determine the type of item to be grasped and determine the corresponding item type based on the gesture recognition results; Use target recognition to obtain the coordinates of the corresponding item class and convert them to the global coordinate system; Finally, the obtained results are input into the robot arm for path planning and implementation of grasping and transfer.

Images