Color attention-based gesture recognition algorithm and system in AR surgical environment

Abstract

This application discloses an improved gesture recognition method and surgical instrument delivery system adapted to the surgical environment. The implementation steps are as follows: First-person view images are captured using AR glasses, and the materials needed for the current surgery are simultaneously projected onto the AR glasses for virtual imaging to assist the surgery. The images are transmitted to the ROS system via TCP / IP and RosConnected. The images are preprocessed, and gesture recognition is performed using a RegNetCA network. After obtaining the gesture recognition result, it enters a judgment gate, guiding a robotic arm to grasp and deliver the corresponding items. Object recognition is performed using the D435i on the Kinova robotic arm, and the object's three-dimensional coordinates are returned using Yolov8. Coordinate transformation is used to convert the object's coordinates in the camera coordinate system to global coordinates. After obtaining the coordinates, the robotic arm reaches the target point to grasp the item and deliver it to the designated location. This invention forms a complete contactless object delivery system, and the accuracy of gesture recognition is improved by incorporating a color-based gesture recognition algorithm.

Description

Technical Field

[0001] This application relates to the fields of AR, machine learning and human-computer interaction, and to a color attention-based gesture recognition algorithm and system in an AR surgical environment. Background Technology

[0002] In modern medical practice, ensuring a sterile environment and controlling infection during surgery is crucial. Traditional methods of transferring surgical instruments often rely on direct handling by medical personnel, which increases the risk of infection in sterile environments or when facing high-risk infectious diseases. In this regard, the combination of augmented reality (AR) technology and robotic arm systems demonstrates significant potential. AR technology can display real-time views of the surgical case, anatomical structures, and preoperative planning information during surgery, providing surgeons with clearer visual references when performing complex procedures. This real-time information presentation not only enhances the surgeon's understanding of the surgical area but also reduces information omissions and errors that may occur during traditional procedures. Simultaneously, robotic arm systems, by providing high-precision operation and stable instrument transfer capabilities, can significantly reduce the infection risks that may be introduced during traditional surgical instrument transfer.

[0003] In the field of gesture recognition, especially egocentric (first-person perspective) gesture recognition, there is still a lack of high-quality gesture datasets adapted to specific scenarios. Existing datasets such as EgoGesture provide a foundation, but they still fall short of fully meeting the needs of specific applications in terms of scale, variability, and realism, particularly in surgical environments where standard gesture recognition datasets have low coverage and applicability. While the hand detection method proposed by Yang and Zhu improves detection accuracy, its effectiveness is limited under complex lighting conditions or with soiled gloves. Furthermore, existing datasets often overlook the specific characteristics of medical staff's hand states during surgery, such as gloved gestures, bloodstains, or other potential interferences. These limitations hinder the development of gesture recognition technology in surgical scenarios. Therefore, datasets tailored for surgical scenarios are crucial for improving gesture recognition accuracy.

[0004] Deep learning is widely used in gesture recognition, primarily employing methods such as Convolutional Neural Networks (CNNs), Long Short-Term Memory Networks (LSTMs), and Transformers based on attention mechanisms to improve recognition accuracy. The 3D CNN method proposed by Ge et al. significantly improves the recognition ability of gesture structures by utilizing the three-dimensional information of depth images. However, CNNs still suffer from accuracy degradation when handling gestures in dynamic and complex backgrounds. LSTMs excel at processing sequential data, but their recognition performance is less than ideal in the complex environmental changes of surgical scenarios. In contrast, Transformers possess stronger parallel processing capabilities and can capture dependencies in sequences, but their computational complexity is high. With continuous optimization of network architecture design, the introduction of RegNet offers new possibilities for improving gesture recognition accuracy in surgical scenarios while maintaining high efficiency. However, current deep learning models still face challenges in dealing with complex lighting and gesture variations in surgical environments.

[0005] In modern surgery, the use of robotic arms has become an important way to reduce direct human contact with and control surgical instruments. Guoguang Du et al. emphasized the crucial role of object localization and attitude estimation in robotic arm grasping; however, existing robotic arm control systems still have certain limitations when handling highly dynamic and complex operations. To achieve precise delivery of surgical instruments in the surgical environment, robotic arms not only need high-precision object recognition and grasping capabilities but also need to incorporate real-time gesture recognition and coordinate transformation mechanisms. Current research has made progress in the autonomous motion control and precise grasping of robotic arms, but there is still room for further improvement when combined with complex surgical environments. This invention improves the real-time grasping control of robotic arms by combining AR technology with gesture recognition, significantly enhancing the accuracy and reliability of operations.

[0006] In the field of human-computer interaction, the combination of augmented reality (AR) technology and robotic arm systems has demonstrated enormous potential. The combination of AR and haptic feedback devices proposed by Yan et al. significantly improved the real-time performance of human-robot collaboration. While Wang et al.'s AR-based gesture recognition system improved operational intuitiveness and accuracy, its application in high-risk environments such as surgery remains immature. In surgical settings, real-time, high-precision gesture recognition and sterile instrument delivery systems remain important directions for technological development. Although YOLOv8 performs excellently in object detection, its application in precise surgical instrument delivery still faces challenges in terms of accuracy and safety. The lack of real-time feedback and precise control support in complex surgical scenarios in existing systems has become a bottleneck for further development. Summary of the Invention

[0007] The purpose of this invention is to propose a contactless surgical instrument transfer method using AR and a robotic arm for the operating room environment. The AR-based surgical environment can assist the surgical system in providing support information to the surgeon. To address the human-computer interaction needs in AR gestures, an AR gesture dataset SHGD is constructed to provide data support for gesture-based AR human-computer interaction. At the same time, this paper provides a RegNetCA algorithm to provide algorithmic support for gesture interaction. An AR surgical environment is simulated, and a robotic arm-assisted transfer surgical system is created to assist the surgeon in transferring surgical instruments during the operation.

[0008] The approach to implementing this invention is as follows: First, input information is captured through an AR headset. This information undergoes data preprocessing. The system determines whether the input data is empty. If empty, no action is taken; otherwise, gesture recognition is performed. A gesture recognition model trained using a dataset and an improved RegNetCA network is used to predict gestures. The color histogram module captures the color features of the input image. Based on the RegNet network structure, multiple convolutional and residual modules extract high-level features. A color attention module enhances attention to specific regions using color information. Classification yields the recognition result. To avoid execution errors and consider time costs, if the predicted results are identical within ten iterations, a judgment and instruction transmission process is initiated. The video refresh rate can reach 10-15. A judgment gate determines which object the robotic arm should grasp. Object recognition is performed using Yolov8 to obtain the object's position. The object's 3D coordinates are obtained by aligning the depth camera with the RGB camera. The object's coordinate system under the current camera is transformed into the global coordinate system through coordinate system transformation. The position is then sent to the robotic arm. The robotic arm plans the optimal running trajectory through path planning and grasps and transmits the object to the designated position.

[0009] The specific steps of this invention are as follows:

[0010] Step 1: Collect a gesture dataset and annotate the gestures and the dataset.

[0011] The collection of gesture images is divided into two scenarios: one is data with gloves on and fake blood on the hands, and the other is data with gloves on but without fake blood on the hands.

[0012] The data collector needs to wear AR glasses to capture first-person images;

[0013] Extract frames from camera video data and categorize the gesture images;

[0014] The image dataset was divided into a 70% training set, a 10% validation set, and a 20% test set.

[0015] Step 2, preprocess the image:

[0016] This includes image cropping and normalization, converting the image to the HSV color space, and then creating a mask for the glove's color;

[0017] The mask is processed by morphological operations, the contour extraction method is used to find the contour closest to the center, and finally the contour area is extracted as the hand area.

[0018] Step 3: Project the materials needed for the current surgery onto the AR eye for virtual imaging to assist the surgery:

[0019] For patient surgical data, the system can use 3D modeling technology to project detailed information such as the appearance, size, and location of the patient's lesions into the doctor's AR view in the form of a 3D model, helping the doctor to accurately locate and use the data during actual surgical procedures.

[0020] Step 4: Establish communication between the AR system and the ROS system via RosConnected.

[0021] The surgeon issues commands via gesture input, and the system transmits the instructions to the ROS system in real time through the RosConnected protocol;

[0022] Step 5: Determine if the gesture data is empty by checking the door:

[0023] Determine if there is a need for gesture control within the current AR field of view;

[0024] If so, perform gesture recognition;

[0025] Step 6: Determine the gesture type using the RegNetCA network.

[0026] The RegNetCA network consists of the RegNet base network and a color attention module;

[0027] As the backbone of the model, RegNet extracts spatial features from images through a series of convolutions, pooling, and non-linear activation operations. The model design defines the width and depth of the convolutional layers to efficiently process complex images. The color attention module generates an attention weight vector based on the color histogram of the input image through a fully connected layer. This weight vector is then fused with the spatial features extracted by RegNet. This allows the network to adjust the weights of each channel according to color features, enhancing its sensitivity to color information. This plays a crucial role in handling complex color interference in surgical scenarios.

[0028] The network is trained to obtain its optimal parameters and saved locally for use in gesture recognition.

[0029] Step 7: Based on the obtained gesture recognition results, proceed to the judgment gate:

[0030] The content of the object being recognized by the robotic arm's camera is determined based on the gesture recognition results;

[0031] If the gesture recognition results are the same ten times in the video frame, then the robotic arm grasping command is executed.

[0032] Step 8: Perform object recognition using Yolov8 to obtain the object coordinates in the global coordinate system.

[0033] The object to be grasped is identified based on the gesture content;

[0034] The acquired images of the object are calibrated and then fed into a Yolov8 model for training to obtain a custom object recognition model.

[0035] The trained object recognition model is used to identify objects and obtain their coordinates. Then, depth-RGB alignment is used to obtain the three-dimensional coordinates of the objects.

[0036] By transforming the coordinates, the object coordinates in the camera coordinate system are transformed to the object coordinates in the global coordinate system;

[0037] Step 9: Based on the converted coordinates, the robotic arm plans its path, grasps the object, and delivers it to the designated location.

[0038] The optimal path for the robotic arm to grasp the target object from its current pose is planned using a path planning algorithm.

[0039] Perform the grasping task and safely deliver the object to the designated location; Attached Figure Description

[0040] Figure 1 This is a system flowchart for AR robot interaction.

[0041] Figure 2 This is a structural diagram of the RegNetCA network. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the accompanying drawings.

[0043] See attached document Figure 1 The steps of the present invention will be described in further detail below.

[0044] Step 1: Collect a gesture dataset and annotate the gestures and the dataset.

[0045] To simulate a real surgical environment, we required participants to wear HoloLens2 glasses, surgical gown sleeves, and surgical gloves. We created our own image dataset containing various hand gestures. This dataset consists of approximately 2000 images from twenty individuals representing seven different hand gestures. Participants wore surgical nitrile gloves under illumination while capturing images. We also used prop blood to smear on the gloves to mimic real-world scenarios, ensuring the diversity and representativeness of the dataset. Hand gesture images were collected in two scenarios: one with gloves and fake blood, and the other without. Participants wore AR glasses for first-person image capture. Frames were extracted from camera video data, and the hand gesture images were categorized. The image dataset was divided into a 70% training set, a 10% validation set, and a 20% test set.

[0046] Step 2, preprocess the image:

[0047] This process includes image cropping and normalization, converting the image to the HSV color space, and then creating a mask for the glove color. The mask is processed through morphological operations, and a contour extraction method is used to find the contour closest to the center. Finally, the contour region is extracted as the hand region.

[0048] Step 3: Project the materials needed for the current surgery onto the AR eye for virtual imaging to assist the surgery:

[0049] For patient surgical data, the system can use 3D modeling technology to project detailed information such as the appearance, size, and location of the patient's lesions into the doctor's AR view in the form of a 3D model, helping the doctor to accurately locate and use the data during actual surgical procedures.

[0050] Step 4: Establish communication between the AR system and the ROS system via RosConnected.

[0051] The surgeon issues commands via gesture input, and the system transmits the instructions to the ROS system in real time through the RosConnected protocol; AR visual information is sent from HoloLens 2 to the ROS system.

[0052] Applications running on HoloLens 2 convert captured visual data and gesture commands into a standardized message format and send this data to ROS nodes on the Ubuntu system via TCP / IP protocol. The ROSconnected framework is responsible for establishing communication links in the ROS environment and publishing these messages to relevant topics. On the Ubuntu system, the ROS nodes are responsible for receiving and processing the data sent from HoloLens 2. The ROS nodes process the received visual information and control commands into control instructions for the robotic arm. These instructions are then sent to the Kinova robotic arm to guide it in performing corresponding actions. In this way, AR visual information and ROS instructions can be seamlessly integrated and synchronized.

[0053] Step 5: Determine if the gesture data is empty by checking the door:

[0054] The system determines whether there is a need for gesture control within the current AR field of view; if so, gesture recognition is performed. The RegNetCA network is used to determine the gesture type. The RegNetCA network consists of a base RegNet network and a color attention module. As the backbone of the model, RegNet extracts spatial features from the image through a series of convolutions, pooling, and non-linear activation operations. The model design defines the width and depth of the convolutional layers to efficiently process complex images. The color attention module generates an attention weight vector based on the color histogram of the input image through fully connected layers. This weight vector is then fused with the spatial features extracted by RegNet. This allows the network to adjust the weights of each channel according to color features, enhancing its sensitivity to color information. This plays a crucial role in handling complex color interference in surgical scenarios. The network is trained to obtain its optimal parameters, which are then saved locally for use in gesture recognition.

[0055] Step 6: Based on the obtained gesture recognition results, proceed to the judgment gate: determine the content of the object recognized by the robotic arm camera based on the gesture recognition results; if the gesture recognition results are the same in ten video frames, then execute the robotic arm grasping command.

[0056] Step 7: Perform object recognition using Yolov8 to obtain the object coordinates in the global coordinate system.

[0057] The YOLOv8 model was trained for object detection. YOLOv8 is an advanced deep learning model capable of efficient and accurate object detection. A labeled image dataset was input into the YOLOv8 model, and hyperparameters were tuned and trained to optimize model performance. During training, the learning rate (lr) was set to 0.01, batch size to 32, and training epochs to 100. The image input size was fixed; each image input to the model was resized to 640x640 pixels, and cross-validation was used to evaluate the model's accuracy and robustness. The model achieved the expected detection accuracy on the validation set. Using the trained model, the system identifies the object to be grasped based on the gesture recognition content; obtains the object's coordinates; uses depth-RGB alignment to obtain the object's 3D coordinates; transforms the object's coordinates from the camera coordinate system to the global coordinate system; the robotic arm performs path planning based on the transformed coordinates, grasps the object, and delivers it to the designated location; the path planning algorithm plans the optimal path from the current pose to the target object grasping position; the grasping task is executed, and the object is safely delivered to the predetermined location.

[0058] The complete RegNetCA network model architecture is as follows: Figure 2 As shown, this includes a color attention module and the RegNet network infrastructure. The input is a preprocessed gesture image, and the output is the final gesture recognition result. Gesture recognition via AR glasses ultimately achieves human-computer interaction.

[0059] The software environment for this invention was selected on a server running Ubuntu 20.04.4, with a GeForce RTX 3090 Ti GPU and 24GB of video memory. Python 3.9 and Pytroch-gpu 1.12.0 were used.

[0060] The various embodiments in this application specification, if they have the same or similar parts, can be referred to each other, especially for system embodiments, since their methods are basically similar to those of method embodiments, and therefore the descriptions are relatively brief. Relevant content can be found in the descriptions of the method embodiments. Of course, the technical solutions in this application specification are not limited to the examples above. Any technical features not described in detail can be implemented based on existing technology, and will not be elaborated here. It should be noted that the above embodiments and accompanying drawings are only used to illustrate the technical solutions of this application and are intended to aid understanding, not to limit this application. For those skilled in the art, various modifications, improvements, additions, or substitutions made without departing from the spirit and scope of this application should be considered within the scope of protection of the claims of this application. Such modifications and changes, if they conform to the spirit of this application, should be within the scope of protection of the claims.

Claims

1. A gesture recognition method for AR robot interaction, characterized in that, include: Capture surgical scene images from a first-person perspective using augmented reality (AR) glasses; The materials needed for the current surgery can be virtually projected onto AR glasses to assist in the surgery; The captured images are transmitted to the ROS system via TCP / IP and ROSConnected protocols; The image is preprocessed, and gesture recognition is performed using the RegNetCA network; wherein the RegNetCA network is customized and improved to enhance the accuracy of gesture recognition in surgical scenarios, including: The process involves creating a mask for the glove's color, performing morphological processing, extracting the contour closest to the center, and finally obtaining the processed image. By separating the color channels of the input image and calculating the color histogram, the distribution of different colors in each channel is quantified. The feature weights of each color channel are learned through a two-layer fully connected neural network, and the attention weights of each channel are output. The feature map is then dynamically adjusted using these weights. Based on the gesture recognition results, the system enters a judgment gate and controls the robotic arm to grasp the corresponding item. Specifically, this involves determining the confidence level of the recognized gesture through a threshold to decide whether to perform the corresponding robotic arm action. The gesture recognition results are obtained by recognition. Only when the same gesture is recognized in ten attempts will it enter the judgment gate to determine which object to grab, and thus perform the corresponding robotic arm action. Object recognition is performed using the D435i camera on the Kinova robotic arm; this includes: training a custom object using Yolov8, recognizing the object, fixing it to the corresponding object, and obtaining the object's two-dimensional coordinates. Returns the 3D coordinates of the object based on the Yolov8 model, and transforms the object's coordinates in the camera coordinate system to the global coordinate system; including: By retrieving the depth image of the object and aligning it with RGB, the three-dimensional point coordinates of the object are obtained; the transformation matrix between the camera and the robotic arm is obtained through the calibration process, and a coordinate transformation algorithm is applied to convert the object coordinates under the camera into coordinates under the robotic arm coordinate system. The robotic arm plans its path based on the converted coordinates, grasps the object, and delivers it to a designated location; this includes: The path planning algorithm is used to pre-determine the movement path of the robotic arm to ensure the accuracy and speed of the robotic arm's movement; The system assigns a fixed location to the doctor and, when the doctor gives a specific gesture, delivers the corresponding item to that location.

2. The method according to claim 1, characterized in that, The AR glasses described herein display surgical materials in real time through virtual imaging to assist in surgical procedures, including: Real-time images of the surgical environment are captured using the camera on the AR glasses; Based on the surgical procedure or the doctor's needs, the required surgical material information is overlaid on the AR glasses' display interface through virtual imaging.

3. The method according to claim 2, characterized in that, The steps for image transmission via TCP / IP and RosConnected include image compression to reduce network latency, including: Before image transmission, the acquired images are compressed using an image compression algorithm to reduce the size of the image files; Within the ROSConnected communication framework, network optimization techniques are used to further reduce transmission latency and ensure the real-time performance of image data.

4. The method according to claim 1, characterized in that, A complete contactless object transfer system has been formed, including: We collected a dataset of gestures in a simulated surgical environment and trained a gesture recognition model using a gesture recognition algorithm that incorporates color. Wearing AR for surgical assistance information presentation and first-person perspective gesture input; The gesture detection results are evaluated. If the gestures detected in the video are consistent ten times, the results are input into the target recognition. Determine the type of item to be grasped and identify the corresponding item type based on the gesture recognition results; Utilize target recognition to obtain the coordinates of the corresponding item category, and then transform them to the global coordinate system; The final result is input into the robotic arm for path planning, grasping, and transfer.

Images