Gesture control system and interaction method for sterile operation

Abstract

The invention provides a gesture control system and an interaction method for a sterile operation. The gesture control system comprises a visualization module and an interaction module, the visualization module is composed of a personal computer provided with visualization software and a display screen and used for loading and displaying the three-dimensional image model of the patient; and the hardware of the interaction module is an Ultra < 3 > Di three-dimensional hand tracking camera, and the interaction module is connected to the personal computer and is used for capturing hand actions. The system is not only suitable for surgical operations, but also can be used for classroom education of medical students, and helps the medical students to more intuitively understand the anatomical structures of organs. Meanwhile, due to the modular architecture and expandability of the system, a wide space is provided for future scientific research and development and technology upgrading, and the refined and intelligent progress of surgical operations can be promoted.

Description

Technical Field

[0001] This invention belongs to the field of medical human-computer interaction, specifically a gesture control system and interaction method for aseptic surgery. Background Technology

[0002] In traditional open minimally invasive surgery, because the surgical field cannot be directly visualized, surgeons need to project the surgical area onto a monitor using a laparoscopic system equipped with a camera. However, to ensure surgical precision, the surgeon still needs to repeatedly review the patient's CT, MRI images, or other medical records during the operation. Therefore, timely and efficient access to the patient's imaging medical records during surgery can effectively reduce operation time and increase surgical safety. In the current operating room environment, access to medical electronic imaging systems is mainly achieved by the surgeon requesting assistance from staff (circulating nurses, anesthesiologists, etc.). However, due to the lack of experience of staff in using imaging systems, operational errors are frequent, sometimes requiring the surgeon or assistant to leave the sterile area to operate the computer imaging system directly, followed by re-sterilization before rejoining the surgery. Obviously, this severely impacts the surgical process.

[0003] In surgery, precise 3D image visualization is crucial for surgical success. Traditionally, surgeons rely on contact devices (such as mice and keyboards) to operate and browse 3D image models. This not only increases the risk of germ transmission in the operating room but can also prolong surgical time due to frequent hand disinfection and the wearing of sterile gloves. Furthermore, the limited space for contact devices near the operating table can affect the surgeon's efficiency and surgical safety. With the continuous development of motion-sensing technology, non-contact gesture control has emerged as a potential solution. However, existing motion-sensing devices such as Kinect and Leap Motion have limitations in tracking performance, environmental adaptability, and development support, making it difficult to meet the high precision, stability, and flexibility requirements of surgery. Therefore, developing a gesture-controlled 3D image visualization system based on advanced motion-sensing technology has become an urgent need to improve surgical efficiency and safety. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a gesture control system and interaction method for aseptic surgery. This addresses the issue that in the prior art, surgeons rely on contact devices (such as mice and keyboards) to operate and browse 3D image models. This not only increases the risk of germ transmission in the operating room but may also prolong the operation time due to frequent hand disinfection and the wearing of sterile gloves. Furthermore, the limited operating space for contact devices next to the operating table may affect the surgeon's operational efficiency and surgical safety.

[0005] A gesture control system and interaction method for aseptic surgery, including a visualization module and an interaction module;

[0006] The visualization module consists of a personal computer with visualization software and a display screen, used to load and display the patient's three-dimensional image model;

[0007] The interaction module, consisting of an Ultraleap 3Di stereo hand tracking camera connected to the personal computer, is used to capture hand movements.

[0008] The interaction module is configured as follows:

[0009] Hand depth information is obtained using a binocular vision depth extraction algorithm;

[0010] Pre-detection tracking (TBD) technology is used to track hand joint information;

[0011] Hand pose estimation and gesture recognition based on a standard 21-keypoint model of the hand and a convolutional neural network (CNN) model;

[0012] The system, consisting of the visualization module and the interaction module, generates a virtual three-dimensional hand projection when it detects that the user has entered the operation state, and realizes the functions of selection, rotation, translation, scaling and transparency adjustment through three basic gestures.

[0013] Preferably, the convolutional neural network (CNN) model comprises a main network and an expert network, wherein:

[0014] The main network is trained based on a general hand pose dataset;

[0015] The expert network is trained on a dataset of specific surgical gesture categories, which include opening the hand, clenching the fist, and grasping actions.

[0016] Preferably, the Ultraleap 3Di stereo hand tracking camera has the ability to withstand complex lighting conditions and an IP54 protection rating to be suitable for the operating room environment.

[0017] Preferably, the visualization module provides a user interface control for undoing operation commands, which undoes the previous command in response to an undo gesture or interface operation.

[0018] Preferably, a gesture control interaction method for aseptic surgery includes the following steps:

[0019] S1. Detect the user's hand entering the operation state and generate a virtual 3D hand projection on the visual interface;

[0020] S2. Capture binocular images of the hand using the Ultraleap 3Di stereo hand tracking camera;

[0021] S3. Use a binocular vision depth extraction algorithm to obtain hand depth information and use TBD technology to track the coordinates of 21 key joints;

[0022] S4. Recognizing gesture types based on a CNN model, wherein the CNN model includes a main network and an expert network trained for specific surgical gestures;

[0023] S5. Execute the corresponding operation command based on the recognition result:

[0024] Responding to a selection gesture triggers the selection of the target area;

[0025] Respond to rotation / translation gestures to control the rotation or translation of the 3D model;

[0026] Responds to zoom gestures, adjusting the display scale of the 3D model;

[0027] Respond to transparency adjustment gestures to modify the transparency of the target area.

[0028] Preferably, the user is a surgeon wearing sterile gloves, and all operations are performed in a non-contact manner.

[0029] Preferably, in response to an undo command, the result of the previous operation is rolled back.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. This invention improves surgical efficiency and safety: Based on the Ultraleap 3Di gesture-controlled 3D image visualization system, non-contact gesture operation significantly reduces the number of times doctors need to scrub their hands for disinfection and wear sterile gloves during surgery, thereby reducing the risk of germ transmission and shortening surgical time. Simultaneously, the system's high-precision hand tracking and recognition capabilities ensure the accuracy and stability of doctors' manipulation of the 3D image model, improving surgical safety.

[0032] 2. This invention optimizes the surgical experience: Ultraleap 3Di's ultra-wide field of view and optimal tracking range allow surgeons to freely perform gesture operations near the operating table without worrying about limited operating space. Furthermore, the system's high frame rate and resistance to complex lighting conditions ensure the accuracy and real-time performance of gesture recognition under varying lighting conditions, further optimizing the surgeon's experience.

[0033] 3. This invention's system is not only suitable for surgical procedures but can also be used in medical students' classroom education, helping them to understand organ anatomy more intuitively. Furthermore, the system's modular architecture and scalability provide ample room for future scientific research and technological upgrades, contributing to the advancement of precision and intelligence in surgical procedures. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the key hand points model of the present invention;

[0035] Figure 2 This is a schematic diagram of the convolutional neural network structure of the present invention;

[0036] Figure 3 These are schematic diagrams of the three gestures of this invention;

[0037] Figure 4 This is a schematic diagram of the gesture tracking process of the present invention;

[0038] Figure 5 This is a task completion time diagram of the contact device and gesture control system of the present invention. Detailed Implementation

[0039] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0040] This invention provides a gesture control system and interaction method for aseptic surgery, including a visualization module and an interaction module;

[0041] The visualization module, consisting of a personal computer with visualization software and a display screen, is used to load and display the patient's three-dimensional image model. The visualization module provides user interface controls for undoing operation commands, responding to undo gestures or interface operations to undo the previous command.

[0042] The interaction module, with hardware consisting of an Ultraleap 3Di stereo hand tracking camera, is designed to withstand complex lighting conditions and has an IP54 protection rating to adapt to the operating room environment. It is connected to the personal computer to capture hand movements.

[0043] The interaction module is configured as follows:

[0044] Hand depth information is obtained using a binocular vision depth extraction algorithm;

[0045] Pre-detection tracking (TBD) technology is used to track hand joint information;

[0046] Hand pose estimation and gesture recognition are performed based on a standard 21-keypoint hand model and a convolutional neural network (CNN) model. The CNN model consists of a main network and an expert network, wherein:

[0047] The main network is trained based on a general hand pose dataset;

[0048] The expert network is trained on a dataset of specific surgical gesture categories, which include open hand, clenched fist, and grasping actions.

[0049] The system, consisting of the visualization module and the interaction module, generates a virtual three-dimensional hand projection when it detects that the user has entered the operation state, and realizes the functions of selection, rotation, translation, scaling and transparency adjustment through three basic gestures.

[0050] (a) Select gesture: Extend the index finger of one hand perpendicular to the screen, and close the other four fingers together. Activate the click by pushing the gesture in the air.

[0051] (b) Rotation / Translation Gesture: Pinch the thumb and forefinger of one hand together and rotate the wrist to achieve rotation or translation gestures;

[0052] (c) Zooming gesture: Pinch your thumbs and index fingers together and zoom in or out by pulling or pushing your hands together.

[0053] (d) Transparency adjustment gesture: Same as the selection gesture, adjust the transparency of the target by clicking.

[0054] After tracking and recognizing the surgeon's gestures, operations such as selection, rotation, and scaling of the image can be performed based on the gesture definition. The formula for exchanging the coordinates of the operator's hand space and the virtual model space is:

[0055]

[0056] in, B P represents the coordinates of the user's hand in the real-world coordinate system. P is the rotational component from the user's hand space to the virtual model space. BORG For translation components, A Let P be the coordinates of the user's hand in the virtual model space. Interaction between the user's gestures and the virtual model can be achieved through spatial transformation using a homogeneous matrix. This equation can be simplified to:

[0057]

[0058] The network loss function is:

[0059]

[0060] Where C is the number of categories, y i It's a real label. The probability predicted by the model

[0061] A gesture control interaction method for aseptic surgery includes the following steps:

[0062] S1. Detect the user's hand entering the operation state. The user is a surgeon wearing sterile gloves. All operations are completed in a non-contact state. A virtual three-dimensional hand projection is generated on the visualization interface.

[0063] S2. Capture binocular images of the hand using the Ultraleap 3Di stereo hand tracking camera;

[0064] S3. Use a binocular vision depth extraction algorithm to obtain hand depth information and use TBD technology to track the coordinates of 21 key joints;

[0065] S4. Recognizing gesture types based on a CNN model, wherein the CNN model includes a main network and an expert network trained for specific surgical gestures;

[0066] S5. Execute the corresponding operation command based on the recognition result:

[0067] Responding to a selection gesture triggers the selection of the target area;

[0068] Respond to rotation / translation gestures to control the rotation or translation of the 3D model;

[0069] Responds to zoom gestures, adjusting the display scale of the 3D model;

[0070] Responding to transparency adjustment gestures, the transparency of the target area is modified;

[0071] In response to an undo command, roll back the results of the previous operation.

[0072] The implementation of this embodiment is as follows: When the Ultraleap 3Di stereo hand tracking camera detects the surgeon's gestures and enters the operation state, the operation state can be set according to the actual situation. For example, when the surgeon's hands are 45 cm above the sensor, a pair of virtual 3D hands will be displayed on the PC and screen. As the surgeon makes different gestures, the sensor will recognize these gestures in real time and record the 3D position of the hands in its coordinate frame. The state and movement of the hands are processed by the command module in the sensor and transmitted to the visualization module via a data cable. The visualization module responds to the commands and executes the corresponding operations. Five functions are achieved through three basic gestures: selection, rotation, translation, scaling, and transparency adjustment.

[0073] (1) Selection: The operator needs to extend the index finger and hold it perpendicular to the screen, while keeping the other four fingers closed. Point the tip of the index finger at the part of the screen to be selected, and keep the hand still in the air. Activate the selection by using the "air push" gesture, that is, move the hand toward the screen to indicate "click", and then select the specified target.

[0074] (2) Rotation: The operator needs to align the thumb and index finger with the part to be rotated and pinch them together, while the other three fingers are naturally spread out. Then, by rotating the wrist up, down, left or right, the operator can achieve rotation in the following ways: upward, downward, clockwise or counterclockwise.

[0075] (3) Translation: The operator needs to align the thumb and forefinger with the model to be translated and pinch them together, while the other three fingers are naturally spread out. Then, by using the gestures of translating upward, downward, leftward, rightward, forward or backward, the operator can achieve the translation function of moving upward, downward, leftward, rightward, far away or near.

[0076] (4) Zooming: The operator needs to simultaneously align the thumbs and index fingers of both hands with the area to be zoomed or sized and pinch them together, while the other fingers are naturally clenched into fists. While maintaining this position, both hands can be pulled outward or pushed inward simultaneously to enlarge or shrink the specified area.

[0077] (5) Transparency Adjustment: Similar to the gesture for executing the "Select" command, the operator only needs to "click" the specified area once to reduce its transparency, thereby revealing its hidden structure. Clicking again will completely hide the area.

[0078] If the system incorrectly interprets an operator's gesture and executes an incorrect command, the erroneous command can be undone by clicking the "Cancel" button on the screen. The Ultraleap 3Di system undergoes initial calibration at the factory, and user calibration is not required. During normal use, the system typically does not require recalibration. However, if significant changes occur in the operating environment, such as drastic changes in lighting, recalibration may be necessary to maintain tracking accuracy. The system recalibration process generally takes only 1-2 minutes.

[0079] Based on the above implementation methods, the applicant designed the following experiments and evaluated the implementation methods:

[0080] The experimental design selected 10 thoracic surgeons with 3 to 7 years of surgical experience as experimental subjects. Before the experiment, the patients' 3D image data was transferred to "Hand Control View". Four tasks were set in sequence, including translation, rotation, hiding specific structures, scaling and resetting the model. The experiment began after the preliminary preparations were completed.

[0081] The first phase involved thoracic surgeons using gestures to control a 3D image visualization system to complete designated tasks, recording the time each surgeon needed from placing their hands in front of the sensors to picking up the thoracoscope again after completing all tasks.

[0082] The second phase involved thoracic surgeons using traditional contact devices (mouse and keyboard) to complete the same task. The time taken for each surgeon was recorded from the moment they removed their sterile gloves, to completing the task using the traditional devices, to re-washing and disinfecting their hands, putting on surgical gowns and sterile gloves, and finally picking up the thoracoscope again.

[0083] The experimental results showed that the time required to complete the specified task using the contact device (M=361.46, SD=9.59) was greater than the time required to complete the specified task using the gesture-controlled 3D image visualization system (M=77.15, SD=7.41), and this difference was statistically significant (t(95)=74.92, p<0.05, 95%, CI[275.73,292.90]), indicating that the gesture-controlled 3D image visualization system based on Ultraleap3Di can significantly improve the surgeon's operational efficiency and reduce the operation time, as shown in Table 5.

[0084] In the gesture recognition accuracy verification experiment, the system demonstrated high gesture recognition accuracy in both basic operations and complex scenarios. In basic operations, the average success rate for the five core functions (selection, rotation, translation, scaling, and transparency adjustment) was 95.80%. In complex scenario tests, the success rates for rapid continuous operations, interference environment A (200 lux), and interference environment B (50 lux) were 92.50%, 91.25%, and 90.00%, respectively. The following is a comparison table of the system's success rates across different dimensions:

[0085]

[0086]

[0087] After the scale evaluation experiment, participants were asked to evaluate the system overall based on four criteria: fluency, effectiveness, ease of use, and potential. The rating scale ranged from 1 to 5 (5 = excellent, 4 = good, 3 = average, 2 = slightly poor, 1 = very poor). The system's average fluency score was 3.1, its average effectiveness score was 4.6, its average ease of use score was 4.2, and its average potential score was 4.8, indicating that the system is easy to learn and use, can effectively shorten operation time, and has broad application potential in future surgical procedures.

[0088] The embodiments of the present invention are given for the purposes of illustration and description. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A gesture control system for aseptic surgery, characterized in that, Includes a visualization module and an interactive module; The visualization module consists of a personal computer with visualization software and a display screen, used to load and display the patient's three-dimensional image model; The interaction module, consisting of an Ultraleap 3Di stereo hand tracking camera connected to the personal computer, is used to capture hand movements. The interaction module is configured as follows: Hand depth information is obtained using a binocular vision depth extraction algorithm; Pre-detection tracking (TBD) technology is used to track hand joint information; Hand pose estimation and gesture recognition based on a standard 21-keypoint model of the hand and a convolutional neural network (CNN) model; The system, consisting of the visualization module and the interaction module, generates a virtual three-dimensional hand projection when it detects that the user has entered the operation state, and realizes the functions of selection, rotation, translation, scaling and transparency adjustment through three basic gestures.

2. The gesture control system for aseptic surgery as described in claim 1, characterized in that: The convolutional neural network (CNN) model comprises a main network and an expert network, wherein: The main network is trained based on a general hand pose dataset; The expert network is trained on a dataset of specific surgical gesture categories, which include opening the hand, clenching the fist, and grasping actions.

3. The gesture control system for aseptic surgery as described in claim 1, characterized in that: The Ultraleap 3Di stereo hand tracking camera is equipped with resistance to complex lighting conditions and an IP54 protection rating to adapt to the operating room environment.

4. The gesture control system for aseptic surgery as described in claim 1, characterized in that: The visualization module provides a user interface control for undoing operation commands, which undoes the previous command in response to an undo gesture or interface operation.

5. A gesture control interaction method for aseptic surgery, applied to the system described in any one of claims 1-4, characterized in that, Including the following steps: S1. Detect the user's hand entering the operation state and generate a virtual 3D hand projection on the visual interface; S2. Capture binocular images of the hand using the Ultraleap 3Di stereo hand tracking camera; S3. Use a binocular vision depth extraction algorithm to obtain hand depth information and use TBD technology to track the coordinates of 21 key joints; S4. Recognizing gesture types based on a CNN model, wherein the CNN model includes a main network and an expert network trained for specific surgical gestures; S5. Execute the corresponding operation command based on the recognition result: Responding to a selection gesture triggers the selection of the target area; Respond to rotation / translation gestures to control the rotation or translation of the 3D model; Responds to zoom gestures, adjusting the display scale of the 3D model; Respond to transparency adjustment gestures to modify the transparency of the target area.

6. The gesture control interaction method for aseptic surgery as described in claim 5, characterized in that: The user is a surgeon wearing sterile gloves, and all operations are performed in a non-contact manner.

7. The gesture control interaction method for aseptic surgery as described in claim 5, characterized in that: The response to the undo command rolls back the result of the previous operation.

Images