Voice control intelligent instrument transfer robot system for operation assistance and control method

Through the voice-controlled intelligent instrument transmission robot system, multi-joint robotic arms and deep learning technology can be used to achieve accurate identification and transmission of surgical instruments, solving the problems of insufficient experience of instrument nurses and transmitting risks, improving surgical efficiency and safety, and through automated recycling and disinfection processes improve the cleanliness and safety of the medical environment.

CN118924433APending Publication Date: 2024-11-12AFFILIATED CHILDRENS HOSPITAL OF CAPITAL INST OF PEDIATRICS
View PDF 0 Cites 12 Cited by

Patent Information

Application Number
CN202410995479.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In minimally invasive surgical procedures, the device nurses have incorrect and inefficient transmission equipment due to insufficient experience or long-term work, and there is a risk of cutting injuries and infection in the process of surgical instrument transmission. The existing technology lacks effective surgical auxiliary equipment nurse robots Solution.

Method used

A voice-controlled intelligent instrument transmission robot system was designed, using multi-joint robotic arms, visual recognition modules and voice recognition modules, and using voice passwords to recognize surgical instruments, and using three-dimensional lidar and deep learning technology to achieve accurate identification, pickup and transmission of instruments. Combining real-time force control technology ensures safety and efficiency.

Benefits of technology

It realizes the rapid and precise transmission of surgical instruments, reduces the risk of injury and infection of medical staff, improves surgical efficiency and safety, reduces the physical and mental burden of doctors, improves the quality and success rate of surgery, and uses automated recycling and disinfection. The process improves the cleanliness and safety of the medical environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118924433A_ABST
    Figure CN118924433A_ABST
Patent Text Reader

Abstract

The invention discloses a sound control intelligent instrument transfer robot system for operation assistance and a control method, and belongs to the technical field of intelligent robots. The method comprises the following steps: sensing and recognizing a voice command of a doctor through a voice control recognition module carried on the mechanical arm; after the mechanical arm obtains the password of the doctor, the needed surgical instrument is recognized according to the two-dimensional code information marked on the surgical instrument; extracting instrument grabbing position and grabbing attitude information from a pre-stored instrument two-dimensional code database, performing attitude detection on the surgical instrument by using the three-dimensional point cloud data, and picking up the surgical instrument; recognizing and positioning gestures of doctors through deep learning; when it is detected that the doctor is in a hand stretching state, the target instrument is transferred to the hand of the doctor; and after it is detected that the instrument is taken away, the mechanical arm returns to the initial position to wait, rapid and accurate transferring of the surgical instrument is achieved, and the mechanical arm has important significance in actual clinical application such as surgical operation medical care assistance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to a voice-controlled intelligent instrument delivery robot system and a control method for surgical assistance. Background Art

[0002] In minimally invasive surgery, instrument nurses are not only good partners for surgeons, but also an indispensable force for the entire surgical team. High-level instrument nurses can assist the surgeon in completing high-quality surgery better, faster, and more efficiently. Generally, there are 20 to 30 small surgical instruments, and large and medium-sized surgeries require hundreds of instruments. Therefore, instrument nurses must not only have rich professional knowledge and skilled operating skills, but also know all surgical instruments and their locations. During the operation, instrument nurses must always remain highly alert and work closely with the surgeon. As soon as the doctor reaches out, the instrument nurse must accurately deliver the required instruments so that every doctor can focus on performing the operation. Currently, instrument nurses are mainly faced with the following problems: 1) Due to lack of experience, young instrument nurses may not deliver instruments correctly or properly, are unfamiliar with cooperating with the leading doctors, and have low efficiency, which affects the surgical rhythm and effect of the surgeon; 2) Instrument nurses need to stand for a long time during the operation, pay attention to every step of the operation, and deliver the correct surgical instruments. The work process is full of pressure and labor intensity; 3) Surgical instruments are usually sharp. During the operation, surgical instruments will be passed from hand to hand between the instrument nurse and the surgeon, and the attention of the instrument nurse and the surgeon is generally focused on the patient, ignoring personal protection, which makes it easy for knife cuts to occur, leading to the risk of spreading blood-borne diseases.

[0003] With the continuous development of medical technology, various medical robots are being used more and more widely. At present, the research on medical robots mainly focuses on surgical robots, rehabilitation robots, nursing robots and micro-robots, while there are fewer instrument nurse robots used for surgical cooperation. The present invention proposes a voice-controlled intelligent instrument delivery robot system for surgical assistance, which replaces the instrument nurse to complete the intelligent identification, picking up, delivery of surgical instruments and interaction with medical staff according to the voice commands of the surgeon during surgery. It can effectively solve the shortage of skilled instrument nurses in surgical operations, reduce the risk of injury and infection of medical staff, and improve and enhance the level of primary medical and health care. Summary of the invention

[0004] The purpose of the present invention is to provide a voice-controlled intelligent instrument delivery robot system and control method for surgical assistance to solve the problems raised in the above-mentioned background technology.

[0005] In order to solve the above technical problems, the present invention provides a voice-controlled intelligent instrument delivery robot system for surgical assistance, comprising: a mechanical arm having multiple joints and each joint having a torque sensor, a controller for force control of the mechanical arm at a frequency of 1000 Hz per second, a visual recognition module, a voice recognition module for recognizing the doctor's command and obtaining the command of the requested surgical instrument, and a storage tray for storing sterile items and surgical instruments;

[0006] Visual recognition module: One camera is installed on the end effector of the robotic arm, and uses a three-dimensional laser radar to identify the QR code information on the surgical instrument and the three-dimensional spatial position of the surgical instrument; the other camera is fixed in the space outside the robotic arm body, and the position of the camera and the world coordinate system remains unchanged, which is used to detect and identify the doctor's gesture posture.

[0007] Furthermore, during laparoscopic surgery or open surgery, the robot completes the tasks of identifying, picking up, and delivering surgical instruments according to the doctor's voice commands, and through real-time force control technology, ensures high reliability and high safety in the human-computer interaction process. The specific process includes the following steps:

[0008] S1. Sense the instrument request signal: The voice control sensor on the robotic arm is used to sense the instrument request signal, detect the doctor's voice command through the voice control interaction function, and obtain the surgical instrument information required by the doctor in a contactless human-computer interaction manner;

[0009] S2. Identify instruments: Each surgery has a set of exclusive surgical instruments. Generally, there are 20 to 30 small surgical instruments, and large and medium-sized surgeries require hundreds of instruments. In addition, the same type of surgical instruments may have different specifications or similar appearances. The surgical instruments are distinguished by the exclusive QR codes on the surgical instruments, which are applicable to all surgeries, and a database of surgical instrument-related information is established. After the robotic arm obtains the doctor's voice command, it scans and identifies the instrument QR code through the visual recognition module it carries, and identifies the surgical instrument that matches the command based on the instrument QR code database information.

[0010] S3. Picking up instruments: Use a 3D laser radar camera to collect 3D point cloud data of the required surgical instruments, and estimate the position and posture of the surgical instruments in 3D space; there are certain risks in the process of transferring surgical instruments. During the transfer process, the safe operating part needs to be facing the doctor, and the required surgical instruments need to be safely transferred to the doctor's hands. In order to implement the transfer process in accordance with the specification requirements, the 3D model of the surgical instruments, the grasping position and grasping posture data are recorded in the instrument QR code database. During the picking process, the target surgical instruments are picked up by clamping according to the set grasping position and grasping posture;

[0011] S4, transfer equipment: real-time force control sensing technology is used in the transfer process to enable the system to have good human-machine interaction capabilities, prevent misoperation and avoid collisions; the steps are:

[0012] ①. Surgical identification and positioning:

[0013] A. Key point detection: detect the doctor's hand and obtain an image of the hand area; identify the pixel coordinates of the key points of the hand skeleton and palm in the image;

[0014] B. Posture estimation: Based on deep learning, the position and posture of the doctor's gesture are estimated, and the state of the doctor's palm is determined, including the "extended hand" state and the "clenched fist" state;

[0015] ②Delivering instruments: When the doctor is detected to be in the "reaching out" state, the optimal obstacle-avoiding motion path from the starting point to the target point is designed and planned to deliver the instrument to the doctor's hand;

[0016] ③. Detection of "clenched fist" gesture: When the doctor's gesture is detected to be in the "clenched fist" state, it means that the doctor is ready to pick up the instrument;

[0017] ④. Sensing force changes: The force data of the end fixture is directly obtained through the torque sensor mounted on the robotic arm, and the pressure of the contact between the surgical instrument and the doctor's hand is used to determine whether the external force has changed;

[0018] ⑤. End of transfer: When a change in external force is detected, it means that the doctor has caught the instrument, and the robotic arm releases the surgical instrument, completing the transfer of the instrument;

[0019] S5, return to the initial position: when it is detected that the doctor has taken away the surgical instruments, return to the storage tray at the initial position and wait.

[0020] Furthermore, in S4, the specific steps of designing and planning the optimal obstacle avoidance motion path from the starting point to the target point are:

[0021] Ⅰ. Perception and positioning: First, the visual recognition module, including the 3D lidar camera installed on the end effector of the robotic arm and the camera fixed in the space outside the robotic arm, is used to accurately identify and locate the operating room environment and the doctor's hand position; second, the doctor's hand image is processed through deep learning technology to identify the pixel coordinates of the key points of the hand skeleton and palm, and estimate the position and posture of the gesture;

[0022] II. Path planning: Based on the perceived doctor's hand position and the preset target point, the robot system uses a heuristic algorithm to search for a safe path in the unit; the grid method uses the basic element as the minimum grid granularity, divides the map into grids, and determines the safe path based on the traffic factor of each grid;

[0023] III. Real-time force control perception: During the delivery of the device, the robot uses the torque sensor on the robotic arm to sense the motion state and force of the robotic arm in real time;

[0024] IV. Motion execution: Based on the planned optimal obstacle avoidance motion path, the robot controls the mechanical arm to move according to the preset trajectory and speed to deliver the surgical instrument to the doctor;

[0025] V. End of transfer and feedback: When the robot detects that the doctor has caught the instrument, it will immediately stop moving and release the surgical instrument. After completing the instrument transfer, the robot will send a feedback signal to the control system to confirm that the transfer task is completed and prepare to execute the next instruction.

[0026] Furthermore, in the process of perception and positioning, in addition to using 3D lidar cameras and fixed cameras, infrared sensors and depth cameras are also introduced to improve the accuracy and robustness of perception of the environment and the doctor's hand position through the fusion of multimodal data; based on deep learning technology, a more advanced dynamic gesture recognition algorithm is developed to track the movement trajectory and posture changes of the doctor's hand in real time; the development steps of a more advanced dynamic gesture recognition algorithm are as follows:

[0027] The first step is data collection and preprocessing;

[0028] Step 2: Deep learning model design;

[0029] Step 3: Model training and optimization;

[0030] Step 4: Real-time tracking and intention judgment.

[0031] Furthermore, in the path planning process, time-efficiency path planning is considered: on the basis of the grid obstacle avoidance control method, time is introduced as one of the optimization objectives to find a path that can avoid obstacles and reach the target point as quickly as possible, thereby improving the efficiency of instrument delivery; and learning-based path planning is carried out: using machine learning or reinforcement learning technology, the robot optimizes the path planning algorithm through continuous learning and practice, so that it can better adapt to various complex surgical environments and doctors' operating habits.

[0032] Furthermore, in the real-time force control perception process, while installing torque sensors on the robotic arms, a compliant control strategy is introduced, so that the robot can automatically adjust the force and speed when contacting the doctor or other objects, achieving smoother and safer interaction; combining the doctor's operating habits and historical data, a predictive force control perception algorithm is developed, so that the robot can predict the doctor's possible force changes in advance and actively adjust its own motion state.

[0033] Furthermore, the motion trajectory is adaptively adjusted during the execution of the movement. That is, during the robot's movement, the motion trajectory and speed are adaptively adjusted according to the real-time monitored environmental changes and the position changes of the doctor's hands, ensuring that the instrument can be delivered to the doctor's hands more accurately; develop collaborative operation functions between the robot and the doctor, so that the robot can adjust the position and posture of the instrument according to the doctor's instructions or hints, and achieve closer human-machine collaboration.

[0034] Furthermore, at the end of the transmission and feedback process, an emergency stop button or sensor is designed. When the robot detects an abnormal situation or the doctor issues a stop command, it can stop moving immediately to avoid harm to the doctor or patient; at the same time, it has fault self-diagnosis and recovery.

[0035] Furthermore, it also includes auxiliary recovery of the device, and the specific steps are:

[0036] 1) Robotic arm and gripping device: The robotic arm is equipped with a precise and flexible gripping device;

[0037] 2) Recycling bins and disinfection equipment: A special recycling bin is set up in the operating room to store used instruments. At the same time, automatic or semi-automatic disinfection equipment can be set up to perform preliminary cleaning and disinfection on the recycled instruments;

[0038] 3) Sensor and recognition system: Use visual recognition, infrared sensor and other technologies to accurately identify and locate the doctor's hand position and the position and status of surgical instruments;

[0039] 4) Real-time detection and judgment: The robot uses sensors to detect the doctor's hand movements and status in real time. When it detects that the doctor is reaching out and preparing to use a new instrument, the robot will quickly grab a sterilized instrument from the sterilization equipment and deliver it to the doctor along the best obstacle-avoiding path.

[0040] 5) Used instrument recycling: After the doctor uses the instrument, the robot quickly identifies and grabs the used instrument and puts it into the recycling bin;

[0041] 6) Disinfection and reuse: After preliminary cleaning and disinfection, the instruments in the recycling box are put back into the disinfection equipment for more thorough disinfection. The disinfected instruments are used for delivery by the robot again and recycled.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The voice control recognition module on the robotic arm senses and recognizes the doctor's voice commands; after obtaining the doctor's commands, the robotic arm identifies the required surgical instruments based on the QR code information marked on the surgical instruments; extracts the instrument grabbing position and grabbing posture information from the pre-stored instrument QR code database, uses three-dimensional point cloud data to perform posture detection on the surgical instruments, and picks up the surgical instruments; through deep learning, the doctor's gestures are identified and located; when it is detected that the doctor is in the "reaching out" state, the target instrument is passed to the doctor's hand; after detecting that the instrument has been taken away, the robotic arm returns to the initial position to wait, thereby realizing the rapid and accurate delivery of surgical instruments, which is of great significance in actual clinical applications such as surgical medical assistance.

[0044] The voice-controlled intelligent instrument delivery robot can quickly respond to the doctor's voice commands, accurately identify and deliver the required surgical instruments, reducing the time the doctor spends searching for and delivering instruments during surgery, thereby speeding up the surgical process. Through precise positioning and obstacle avoidance functions, the robot can ensure that it avoids collisions with the doctor or other surgical instruments during the delivery of instruments, reducing the risk of accidental injury. At the same time, the robot can also monitor the doctor's operating status in real time. Once an abnormality is detected, it can stop moving immediately to ensure the safety of the operation. The application of the voice-controlled intelligent instrument delivery robot allows doctors to focus on the surgical operation itself without being distracted by the delivery and management of instruments, thereby reducing the doctor's physical and mental burden and helping the doctor maintain a better surgical state. Through real-time force control sensing technology, the robot can accurately control the delivery force and speed of the instrument, avoiding instrument damage or surgical errors caused by improper operation, thereby improving the quality and success rate of the operation. The research and development and application of the voice-controlled intelligent instrument delivery robot have promoted the innovation and development of medical technology and provided more possibilities and development space for future surgical assistance robots.

[0045] Through the robot's assisted recycling function, the instruments used in the operating room can be quickly and accurately placed in the recycling bin, avoiding the scattering and confusion of instruments on the operating table and maintaining the cleanliness of the operating room; at the same time, it reduces the manual handling of instruments by doctors or nurses, reduces the risk of cross infection, and improves the safety of surgery; the robot can automatically identify and grab used instruments and put them into the recycling bin, simplifying the instrument recycling process; at the same time, after preliminary cleaning and disinfection, the instruments in the recycling bin can be re-entered into the disinfection equipment for more thorough disinfection to prepare for the next operation. This automated management process not only improves efficiency, but also reduces the possibility of human error; through the robot's assisted recycling function, it can replace some manual operations, reduce the workload of medical staff, and reduce the labor cost of the operating room; medical staff can devote more energy to surgical operations, improving work efficiency and surgical quality; through the robot's assisted recycling and disinfection, surgical instruments can be better recycled; this not only reduces the waste of instruments, but also reduces medical costs; at the same time, because the instruments have undergone strict disinfection, their safety in use is also guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A system module structure diagram of a voice-controlled intelligent instrument delivery robot system and control method for surgical assistance according to the present invention;

[0047] Figure 2 This is a flow chart of a voice-controlled intelligent instrument delivery robot system and control method for surgical assistance according to the present invention;

[0048] Figure 3 A two-dimensional code database diagram of a voice-controlled intelligent instrument delivery robot system and control method for surgical assistance according to the present invention;

[0049] Figure 4 A schematic diagram of picking up surgical instruments in a voice-controlled intelligent instrument transfer robot system and control method for surgical assistance according to the present invention;

[0050] Figure 5 A schematic diagram of a delivery instrument in a voice-controlled intelligent instrument delivery robot system and control method for surgical assistance according to the present invention;

[0051] Figure 6 This is a schematic diagram of the principle of detecting the "fist" gesture in a voice-controlled intelligent instrument delivery robot system and control method for surgical assistance in the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] See also Figure 1-Figure 6 , the present invention provides a technical solution:

[0054] See also Figure 1-Figure 6 As shown, a voice-controlled intelligent instrument delivery robot system for surgical assistance includes: a robotic arm with multiple joints and a torque sensor on each joint, a controller for force control of the robotic arm at a frequency of 1000 Hz per second, a visual recognition module, a voice recognition module for recognizing the doctor's command and obtaining the command of the requested surgical instrument, and a storage tray for storing sterile items and surgical instruments;

[0055] Visual recognition module: One camera is installed on the end effector of the robotic arm, and uses a three-dimensional laser radar to identify the QR code information on the surgical instrument and the three-dimensional spatial position of the surgical instrument; the other camera is fixed in the space outside the robotic arm body, and the position of the camera and the world coordinate system remains unchanged, which is used to detect and identify the doctor's gesture posture.

[0056] Furthermore, during laparoscopic surgery or open surgery, the robot completes the tasks of identifying, picking up, and delivering surgical instruments according to the doctor's voice commands, and through real-time force control technology, ensures high reliability and high safety in the human-computer interaction process. The specific process includes the following steps:

[0057] S1. Sense the instrument request signal: The voice control sensor on the robotic arm is used to sense the instrument request signal, detect the doctor's voice command through the voice control interaction function, and obtain the surgical instrument information required by the doctor in a contactless human-computer interaction manner;

[0058] S2. Identify instruments: Each surgery has a set of exclusive surgical instruments. Generally, there are 20 to 30 small surgical instruments, and large and medium-sized surgeries require hundreds of instruments. In addition, the same surgical instruments may have different specifications or similar appearances. In order to accurately identify the required surgical instruments, this system uses the exclusive QR code on the surgical instruments to distinguish the surgical instruments, which can be extended to all surgeries. Surgical instruments are generally sharp and dangerous medical instruments. Therefore, in addition to basic surgical instrument information, the grasping position and grasping posture of the surgical instruments are extremely important for the accuracy and safety of intraoperative instrument delivery. By establishing a database of surgical instrument-related information, the difficulty in identifying surgical instruments and the safety of intraoperative instrument delivery are solved; after the robotic arm obtains the doctor's voice command, it scans and identifies the instrument QR code through the visual recognition module on board, and identifies the surgical instrument matching the command based on the instrument QR code database information;

[0059] S3. Picking up instruments: Use a 3D laser radar camera to collect 3D point cloud data of the required surgical instruments, and estimate the position and posture of the surgical instruments in 3D space; there are certain risks in the process of transferring surgical instruments. During the transfer process, the safe operating part needs to be facing the doctor, and the required surgical instruments need to be safely transferred to the doctor's hands. In order to implement the transfer process in accordance with the specification requirements, the 3D model of the surgical instruments, the grasping position and grasping posture data are recorded in the instrument QR code database. During the picking process, the target surgical instruments are picked up by clamping according to the set grasping position and grasping posture;

[0060] S4, transfer equipment: real-time force control sensing technology is used in the transfer process to enable the system to have good human-machine interaction capabilities, prevent misoperation and avoid collisions; the steps are:

[0061] ①. Surgical identification and positioning:

[0062] A. Key point detection: Detect the doctor's hand and obtain an image of the hand area; identify the pixel coordinates of the key points of the hand skeleton and palm in the image, that is, Figure 5 Coordinates of 21 joints in the middle palm;

[0063] B. Posture estimation: Based on deep learning, the position and posture of the doctor's gesture are estimated, and the state of the doctor's palm is determined, including the "extended hand" state and the "clenched fist" state;

[0064] ②Delivering instruments: When the doctor is detected to be in the "reaching out" state, the optimal obstacle-avoiding motion path from the starting point to the target point is designed and planned to deliver the instrument to the doctor's hand;

[0065] ③. Detection of "clenched fist" gesture: When the doctor's gesture is detected to be in the "clenched fist" state, it means that the doctor is ready to pick up the instrument;

[0066] ④. Sensing force changes: The force data of the end fixture is directly obtained through the torque sensor mounted on the robotic arm, and the pressure of the contact between the surgical instrument and the doctor's hand is used to determine whether the external force has changed;

[0067] ⑤. End of transfer: When a change in external force is detected, it means that the doctor has caught the instrument, and the robotic arm releases the surgical instrument, completing the transfer of the instrument;

[0068] S5, return to the initial position: when it is detected that the doctor has taken away the surgical instruments, return to the storage tray at the initial position and wait.

[0069] Furthermore, in S4, the specific steps of designing and planning the optimal obstacle avoidance motion path from the starting point to the target point are:

[0070] Ⅰ. Perception and positioning: First, the visual recognition module, including the 3D lidar camera installed on the end effector of the robotic arm and the camera fixed in the space outside the robotic arm, is used to accurately identify and locate the operating room environment and the doctor's hand position; second, the doctor's hand image is processed through deep learning technology to identify the pixel coordinates of the key points of the hand skeleton and palm, and estimate the position and posture of the gesture;

[0071] II. Path planning: Based on the perceived position of the doctor's hand and the preset target point (i.e., the expected position of the doctor's hand), the robot system uses heuristic algorithms (such as the grid method obstacle avoidance control method) to search for a safe path in the unit; the grid method uses the basic element as the minimum grid granularity, divides the map into grids, and determines the safe path based on the traffic factor of each grid (the free area is 0, the obstacle area or the area containing obstacles is 1); secondly, when planning the path, the robot will consider possible obstacles in the operating room, such as the operating table, other medical equipment, etc., and ensure the safety and efficiency of the path by avoiding these obstacles;

[0072] III. Real-time force control perception: During the delivery of the instrument, the robot senses the motion state and force of the robotic arm in real time through the torque sensor on the robotic arm. This helps ensure that the robot remains stable during the movement and adjusts the movement trajectory or speed as needed to avoid collisions with doctors or other objects.

[0073] IV. Motion execution: Based on the planned optimal obstacle avoidance motion path, the robot controls the mechanical arm to move according to the preset trajectory and speed to deliver the surgical instrument to the doctor's hand; secondly, during the movement, the robot will continuously monitor environmental changes and the position of the doctor's hand, and adjust the path and motion parameters in real time as needed to ensure that the instrument can be accurately and safely delivered to the doctor's hand;

[0074] V. End of transfer and feedback: When the robot detects that the doctor has caught the instrument (determined by the change in external force sensed by the torque sensor), the robot will immediately stop moving and release the surgical instrument; after completing the instrument transfer, the robot will send a feedback signal to the control system to confirm that the transfer task is completed and prepare to execute the next instruction.

[0075] Furthermore, in the process of perception and positioning, in addition to using 3D lidar cameras and fixed cameras, infrared sensors and depth cameras are also introduced to improve the accuracy and robustness of perception of the environment and the doctor's hand position through the fusion of multimodal data; based on deep learning technology, a more advanced dynamic gesture recognition algorithm is developed to track the movement trajectory and posture changes of the doctor's hand in real time, so as to more accurately judge the doctor's hand intention and target position; the development steps of a more advanced dynamic gesture recognition algorithm are as follows:

[0076] Step 1: Data collection and preprocessing: First, a large amount of doctor's hand motion data needs to be collected, including motion trajectories and posture changes of different gestures, different speeds, and different directions; these data should cover various possible environments and lighting conditions to ensure the robustness of the algorithm; during the data collection process, it is necessary to use 3D lidar cameras, fixed cameras, infrared sensors, and depth cameras simultaneously for multimodal data acquisition; each sensor has its unique advantages: 3D lidar cameras can provide accurate spatial information, fixed cameras can provide rich color and texture information, infrared sensors can provide effective supplements when lighting conditions are poor, and depth cameras can provide depth information on the surface of objects; the data preprocessing stage includes data cleaning (removing noise and invalid data), data annotation (providing labels for gesture categories and trajectories), and data enhancement (increasing data diversity through operations such as rotation, scaling, and translation);

[0077] Step 2: Deep learning model design: Next, a deep learning model needs to be designed to process these multimodal data. The model architecture can be a hybrid model that integrates multiple sub-networks, each of which is responsible for processing data of one modality. For example, a convolutional neural network (CNN) can be designed to process image data provided by fixed cameras and depth cameras to extract color and texture features. At the same time, a recurrent neural network (RNN) or long short-term memory network (LSTM) can be designed to process time series data, such as hand motion trajectories provided by 3D lidar cameras and infrared sensors. The outputs of these sub-networks can be fused in some way (such as feature concatenation, weighted fusion, etc.) and then passed to a classifier or regressor to identify gesture categories or predict hand motion trajectories.

[0078] Step 3: Model training and optimization: In the model training phase, you need to use labeled data sets to train the model, and adjust the model parameters through optimization methods such as back propagation algorithm and gradient descent to minimize the prediction error. In order to improve the performance of the model, you can adopt some optimization strategies, such as using a more complex network structure, adding regularization terms to prevent overfitting, and adopting a learning rate decay strategy. In addition, you can also use transfer learning technology to use pre-trained model parameters as initial values ​​to accelerate the training process and improve model performance.

[0079] Step 4: Real-time tracking and intention judgment: After the model training is completed, it can be deployed to the actual system for real-time gesture recognition. By continuously collecting new multimodal data and inputting it into the trained model, the doctor's hand movement trajectory and posture changes can be tracked in real time. Based on these real-time data, the doctor's hand extension intention and target position can be further judged. This can be achieved by analyzing the speed, direction, acceleration and other characteristics of the hand movement. For example, when the hand moves quickly in a certain direction, it may mean that the doctor wants to grab a certain instrument; and when the hand stays in a certain position for a long time, it may mean that the doctor wants to place the instrument at that position.

[0080] Specific example: Fusion model of convolutional neural network and long short-term memory network:

[0081] Suppose a model that integrates convolutional neural networks (CNNs) and long short-term memory networks (LSTMs) is designed to process multimodal data; CNN is responsible for extracting features from images from fixed cameras and depth cameras, while LSTM is responsible for processing time series data from three-dimensional lidar cameras and infrared sensors; the CNN part can use classic image classification network structures, such as ResNet, VGG, etc., to extract features from images; the LSTM part receives time series data as input and captures temporal dependencies through recurrent units; in the fusion stage, the image features extracted by CNN and the time series features extracted by LSTM can be spliced ​​or weighted fused, and then passed to a fully connected layer for classification or regression; the model can simultaneously use image and time series information for gesture recognition; after training and optimization with a large amount of labeled data, the model can achieve real-time tracking of the doctor's hand movement trajectory and posture changes, and accurately judge the doctor's hand extension intention and target position; it helps the robotic arm to more accurately execute the doctor's instructions during surgery, improving the efficiency and safety of surgery.

[0082] Furthermore, in the process of path planning, time efficiency is considered: on the basis of the grid method obstacle avoidance control method, time is introduced as one of the optimization objectives to find a path that can avoid obstacles and reach the target point as quickly as possible, thereby improving the efficiency of instrument delivery; and learning-based path planning is performed: using machine learning or reinforcement learning technology, the robot optimizes the path planning algorithm through continuous learning and practice, so that it can better adapt to various complex surgical environments and doctors' operating habits; specifically:

[0083] Path planning considering time efficiency:

[0084] Introducing time cost into the grid method: Based on the grid method obstacle avoidance control method, in addition to assigning a pass factor to each grid, a time cost factor can also be introduced; the time cost factor can be set according to the time required for the robot to pass through the grid, for example, considering the robot's speed, the size of the grid, and the impact of possible obstacles; in the path search process, in addition to finding a path with good passability, the time cost must also be considered, so that the final selected path can avoid obstacles and minimize the time to reach the target point;

[0085] Heuristic search optimization: Use heuristic search algorithms such as the A* algorithm to comprehensively consider distance and time costs during the search process; by setting the heuristic function reasonably, the search process can be guided to a path with higher time efficiency;

[0086] Dynamically adjust the grid granularity: dynamically adjust the grid granularity according to the surgical environment and the doctor's operation requirements; use a smaller grid granularity in areas with dense obstacles or where high-precision obstacle avoidance is required; use a larger grid granularity in areas that are relatively open or where accuracy requirements are not high, to improve the efficiency of path planning;

[0087] Learning Path Planning:

[0088] Path optimization based on machine learning: Use machine learning algorithms, such as deep learning and support vector machines, to learn historical path data to extract features and rules that help optimize the path; through training models, the robot can predict the optimal path under different surgical environments and doctor's operating habits;

[0089] Application of reinforcement learning in path planning: Reinforcement learning technology is used to allow the robot to continuously learn and optimize the path planning strategy during the interaction with the environment; by defining appropriate state space, action space and reward function, the robot can gradually improve its path planning performance according to the feedback from the environment to adapt to various complex surgical environments and doctors' operating habits; reinforcement learning algorithms such as Qlearning and SARSA can be used to learn the optimal path planning strategy; by continuously updating the value function or strategy function, the robot can gradually find a path that can avoid obstacles and reach the target point as quickly as possible;

[0090] Online learning and real-time adjustment: When the robot is performing surgical tasks, it collects new environmental data and doctor operation data in real time through online learning technology, and adjusts and optimizes the path planning algorithm in real time based on this data. This can be achieved by setting a certain learning rate and update strategy to ensure that the robot can continuously improve the efficiency and accuracy of path planning in a constantly changing surgical environment.

[0091] Furthermore, in the real-time force control perception process, while installing torque sensors on the robotic arms, a compliant control strategy is introduced, so that the robot can automatically adjust the force and speed when in contact with doctors or other objects, achieving smoother and safer interaction; combined with the doctor's operating habits and historical data, a predictive force control perception algorithm is developed, so that the robot can predict the doctor's possible force changes in advance and actively adjust its own motion state to adapt to the doctor's operating needs; specifically:

[0092] Implementation of compliant control strategy:

[0093] Integration of torque sensors: High-precision torque sensors are installed at key parts of the robot arm to monitor the force and torque changes when the robot arm contacts the outside world in real time. These sensors can convert contact force into electrical signals, which can then be read and processed by the control system.

[0094] Application of compliant control algorithms: Combining the robot's dynamic model and force sensor data, compliant control algorithms such as impedance control and adaptive control can be implemented. Impedance control can change the response characteristics of the robot to external forces by adjusting the impedance parameters of the robot arm to achieve smoother interaction. Adaptive control strategies can dynamically adjust the control parameters of the robot arm based on real-time contact force information to better adapt to the movement of doctors or other objects.

[0095] Establishment of safety mechanism: During the compliant control process, force threshold and speed threshold are set to ensure that the robot arm can automatically slow down or stop when it is subjected to excessive external force or moves too fast, to avoid harm to doctors or patients;

[0096] Development of predictive force control perception algorithms:

[0097] Data collection and processing: By long-term recording of doctors’ operating habits and historical data, including changes in doctors’ force and speed during surgery, as well as the response of the robotic arm, etc., the collected data is cleaned, labeled, and preprocessed to extract features related to force control perception;

[0098] Selection and training of algorithm models: Select appropriate machine learning or deep learning models, such as neural networks, support vector machines, etc., to predict possible changes in doctors’ strength; use historical data to train the model so that the model can learn the relationship between doctors’ operating patterns and changes in strength;

[0099] Prediction and proactive adjustment: In real-time operation, the trained model is used to predict the force according to the current doctor's operation behavior and the state of the robotic arm. Based on the prediction results, the robotic arm can adjust its own motion state in advance, such as adjusting the motion trajectory, speed or strength, to adapt to the doctor's upcoming operation;

[0100] Feedback and optimization: In actual applications, by continuously collecting new operation data and force control perception results, the predictive force control perception algorithm is iterated and optimized to improve prediction accuracy and response speed.

[0101] Furthermore, the robot can adjust the motion trajectory adaptively during the motion execution process, that is, according to the real-time monitored environmental changes and the position changes of the doctor's hands, the robot can adaptively adjust the motion trajectory and speed to ensure that the instrument can be delivered to the doctor more accurately; develop the collaborative operation function between the robot and the doctor, so that the robot can adjust the position and posture of the instrument according to the doctor's instructions or hints, and achieve closer human-machine collaboration; specifically:

[0102] Adaptive motion trajectory adjustment:

[0103] Real-time monitoring of the environment and hand position: Use multimodal sensors such as 3D lidar cameras, fixed cameras, infrared sensors, and depth cameras to obtain real-time environmental information and the position and posture data of the doctor's hands; use deep learning algorithms and computer vision technology to process and analyze these data to extract useful information;

[0104] Trajectory planning and adjustment: Based on the real-time monitored environmental changes and changes in the doctor's hand position, the robot needs to re-plan or adjust its motion trajectory in real time; adopt path planning-based algorithms, such as the A* algorithm, RRT (rapidly expanding random tree) algorithm, etc., combined with real-time monitoring data to generate new motion trajectories; at the same time, consider the robot's dynamic characteristics and kinematic constraints to ensure that the adjusted trajectory meets both the doctor's needs and the robot's motion capabilities.

[0105] Real-time feedback and adjustment: As the robot executes its motion trajectory, it continuously acquires new real-time monitoring data, and evaluates and adjusts the motion trajectory in real time. It sets evaluation indicators (such as path length, arrival time, distance from the doctor's hand, etc.) to judge the quality of the current trajectory and make fine adjustments as needed.

[0106] Collaborative operation functions between robots and doctors:

[0107] Voice and gesture recognition: Through voice recognition technology, the robot can recognize the doctor's voice commands, such as "move forward" and "turn left". At the same time, combined with gesture recognition algorithms, the robot can recognize the doctor's hand movements and postures, thereby understanding the doctor's intentions.

[0108] Intention understanding and response: The robot needs to be able to accurately understand the doctor's intention and adjust the position and posture of the instrument according to the intention; this can be achieved by converting the results of voice and gesture recognition into the robot's control instructions; for example, when the doctor issues a voice command of "grasp" and makes a grasping gesture, the robot can perform the grasping action;

[0109] Feedback and collaboration: The robot can provide feedback to the doctor through voice, light or other means, such as current status, execution results, etc.; at the same time, the robot should be able to make further adjustments and optimizations based on the doctor's real-time feedback or new instructions to achieve closer human-machine collaboration.

[0110] Furthermore, at the end of the transmission and feedback process, an emergency stop button or sensor is designed. When the robot detects an abnormal situation or the doctor issues a stop command, it can stop moving immediately to avoid causing harm to the doctor or patient; at the same time, it has fault self-diagnosis and recovery; specifically: the emergency stop button should be designed as a large red button that is easy to identify and operate, ensuring that the doctor can quickly find and press it in an emergency. At the same time, the button should have waterproof and dustproof properties to meet the usage requirements in different environments; the emergency stop button achieves the purpose of immediately stopping the operation of the equipment by disconnecting the circuit; when the button is pressed, the internal contacts will quickly disconnect the robot's main power supply or control signal, causing the robot to immediately stop all actions; the emergency stop button should be designed to require a certain amount of pressure to trigger, so as to avoid accidental shutdown caused by accidental touch; in addition to the emergency stop button, integrated safety sensors (such as gratings, pressure sensors, etc.) can also be used to detect whether there are abnormalities around the robot situations or specific actions of the doctor; when the sensor detects an abnormal situation or receives a stop command from the doctor, the robot's emergency stop mechanism can be triggered; the robot has a fault self-diagnosis function, and monitors the robot's operating status in real time through sensors and control systems; when a fault occurs, the system can automatically detect and identify the type of fault, such as sensor failure, mechanical structure failure or control system failure; the diagnostic system records the fault information and reports the fault type and location to the operator or maintenance personnel for timely repair; in some cases, the robot may have an automatic recovery function; for example, when a sensor failure is detected, the robot can try to recalibrate or replace a spare sensor; however, for faults involving safety or critical functions, it is usually recommended to shut down and have professionals perform repairs; by combining the use of emergency stop buttons and sensors, as well as fault self-diagnosis and recovery functions, the safety and reliability of the voice-controlled intelligent instrument delivery robot during surgery can be greatly improved.

[0111] Furthermore, it also includes auxiliary recovery of the device, and the specific steps are:

[0112] 1) Robotic arm and gripping device: The robotic arm is equipped with a precise and flexible gripping device; specifically:

[0113] Jaw design: The gripping device can be designed as an adjustable jaw, which can be opened and closed electrically or pneumatically to accommodate instruments of different sizes. The inside of the jaw can be covered with soft rubber or silicone material to increase the stability of the grip and reduce damage to the instrument.

[0114] Sensor integration: Integrate force sensors and position sensors on the robot arm and gripper to monitor the gripping force and position in real time, ensuring stable gripping while avoiding over-gripping;

[0115] Adaptive control: Through advanced control algorithms, the robot arm can adaptively adjust the grasping mode and strength according to the shape and size of the instrument to ensure stable and lossless grasping; the control algorithm includes fuzzy control algorithm and reinforcement learning algorithm; fuzzy control algorithm is a control method based on fuzzy set theory, fuzzy language variables and fuzzy logic reasoning, which realizes effective control of complex systems by simulating human fuzzy reasoning and decision-making process; during the grasping process of the robot arm, the fuzzy control algorithm can output the corresponding grasping mode and strength control signal according to the input fuzzy information such as the shape and size of the instrument through pre-set fuzzy rules and reasoning mechanism; in this way, the robot arm can adaptively adjust its grasping strategy according to the characteristics of the instrument; the fuzzy control algorithm does not need to establish an accurate mathematical model, has strong robustness to changes in system parameters, and is suitable for systems that are difficult to describe with accurate mathematical models; reinforcement learning algorithm is an algorithm that learns by trial and error, which makes intelligent In the process of interaction with the environment, the body learns the optimal behavior strategy by maximizing the cumulative reward; in the adaptive grasping of the robot arm, the reinforcement learning algorithm can be applied to the learning and optimization of the grasping strategy; the robot arm may try different grasping methods and strengths in the initial stage, and gradually learn the optimal grasping strategy for different devices through continuous trial and error and receiving feedback from the environment (such as whether the grasping is successful, whether the device is damaged, etc.); the reinforcement learning algorithm has the ability of self-learning and self-adaptation, and can cope with environmental changes and uncertainties; through a large amount of trial and error and feedback, the robot arm can gradually improve its grasping accuracy and stability; both algorithms can be used in combination with machine vision technology to provide necessary input data for adaptive control by identifying and analyzing the shape, size and other information of the device; in addition, other advanced control algorithms, such as neural network control, genetic algorithm, etc., can be selected or combined according to actual needs and system characteristics to achieve more efficient adaptive grasping;

[0116] Machine vision: Combined with machine vision technology, it can realize automatic recognition and positioning of surgical instruments. Through image processing and recognition algorithms, the robotic arm can accurately identify the position and shape of the instrument, so as to perform precise grasping.

[0117] Example: Take the intelligent robotic arm used in a hospital operating room as an example. The robotic arm is equipped with a high-precision and flexible grasping device that can stably grasp and release various surgical instruments. During the operation, the doctor only needs to place the required instruments in the designated position, and the robotic arm can automatically identify and grasp the instruments and then deliver them to the doctor. At the same time, the robotic arm can also flexibly adjust according to the doctor's instructions or gestures to meet the needs of different surgical scenarios.

[0118] In addition, the gripping device of the robot arm uses specially designed grippers and soft material covering to ensure that the instrument will not be damaged during the gripping process. At the same time, through the integration of force sensors and position sensors, the robot arm can monitor the gripping force and position in real time to ensure stable gripping while avoiding over-gripping.

[0119] 2) Recycling bins and disinfection equipment: A special recycling bin should be set up in the operating room to store used instruments. The recycling bin should be sealed to prevent the spread of contaminants. At the same time, automatic or semi-automatic disinfection equipment can be set up to perform preliminary cleaning and disinfection on the recycled instruments.

[0120] 3) Sensor and recognition system: Use visual recognition, infrared sensors and other technologies to accurately identify and locate the doctor's hand position and the position and status of surgical instruments; help the robot accurately determine when to retrieve and deliver the instruments; specifically:

[0121] The visual recognition system uses a high-resolution camera combined with a deep learning algorithm to capture and analyze real-time images of scenes in the operating room. When the doctor reaches out to use a new instrument, the system can recognize the doctor's hand position and posture using voice control, and detect surgical instruments on the operating table or in the storage tray.

[0122] Example: Multiple surgical instruments, such as forceps and scissors, are laid out flat on a storage tray. Each surgical instrument has a QR code on its handheld part to uniquely identify and track the instrument's information. When a doctor needs an instrument, he or she only needs to extend his or her hand (or use a voice control sensor) to signal. After the camera captures the doctor's movements, the visual recognition system quickly identifies the doctor's target instrument and calculates the best delivery path.

[0123] Infrared sensors are installed on the robot arm and gripping device to accurately identify and locate the tip of the surgical instrument. These sensors can detect and provide feedback on the movement and position of the robot in real time, ensuring that the robot can accurately grasp the tip of the surgical instrument.

[0124] Example: When the doctor selects an instrument, the robotic arm starts to move, and the infrared sensor detects and locates the tip of the instrument on the storage tray in real time. Once the robotic arm moves to the correct position, it gently grasps the tip of the instrument, then turns 90 degrees to face the handheld part of the instrument toward the doctor;

[0125] During the gripping process of the robot, a QR code scanner is also integrated. When the robot grips an instrument, it automatically scans the QR code on the instrument, reads and verifies the instrument information, and ensures that the instrument delivered to the doctor is correct and sterilized.

[0126] Example: When the robot grasps an instrument, the QR code scanner quickly scans the QR code on the instrument, and the system verifies the instrument number, type, disinfection status and other information. If everything is normal, the robot will deliver the instrument to the doctor;

[0127] Through such integration and expansion, the control method of the voice-controlled intelligent instrument delivery robot not only has efficient and accurate delivery capabilities, but also increases the strict management and tracking of surgical instruments, further improving the safety and work efficiency of the operating room;

[0128] 4) Real-time detection and judgment: The robot uses sensors to detect the doctor's hand movements and status in real time. When it detects that the doctor is reaching out and preparing to use a new instrument, the robot will quickly grab a sterilized instrument from the sterilization equipment and deliver it to the doctor along the best obstacle-avoiding path.

[0129] 5) Used instrument recycling: After the doctor has used the instrument, the robot quickly identifies and grabs the used instrument and then puts it into the recycling bin; the recycling bin is designed to close automatically to reduce the risk of contamination;

[0130] 6) Disinfection and reuse: After preliminary cleaning and disinfection, the instruments in the recycling bin can be put back into the disinfection equipment for more thorough disinfection. The disinfected instruments can be used again by the robot for delivery, thus realizing the recycling of instruments.

[0131] It should be noted that the steps for precise rotation angle control of the robot arm during the instrument recovery process are as follows: high-precision motors and encoders are selected to ensure that the robot arm joints can rotate accurately and stably; the sensor and recognition system receives and parses instructions, calculates the required rotation angle according to the target position and posture, monitors the rotation angle of the joint in real time, and adjusts it according to the data fed back by the encoder to ensure that the actual angle is consistent with the target angle. At the same time, the sensor and recognition system is preset with control algorithms, such as adaptive control and fuzzy control, to improve the accuracy and stability of rotation angle control. These algorithms can dynamically adjust the control strategy according to real-time environmental information and the motion state of the robot arm to cope with various complex situations; Example: Taking the intelligent robot arm used in the operating room of a hospital as an example, the robot arm is equipped with high-precision The robot arm is equipped with a motor and encoder, as well as a sensor and recognition system. During the process of recovering the instrument, the doctor issues instructions through the operation interface. After receiving the instructions, the control system of the robotic arm uses the sensor and recognition system to identify the position and posture of the instrument. Then, based on the recognition result and the kinematic model of the robotic arm, the precise rotation angle to be achieved is calculated. Through motor drive and encoder feedback, the joints of the robotic arm accurately rotate to the specified angle, thereby achieving precise grasping and recovery of the instrument. In addition, the robotic arm also adopts an adaptive control algorithm, which can dynamically adjust the control strategy according to real-time environmental information and motion status to adapt to different surgical instruments and grasping requirements. At the same time, the robotic arm is also equipped with safety mechanisms such as collision detection and emergency stop to ensure that no harm is caused to the doctor or the surrounding environment during the recovery process.

Claims

1. A voice-controlled intelligent instrument delivery robot system for surgical assistance, characterized in that: include: A robotic arm with multiple joints and torque sensors on each joint, a controller that controls the force of the robotic arm at a frequency of 1000 Hz per second, a visual recognition module, a voice recognition module that recognizes the doctor's command and obtains the command for the requested surgical instrument, and a storage tray for sterile items and surgical instruments; Visual recognition module: One camera is installed on the end effector of the robotic arm, and uses a three-dimensional laser radar to identify the QR code information on the surgical instrument and the three-dimensional spatial position of the surgical instrument; the other camera is fixed in the space outside the robotic arm body, and the position of the camera and the world coordinate system remains unchanged, which is used to detect and identify the doctor's gesture posture.

2. A control method for a voice-controlled intelligent instrument delivery robot for surgical assistance, characterized in that: During laparoscopic or open surgery, the system completes the tasks of identifying, picking up, and delivering surgical instruments according to the doctor's voice commands. The real-time force control technology ensures high reliability and safety in the human-computer interaction process. The specific process includes the following steps: S1. Sense the instrument request signal: The voice control sensor on the robotic arm is used to sense the instrument request signal, detect the doctor's voice command through the voice control interaction function, and obtain the surgical instrument information required by the doctor in a contactless human-computer interaction manner; S2. Identify instruments: Each surgery has a set of exclusive surgical instruments. Generally, there are 20 to 30 small surgical instruments, and large and medium-sized surgeries require hundreds of instruments. In addition, the same type of surgical instruments may have different specifications or similar appearances. The surgical instruments are distinguished by the exclusive QR codes on the surgical instruments, which are applicable to all surgeries, and a database of surgical instrument-related information is established. After the robotic arm obtains the doctor's voice command, it scans and identifies the instrument QR code through the visual recognition module it carries, and identifies the surgical instrument that matches the command based on the instrument QR code database information. S3. Picking up instruments: Use a 3D laser radar camera to collect 3D point cloud data of the required surgical instruments, and estimate the position and posture of the surgical instruments in 3D space; there are certain risks in the process of transferring surgical instruments. During the transfer process, the safe operating part needs to be facing the doctor, and the required surgical instruments need to be safely transferred to the doctor's hands. In order to implement the transfer process in accordance with the specification requirements, the 3D model of the surgical instruments, the grasping position and grasping posture data are recorded in the instrument QR code database. During the picking process, the target surgical instruments are picked up by clamping according to the set grasping position and grasping posture; S4, transfer equipment: real-time force control sensing technology is used in the transfer process to enable the system to have good human-machine interaction capabilities, prevent misoperation and avoid collisions; the steps are: ①. Surgical identification and positioning: A. Key point detection: detect the doctor's hand and obtain an image of the hand area; identify the pixel coordinates of the key points of the hand skeleton and palm in the image; B. Posture estimation: Based on deep learning, the position and posture of the doctor's gesture are estimated, and the state of the doctor's palm is determined, including the "extended hand" state and the "clenched fist" state; ②Delivering instruments: When the doctor is detected to be in the "reaching out" state, the optimal obstacle-avoiding motion path from the starting point to the target point is designed and planned to deliver the instrument to the doctor's hand; ③. Detect "clenched fist" gesture: When the doctor's gesture is detected to be in the "clenched fist" state, it means that the doctor is ready to pick up the instrument; ④. Sensing force changes: The force data of the end fixture is directly obtained through the torque sensor mounted on the robotic arm, and the pressure of the contact between the surgical instrument and the doctor's hand is used to determine whether the external force has changed; ⑤. End of transfer: When a change in external force is detected, it means that the doctor has caught the instrument, and the robotic arm releases the surgical instrument, completing the transfer of the instrument; S5, return to the initial position: when it is detected that the doctor has taken away the surgical instruments, return to the storage tray at the initial position and wait.

3. A method for controlling a voice-controlled intelligent instrument delivery robot for surgical assistance as claimed in claim 2, characterized in that: In S4, the specific steps of designing and planning the optimal obstacle avoidance motion path from the starting point to the target point are: Ⅰ. Perception and positioning: First, the visual recognition module, including the 3D lidar camera installed on the end effector of the robotic arm and the camera fixed in the space outside the robotic arm, is used to accurately identify and locate the operating room environment and the doctor's hand position; second, the doctor's hand image is processed through deep learning technology to identify the pixel coordinates of the key points of the hand skeleton and palm, and estimate the position and posture of the gesture; II. Path planning: Based on the perceived doctor's hand position and the preset target point, the robot system uses a heuristic algorithm to search for a safe path in the unit; the grid method uses the basic element as the minimum grid granularity, divides the map into grids, and determines the safe path based on the traffic factor of each grid; III. Real-time force control perception: During the delivery of the device, the robot uses the torque sensor on the robotic arm to sense the motion state and force of the robotic arm in real time; IV. Motion execution: Based on the planned optimal obstacle avoidance motion path, the robot controls the mechanical arm to move according to the preset trajectory and speed to deliver the surgical instrument to the doctor; V. End of transfer and feedback: When the robot detects that the doctor has caught the instrument, it will immediately stop moving and release the surgical instrument. After completing the instrument transfer, the robot will send a feedback signal to the control system to confirm that the transfer task is completed and prepare to execute the next instruction.

4. A method for controlling a voice-controlled intelligent instrument delivery robot for surgical assistance as claimed in claim 3, characterized in that: In the process of perception and positioning, in addition to using 3D lidar cameras and fixed cameras, infrared sensors and depth cameras are also introduced to improve the accuracy and robustness of perception of the environment and the doctor's hand position through the fusion of multimodal data; based on deep learning technology, a more advanced dynamic gesture recognition algorithm is developed to track the movement trajectory and posture changes of the doctor's hand in real time; the development steps of a more advanced dynamic gesture recognition algorithm are as follows: The first step is data collection and preprocessing; Step 2: Deep learning model design; Step 3: Model training and optimization; Step 4: Real-time tracking and intention judgment.

5. The control method of a voice-controlled intelligent instrument delivery robot for surgical assistance according to claim 3, characterized in that: In the process of path planning, consider time efficiency: Based on the grid obstacle avoidance control method, introduce time as one of the optimization goals to find a path that can avoid obstacles and reach the target point as quickly as possible, thereby improving the efficiency of device delivery; And perform learning-based path planning: Using machine learning or reinforcement learning technology, the robot can optimize the path planning algorithm through continuous learning and practice, so that it can better adapt to various complex surgical environments and doctors' operating habits.

6. A method for controlling a voice-controlled intelligent instrument delivery robot for surgical assistance as claimed in claim 3, characterized in that: In the real-time force control perception process, torque sensors are installed on the robotic arm and a compliant control strategy is introduced, so that the robot can automatically adjust the force and speed when in contact with doctors or other objects, achieving smoother and safer interaction; By combining doctors' operating habits and historical data, a predictive force control perception algorithm is developed, which enables the robot to predict possible changes in doctors' strength in advance and actively adjust its own movement state.

7. A method for controlling a voice-controlled intelligent instrument delivery robot for surgical assistance as claimed in claim 3, characterized in that: Adaptive motion trajectory adjustment during the execution of movement, that is, during the robot's movement, the motion trajectory and speed are adaptively adjusted according to the real-time monitored environmental changes and the position changes of the doctor's hands, ensuring that the instrument can be delivered to the doctor's hands more accurately; develop collaborative operation functions between the robot and the doctor, so that the robot can adjust the position and posture of the instrument according to the doctor's instructions or hints, and achieve closer human-machine collaboration.

8. The control method of a voice-controlled intelligent instrument delivery robot for surgical assistance as claimed in claim 3, characterized in that: At the end of the transmission and feedback process, an emergency stop button or sensor is designed. When the robot detects an abnormal situation or the doctor issues a stop command, it can stop moving immediately to avoid harm to the doctor or patient; it also has fault self-diagnosis and recovery.

9. A method for controlling a voice-controlled intelligent instrument delivery robot for surgical assistance as claimed in claim 2, characterized in that: It also includes auxiliary recovery of the device, the specific steps are: 1) Robotic arm and grasping device: The robotic arm is equipped with a precise and flexible grasping device (manipulator); 2) Recycling bins and disinfection equipment: A special recycling bin is set up in the operating room to store used instruments. At the same time, automatic or semi-automatic disinfection equipment can be set up to perform preliminary cleaning and disinfection on the recycled instruments; 3) Sensor and recognition system: Use visual recognition, infrared sensor and other technologies to accurately identify and locate the doctor's hand position and the position and status of surgical instruments; 4) Real-time detection and judgment: The robot uses sensors to detect the doctor's hand movements and status in real time. When it detects that the doctor is reaching out and preparing to use a new instrument, the robot will quickly grab a sterilized instrument from the sterilization equipment and deliver it to the doctor along the best obstacle-avoiding path. 5) Used instrument recycling: After the doctor uses the instrument, the robot quickly identifies and grabs the used instrument and puts it into the recycling bin; 6) Disinfection and reuse: After preliminary cleaning and disinfection, the instruments in the recycling bin are put back into the disinfection equipment for more thorough disinfection; The sterilized instruments are then delivered again by the robot and recycled.

Citation Information

Cited By

  • Intelligent auxiliary system for microsurgery

    CN119523654A

  • Mechanical arm remote control method and system based on artificial intelligence and robot technology

    CN119610143A

  • Method and system for remote control of robotic arms based on artificial intelligence and robotics technology

    CN119610143B

  • Surgical robot mechanical arm control method based on binocular structured light camera and large model

    CN119858160A

  • Method, device and equipment for calculating posture of oral cavity operation auxiliary transmitter and medium

    CN120114216A