A surgical robot vision obstacle avoidance control method and device for surgical robot body intelligence

Abstract

This application provides a visual obstacle avoidance control method and apparatus for surgical robots with embodied intelligence. The method includes: acquiring a real surgical scene video stream and preoperative planning information; parsing the real surgical scene video stream to obtain obstacle information; constructing a virtual operating environment based on the obstacle information, preoperative planning information, and the surgical scene video stream; predicting a collision-free robotic arm trajectory based on the virtual operating environment; and controlling the robotic arm to execute the robotic arm trajectory based on the real surgical scene video stream, and displaying the result in real time in the virtual operating environment. In this application, the virtual operating environment fully simulates the real surgical scene, thereby predicting a collision-free robotic arm trajectory and avoiding collision risks during actual operation, thus greatly simplifying the obstacle avoidance scheme.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method and apparatus for visual obstacle avoidance control of surgical robots for embodied intelligence. Background Technology

[0002] Embodied intelligence refers to intelligent agents with bodies that can interact with the physical world, such as robots and autonomous vehicles. By processing various sensor data inputs, the control center, such as a large model, generates motion commands to drive the intelligent agent, replacing the traditional motion-driven method based on rules or mathematical formulas, and achieving a deep integration of virtual and reality.

[0003] Surgical robots can serve as a perfect carrier of embodied intelligence, enabling the realization of surgical robot functions, such as obstacle avoidance.

[0004] However, in the past, surgical robots used a method of first planning the path of the robotic arm when using it for obstacle avoidance. Then, when the robotic arm was actually moving along the planned path, it would avoid collisions in real time if a collision risk was detected. This obstacle avoidance method is too cumbersome. Summary of the Invention

[0005] The problem this application addresses is that current obstacle avoidance methods are too cumbersome.

[0006] To address the aforementioned problems, the first aspect of this application provides a visual obstacle avoidance control method for surgical robots with embodied intelligence, comprising:

[0007] Obtain real surgical scene video streams and preoperative planning information;

[0008] Analyze real surgical scene video streams to obtain obstacle information;

[0009] A virtual operating environment is constructed based on the obstacle information, preoperative planning information, and the surgical scene video stream;

[0010] Based on the virtual operating environment, predict the collision-free movement trajectory of the robotic arm;

[0011] Based on the video stream of a real surgical scene, the robotic arm is controlled to execute the robotic arm movement trajectory and displayed in real time in the virtual operating environment.

[0012] A second aspect of this application provides a visual obstacle avoidance control device for surgical robots with embodied intelligence, comprising:

[0013] The information acquisition module is used to acquire real surgical scene video streams and preoperative planning information;

[0014] The video parsing module is used to parse the video stream of a real surgical scene and obtain obstacle information;

[0015] A virtual construction module is used to construct a virtual operating environment based on the obstacle information, preoperative planning information, and the surgical scene video stream;

[0016] The trajectory prediction module is used to predict the collision-free movement trajectory of the robotic arm based on the virtual operating environment.

[0017] The robotic arm control module is used to control the robotic arm to execute the robotic arm movement trajectory based on the real surgical scene video stream, and display it in real time in the virtual operating environment.

[0018] A third aspect of this application provides an electronic device comprising: a memory and a processor;

[0019] The memory is used to store programs;

[0020] The processor, coupled to the memory, is used to execute the program for:

[0021] Obtain real surgical scene video streams and preoperative planning information;

[0022] Analyze real surgical scene video streams to obtain obstacle information;

[0023] A virtual operating environment is constructed based on the obstacle information, preoperative planning information, and the surgical scene video stream;

[0024] Based on the virtual operating environment, predict the collision-free movement trajectory of the robotic arm;

[0025] Based on the video stream of a real surgical scene, the robotic arm is controlled to execute the robotic arm movement trajectory and displayed in real time in the virtual operating environment.

[0026] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the above-described surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots.

[0027] In this application, a virtual operating environment is used to fully simulate a real surgical scenario, thereby predicting the collision-free movement trajectory of the robotic arm and avoiding the risk of collision during actual operation, thus greatly simplifying the obstacle avoidance scheme. Attached Figure Description

[0028] Figure 1 This is a flowchart of a visual obstacle avoidance control method for surgical robot embodied intelligence according to an embodiment of this application;

[0029] Figure 2 This is a structural block diagram of a surgical robot visual obstacle avoidance control device for embodied intelligence of a surgical robot according to an embodiment of this application;

[0030] Figure 3 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0031] To make the above-mentioned objects, features, and advantages of this application more apparent and understandable, specific embodiments of this application will be described in detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0032] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0033] Surgical robots can serve as a perfect embodiment of embodied intelligence, enabling the realization of their functions. In practice, past operational experience with surgical robots can be summarized first, and then a new control center can be generated based on this experience.

[0034] However, for surgical robots, the past experience they learn is the experience of human control of the surgical robot, and the application scenario based on this experience is the scenario of automatic control of the surgical robot. There are differences in the usage environment between their past experience and the application scenario. How to reduce the impact of this difference in the process of generating a new control center is a problem that is currently difficult to solve.

[0035] To address the aforementioned issues, this application provides a novel skeleton registration scheme that uses digital twins for coarse registration, thereby resolving the problem of low accuracy in current feature point registration.

[0036] This application provides a visual obstacle avoidance control method for surgical robots with embodied intelligence. The specific solution of this method is as follows: Figures 1-2 As shown, this method can be executed by a surgical robot visual obstacle avoidance control device for embodied intelligence of surgical robots, which can be integrated into electronic devices such as computers, servers, computer clusters, and data centers. Combined with... Figure 1The diagram shows a flowchart of a surgical robot visual obstacle avoidance control method for embodied intelligence of a surgical robot according to an embodiment of this application; wherein, the surgical robot visual obstacle avoidance control method for embodied intelligence of a surgical robot includes:

[0037] S101, acquire real surgical scene video stream and preoperative planning information;

[0038] In this application, the actual surgical scene video stream is an intraoperative video acquired by a binocular camera installed on the operating table. This video is acquired in real time so that the robotic arm can be controlled in real time based on the surgical scene video stream.

[0039] In this application, the preoperative planning information is planning information determined based on the patient's medical images taken before surgery. In osteotomy and joint replacement surgery, the preoperative planning information generally includes the osteotomy location, osteotomy section, and the robotic arm preparation position for osteotomy.

[0040] Furthermore, since determining the preoperative planning information requires first segmenting the patient's medical images and creating a three-dimensional skeletal model, and then generating the osteotomy location, osteotomy section, and robotic arm preparation position based on the skeletal model, the preoperative planning information in this application also includes the corresponding skeletal three-dimensional model.

[0041] S102, analyze the real surgical scene video stream to obtain obstacle information;

[0042] In a real-world surgical scenario, not only are the critical surgical sites included, but also other non-surgical components such as instruments, equipment, doctors, and nurses. These can become obstacles hindering the movement of the robotic arm. By analyzing the video and obtaining information about these obstacles, subsequent path planning for the robotic arm can be performed.

[0043] S103, construct a virtual operating environment based on the obstacle information, preoperative planning information, and surgical scene video stream;

[0044] S104, based on the virtual operating environment, predicts the collision-free movement trajectory of the robotic arm;

[0045] S105, based on the real surgical scene video stream, control the robotic arm to execute the robotic arm movement trajectory and display it in real time in the virtual operating environment.

[0046] In this application, a virtual operating environment is used to fully simulate a real surgical scenario, thereby predicting the collision-free movement trajectory of the robotic arm and avoiding the risk of collision during actual operation, thus greatly simplifying the obstacle avoidance scheme.

[0047] In this application, collision detection of the robotic arm's motion trajectory is performed through a virtual operating environment, thereby detecting motion trajectories with collision risks and retaining collision-free robotic arm motion trajectories.

[0048] In one implementation, the step of parsing the real surgical scene video stream to obtain obstacle information includes:

[0049] Acquire a real surgical scene video stream and segment it into continuously set depth image frames;

[0050] A depth image frame is input into an obstacle recognition model to obtain obstacle recognition results; the obstacle recognition results include static obstacles and dynamic obstacles.

[0051] By inputting continuous depth image frames with labeled obstacle identification results into the spatiotemporal prediction model, the spatial motion trajectory of dynamic obstacles can be obtained.

[0052] In one implementation, the obstacle information includes static information and dynamic information; the dynamic information includes predicted motion information of the obstacle over future time intervals.

[0053] The step of inputting consecutive depth image frames labeled with obstacle recognition results into the spatiotemporal prediction model to obtain obstacle prediction results includes:

[0054] Based on the obstacle recognition results, the location key points of dynamic obstacles are marked in the depth image frame;

[0055] A continuous feature map is generated based on the location key points of consecutive depth image frames;

[0056] The continuous feature map is input into the spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.

[0057] In this application, obstacle detection results from depth images are used to locate key points (such as center points, edge points, or feature corner points) of dynamic obstacles in each frame. Obstacles can be accurately labeled using object detection models (such as YOLO or SSD) combined with depth information.

[0058] In this application, obstacle contour information is extracted from depth images, and key points are marked at the edges or center of the obstacles. These key points reflect the location and shape characteristics of the obstacles.

[0059] In this application, key points are ensured to have depth information in order to accurately locate the position information of dynamic obstacles in three-dimensional space.

[0060] In this application, a continuous feature map is generated by combining key point information from multiple frames, enabling it to reflect the motion changes of obstacles in a time series.

[0061] In this application, the model outputs the spatial trajectory of the obstacle, including the obstacle's position and direction of movement at future moments. The trajectory can be represented as a series of coordinate points or curves, intuitively reflecting the obstacle's future position.

[0062] In one implementation, the spatiotemporal prediction model is a gated dilated causal convolutional neural network.

[0063] In one implementation, generating a continuous feature map based on the location key points of consecutive depth image frames includes:

[0064] Based on the aforementioned key locations, construct a domain space graph structure;

[0065] Based on the correlation between the location key points of the same dynamic obstacle, a weighted adjacency matrix of the domain space graph structure is constructed;

[0066] Based on the domain spatial graph structure of consecutive depth image frames and the corresponding weighted adjacency matrix, the feature representation of the domain spatial graph structure at different times is generated, which is the continuous feature map.

[0067] In this application, the key points in each depth image frame are regarded as nodes in a graph structure. Each node contains the key point's position coordinates, depth value, and other possible features (such as velocity, direction, etc.).

[0068] In this application, key points in a depth image frame are combined into a domain spatial graph structure, which reflects the spatial layout of obstacles and the geometric relationships between key points.

[0069] In this application, an undirected graph can be used to represent the domain space graph structure of obstacles, where each node represents a key point and the edges between nodes represent the adjacency relationship between key points.

[0070] In this application, the adjacency relationship of each key point is defined based on the association relationship between key points of the same dynamic obstacle. The association relationship can be spatial distance, motion correlation between key points, etc.

[0071] In this application, weights are assigned to the edges of the adjacency matrix based on the association relationships. The weights can be defined in the following ways:

[0072] Spatial distance weighting: The closer the distance between two key points, the higher the weight.

[0073] Direction weighting: Assign corresponding directional weights to adjacency relationships based on the movement direction of key points.

[0074] Velocity weighting: The higher the velocity similarity between adjacent key points, the higher the weight.

[0075] In this application, an adjacency matrix containing weight information is generated, where each element in the matrix represents the weighted relationship between two key points in the graph structure.

[0076] In this application, the domain spatial graph structure of continuous depth image frames and the corresponding weighted adjacency matrix are combined to form a time-series graph structure. The graph structure in each time step represents the key point relationships of the same obstacle.

[0077] In this application, in the graph structure of each frame, feature representations of each node are generated based on the weights of the weighted adjacency matrix. The feature representations can be a combination of various information such as the location, depth, and velocity of keypoints.

[0078] In this application, by varying the domain space graph structure and adjacency matrix over multiple consecutive time steps, feature representations of dynamic obstacles at different times are generated, forming continuous graph features. This feature sequence is the continuous feature graph, reflecting the motion trajectory of the dynamic obstacles.

[0079] In one implementation, constructing a virtual operating environment based on the obstacle information, preoperative planning information, and the surgical scene video stream includes:

[0080] Obtain the parameters of the robotic arm and construct a model of the robotic arm;

[0081] Based on the hand-eye calibration method, the mapping relationship between the binocular camera coordinate system and the robotic arm coordinate system is determined;

[0082] Acquire outside-field-of-view measurement information from the binocular camera, including obstacle information;

[0083] Based on the mapping relationship, the out-of-field measurement information, the obstacle information, the robotic arm model, the preoperative planning information, and the surgical scene video stream are unified into the same coordinate system;

[0084] A virtual operating environment is constructed based on the field-of-view measurement information, obstacle information, robotic arm model, preoperative planning information, and surgical scene video stream in the same coordinate system.

[0085] In this application, the structural parameters of the robotic arm include the number of joints, the range of motion of each joint, the link length, and inertial parameters.

[0086] In this application, the Denavit-Hartenberg parametric method or other robotic arm modeling methods can be used to construct a robotic arm model to accurately simulate the movement of the robotic arm.

[0087] In this application, hand-eye calibration is performed by obtaining the reference point of the robotic arm from the perspective of the binocular camera and the corresponding point in the robotic arm coordinate system, and then calculating the transformation matrix through least squares fitting or optimization algorithms. The specific method for obtaining the matrix is ​​not described in detail in this application.

[0088] In this application, the binocular camera can be rotated to obtain the outside-field measurement information of the binocular camera using the aforementioned obstacle information acquisition method; the outside-field measurement information and the inside-field measurement information can be stitched together by key point recognition to form a complete intraoperative environment.

[0089] Preferably, the binocular camera needs to periodically update the measurement information outside the field of view to ensure that it can continuously capture changes in the position of obstacles during the operation, providing support for real-time updates of the virtual environment.

[0090] In this application, out-of-field measurement information, obstacle information, robotic arm model, preoperative planning information, and surgical scene video stream are integrated into a unified coordinate system to ensure that all data have a consistent spatial reference in the virtual environment.

[0091] In this application, a virtual surgical scene is constructed in the same coordinate system, including a robotic arm model, preoperative planning information (such as important anatomical structures), obstacles, and dynamic objects in a real-time video stream.

[0092] In this application, the measurement information outside the field of view of the binocular camera and the video stream of the surgical scene are collected in real time and updated to the virtual operating environment, so that the environment can dynamically reflect the changes of the actual surgical scene.

[0093] In this application, the real-time position of the robotic arm, obstacles, and target area are displayed intuitively in a virtual operating environment. Users or control systems can monitor the robotic arm's movement trajectory and potential collision risks within this environment.

[0094] In one implementation, the preoperative planning information includes osteotomy surface information, osteotomy path information, and osteotomy preparation position information.

[0095] In one implementation, the construction of a virtual operating environment based on out-of-field measurement information in the same coordinate system, obstacle information, robotic arm model, preoperative planning information, and surgical scene video stream includes:

[0096] Based on the out-of-field measurement information and the obstacle information, an obstacle model and corresponding coordinates are generated;

[0097] Based on the preoperative planning information, generate the osteotomy surface model, osteotomy path point coordinates, and osteotomy preparation position model;

[0098] Construct a virtual operating environment and project the robotic arm model, obstacle model, osteotomy surface model, osteotomy path point coordinates, and osteotomy preparation position into the virtual operating environment;

[0099] The virtual operating environment is generated from the perspective of a binocular camera and projected onto the surgical scene video stream.

[0100] In this application, a robotic arm model is projected onto a virtual environment to display the current position and motion state of the robotic arm.

[0101] In this application, obstacle models are projected into a virtual environment to ensure that the spatial position of the obstacles is consistent with the actual scene.

[0102] In this application, the projected osteotomy surface model and path point coordinates are used to display the preoperatively planned osteotomy location and path.

[0103] This application shows an osteotomy preparation position model to assist the robotic arm in correct positioning before surgery.

[0104] In this application, a virtual viewpoint of a binocular camera is set in a virtual environment to capture images of the virtual scene at the same angle and position as the actual binocular camera.

[0105] In this application, the images captured by the virtual camera are used to generate a real-time virtual operating environment display, which includes information such as robotic arm, obstacles, osteotomy surfaces, path points, and preparation positions.

[0106] In this application, the generated virtual display is overlaid on the surgical scene video stream, allowing the surgical team to see the overlay effect of virtual information in real-time video. Augmented reality (AR) technology can be used to accurately project virtual information onto the actual scene, helping surgeons to accurately complete surgical procedures.

[0107] In one implementation, predicting the collision-free robotic arm trajectory based on the virtual operating environment includes:

[0108] Obtain the initial position information and osteotomy preparation position information of the robotic arm;

[0109] Based on the initial position information and osteotomy preparation position information, multiple motion paths of the robotic arm within future time intervals are generated using a random tree algorithm;

[0110] Multiple motion paths are projected into the virtual operating environment for collision testing at future time intervals;

[0111] One of the multiple motion paths that passed the collision test is selected as the motion trajectory of the robotic arm.

[0112] In this application, the osteotomy preparation position information is the end point position of the robotic arm; the motion path of the initial position information and the osteotomy preparation position information is generated by a random tree algorithm.

[0113] In this application, the future time interval is a period of time in the future.

[0114] It should be noted that if the initial position information and osteotomy preparation position information of the robotic arm are too far apart to be generated at once, the specific motion path of the robotic arm to reach a certain intermediate position in the future can also be generated directly.

[0115] In this application, multiple generated motion paths are projected into a virtual operating environment, and collision detection is performed on each path. This ensures that there are no obstacles interfering with any point along the path in space, thus preventing the robotic arm from colliding with other objects.

[0116] In this application, the optimal path is selected from multiple paths that pass the collision test as the final motion trajectory of the robotic arm. The selection criteria can be based on factors such as path smoothness, motion time, and energy consumption.

[0117] In this application, the specific process of the random tree algorithm is as follows: initialize the random tree, generate the source point, the target point, and the radius of the target point, and generate an obstacle matrix; generate random points; find the point on the tree closest to the random point; extend the nearest point towards the random point using the distance from the nearest point to the random point to obtain a new point; if the straight line segment from the nearest point to the new point does not pass through an obstacle, the new point is not empty; otherwise, the new point is empty; if the new point is empty, return to the step of generating random points; add the new point to the tree as a node, and use the nearest point as the parent node of the new point, and record it; if the distance from the new point to the target point is less than a preset distance, the algorithm ends; otherwise, return to the step of generating random points.

[0118] In this application, collision detection is performed by setting up a virtual environment, which greatly reduces the limitations and difficulty of generating motion paths using the random tree algorithm, while ensuring accuracy.

[0119] This application provides a surgical robot visual obstacle avoidance control device for embodied intelligence of surgical robots, used to execute the surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots described above. The following is a detailed description of the surgical robot visual obstacle avoidance control device for embodied intelligence of surgical robots.

[0120] like Figure 2 As shown, the surgical robot visual obstacle avoidance control device for embodied intelligence of surgical robots includes:

[0121] Information acquisition module 101 is used to acquire real surgical scene video streams and preoperative planning information;

[0122] The video parsing module 102 is used to parse the real surgical scene video stream and obtain obstacle information;

[0123] The virtual construction module 103 is used to construct a virtual operating environment based on the obstacle information, preoperative planning information, and the surgical scene video stream;

[0124] The trajectory prediction module 104 is used to predict the collision-free movement trajectory of the robotic arm based on the virtual operating environment.

[0125] The robotic arm control module 105 is used to control the robotic arm to execute the robotic arm movement trajectory based on the real surgical scene video stream, and display it in real time in the virtual operating environment.

[0126] In one implementation, the video parsing module 102 is further configured to:

[0127] A real surgical scene video stream is acquired and segmented into continuously set depth image frames; the depth image frames are input into an obstacle recognition model to obtain obstacle recognition results; the obstacle recognition results include static obstacles and dynamic obstacles; the continuous depth image frames labeled with obstacle recognition results are input into a spatiotemporal prediction model to obtain the spatial motion trajectory of dynamic obstacles.

[0128] In one implementation, the obstacle information includes static information and dynamic information; the dynamic information includes predicted motion information of the obstacle over future time intervals.

[0129] In one implementation, the virtual building module 103 is further configured to:

[0130] Obtain robotic arm parameters and construct a robotic arm model; determine the mapping relationship between the binocular camera coordinate system and the robotic arm coordinate system according to the hand-eye calibration method; obtain the field-of-view measurement information of the binocular camera, which includes obstacle information; unify the field-of-view measurement information, obstacle information, robotic arm model, preoperative planning information, and surgical scene video stream into the same coordinate system according to the mapping relationship; construct a virtual operating environment based on the field-of-view measurement information, obstacle information, robotic arm model, preoperative planning information, and surgical scene video stream under the same coordinate system.

[0131] In one implementation, the preoperative planning information includes osteotomy surface information, osteotomy path information, and osteotomy preparation position information.

[0132] In one implementation, the virtual building module 103 is further configured to:

[0133] Based on the out-of-field measurement information and the obstacle information, an obstacle model and corresponding coordinates are generated; based on the preoperative planning information, an osteotomy surface model, osteotomy path point coordinates, and osteotomy preparation position model are generated; a virtual operating environment is constructed, and the robotic arm model, obstacle model, osteotomy surface model, osteotomy path point coordinates, and osteotomy preparation position are projected into the virtual operating environment; a display screen of the virtual operating environment is generated from the perspective of a binocular camera and projected into the surgical scene video stream.

[0134] In one implementation, the trajectory prediction module 104 is further configured to:

[0135] The initial position information and osteotomy preparation position information of the robotic arm are obtained; based on the initial position information and osteotomy preparation position information, multiple motion paths of the robotic arm within future time intervals are generated using a random tree algorithm; the multiple motion paths are projected onto the virtual running environment to conduct collision tests within the future time intervals; one of the multiple motion paths that pass the collision test is selected as the motion trajectory of the robotic arm.

[0136] The surgical robot visual obstacle avoidance control device for embodied intelligence of surgical robots provided in the above embodiments of this application corresponds to the surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots provided in the embodiments of this application. Therefore, the specific contents of the device correspond to the surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots. The specific contents can be referred to the records in the surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots, which will not be repeated in this application.

[0137] The surgical robot visual obstacle avoidance control device for embodied intelligence of surgical robots provided in the above embodiments of this application and the surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by their stored applications.

[0138] The above describes the internal functions and structure of a visual obstacle avoidance control device for embodied intelligence in surgical robots, such as... Figure 3 As shown, in practice, the surgical robot visual obstacle avoidance control device for the embodied intelligence of the surgical robot can be implemented as an electronic device, including: a memory 301 and a processor 303.

[0139] Memory 301 can be configured to store a program.

[0140] Additionally, memory 301 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phonebook data, messages, pictures, videos, etc.

[0141] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0142] Processor 303, coupled to memory 301, is used to execute programs in memory 301 for:

[0143] Obtain real surgical scene video streams and preoperative planning information;

[0144] Analyze real surgical scene video streams to obtain obstacle information;

[0145] A virtual operating environment is constructed based on the obstacle information, preoperative planning information, and the surgical scene video stream;

[0146] Based on the virtual operating environment, predict the collision-free movement trajectory of the robotic arm;

[0147] Based on the video stream of a real surgical scene, the robotic arm is controlled to execute the robotic arm movement trajectory and displayed in real time in the virtual operating environment.

[0148] In one implementation, the processor 303 is further configured to:

[0149] A real surgical scene video stream is acquired and segmented into continuously set depth image frames; the depth image frames are input into an obstacle recognition model to obtain obstacle recognition results; the obstacle recognition results include static obstacles and dynamic obstacles; the continuous depth image frames labeled with obstacle recognition results are input into a spatiotemporal prediction model to obtain the spatial motion trajectory of dynamic obstacles.

[0150] In one implementation, the obstacle information includes static information and dynamic information; the dynamic information includes predicted motion information of the obstacle over future time intervals.

[0151] In one implementation, the processor 303 is further configured to:

[0152] Obtain robotic arm parameters and construct a robotic arm model; determine the mapping relationship between the binocular camera coordinate system and the robotic arm coordinate system according to the hand-eye calibration method; obtain the field-of-view measurement information of the binocular camera, which includes obstacle information; unify the field-of-view measurement information, obstacle information, robotic arm model, preoperative planning information, and surgical scene video stream into the same coordinate system according to the mapping relationship; construct a virtual operating environment based on the field-of-view measurement information, obstacle information, robotic arm model, preoperative planning information, and surgical scene video stream under the same coordinate system.

[0153] In one implementation, the preoperative planning information includes osteotomy surface information, osteotomy path information, and osteotomy preparation position information.

[0154] In one implementation, the processor 303 is further configured to:

[0155] Based on the out-of-field measurement information and the obstacle information, an obstacle model and corresponding coordinates are generated; based on the preoperative planning information, an osteotomy surface model, osteotomy path point coordinates, and osteotomy preparation position model are generated; a virtual operating environment is constructed, and the robotic arm model, obstacle model, osteotomy surface model, osteotomy path point coordinates, and osteotomy preparation position are projected into the virtual operating environment; a display screen of the virtual operating environment is generated from the perspective of a binocular camera and projected into the surgical scene video stream.

[0156] In one implementation, the processor 303 is further configured to:

[0157] The initial position information and osteotomy preparation position information of the robotic arm are obtained; based on the initial position information and osteotomy preparation position information, multiple motion paths of the robotic arm within future time intervals are generated using a random tree algorithm; the multiple motion paths are projected onto the virtual running environment to conduct collision tests within the future time intervals; one of the multiple motion paths that pass the collision test is selected as the motion trajectory of the robotic arm.

[0158] In this application, the processor is also specifically used to execute all the processes and steps of the above-mentioned surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots. For details, please refer to the records in the surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots, which will not be repeated in this application.

[0159] In this application, Figure 3 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 3 The components shown.

[0160] The electronic device provided in this embodiment is based on the same inventive concept as the surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots provided in the embodiments of this application, and has the same beneficial effects as the methods adopted, run or implemented by the application stored therein.

[0161] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0166] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0167] This application also provides a computer-readable storage medium corresponding to the surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots provided in the foregoing embodiments, wherein a computer program (i.e., a program product) is stored thereon, and the computer program, when run by a processor, executes the surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots provided in any of the foregoing embodiments.

[0168] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0169] The computer-readable storage medium provided in the above embodiments of this application and the surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0170] It should be noted that numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0171] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0172] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A visual obstacle avoidance control method for surgical robots with embodied intelligence, characterized in that, include: Obtain real surgical scene video streams and preoperative planning information; Analyze real surgical scene video streams to obtain obstacle information; A virtual operating environment is constructed based on the obstacle information, preoperative planning information, and the surgical scene video stream; Based on the virtual operating environment, predict the collision-free movement trajectory of the robotic arm; Based on the video stream of a real surgical scene, the robotic arm is controlled to execute the robotic arm movement trajectory and displayed in real time in the virtual operating environment; The process of parsing the real surgical scene video stream to obtain obstacle information includes: Acquire a real surgical scene video stream and segment it into continuously set depth image frames; A depth image frame is input into an obstacle recognition model to obtain obstacle recognition results; the obstacle recognition results include static obstacles and dynamic obstacles. By inputting continuous depth image frames with labeled obstacle recognition results into the spatiotemporal prediction model, the spatial motion trajectory of dynamic obstacles can be obtained. The step of inputting continuous depth image frames labeled with obstacle recognition results into a spatiotemporal prediction model to obtain the spatial motion trajectory of dynamic obstacles includes: Based on the obstacle recognition results, the location key points of dynamic obstacles are marked in the depth image frame; A continuous feature map is generated based on the location key points of consecutive depth image frames; The continuous feature map is input into the spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle; The step of generating a continuous feature map based on the location key points of consecutive depth image frames includes: Based on the aforementioned key locations, construct a domain space graph structure; Based on the correlation between the location key points of the same dynamic obstacle, a weighted adjacency matrix of the domain space graph structure is constructed; Based on the domain spatial graph structure of consecutive depth image frames and the corresponding weighted adjacency matrix, the feature representation of the domain spatial graph structure at different times is generated, which is the continuous feature map.

2. The surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots according to claim 1, characterized in that, The obstacle information includes static information and dynamic information; the dynamic information includes the predicted motion information of the obstacle in future time intervals.

3. The surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots according to claim 2, characterized in that, The step of constructing a virtual operating environment based on the obstacle information, preoperative planning information, and the surgical scene video stream includes: Obtain the parameters of the robotic arm and construct a model of the robotic arm; Based on the hand-eye calibration method, the mapping relationship between the binocular camera coordinate system and the robotic arm coordinate system is determined; Acquire outside-field-of-view measurement information from the binocular camera, including obstacle information; Based on the mapping relationship, the out-of-field measurement information, the obstacle information, the robotic arm model, the preoperative planning information, and the surgical scene video stream are unified into the same coordinate system; A virtual operating environment is constructed based on the field-of-view measurement information, obstacle information, robotic arm model, preoperative planning information, and surgical scene video stream in the same coordinate system.

4. The surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots according to claim 3, characterized in that, The preoperative planning information includes osteotomy surface information, osteotomy path information, and osteotomy preparation position information.

5. The surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots according to claim 4, characterized in that, The virtual operating environment is constructed based on the field-of-view measurement information in the same coordinate system, the obstacle information, the robotic arm model, the preoperative planning information, and the surgical scene video stream, including: Based on the out-of-field measurement information and the obstacle information, an obstacle model and corresponding coordinates are generated; Based on the preoperative planning information, generate the osteotomy surface model, osteotomy path point coordinates, and osteotomy preparation position model; Construct a virtual operating environment and project the robotic arm model, obstacle model, osteotomy surface model, osteotomy path point coordinates, and osteotomy preparation position into the virtual operating environment; The virtual operating environment is generated from the perspective of a binocular camera and projected onto the surgical scene video stream.

6. The surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots according to claim 1, characterized in that, The step of predicting the collision-free robotic arm trajectory based on the virtual operating environment includes: Obtain the initial position information and osteotomy preparation position information of the robotic arm; Based on the initial position information and osteotomy preparation position information, multiple motion paths of the robotic arm within future time intervals are generated using a random tree algorithm; Multiple motion paths are projected into the virtual operating environment for collision testing at future time intervals; One of the multiple motion paths that passed the collision test is selected as the motion trajectory of the robotic arm.

7. A visual obstacle avoidance control device for surgical robots with embodied intelligence, characterized in that, include: The information acquisition module is used to acquire real surgical scene video streams and preoperative planning information; The video parsing module is used to parse the video stream of a real surgical scene and obtain obstacle information; A virtual construction module is used to construct a virtual operating environment based on the obstacle information, preoperative planning information, and the surgical scene video stream; The trajectory prediction module is used to predict the collision-free movement trajectory of the robotic arm based on the virtual operating environment. The robotic arm control module is used to control the robotic arm to execute the robotic arm movement trajectory based on the real surgical scene video stream, and display it in real time in the virtual operating environment; The process of parsing the real surgical scene video stream to obtain obstacle information includes: Acquire a real surgical scene video stream and segment it into continuously set depth image frames; A depth image frame is input into an obstacle recognition model to obtain obstacle recognition results; the obstacle recognition results include static obstacles and dynamic obstacles. By inputting continuous depth image frames with labeled obstacle recognition results into the spatiotemporal prediction model, the spatial motion trajectory of dynamic obstacles can be obtained. The step of inputting continuous depth image frames labeled with obstacle recognition results into a spatiotemporal prediction model to obtain the spatial motion trajectory of dynamic obstacles includes: Based on the obstacle recognition results, the location key points of dynamic obstacles are marked in the depth image frame; A continuous feature map is generated based on the location key points of consecutive depth image frames; The continuous feature map is input into the spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle; The step of generating a continuous feature map based on the location key points of consecutive depth image frames includes: Based on the aforementioned key locations, construct a domain space graph structure; Based on the correlation between the location key points of the same dynamic obstacle, a weighted adjacency matrix of the domain space graph structure is constructed; Based on the domain spatial graph structure of consecutive depth image frames and the corresponding weighted adjacency matrix, the feature representation of the domain spatial graph structure at different times is generated, which is the continuous feature map.

8. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is used to execute the program for: Obtain real surgical scene video streams and preoperative planning information; Analyze real surgical scene video streams to obtain obstacle information; A virtual operating environment is constructed based on the obstacle information, preoperative planning information, and the surgical scene video stream; Based on the virtual operating environment, predict the collision-free movement trajectory of the robotic arm; Based on the video stream of a real surgical scene, the robotic arm is controlled to execute the robotic arm movement trajectory and displayed in real time in the virtual operating environment; The process of parsing the real surgical scene video stream to obtain obstacle information includes: Acquire a real surgical scene video stream and segment it into continuously set depth image frames; A depth image frame is input into an obstacle recognition model to obtain obstacle recognition results; the obstacle recognition results include static obstacles and dynamic obstacles. By inputting continuous depth image frames with labeled obstacle recognition results into the spatiotemporal prediction model, the spatial motion trajectory of dynamic obstacles can be obtained. The step of inputting continuous depth image frames labeled with obstacle recognition results into a spatiotemporal prediction model to obtain the spatial motion trajectory of dynamic obstacles includes: Based on the obstacle recognition results, the location key points of dynamic obstacles are marked in the depth image frame; A continuous feature map is generated based on the location key points of consecutive depth image frames; The continuous feature map is input into the spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle; The step of generating a continuous feature map based on the location key points of consecutive depth image frames includes: Based on the aforementioned key locations, construct a domain space graph structure; Based on the correlation between the location key points of the same dynamic obstacle, a weighted adjacency matrix of the domain space graph structure is constructed; Based on the domain spatial graph structure of consecutive depth image frames and the corresponding weighted adjacency matrix, the feature representation of the domain spatial graph structure at different times is generated, which is the continuous feature map.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the surgical robot visual obstacle avoidance control method for embodied intelligence of surgical robots as described in any one of claims 1-6.

Images