Visual obstacle avoidance control method and device for surgical robot with intelligent body

Abstract

The invention provides an intelligent surgical robot visual obstacle avoidance control method and device for a surgical robot. The method comprises the following steps: acquiring a real surgical scene video stream and preoperative planning information; analyzing the real operation scene video stream to obtain obstacle information; constructing a virtual operation environment according to the obstacle information, the preoperative planning information and the operation scene video stream; according to the virtual operation environment, a collision-free mechanical arm movement track is predicted; and according to the real operation scene video stream, the mechanical arm is controlled to execute the mechanical arm motion trail, and the mechanical arm motion trail is displayed in the virtual operation environment in real time. According to the method and the device, the real operation scene is completely simulated through the virtual operation environment, so that the collision-free motion track of the mechanical arm is predicted, the collision risk of the mechanical arm in the actual operation process is avoided, and the obstacle avoidance scheme is greatly simplified.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a method and device for visual obstacle avoidance control of a surgical robot with embodied intelligence. Background Art

[0002] Embodied intelligence refers to intelligent entities that have bodies and support interaction with the physical world, such as robots and driverless cars. By processing multiple sensor data inputs, a control center such as a large model generates motion commands to drive the intelligent entity, replacing the traditional motion-driving method based on rules or mathematical formulas, and achieving a deep integration of virtual and reality.

[0003] Surgical robots can serve as the perfect carrier of embodied intelligence, and realize the functions of surgical robots based on embodied intelligence, such as obstacle avoidance.

[0004] However, in the past, surgical robots used the solution of first planning the path of the robotic arm when using the robotic arm to avoid obstacles, and then controlling the robotic arm to avoid the obstacle in real time when a collision risk was detected during the actual control of the robotic arm to move along the planned path. This obstacle avoidance method is too cumbersome. Summary of the invention

[0005] The problem that this application solves is that current obstacle avoidance methods are too cumbersome.

[0006] To solve the above problems, the first aspect of the present application provides a visual obstacle avoidance control method for a surgical robot with embodied intelligence, comprising:

[0007] Obtain real surgical scene video stream and preoperative planning information;

[0008] Analyze the real surgical scene video stream to obtain obstacle information;

[0009] Constructing a virtual operating environment according to the obstacle information, the preoperative planning information and the surgical scene video stream;

[0010] Predict collision-free robot arm motion trajectory based on the virtual operating environment;

[0011] According to the real surgical scene video stream, the robot arm is controlled to execute the robot arm motion trajectory and displayed in real time in the virtual operating environment.

[0012] The second aspect of the present application provides a visual obstacle avoidance control device for a surgical robot with embodied intelligence, comprising:

[0013] An information acquisition module, which is used to obtain the real surgical scene video stream and preoperative planning information;

[0014] A video analysis module is used to analyze the real surgical scene video stream and obtain obstacle information;

[0015] A virtual construction module, which is used to construct a virtual operating environment according to the obstacle information, the preoperative planning information and the surgical scene video stream;

[0016] A trajectory prediction module, which is used to predict the collision-free robot arm motion trajectory based on the virtual operating environment;

[0017] The robot control module is used to control the robot to execute the robot motion trajectory according to the real surgical scene video stream, and display it in real time in the virtual operating environment.

[0018] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;

[0019] The memory is used to store programs;

[0020] The processor, coupled to the memory, is configured to execute the program to:

[0021] Obtain real surgical scene video stream and preoperative planning information;

[0022] Analyze the real surgical scene video stream to obtain obstacle information;

[0023] Constructing a virtual operating environment according to the obstacle information, the preoperative planning information and the surgical scene video stream;

[0024] Predict collision-free robot arm motion trajectory based on the virtual operating environment;

[0025] According to the real surgical scene video stream, the robot arm is controlled to execute the robot arm motion trajectory and displayed in real time in the virtual operating environment.

[0026] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned surgical robot visual obstacle avoidance control method for surgical robot embodied intelligence.

[0027] In this application, a real surgical scene is completely simulated through a virtual operating environment, so as to predict the collision-free motion trajectory of the robotic arm, avoid the risk of collision of the robotic arm during actual operation, and thus greatly simplify the obstacle avoidance solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of a visual obstacle avoidance control method for a surgical robot with embodied intelligence according to an embodiment of the present application;

[0029] Figure 2 is a structural block diagram of a surgical robot visual obstacle avoidance control device for surgical robot embodied intelligence according to an embodiment of the present application;

[0030] Figure 3 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0033] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0034] The surgical robot can be used as a perfect carrier of embodied intelligence to realize the functions of the surgical robot based on embodied intelligence. In the specific implementation process, the past experience of operating the surgical robot can be summarized first, and then a new control center can be generated based on the summarized experience.

[0035] However, for surgical robots, the past experience they learn is the experience of manual control of surgical robots, and the scenarios based on the application of experience are scenarios for automatic control of surgical robots. There are differences in the usage environment between their past experience and the application scenarios. How to reduce the impact of this difference in the process of generating a new control center is a problem that is currently difficult to solve.

[0036] In response to the above problems, the present application provides a new skeleton registration solution, which performs coarse registration through digital twins to solve the problem of low accuracy of current feature point registration.

[0037] The present application embodiment provides a visual obstacle avoidance control method for a surgical robot with embodied intelligence, and the specific scheme of the method is as follows: Figure 1-Figure 2 As shown, the method can be performed by a surgical robot visual obstacle avoidance control device for surgical robot embodied intelligence, and the surgical robot visual obstacle avoidance control device for surgical robot embodied intelligence can be integrated into electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1As shown, it is a flow chart of a visual obstacle avoidance control method for a surgical robot with embodied intelligence according to an embodiment of the present application; wherein the visual obstacle avoidance control method for a surgical robot with embodied intelligence comprises:

[0038] S101, obtaining a real surgical scene video stream and preoperative planning information;

[0039] In the present application, the real surgical scene video stream is an intraoperative video obtained by a binocular camera installed on the operating table. The video is obtained in real time to facilitate real-time control of the robotic arm based on the surgical scene video stream.

[0040] In the present application, the preoperative planning information is the planning information determined based on the medical images of the patient taken before the operation. In osteotomy and joint replacement surgery, the preoperative planning information generally includes the osteotomy position, osteotomy section and the osteotomy robot arm preparation position.

[0041] In addition, when determining the planning information before surgery, it is necessary to first segment the patient's medical image and build a three-dimensional bone model, and then generate the osteotomy position, osteotomy section, and osteotomy robot arm preparation position based on the three-dimensional bone model. Therefore, in this application, the preoperative planning information also includes the corresponding three-dimensional bone model.

[0042] S102, parsing the real surgical scene video stream to obtain obstacle information;

[0043] The real scene during surgery not only includes the key positions that require surgery, but also other non-surgical parts, such as instruments, equipment, doctors and nurses, which will become obstacles that hinder the movement of the robotic arm. By analyzing the video and obtaining information about obstacles, the path planning of the subsequent robotic arm can be carried out.

[0044] S103, constructing a virtual operating environment according to the obstacle information, the preoperative planning information and the surgical scene video stream;

[0045] S104, predicting a collision-free robot arm motion trajectory according to the virtual operating environment;

[0046] S105, according to the real surgical scene video stream, controlling the robot arm to execute the robot arm motion trajectory, and displaying it in real time in the virtual operating environment.

[0047] In this application, a real surgical scene is completely simulated through a virtual operating environment, so as to predict the collision-free motion trajectory of the robotic arm, avoid the risk of collision of the robotic arm during actual operation, and thus greatly simplify the obstacle avoidance solution.

[0048] In the present application, collision detection is performed on the robot arm motion trajectory through a virtual operating environment, so that motion trajectories with collision risks can be detected and collision-free robot arm motion trajectories can be retained.

[0049] In one embodiment, parsing the real surgical scene video stream to obtain obstacle information includes:

[0050] Obtain a real surgical scene video stream and segment it into continuously set depth image frames;

[0051] Inputting the depth image frame into the obstacle recognition model to obtain an obstacle recognition result; the obstacle recognition result includes static obstacles and dynamic obstacles;

[0052] The continuous depth image frames with labeled obstacle recognition results are input into the spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.

[0053] In one implementation, the obstacle information includes static information and dynamic information; the dynamic information includes predicted movement information of the obstacle in a future time interval.

[0054] The step of inputting the continuous depth image frames with the obstacle recognition results marked thereon into the spatiotemporal prediction model to obtain the obstacle prediction results comprises:

[0055] Based on the obstacle recognition results, the key points of the dynamic obstacles are marked in the depth image frame;

[0056] Generate a continuous feature map according to the position key points of continuous depth image frames;

[0057] The continuous feature map is input into a spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.

[0058] In this application, the obstacle detection results of the depth image are used to locate the key points (such as center points, edge points or feature corner points) of the dynamic obstacles in each frame. The object detection model (such as YOLO or SSD) can be used in combination with the depth information to accurately mark the obstacles.

[0059] In this application, the obstacle contour information in the depth image is extracted, and key points are marked at the edge or center of the obstacle. These key points reflect the location and shape characteristics of the obstacle.

[0060] In this application, it is ensured that the key points have depth information so as to accurately locate the position information of dynamic obstacles in three-dimensional space.

[0061] In the present application, a continuous feature map is generated by combining key point information of multiple frames, so that it can reflect the movement changes of obstacles in the time series.

[0062] In this application, the model outputs the spatial motion trajectory of the obstacle, including the position and motion direction of the obstacle at the future moment. The trajectory can be expressed as a series of coordinate points or curves, which intuitively reflects the future position of the obstacle.

[0063] In one embodiment, the spatiotemporal prediction model is a gated dilated causal convolutional neural network.

[0064] In one implementation, generating a continuous feature map according to the position key points of continuous depth image frames includes:

[0065] Constructing a domain space graph structure according to the position key points;

[0066] Based on the correlation between the key points of the same dynamic obstacle, a weighted adjacency matrix of the domain space graph structure is constructed;

[0067] According to the domain space graph structure of continuous depth image frames and the corresponding weighted adjacency matrix, feature representations of the domain space graph structure at different times are generated, namely, the continuous feature graph.

[0068] In this application, the position key points in each depth image frame are regarded as nodes in the graph structure. Each node contains the position coordinates, depth value and other possible features (such as speed, direction, etc.) of the key point.

[0069] In the present application, key points in a depth image frame are combined into a domain space graph structure, which reflects the spatial layout of obstacles and the geometric relationship between the key points.

[0070] In the present application, an undirected graph may be used to represent the domain space graph structure of obstacles, each node represents a key point, and the edges between nodes represent the adjacency relationship between the key points.

[0071] In this application, the adjacency relationship of each key point is defined based on the association relationship between the key points of the same dynamic obstacle. The association relationship can be spatial distance, motion correlation between key points, etc.

[0072] In this application, weights are assigned to the edges of the adjacency matrix based on the association relationship. The weights can be defined in the following ways:

[0073] Spatial distance weighting: The closer the distance between two key points, the higher the weight.

[0074] Direction weighting: According to the movement direction of the key point, the corresponding direction weight is assigned to the adjacency relationship.

[0075] Speed ​​weighting: The higher the speed similarity of adjacent key points, the higher the weight.

[0076] In this application, an adjacency matrix containing weight information is generated, and each element in the matrix represents the weighted relationship between two key points in the graph structure.

[0077] In this application, the domain space graph structure of the continuous depth image frames and the corresponding weighted adjacency matrix are combined to form a time series graph structure. The graph structure in each time step represents the key point relationship of the same obstacle.

[0078] In this application, in the graph structure of each frame, the feature representation of each node is generated according to the weight of the weighted adjacency matrix. The feature representation can be a combination of multiple information such as the position, depth, and speed of the key point.

[0079] In this application, the feature representation of dynamic obstacles at different times is generated by changing the domain space graph structure and the adjacency matrix at multiple consecutive time steps to form a continuous graph feature. This feature sequence is a continuous feature graph, which reflects the movement trajectory of the dynamic obstacle.

[0080] In one embodiment, constructing a virtual operating environment according to the obstacle information, the preoperative planning information and the surgical scene video stream includes:

[0081] Get the robot arm parameters and build the robot arm model;

[0082] According to the hand-eye calibration method, determine the mapping relationship between the binocular camera coordinate system and the robotic arm coordinate system;

[0083] Acquire out-of-field measurement information of the binocular camera, wherein the out-of-field measurement information includes obstacle information;

[0084] According to the mapping relationship, the out-of-field measurement information, the obstacle information, the robot arm model, the preoperative planning information, and the surgical scene video stream are unified into the same coordinate system;

[0085] A virtual operating environment is constructed based on the out-of-field measurement information, the obstacle information, the robotic arm model, the preoperative planning information, and the surgical scene video stream in the same coordinate system.

[0086] In this application, the structural parameters of the robotic arm include the number of joints, the range of motion of each joint, the length of the connecting rod, the inertia parameters, etc.

[0087] In the present application, the Denavit-Hartenberg parameter method or other robot modeling methods can be used to construct a robot model to accurately simulate the movement of the robot.

[0088] In this application, hand-eye calibration is performed by obtaining the manipulator reference point under the binocular camera view and the corresponding point under the manipulator coordinate system, and calculating the transformation matrix through least squares fitting or optimization algorithm. The specific acquisition method is not described in detail in this application.

[0089] In the present application, the binocular camera can be rotated and the above-mentioned obstacle information acquisition method can be used to obtain the measurement information outside the field of view of the binocular camera; the measurement information outside the field of view and the measurement information inside the field of view can be spliced ​​by key point recognition to form a complete intraoperative environment.

[0090] Preferably, the binocular camera needs to regularly update the out-of-field measurement information to ensure that the position changes of obstacles can be continuously captured during the operation, providing support for real-time updates of the virtual environment.

[0091] In this application, out-of-field measurement information, obstacle information, robotic arm model, preoperative planning information, and surgical scene video stream are integrated into a unified coordinate system to ensure that all data have a consistent spatial reference in the virtual environment.

[0092] In this application, a virtual surgical scene is constructed in the same coordinate system, including a robotic arm model, preoperative planning information (such as important anatomical structures), obstacles, and dynamic objects in a real-time video stream.

[0093] In this application, the measurement information outside the field of view of the binocular camera and the video stream of the surgical scene are collected in real time and updated to the virtual operating environment so that the environment can dynamically reflect the changes in the actual surgical scene.

[0094] In this application, the real-time position, obstacles and target areas of the robot arm are intuitively displayed in a virtual operating environment. The user or control system can monitor the movement trajectory and potential collision risks of the robot arm in this environment.

[0095] In one embodiment, the preoperative planning information includes osteotomy surface information, osteotomy path information, and osteotomy preparation position information.

[0096] In one embodiment, the virtual operating environment is constructed based on the out-of-field measurement information, the obstacle information, the robot model, the preoperative planning information, and the surgical scene video stream in the same coordinate system, including:

[0097] Generate an obstacle model and corresponding coordinates according to the out-of-field measurement information and the obstacle information;

[0098] Generate osteotomy surface model, osteotomy path point coordinates and osteotomy preparation position model according to preoperative planning information;

[0099] Construct a virtual operating environment, and project the robot arm model, obstacle model, osteotomy surface model, osteotomy path point coordinates and osteotomy preparation position into the virtual operating environment;

[0100] The display screen of the virtual operating environment is generated from the perspective of the binocular camera and projected into the video stream of the surgical scene.

[0101] In this application, the robot arm model is projected into the virtual environment to display the current robot arm position and motion status.

[0102] In this application, the obstacle model is projected into the virtual environment to ensure that the spatial position of the obstacle is consistent with the actual scene.

[0103] In this application, the osteotomy surface model and path point coordinates are projected to display the preoperatively planned osteotomy position and path.

[0104] In this application, an osteotomy preparation position model is displayed to assist in the correct positioning of the robotic arm before surgery.

[0105] In the present application, a virtual viewing angle of a binocular camera is set in a virtual environment to capture images of a virtual scene at an angle and position consistent with an actual binocular camera.

[0106] In the present application, the images captured by the virtual camera are used to generate a real-time virtual operating environment display, including information such as the robotic arm, obstacles, osteotomy surfaces, path points and preparation positions.

[0107] In this application, the generated virtual display screen is superimposed on the video stream of the surgical scene, so that the surgical team can see the superimposed effect of virtual information in the real-time video. Augmented reality (AR) technology can be used to accurately project virtual information onto the actual scene, helping the surgeon to accurately complete the surgical steps.

[0108] In one embodiment, predicting a collision-free robot arm motion trajectory based on a virtual operating environment includes:

[0109] Obtaining the initial position information of the robotic arm and the osteotomy preparation position information;

[0110] According to the initial position information and the osteotomy preparation position information, a plurality of motion paths of the robot arm in the future time interval are generated by a random tree algorithm;

[0111] Projecting a plurality of said motion paths into said virtual operating environment to perform collision testing within a future time interval;

[0112] One of the multiple motion paths that pass the collision test is selected as the motion trajectory of the robot arm.

[0113] In the present application, the osteotomy preparation position information is the end position of the robotic arm; the initial position information and the motion path of the osteotomy preparation position information are generated by a random tree algorithm.

[0114] In this application, the future time interval refers to a period of time in the future.

[0115] It should be noted that if the initial position information of the robotic arm and the osteotomy preparation position information of the robotic arm are too far apart and cannot be generated at one time, the specific motion path of the robotic arm to reach a certain intermediate position within a certain period of time in the future can also be directly generated.

[0116] In this application, the generated multiple motion paths are projected into the virtual operating environment, and collision detection is performed on each path to ensure that there are no obstacles at all points in the path in space to avoid collisions of the robot arm.

[0117] In the present application, an optimal path is selected from multiple paths through collision testing as the final motion trajectory of the robot arm. The selection criteria can be based on factors such as path smoothness, motion time, and energy consumption.

[0118] In the present application, the specific process of the random tree algorithm is: initialize the random tree, generate source points, target points, and target point radius, and generate an obstacle matrix; generate random points; find the point on the tree closest to the random point; use the distance from the nearest point to the random point to extend the nearest point to the random point to obtain a new point; if the straight line segment from the nearest point to the new point does not pass through an obstacle, the new point is not empty, otherwise, the new point is empty; if the new point is empty, return to the step to generate a random point; add the new point to the tree as a node, and use the nearest point as the parent node of the new point, and record it; if the distance from the new point to the target point is less than the preset distance, the algorithm ends, otherwise return to the step to generate a random point.

[0119] In the present application, a virtual environment is set up for collision detection, thereby greatly reducing the limitations and difficulty of the random tree algorithm in generating motion paths, and ensuring accuracy.

[0120] An embodiment of the present application provides a surgical robot visual obstacle avoidance control device for surgical robot embodied intelligence, which is used to execute the surgical robot visual obstacle avoidance control method for surgical robot embodied intelligence described in the above content of the present application. The surgical robot visual obstacle avoidance control device for surgical robot embodied intelligence is described in detail below.

[0121] like Figure 2 As shown, the visual obstacle avoidance control device for a surgical robot with embodied intelligence comprises:

[0122] An information acquisition module 101 is used to acquire a real surgical scene video stream and preoperative planning information;

[0123] A video analysis module 102 is used to analyze the real surgical scene video stream to obtain obstacle information;

[0124] A virtual construction module 103, which is used to construct a virtual operating environment according to the obstacle information, the preoperative planning information and the surgical scene video stream;

[0125] A trajectory prediction module 104, which is used to predict a collision-free robot arm motion trajectory based on a virtual operating environment;

[0126] The robot control module 105 is used to control the robot to execute the robot motion trajectory according to the real surgical scene video stream, and display it in real time in the virtual operating environment.

[0127] In one implementation, the video parsing module 102 is further configured to:

[0128] A real surgical scene video stream is obtained and divided into continuously set depth image frames; the depth image frames are input into an obstacle recognition model to obtain obstacle recognition results; the obstacle recognition results include static obstacles and dynamic obstacles; the continuous depth image frames marked with obstacle recognition results are input into a spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.

[0129] In one implementation, the obstacle information includes static information and dynamic information; the dynamic information includes predicted movement information of the obstacle in a future time interval.

[0130] In one embodiment, the virtual construction module 103 is further used to:

[0131] Acquire the parameters of the robotic arm and construct a robotic arm model; determine the mapping relationship between the binocular camera coordinate system and the robotic arm coordinate system according to the hand-eye calibration method; acquire the out-of-field measurement information of the binocular camera, wherein the out-of-field measurement information includes obstacle information; according to the mapping relationship, unify the out-of-field measurement information, the obstacle information, the robotic arm model, preoperative planning information, and the surgical scene video stream into the same coordinate system; construct a virtual operating environment based on the out-of-field measurement information, the obstacle information, the robotic arm model, preoperative planning information, and the surgical scene video stream in the same coordinate system.

[0132] In one embodiment, the preoperative planning information includes osteotomy surface information, osteotomy path information, and osteotomy preparation position information.

[0133] In one embodiment, the virtual construction module 103 is further used to:

[0134] According to the out-of-field measurement information and the obstacle information, an obstacle model and corresponding coordinates are generated; according to the preoperative planning information, an osteotomy surface model, osteotomy path point coordinates and an osteotomy preparation position model are generated; a virtual operating environment is constructed, and the robotic arm model, obstacle model, osteotomy surface model, osteotomy path point coordinates and osteotomy preparation position are projected to the virtual operating environment; a display screen of the virtual operating environment is generated with a binocular camera as the perspective, and is projected into the surgical scene video stream.

[0135] In one implementation, the trajectory prediction module 104 is further configured to:

[0136] Acquire the initial position information and osteotomy preparation position information of the robotic arm; generate multiple motion paths of the robotic arm within a future time interval through a random tree algorithm based on the initial position information and the osteotomy preparation position information; project the multiple motion paths into the virtual operating environment to perform collision tests within a future time interval; and select one of the multiple motion paths that pass the collision test as the motion trajectory of the robotic arm.

[0137] The surgical robot visual obstacle avoidance control device for surgical robot embodied intelligence provided in the above-mentioned embodiments of the present application has a corresponding relationship with the surgical robot visual obstacle avoidance control method for surgical robot embodied intelligence provided in the embodiments of the present application. Therefore, the specific content in the device has a corresponding relationship with the surgical robot visual obstacle avoidance control method for surgical robot embodied intelligence. The specific content can be referred to the record in the surgical robot visual obstacle avoidance control method for surgical robot embodied intelligence, and will not be repeated in this application.

[0138] The surgical robot visual obstacle avoidance control device for surgical robot embodied intelligence provided in the above-mentioned embodiments of the present application and the surgical robot visual obstacle avoidance control method for surgical robot embodied intelligence provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0139] The above describes the internal functions and structures of the visual obstacle avoidance control device for surgical robots with embodied intelligence. Figure 3 As shown, in practice, the surgical robot visual obstacle avoidance control device for surgical robot embodied intelligence can be realized as an electronic device, including: a memory 301 and a processor 303.

[0140] The memory 301 may be configured to store programs.

[0141] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0142] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0143] The processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:

[0144] Obtain real surgical scene video stream and preoperative planning information;

[0145] Analyze the real surgical scene video stream to obtain obstacle information;

[0146] Constructing a virtual operating environment according to the obstacle information, the preoperative planning information and the surgical scene video stream;

[0147] Predict collision-free robot arm motion trajectory based on the virtual operating environment;

[0148] According to the real surgical scene video stream, the robot arm is controlled to execute the robot arm motion trajectory and displayed in real time in the virtual operating environment.

[0149] In one implementation, the processor 303 is further configured to:

[0150] A real surgical scene video stream is obtained and divided into continuously set depth image frames; the depth image frames are input into an obstacle recognition model to obtain obstacle recognition results; the obstacle recognition results include static obstacles and dynamic obstacles; the continuous depth image frames marked with obstacle recognition results are input into a spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.

[0151] In one implementation, the obstacle information includes static information and dynamic information; the dynamic information includes predicted movement information of the obstacle in a future time interval.

[0152] In one implementation, the processor 303 is further configured to:

[0153] Acquire the parameters of the robotic arm and construct a robotic arm model; determine the mapping relationship between the binocular camera coordinate system and the robotic arm coordinate system according to the hand-eye calibration method; acquire the out-of-field measurement information of the binocular camera, wherein the out-of-field measurement information includes obstacle information; according to the mapping relationship, unify the out-of-field measurement information, the obstacle information, the robotic arm model, preoperative planning information, and the surgical scene video stream into the same coordinate system; construct a virtual operating environment based on the out-of-field measurement information, the obstacle information, the robotic arm model, preoperative planning information, and the surgical scene video stream in the same coordinate system.

[0154] In one embodiment, the preoperative planning information includes osteotomy surface information, osteotomy path information, and osteotomy preparation position information.

[0155] In one implementation, the processor 303 is further configured to:

[0156] According to the out-of-field measurement information and the obstacle information, an obstacle model and corresponding coordinates are generated; according to the preoperative planning information, an osteotomy surface model, osteotomy path point coordinates and an osteotomy preparation position model are generated; a virtual operating environment is constructed, and the robotic arm model, obstacle model, osteotomy surface model, osteotomy path point coordinates and osteotomy preparation position are projected to the virtual operating environment; a display screen of the virtual operating environment is generated with a binocular camera as the perspective, and is projected into the surgical scene video stream.

[0157] In one implementation, the processor 303 is further configured to:

[0158] Acquire the initial position information and osteotomy preparation position information of the robotic arm; generate multiple motion paths of the robotic arm within a future time interval through a random tree algorithm based on the initial position information and the osteotomy preparation position information; project the multiple motion paths into the virtual operating environment to perform collision tests within a future time interval; and select one of the multiple motion paths that pass the collision test as the motion trajectory of the robotic arm.

[0159] In the present application, the processor is also specifically used to execute all processes and steps of the above-mentioned surgical robot visual obstacle avoidance control method for surgical robot embodied intelligence. The specific content can refer to the records in the surgical robot visual obstacle avoidance control method for surgical robot embodied intelligence, and will not be repeated in this application.

[0160] In this application, Figure 3 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 3 Components shown.

[0161] The electronic device provided in this embodiment is based on the same inventive concept as the surgical robot visual obstacle avoidance control method for surgical robot embodied intelligence provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored in it.

[0162] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0163] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0166] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0167] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0168] The present application also provides a computer-readable storage medium corresponding to the surgical robot visual obstacle avoidance control method for surgical robot embodied intelligence provided in the aforementioned embodiments, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the surgical robot visual obstacle avoidance control method for surgical robot embodied intelligence provided in any of the aforementioned embodiments.

[0169] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0170] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the surgical robot visual obstacle avoidance control method for surgical robot embodied intelligence provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0171] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0172] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0173] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A visual obstacle avoidance control method for a surgical robot with embodied intelligence, characterized in that: include: Obtain real surgical scene video stream and preoperative planning information; Analyze the real surgical scene video stream to obtain obstacle information; Constructing a virtual operating environment according to the obstacle information, the preoperative planning information and the surgical scene video stream; Predict collision-free robot arm motion trajectory based on the virtual operating environment; According to the real surgical scene video stream, the robot arm is controlled to execute the robot arm motion trajectory and displayed in real time in the virtual operating environment.

2. The visual obstacle avoidance control method for a surgical robot with embodied intelligence according to claim 1, characterized in that: The method of parsing the real surgical scene video stream to obtain obstacle information includes: Obtain a real surgical scene video stream and segment it into continuously set depth image frames; Inputting the depth image frame into the obstacle recognition model to obtain an obstacle recognition result; the obstacle recognition result includes static obstacles and dynamic obstacles; The continuous depth image frames with labeled obstacle recognition results are input into the spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.

3. The visual obstacle avoidance control method for a surgical robot with embodied intelligence according to claim 1 or 2, characterized in that: The obstacle information includes static information and dynamic information; the dynamic information includes predicted movement information of the obstacle in a future time interval.

4. The visual obstacle avoidance control method for surgical robots with embodied intelligence according to claim 3 is characterized in that: The step of constructing a virtual operating environment according to the obstacle information, the preoperative planning information and the surgical scene video stream includes: Get the robot arm parameters and build the robot arm model; According to the hand-eye calibration method, determine the mapping relationship between the binocular camera coordinate system and the robotic arm coordinate system; Acquire out-of-field measurement information of the binocular camera, wherein the out-of-field measurement information includes obstacle information; According to the mapping relationship, the out-of-field measurement information, the obstacle information, the robot arm model, the preoperative planning information, and the surgical scene video stream are unified into the same coordinate system; A virtual operating environment is constructed based on the out-of-field measurement information, the obstacle information, the robotic arm model, the preoperative planning information, and the surgical scene video stream in the same coordinate system.

5. The visual obstacle avoidance control method for surgical robots with embodied intelligence according to claim 4, characterized in that: The preoperative planning information includes osteotomy surface information, osteotomy path information and osteotomy preparation position information.

6. The visual obstacle avoidance control method for a surgical robot with embodied intelligence according to claim 5, characterized in that: The virtual operation environment is constructed based on the out-of-field measurement information, the obstacle information, the robot model, the preoperative planning information, and the surgical scene video stream in the same coordinate system, including: Generate an obstacle model and corresponding coordinates according to the out-of-field measurement information and the obstacle information; Generate osteotomy surface model, osteotomy path point coordinates and osteotomy preparation position model according to preoperative planning information; Construct a virtual operating environment, and project the robot arm model, obstacle model, osteotomy surface model, osteotomy path point coordinates and osteotomy preparation position into the virtual operating environment; The display screen of the virtual operating environment is generated from the perspective of the binocular camera and projected into the video stream of the surgical scene.

7. The visual obstacle avoidance control method for a surgical robot with embodied intelligence according to claim 1 or 2, characterized in that: The method of predicting a collision-free robot arm motion trajectory according to the virtual operating environment includes: Obtaining the initial position information of the robotic arm and the osteotomy preparation position information; According to the initial position information and the osteotomy preparation position information, a plurality of motion paths of the robot arm in the future time interval are generated by a random tree algorithm; Projecting a plurality of said motion paths into said virtual operating environment to perform collision testing within a future time interval; One of the multiple motion paths that pass the collision test is selected as the motion trajectory of the robot arm.

8. A visual obstacle avoidance control device for a surgical robot with embodied intelligence, characterized in that: include: An information acquisition module, which is used to obtain the real surgical scene video stream and preoperative planning information; A video analysis module is used to analyze the real surgical scene video stream and obtain obstacle information; A virtual construction module, which is used to construct a virtual operating environment according to the obstacle information, the preoperative planning information and the surgical scene video stream; A trajectory prediction module, which is used to predict the collision-free robot arm motion trajectory based on the virtual operating environment; The robot control module is used to control the robot to execute the robot motion trajectory according to the real surgical scene video stream, and display it in real time in the virtual operating environment.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Obtain real surgical scene video stream and preoperative planning information; Analyze the real surgical scene video stream to obtain obstacle information; Constructing a virtual operating environment according to the obstacle information, the preoperative planning information and the surgical scene video stream; Predict collision-free robot arm motion trajectory based on the virtual operating environment; According to the real surgical scene video stream, the robot arm is controlled to execute the robot arm motion trajectory and displayed in real time in the virtual operating environment.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by the processor to implement the surgical robot visual obstacle avoidance control method for surgical robot embodied intelligence as described in any one of claims 1-7.

Images