Robotic preoperative navigation method, system, storage medium and computer equipment

Abstract

The present invention relates to a preoperative navigation method for a surgical robot, a robotic preoperative navigation system, a storage medium, and a computer device. The method comprises: obtaining environmental information within an operating room and displaying a virtual map based on the environmental information; forming a mark on the virtual map indicating the surgical robot's starting position and a mark on the surgical operation position; and obtaining first interaction information and generating a planned path for the surgical robot in the virtual map based on the first interaction information; the planned path is used for the surgical robot to move from a corresponding starting position within the operating room to a surgical operation position. The above-mentioned preoperative navigation method for a surgical robot, robotic preoperative navigation system, storage medium, and computer device can be applied to navigation within an operating room.

Description

Technical Field

[0001] The present application relates to the field of medical robot technology, and in particular to a robot preoperative navigation method, a robot preoperative navigation system, a storage medium, and a computer device. Background Art

[0002] Traditional techniques use Global Positioning System (GPS) sensors to locate a robot's starting and target positions, as well as the path between them. This method plans the robot's navigation path by matching longitude and latitude coordinates. However, due to weak indoor GPS signals, this method is unsuitable for surgical robot navigation within the operating room. Consequently, it cannot effectively avoid obstacles while the robot moves from its starting position to its surgical location. Summary of the Invention

[0003] Based on this, it is necessary to provide a robot preoperative navigation method, a robot preoperative navigation system, a storage medium and a computer device to address the above problems.

[0004] A robot preoperative navigation method includes: obtaining environmental information in an operating room and displaying a virtual map based on the environmental information; forming a mark of the surgical robot's starting position and a mark of the surgical operation position in the virtual map; and obtaining first interaction information and generating a planned path for the surgical robot in the virtual map based on the first interaction information; the planned path is used for the surgical robot to move from the starting position to the surgical operation position in the operating room accordingly.

[0005] In one embodiment, the virtual map is displayed overlaid on the real scene.

[0006] In one embodiment, the virtual map includes grid markings, and the mark of the starting position and the mark of the surgical operation position are both located at the intersection of the grid markings; the planned path connects the mark of the starting position and the mark of the surgical operation position through the intersection marked as no obstacle in the grid markings.

[0007] In one embodiment, obtaining environmental information in an operating room and displaying a virtual map based on the environmental information includes: obtaining the environmental information using multiple pixel images taken at different locations in the operating room; constructing and displaying a virtual map with grid markers based on the environmental information; wherein some intersections in the grid markers display the environmental information.

[0008] In one embodiment, the environmental information includes the position coordinate information of the obstacles in the operating room and the starting position; the constructing and displaying the virtual map with grid markings based on the environmental information includes: based on the position coordinate information of the mark of the surgical operation position, and the position coordinate information of the obstacles and the starting position, expanding the coordinate information of multiple virtual points that do not overlap with each position coordinate information; using the coordinate information of each virtual point and each position coordinate information as the intersection of the grid markings, generating and displaying the virtual map with grid markings.

[0009] In one embodiment, the grid shape in the virtual map with grid identification includes a triangle.

[0010] In one embodiment, generating the planned path of the surgical robot in the virtual map based on the first interaction information includes: determining at least one posture change information relative to the starting position based on the first interaction information to obtain the planned path of the surgical robot; and displaying the planned path in the virtual map.

[0011] In one embodiment, during the process of the surgical robot moving from the starting position to the surgical operation position, the method further includes: obtaining second interaction information, and adjusting the motion state of the surgical robot based on the second interaction information, and / or providing prompt information.

[0012] In one embodiment, the robot preoperative navigation method further includes: projecting the planned path in the operating room between the starting position and the surgical operation position according to a corresponding scale.

[0013] In one embodiment, the need to adjust the motion state of the surgical robot during its movement includes: the distance between the surgical robot and an obstacle is less than a preset safety distance and / or the motion trajectory of the surgical robot deviates from the planned path.

[0014] In one embodiment, the method further includes: controlling the movement of the surgical robot and / or controlling the positioning of the robotic arm of the surgical robot based on the planned path.

[0015] A robotic preoperative navigation system comprises: a control processing device and a human-computer interaction device, wherein the control processing device and the human-computer interaction device are communicatively connected; the human-computer interaction device is used to display a virtual map and to obtain first interaction information of a user; and the control processing device is used to execute the robotic preoperative navigation method as described in any one of the above items.

[0016] In one embodiment, the human-computer interaction device includes an AR device, and the AR device is used to overlay and display the virtual map on a real scene; the AR device is used to overlay and display the virtual map on a real scene.

[0017] In one embodiment, the control processing device includes a position adjustment unit and a camera, the camera is set on the position adjustment unit, the position adjustment unit is used to adjust the shooting angle of the camera, and the camera is used to collect multiple pixel images taken at different positions in the operating room to obtain the environmental information.

[0018] In one embodiment, the human-computer interaction device projects the planned path in the operating room between the starting position and the surgical operation position according to a corresponding proportion.

[0019] In one embodiment, while the surgical robot moves from the starting position to the surgical operation position in the operating room, the control processing device obtains second interaction information based on gesture information, and adjusts the motion state of the surgical robot based on the second interaction information, and / or provides prompt information.

[0020] In one embodiment, the human-computer interaction device obtains the first interaction information by collecting gesture images.

[0021] A storage medium stores a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of any of the above methods.

[0022] A computer device comprises a memory and a processor; a computer program executable on the processor is stored on the processor, and the processor implements the steps of any of the above methods when executing the computer program.

[0023] The above-mentioned robot preoperative navigation method, robot preoperative navigation system, storage medium and computer equipment obtain environmental information in the operating room to display a virtual map of the operating room. The operator uses gestures to plan the navigation path in advance on the virtual map. The surgical robot can automatically and quickly move to the position and avoid obstacles in the room according to the planned path, thereby reducing robot collisions and reducing the robot damage rate; and this method can be applied to indoor navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flow chart of a robot preoperative navigation method provided in one embodiment;

[0026] Figure 2 is a schematic diagram of a robot system provided in one embodiment;

[0027] Figure 3 is a schematic diagram of the preoperative movement process of a surgical robot provided in one embodiment;

[0028] Figure 4 This is a flowchart of the specific steps of step S11 provided in one embodiment;

[0029] Figure 5 is a schematic structural diagram of a control processing device provided in one embodiment;

[0030] Figure 6 This is a flowchart of the specific steps of step S112 provided in one embodiment;

[0031] Figures 7a to 7d A schematic diagram of the principles of a grid-based navigation path planning method provided in one embodiment;

[0032] Figure 8 This is a flowchart of the specific steps of step S12 provided in one embodiment;

[0033] Figure 9a Schematic diagram of the structure of AR glasses with binocular cameras provided in one embodiment;

[0034] Figure 9b Schematic diagram of the relative relationship between the coordinate systems of the AR glasses and the binocular camera provided in one embodiment;

[0035] Figure 9c A schematic diagram of the binocular vision principle provided in one embodiment;

[0036] Figure 10 This is a flowchart of the specific steps of step S122 provided in one embodiment;

[0037] Figure 11a A schematic diagram illustrating a histogram-based segmentation implementation method provided in an embodiment;

[0038] Figure 11bA schematic diagram illustrating a segmentation implementation method based on local area information provided in one embodiment;

[0039] Figure 12 A flowchart of steps further included in the robot preoperative navigation method provided in one embodiment;

[0040] Figure 13 A schematic diagram of adjusting the motion state of a robot using an operator's gestures provided in one embodiment;

[0041] Figure 14 A schematic diagram of a driving principle provided in one embodiment;

[0042] Figure 15 4 is a structural block diagram of a robotic preoperative navigation system provided in one embodiment. DETAILED DESCRIPTION

[0043] To facilitate understanding of the present application, a more comprehensive description of the present application will be provided below with reference to the accompanying drawings. The drawings illustrate preferred embodiments of the present application. However, the present application may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present disclosure.

[0044] Figure 1 FIG. 1 is a flow chart of a robot preoperative navigation method in one embodiment. Figure 1 ,The robotic preoperative navigation method includes the following steps:

[0045] Step S11 , obtaining environmental information in the operating room, and displaying a virtual map based on the environmental information; a mark of the surgical robot's starting position and a mark of the surgical operation position are formed in the virtual map.

[0046] Specifically, the robotic preoperative navigation method can be implemented using a robotic preoperative navigation system, which is used to navigate the preoperative movement of a surgical robot in a robotic system in an operating room. Figure 2 The robotic system is located within the operating room. The robotic system may include a doctor's operating terminal 12, a patient operating terminal (i.e., a surgical robot) 11, a visual platform 13, and surgical instruments 14. In addition to the robotic system, the operating room may also include a patient table 15 and a patient positioned on the table. The robotic arm of the surgical robot 11 can be used to connect to the surgical instruments 14, thereby assisting the doctor in performing the surgery.

[0047] Environmental information within the operating room may include the coordinates of obstacles within the operating room and the starting position. Objects and people other than the surgical robot 11 within the operating room may be considered obstacles during the preoperative movement of the surgical robot 11. The starting position may be the current position of the surgical robot 11. The surgical operation position may be the location of the patient's lesion.

[0048] The robotic preoperative navigation system may include a control processing device and a human-computer interaction device. The control processing device is used to obtain environmental information in the operating room and reconstruct a virtual map of the operating room based on the environmental information. The virtual map can be a planar image or a three-dimensional image. The obstacles and the surgical robot 11 in the virtual map can be similar to the corresponding physical objects, or they can be replaced by corresponding symbols. The obstacles in the virtual map can be called obstacle markers, and the obstacle markers correspond to the obstacles in the operating room of the real scene. The surgical robot 11 in the virtual map can be called a surgical robot marker, and the surgical robot marker corresponds to the surgical robot 11 in the operating room of the real scene. Of course, it is also possible not to mark the surgical robot marker on the virtual map and directly replace it with a marker of the initial position. The distance between each obstacle and the surgical robot 11 in the virtual map can be reduced in direct proportion to the corresponding actual distance.

[0049] The virtual map includes a mark for the starting position of the surgical robot 11 and a mark for the surgical operation position. In this application, the starting position in the operating room corresponds to the mark for the starting position on the virtual map, and the surgical operation position in the operating room corresponds to the mark for the surgical operation position on the virtual map.

[0050] The human-computer interaction device can be communicatively connected to the control processing device. The human-computer interaction device obtains and displays a virtual map from the control processing device, and detects first interaction information input by the user. The human-computer interaction device is configured with, for example, a monocular / binocular camera, a touch screen, or a keyboard and mouse, and utilizes the human-computer interaction device to obtain the first interaction information. The first interaction information includes at least one of the following information set on the virtual map: a mark indicating the surgical operation location, a planned route, and a posture at the surgical operation location. For example, the first interaction information is determined based on at least one image containing a human posture, or by detecting at least one of a swipe, click, or press performed by the user on the human-computer interaction device. For example, the human posture includes a hand posture, an eye posture, or a lower limb posture. By detecting at least one image containing a human posture, the human-computer interaction device obtains a mark indicating the surgical operation location corresponding to the human posture and the posture after reaching the surgical operation location. For example, the human-computer interaction device determines the planned route and surgical operation position by detecting the trajectory swiped from the starting position mark on the touch screen and its trajectory end point; and determines the posture of the surgical robot 11 at the surgical operation position by detecting the posture option displayed at the end position on the touch screen.

[0051] Step S12: obtaining first interaction information, and generating a planned path for the surgical robot in the virtual map based on the first interaction information; the planned path is used for the surgical robot to move from the starting position to the surgical operation position in the operating room accordingly.

[0052] For details, please refer to Figure 3 Before the operation, the surgical robot 11 will be far away from the patient table 15 due to the need to prepare instruments and push the patient in. After a series of preparatory work is completed, the surgical robot 11 needs to move to a position closer to the patient table 15 and use the robotic arm to control the surgical instruments 14. The robotic arm can also control medical imaging equipment such as laparoscopes to assist doctors in successfully completing the operation. Therefore, before the operation, the surgical robot 11 needs to move to a more appropriate position next to the patient to position the robotic arm. In this embodiment, the navigation path is planned in advance before moving the surgical robot 11, and the surgical robot 11 can be controlled to automatically move according to the planned path to place the robotic arm in the correct surgical position and surgical posture.

[0053] The operator can use gestures to connect the mark of the starting position and the mark of the surgical operation position according to the displayed virtual map, and avoid obstacles on the virtual map during the connection process.

[0054] The control processing device generates a planned path for the surgical robot 11 in the virtual map based on the first interaction information. The human-machine interaction device can display the planned path in the virtual map, where the planned path connects the mark of the starting position and the mark of the surgical operation position in the virtual map.

[0055] The surgical robot 11 may be integrated with a robot driver. The control processing device may transmit the planned path to the driver of the surgical robot 11 via wireless communication technologies such as Bluetooth or mobile hotspots (Wi-Fi), so that the driver of the surgical robot 11 can control the surgical robot 11 to move autonomously from a starting position in a real-world scenario to a surgical operation position according to the planned path.

[0056] The robotic preoperative navigation method captures environmental information within the operating room to display a virtual map of the operating room. The operator uses gestures to pre-plan a navigation path on the virtual map. The surgical robot 11 follows the planned path, automatically and quickly moving into position and avoiding obstacles within the operating room. This reduces collisions and damage to the robot 11, making the method applicable to in-operation navigation. Furthermore, the patient control panel is integrated with the surgical robot 11 within the operating room. The planned path is used to control the robot 11 to automatically move from a starting position in the real world to the surgical operation position, eliminating the need for manual movement of the patient control panel. This reduces manual operation costs and saves time and effort.

[0057] In some examples, a virtual map can be superimposed on a real scene to facilitate the operator to plan a navigation path on the virtual map through gestures. In other examples, the virtual map can also be displayed on a display screen.

[0058] Specifically, the human-computer interaction device may include an augmented reality (AR) device. The operator may wear AR glasses or other AR devices. The AR device may be communicatively connected to the control processing device, and the control processing device may transmit a virtual map to the AR device, so that the virtual map is superimposed and displayed on the real scene through the AR device.

[0059] In some examples, the virtual map includes a grid marker. The starting position marker, the surgical operation position marker, and each obstacle can all be located at different intersections of the grid marker; or the starting position marker and the surgical operation position marker can be located at different intersections of the grid marker, with each obstacle distributed on the grid. The operator can connect the starting position marker and the surgical operation position marker at the intersection marked as obstacle-free in the grid marker on the virtual map to form a planned path, so that the surgical robot 11 can avoid obstacles in the operating room when moving along the planned path within the operating room.

[0060] For some examples, see Figure 4 , step S11 may specifically include steps S111 to S112.

[0061] Step S111 , obtaining environmental information using a plurality of pixel images captured at different locations in the operating room.

[0062] For details, please refer to Figure 5 The control processing device may include a camera 311 and a position adjustment unit 312. The camera 311 may be mounted on the position adjustment unit 312, which is used to adjust the camera's shooting angle. Through the multi-degree-of-freedom motion of the position adjustment unit 312, the camera 311 can rotate 360 degrees to scan the operating room environment, thereby capturing a more complete pixel image of the operating room. The control processing device uses the camera 311 to capture multiple pixel images captured at different locations within the operating room, and can obtain environmental information based on these pixel images. The control processing device can directly identify the surgical robot 11, the patient's lesion, and other obstacles in the multiple pixel images, thereby obtaining coordinate information for the starting position, the surgical operation position, and the obstacles. The operator can also be configured to determine the coordinate information for the surgical operation position based on these pixel images and input it to the control processing device via an input device. In other examples, the control processing device may include a depth data measurement device, which can directly measure the coordinates of various obstacles in the operating room, the current position of the surgical robot, and the position of the patient's lesion, thereby obtaining environmental information based on the measurement data.

[0063] Step S112: constructing a virtual map with grid marks based on the environmental information; wherein some intersections in the grid marks display the environmental information.

[0064] Optionally, the shape of the grid in the grid mark can be set according to actual needs. For example, the shape of the grid in the virtual map with the grid mark includes a triangle.

[0065] For some examples, see Figure 6 , step S112 may specifically include step S1121 to step S1122.

[0066] Step S1121 : Based on the position coordinate information of the surgical operation position and the position coordinate information of the obstacle and the starting position, coordinate information of a plurality of virtual points that do not overlap with the position coordinate information is expanded.

[0067] Step S1122: Using the coordinate information of each virtual point and the coordinate information of each position as the intersection of the grid marks, a virtual map with grid marks is generated.

[0068] Specifically, the control processing device can establish an initial virtual map of the operating room based on the environmental information. The initial virtual map can include an image that is proportionally reduced in size with the surgical robot 11 and each obstacle, and the distance between the surgical robot 11 and each obstacle can also be proportionally reduced. Of course, the surgical robot 11 and each obstacle in the initial virtual map can also be replaced by corresponding symbols. The initial virtual map does not contain a grid identifier. Then, the control processing device can establish a grid identifier based on the environmental information, and then integrate the grid identifier into the initial virtual map, so that the mark of the initial position, the mark of the surgical operation position and each obstacle on the initial virtual map overlap with the corresponding intersection on the grid identifier, thereby forming a virtual map with a grid identifier.

[0069] Here we take the triangle shape of the grid in the grid mark as an example to explain the method of forming the grid mark. Figure 7a Based on the coordinate information of the surgical operation position and the coordinate information of the obstacle and the starting position, the coordinate information of the expanded virtual point can be calculated using the point set triangulation (Delaunay) algorithm. A discrete point set is generated based on the coordinate information of the virtual point and the coordinate information of each position. Figure 7b , using the Delaunay algorithm to generate a triangular mesh image or mesh mark from a discrete point set. For further information, please refer to Figure 7c The grid identifier can also be simplified to reduce the amount of calculation and improve the efficiency of generating the subsequent planned path. It should be noted that the discrete points corresponding to the starting position and the surgical operation position in the grid identifier before and after simplification are all located at the intersection of the grids.

[0070] The following details the specific implementation process of the Delaunay algorithm using discrete point sets to generate grid identifiers:

[0071] A. Determine point p3: Assume there are two points p1 and p2. We call p3 a visible point on the line p1p2. We determine p3 based on the following three conditions: (1) p3 is to the right of the edge p1p2 (in clockwise order); (2) p3 is visible to p1, that is, the edge p1p3 does not intersect any constraining edge; and (3) p3 is visible to p2.

[0072] B. Determine the DT point: In a constrained Delaunay triangle, the vertex opposite an edge is called the DT point of that edge. The process of determining the DT point is as follows:

[0073] Step 1. Construct the circumscribed circle C(p1, p2, p3) of Δp1p2p3 and its grid bounding box B(C(p1, p2, p3));

[0074] Step 2. Visit each grid cell within the grid bounding box in turn: Search for unvisited grid cells and mark them as the currently visited grid cells. If a visible point p exists in a grid cell and ∠p1pp2 > ∠p1p3p2, set p3 = p1 and go to Step 1; otherwise, go to Step 3.

[0075] Step 3. If all grid cells in the current grid bounding box have been marked as the currently visited grid cells, that is, there is no visible point in C(p1, p2, p3), then p3 is the DT point of p1p2.

[0076] C. Algorithm Design:

[0077] Step 1. Take any outer boundary edge p1p2.

[0078] Step 2. Calculate the DT point p3 to form the constrained Delaunay triangle Δp1p2p3.

[0079] Step 3. If the newly generated edge p1p3 is not a constraint edge, if it is already in the stack, delete it from it; otherwise, put it into the stack; similarly, p3p2 can be processed.

[0080] Step 4. If the stack is not empty, take out an edge from it and go to Step 3; otherwise, the algorithm stops.

[0081] Then, the grid marks before or after simplification are integrated into the initial virtual map to obtain a virtual map with grid marks.

[0082] In other examples, a grid marker can be directly constructed based on the location coordinates in the environmental information. The grid spacing can be adjusted based on the interval between the location coordinates. In this way, a virtual map with grid markers can be obtained without expanding the virtual points.

[0083] For some examples, see Figure 8 , step S12 specifically includes steps S121 to S124.

[0084] Step S121: Acquire a gesture image.

[0085] Step S122: Obtain first interaction information based on the gesture image.

[0086] Step S123: Based on the first interaction information, determine at least one position change information relative to the starting position to obtain a planned path of the surgical robot.

[0087] Step S124: display the planned path on the virtual map.

[0088] For details, please refer to Figure 7d After the grid mark is integrated into the initial virtual map for display, the operator can use gestures to connect the points they want to connect in sequence starting from the starting position on the virtual map ( Figure 7d The bold line in the middle is the planned route formed by the operator connecting the desired points through gestures, where point A corresponds to the starting position and point B corresponds to the surgical operation position).

[0089] The first interactive information is the operator's hand posture information. Figure 9a , the upper end of the AR device 20 can be connected to the embedded binocular camera 21 through a printed circuit board (PCB) to collect the operator's gesture image and transmit the gesture image to the control processing device. The control processing device obtains the first interaction information based on the gesture image, and determines at least one posture change information relative to the starting position to obtain the planned path of the surgical robot. For example, when the planned path of the surgical robot 11 is a straight line, it is only necessary to determine one posture change information of the surgical robot 11 relative to the starting position. When the planned path of the surgical robot 11 is not a straight line, it is necessary to determine multiple posture change information of the surgical robot 11 during the movement. The control processing device can record all coordinates and routes on the planned path, and transmit the recorded data to the human-computer interaction device, so that the human-computer interaction device superimposes the planned path on the corresponding position on the virtual image for display.

[0090] in, Figure 9a The working principle of the binocular camera is as follows:

[0091] See also Figure 9b and 9c , the relative coordinate relationship between the AR device 20 and the binocular camera 21 is fixed. The camera coordinate system (X5, Y5, Z5) can be mapped to the display coordinate system (X3, Y3, Z3) through the mechanical position. The camera coordinate system (X5, Y5, Z5) and the world coordinate system (X0, Y0, Z0) can be mapped to each other through the rotation matrix R and the translation vector t. The mapping relationship is shown in Equation (1).

[0092]

[0093] Among them, (x c ,y c , z c ) is the coordinate value of point P in the camera coordinate system, (x w ,y w , z w ) is the coordinate value of point P in the world coordinate system.

[0094] Among them, according to Figure 9cThe geometric relationship in can be obtained that point P satisfies equations (2) to (5)

[0095]

[0096]

[0097]

[0098]

[0099] The binocular camera 21 includes a left camera and a right camera. The distance between the left camera and the right camera is b. The distance between the left camera and the right camera and the x-axis is f. The distance from the intersection of the line connecting the point P (x, y, z) and the left camera and the x-axis to the z-axis is x. l , the distance from the intersection of the line connecting point P(x, y, z) and the left camera with the x-axis to the y-axis is y l The distance from the intersection of the line connecting the right camera and point P with the x-axis to the straight line parallel to the z-axis where the right camera is located is x. r , the distance between point P and the straight line where the right camera is located and parallel to the z-axis is (xb).

[0100] For some examples, see Figure 10 , step S122 specifically includes steps S1221 to S1224.

[0101] Step S1221 , pre-processing the gesture image to obtain a gesture contour image.

[0102] Step S1222: extracting geometric moment features of the gesture contour image.

[0103] Step S1223 : Calculate the distance between gesture images at different angles at the same moment based on the geometric moment features of the gesture contour image.

[0104] Step S1224: Recognize the gesture at the moment based on the distance between gesture images at different angles at the same moment to obtain first interaction information.

[0105] Specifically, a recognition algorithm of geometric moment and edge detection can be used for preprocessing. First, the gesture image is binarized to obtain a gesture contour image. Then the geometric moment features of the gesture contour image are extracted. Specifically, four components of the seven vectors can be taken out, the edges of the image can be directly detected based on the grayscale image, and the boundary direction features of the image can be represented by a histogram. Finally, the distance between images is calculated by setting the weights of the geometric moment features, and then the gesture is recognized. This method uses two or more cameras (the left camera and the right camera in the binocular camera 21 in this embodiment) to acquire images at the same time, just like humans use two eyes and insects use multi-eye compound eyes to observe the world. By comparing the differences in images obtained by these different cameras at the same time, an algorithm is used to calculate the depth information, thereby achieving multi-perspective three-dimensional imaging.

[0106] In some examples, the preprocessing method may specifically include performing any one of histogram-based segmentation, local region information-based segmentation, and physical feature-based segmentation on the gesture image. The following provides examples of these three preprocessing methods.

[0107] See also Figure 11a In histogram-based segmentation, by preprocessing the histogram and tracing its contours, we can determine its peak-valley structure and thus find a reasonable segmentation threshold. As long as there are multiple peak structures in the image histogram and an ideal segmentation threshold is set, this method will produce good segmentation results.

[0108] See also Figure 11b In local area segmentation, contour extraction can typically be performed by obtaining the coordinates of boundary points through edge detection. A typical approach is to extract the coordinates of gesture boundary points using the eight-neighborhood search algorithm. Each point has eight adjacent points. If one of these points is used as the starting boundary point, the next boundary point must be within the eight-neighborhood neighborhood of that point. This algorithm then uses tracking to extract a closed contour.

[0109] In the segmentation based on physical features such as color, skin color is extracted through YCbCr color space and skin color modeling based on Gaussian model, and motion information analysis is performed through image difference operation to remove skin-like background in the image. This method ensures the accuracy of gesture segmentation in complex backgrounds.

[0110] For some examples, see Figure 12 , the robot preoperative navigation method also includes steps S13 to S16.

[0111] Step S13: Control the surgical robot to move from the starting position to the surgical operation position in the operating room based on the planned path.

[0112] Specifically, the driver of the surgical robot 11 can control the movement of the surgical robot 11 and / or control the positioning of the robotic arm of the surgical robot 11 based on the planned path, so that the control center of the robotic arm of the surgical robot 11 moves from the current position in the operating room to the surgical operation position in a real scenario.

[0113] Step S14: determining whether the motion state of the surgical robot needs to be adjusted during its motion.

[0114] Specifically, the robot's preoperative navigation system can automatically determine whether the motion state of the surgical robot 11 needs to be adjusted during its movement. When it is determined that the motion state needs to be adjusted, a prompt can be issued to the operator so that the operator can execute step S15. The operator can also determine whether the motion state of the surgical robot 11 needs to be adjusted during its movement. When it is determined that the motion state needs to be adjusted, step S15 can be executed. The need to adjust the motion state during the movement of the surgical robot 11 may include the distance between the surgical robot 11 and the obstacle being less than a preset safety distance and / or the motion trajectory of the surgical robot 11 deviating from the planned path, etc. If it is determined that the motion state does not need to be adjusted during the movement of the surgical robot 11, step S16 can be executed, and the surgical robot 11 continues to move along the planned path until it reaches the surgical operation position.

[0115] Step S15, obtain the second interaction information, and adjust the motion state of the surgical robot based on the second interaction information, and / or give prompt information. The second interaction information is detected by the human-computer interaction device, and is used to adjust part of the path of the surgical robot 11 during the movement of the surgical robot 11. The acquisition method is the same or similar to that of the first interaction information. In order to distinguish the first interaction information, in some examples, the second interaction information and the first interaction information are in different working modes of the surgical robot 11. For example, the first interaction information is obtained when the surgical robot 11 works in the stop mode, and the second interaction information is obtained when the surgical robot 11 works in the moving mode. In other examples, the second interaction information and the first interaction information are information represented by different detection signals or different image features. For example, the second interaction information is at least one image containing a left turn (or right turn) gesture. For another example, the second interaction information is the information generated by clicking the left turn (or right turn) button.

[0116] For details, please refer to Figure 13 and 9a, when it is determined that the motion state of the surgical robot 11 needs to be adjusted during its movement, the operator issues an adjustment gesture. A binocular camera 21 (i.e., a binocular vision module) can be set on the AR device 20 worn by the operator to collect the operator's adjustment gesture image, and transmit the adjustment gesture image to the control processing device. After the control processing device obtains the adjustment gesture image, it can obtain second interaction information based on the adjustment gesture image, and adjust the motion state of the surgical robot 11 based on the second interaction information. In other examples, the control processing device can give prompt information based on the second interaction information. For example, when the distance between the surgical robot 11 and the obstacle is less than a preset safety distance, the control processing device can issue a voice prompt. In other examples, steps S13 and S14 may not be executed by the control processing device, and the control processing device executes step S15 during the process of the surgical robot 11 moving from the starting position to the surgical operation position.

[0117] In some examples, when the robot preoperative navigation system determines in step S14 whether the motion state of the surgical robot 11 needs to be adjusted during the movement, the robot preoperative navigation method may also include: obtaining the position information of the surgical robot 11, and determining whether the motion state of the surgical robot 11 needs to be adjusted based on the position information of the surgical robot 11; if the motion state of the surgical robot 11 needs to be adjusted during the movement, outputting a prompt message.

[0118] Specifically, the position information of the surgical robot 11 may include the distance between the surgical robot 11 and the nearest obstacle during the real-time movement of the surgical robot 11. A distance measuring device such as an ultrasonic distance measuring device may be provided on the surgical robot 11 to obtain the position information of the surgical robot 11 and transmit the position information of the surgical robot 11 to the control processing device. After obtaining the position information of the surgical robot 11, the control processing device determines whether the motion state of the surgical robot 11 needs to be adjusted based on the position information of the surgical robot 11. For example, the configuration may determine that the motion state of the surgical robot 11 needs to be adjusted when the distance between the surgical robot 11 and the nearest obstacle is less than a preset safety distance. The surgical robot 11 or the control processing device may also be provided with a prompt device such as an alarm light or a buzzer. When the control processing device determines that the motion state of the surgical robot 11 needs to be adjusted during the movement, a prompt message may be output through the prompt device, so that the operator knows that an adjustment gesture is needed to control the surgical robot 11 to adjust its motion state to achieve the purpose of obstacle avoidance.

[0119] When the operator uses the adjustment gesture to control the surgical robot 11 to adjust its motion state, he or she can specifically control the direction of movement of the surgical robot 11, such as controlling the robot to turn left, turn right, move backward, move forward, etc. If the operator does not receive a prompt, the control processing device can control the surgical robot 11 to continue moving along the planned path until it reaches the surgical operation position. In other examples, the ranging device can further measure the position coordinates of the surgical robot 11 in real time and transmit the position coordinates of the surgical robot 11 to the control processing device. The control processing device can determine whether the position coordinates of the surgical robot 11 are on the planned path. If it deviates from the planned path, it can also control the prompt device to issue a prompt.

[0120] In some examples, the ranging device uses ultrasonic ranging to collect the robot's position information. The specific principles are as follows:

[0121] Ultrasonic ranging is achieved by using the ultrasonic pulse echo transit time method. Assuming that the time from the ultrasonic pulse being emitted by the sensor to the time it is received is t, and the propagation speed of ultrasonic waves in the air is c, the distance D from the sensor to the obstacle can be calculated using formula (6):

[0122] D=ct / 2 (6)

[0123] It can be understood that by using IO (Trig (control end)) to trigger the ranging and give a high-level signal of at least 10us, the module will automatically transmit 8 40khz square waves and automatically detect whether there is a signal returned. If so, the ultrasonic wave will output a high level. The duration of the high level is the round-trip time of the ultrasonic wave (which can be calculated using a timer). The test distance = (high level time * sound speed (340M / S)) / 2.

[0124] During the movement of the surgical robot 11, the prompt device can be configured to issue an alarm message when the distance between the surgical robot 11 and the nearest obstacle detected by the ranging device is less than a set safety distance. The operator can then adjust the movement direction of the patient's operating terminal through gestures. For example, if the distance is less than 5cm, it is determined to be an information prompt distance. When the distance between the patient's operating terminal and the obstacle is less than 5cm, the buzzer in the prompt device will automatically sound and the red light will flash. At this time, it is necessary to adjust the movement direction through gestures.

[0125] In some examples, when the operator determines in step S14 whether the surgical robot 11 needs to adjust its motion state during movement, the robot preoperative navigation method may also include: projecting the planned path in the operating room between the starting position and the surgical operation position according to the corresponding proportion.

[0126] Specifically, the control processing device generates a virtual map based on a real image of the operating room at a certain scale. The planned path displayed on the virtual map should also be scaled down by a certain scale compared to the actual path used when the surgical robot 11 is actually controlled to move between the starting position and the surgical operation position in the operating room. Therefore, when projecting the planned path into the real scene, it is necessary to amplify the planned path on the virtual map by a certain scale so that the planned path projected into the real scene can connect the starting position and the surgical operation position in the operating room. In this way, the operator can observe in real time whether the surgical robot 11 deviates from the planned path during its movement. If so, the operator can issue an adjustment gesture to control the surgical robot 11 to adjust its movement state.

[0127] For some examples, see Figure 14 The bottom of the surgical robot 11 can be provided with a mobile platform 111 and a driving wheel 112. The principle of the driver driving the robot to move is as follows: given a point P between two wheels (the distance between each wheel and the point P is 1), the wheel radius r, the angle θ between the robot direction and the X-axis direction, and the rotation speed of each wheel and The total velocity of the surgical robot 11 in the global reference frame is predicted by the forward kinematics model.

[0128] It should be understood that although Figure 1 、 4 The steps in the flowcharts of , 6, 8, 10 and 12 are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 、 4 , 6, 8, 10 and at least part of the steps 12 may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0129] This application also provides a robot preoperative navigation system. Figure 2 and Figure 15The robot preoperative navigation system includes a control processing device 31 and a human-computer interaction device 32. The control processing device 31 and the human-computer interaction device 32 are communicatively connected. The control processing device 31 is used to obtain environmental information in the operating room, and to generate a virtual map based on the environmental information. The virtual map is formed with a mark of the starting position of the surgical robot 11 and a mark of the surgical operation position. The human-computer interaction device 32 is used to display the virtual map and to obtain the user's first interaction information. The control processing device 31 is also used to generate a planned path for the surgical robot 11 in the virtual map based on the first interaction information. The planned path is used for the surgical robot 11 to move from the corresponding starting position in the operating room to the surgical operation position.

[0130] For some examples, see also Figure 9a The human-computer interaction device 32 includes an AR device 20, which is used to overlay and display a virtual map on a real scene.

[0131] In some examples, the human-computer interaction device 32 is configured to capture gesture images to obtain the first interaction information. Specifically, the AR device 20 may be provided with a binocular camera 21, which is configured to capture gesture images to obtain the first interaction information.

[0132] For some examples, see Figure 5 The control processing device 31 includes a position adjustment unit 312 and a camera 311. The camera 311 is set on the position adjustment unit 312. The position adjustment unit 312 is used to adjust the shooting angle of the camera 311. The camera 311 is used to collect multiple pixel images taken at different positions in the operating room to obtain environmental information.

[0133] In some examples, while the surgical robot 11 moves from the starting position to the surgical operation position in the operating room, the control processing device obtains second interaction information based on the gesture information, and adjusts the motion state of the surgical robot 11 based on the second interaction information, and / or gives prompt information.

[0134] Optionally, the control processing device 31 is also communicatively connected to the surgical robot 11, and the control processing device 31 is also used to control the surgical robot 11 to move from a starting position to a surgical operation position in the operating room based on a planned path; if the motion state of the surgical robot 11 needs to be adjusted during the movement, the binocular camera 21 obtains a gesture image, and the control processing device 31 obtains second interaction information based on the gesture information, and adjusts the motion state of the surgical robot 11 based on the second interaction information, and / or gives prompt information.

[0135] In some examples, the robotic preoperative navigation system also includes a distance measuring device (not shown in the figure), which is arranged on the surgical robot 11 and is used to measure the distance between the surgical robot 11 and the obstacle in real time; the control processing device 31 is also communicated with the distance measuring device, and the control processing device 31 is also used to determine whether the distance between the surgical robot 11 and the obstacle is less than the preset safety distance, and output a prompt message when the distance between the surgical robot 11 and the obstacle is less than the preset safety distance.

[0136] In some examples, the human-computer interaction device 32 is used to project the planned path in a corresponding proportion between the starting position and the surgical operation position in the operating room.

[0137] Furthermore, the robotic preoperative navigation system can also perform any step in the above-mentioned robotic preoperative navigation method. For the specific definition of the robotic preoperative navigation system, please refer to the definition of the robotic preoperative navigation method above, which will not be repeated here. The various modules in the above-mentioned robotic preoperative navigation system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. Among them, the computer device may include an AR device 20, a control processing device 31, a driver for the surgical robot 11, etc.

[0138] The present application also provides a storage medium having a computer program stored thereon, which implements the steps of the method described in any of the above embodiments when the computer program is executed by a processor.

[0139] The present application also provides a computer device, including a memory and a processor; a computer program that can be run on the processor is stored on the processor, and when the processor executes the computer program, the steps of the method described in any of the above embodiments are implemented.

[0140] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0141] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A robot preoperative navigation method, characterized in that: include: Acquiring environmental information in the operating room and displaying a virtual map based on the environmental information; The virtual map includes a mark for the surgical robot's starting position and a mark for the surgical operation position; and Acquiring first interaction information, and generating a planned path for the surgical robot in the virtual map based on the first interaction information; the planned path is used for the surgical robot to move from a starting position to a surgical operation position in the operating room; Generating a planned path for the surgical robot in the virtual map based on the first interaction information includes: Determining, based on the first interaction information, at least one position change information relative to the starting position to obtain a planned path for the surgical robot, wherein the first interaction information is generated by an operator connecting a marker of the starting position and a marker of the surgical operation position using a gesture according to a displayed virtual map, while avoiding obstacles on the virtual map during the connection process; The planned path is displayed on the virtual map.

2. The robot preoperative navigation method according to claim 1, characterized in that: The virtual map is superimposed and displayed on the real scene.

3. The method according to claim 1, characterized in that The virtual map includes a grid mark, and the mark of the starting position and the mark of the surgical operation position are both located at the intersection of the grid marks; the planned path connects the mark of the starting position and the mark of the surgical operation position through the intersection marked as free of obstacles in the grid marks.

4. The robot preoperative navigation method according to claim 1, characterized in that: The acquiring of environmental information in the operating room and displaying a virtual map based on the environmental information includes: Obtaining the environmental information using a plurality of pixel images captured at different locations within the operating room; A virtual map with grid marks is constructed and displayed based on the environmental information; wherein some intersections in the grid marks display the environmental information.

5. The robot preoperative navigation method according to claim 4, characterized in that: The environmental information includes the coordinate information of the obstacles in the operating room and the starting position; and the step of constructing and displaying a virtual map with grid markings based on the environmental information includes: Based on the position coordinate information of the surgical operation position and the position coordinate information of the obstacle and the starting position, coordinate information of a plurality of virtual points that do not overlap with the position coordinate information is expanded; The virtual map with the grid mark is generated and displayed using the coordinate information of each virtual point and the coordinate information of each position as the intersection of the grid mark.

6. The robot preoperative navigation method according to claim 4, characterized in that: The grid shape in the virtual map with the grid mark includes a triangle.

7. The robot preoperative navigation method according to any one of claims 1 to 6, characterized in that: During the process of the surgical robot moving from the starting position to the surgical operation position, the method further includes: Acquire second interaction information, and adjust the motion state of the surgical robot and / or provide prompt information based on the second interaction information.

8. The robot preoperative navigation method according to claim 2, characterized in that: The robot preoperative navigation method further includes: The planned path is projected in the operating room between the starting position and the surgical operation position according to a corresponding scale.

9. The robot preoperative navigation method according to claim 7, characterized in that: The need to adjust the motion state of the surgical robot during its movement includes: the distance between the surgical robot and the obstacle is less than a preset safety distance and / or the motion trajectory of the surgical robot deviates from the planned path.

10. The robot preoperative navigation method according to claim 1, characterized in that: Also includes: The movement of the surgical robot and / or the positioning of the robotic arm of the surgical robot are controlled based on the planned path.

11. A robotic preoperative navigation system, characterized in that: include: A control processing device and a human-computer interaction device, wherein the control processing device and the human-computer interaction device are communicatively connected; The human-computer interaction device is used to display a virtual map and to obtain first interaction information from a user; The control processing device is used to execute the robot preoperative navigation method according to any one of claims 1 to 10.

12. The robotic preoperative navigation system according to claim 11, characterized in that: The human-computer interaction device includes an AR device, and the AR device is used to overlay and display the virtual map on a real scene.

13. The robotic preoperative navigation system according to claim 11, characterized in that: The control processing device includes a position adjustment unit and a camera. The camera is set on the position adjustment unit. The position adjustment unit is used to adjust the shooting angle of the camera. The camera is used to collect multiple pixel images taken at different positions in the operating room to obtain the environmental information.

14. The robotic preoperative navigation system according to claim 11, characterized in that: The human-computer interaction device projects the planned path in the operating room between the starting position and the surgical operation position according to a corresponding proportion.

15. The robotic preoperative navigation system according to any one of claims 11 to 14, characterized in that: During the movement of the surgical robot from the starting position to the surgical operation position in the operating room; The control processing device obtains second interaction information based on the gesture information, and adjusts the motion state of the surgical robot and / or provides prompt information based on the second interaction information.

16. The robotic preoperative navigation system according to claim 11, characterized in that: The human-computer interaction device obtains the first interaction information by collecting gesture images.

17. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

18. A computer device comprising a memory and a processor; the processor stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

Images