Computer-readable storage medium, electronic device, path planning, and robot system

Abstract

The present invention provides a computer-readable storage medium, an electronic device, a path planning and a robot system. A program is stored on the computer-readable storage medium. When the program is executed, the following steps are performed: establishing a first three-dimensional model according to the first image information of the surgical area; establishing a second three-dimensional model according to the second image information of the local area of the surgical area, where the second three-dimensional model is used to plan the expected surgical operation position; mapping the expected surgical operation position to the first three-dimensional model, and obtaining the target surgical operation position on the first three-dimensional model; generating a movement path of the surgical instrument according to the starting position of the surgical instrument, the first three-dimensional model and the target surgical operation position. When the computer-readable storage medium is applied to a surgical robot system, it combines the information of the entire preoperative range of the surgical area and the intraoperative local information to perform path planning of the surgical instrument, making the planned movement path more in line with the actual situation and improving the surgical efficiency and safety.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and particularly to a computer-readable storage medium, an electronic device, a path planning, and a robot system. Background Art

[0002] The design concept of a surgical robot is to precisely perform complex surgical operations in a minimally invasive manner. Surgical robots have emerged in the face of various limitations in traditional surgical operations. Surgical robots break through the limitations of the human eye and can use stereoscopic imaging technology to present the internal organs of the human body more clearly to the operator. And for some narrow areas where a person's hand cannot reach, the surgical robot can still control the surgical instrument to complete moving, swinging, clamping, and 360° rotation, and can avoid jitter, improve the surgical precision, and further achieve the advantages of small incisions, less bleeding, fast postoperative recovery, and greatly shortening the postoperative hospitalization time of the surgical subject. Therefore, surgical robots are deeply favored by the majority of doctors and patients and are widely used in various clinical surgeries.

[0003] In the prior art, when using a surgical robot to perform a surgical operation, the movement path of the surgical instrument is usually planned by using preoperative medical images such as CT images, MRI images, B-ultrasound images, etc., and then the doctor controls the movement of the tool arm of the surgical robot to drive the surgical instrument connected to the end of the tool arm to move along the movement path to the surgical operation position and perform the corresponding surgical operation. This solution only plans the movement path of the surgical instrument by using preoperative medical images and does not consider that the actual situation of the patient during the operation may be different from the situation shown in the medical images. For example, the actual surgical boundary is larger than the boundary shown in the medical images, and the movement path that was not blocked in the medical images is blocked in the patient's body, resulting in the pre-planned movement path not being able to adapt to the actual situation. Summary of the Invention

[0004] The purpose of the present invention is to provide a computer-readable storage medium, an electronic device, a path planning, and a robot system, aiming to plan the movement path of the surgical instrument that conforms to the actual situation during the operation, so as to improve the efficiency and safety of the surgical operation.

[0005] To achieve the above object, the present invention provides a computer-readable storage medium, on which a program is stored. When the program is executed, the following steps are performed:

[0006] Establish a first three-dimensional model according to the first image information of the surgical area before the operation;

[0007] Establish a second three-dimensional model according to the second image information of the surgical area during the operation, and the second three-dimensional model is used to plan the expected surgical operation position;

[0008] Map the expected surgical operation position to the first three-dimensional model, and obtain the target surgical operation position on the first three-dimensional model;

[0009] Generate a movement path of the surgical instrument based on the starting position of the surgical instrument, the first three-dimensional model, and the target surgical operation position, so that when the surgical instrument moves along the movement path, it can move from the starting position to the target surgical operation position.

[0010] Optionally, the first image information is the image information within the entire range of the surgical area; the second image information is the image information of a local area of the surgical area.

[0011] Optionally, the step of establishing the first three-dimensional model according to the first image information of the surgical area before surgery specifically includes:

[0012] Perform gray-scale normalization processing and binary processing on the first image information, and extract feature values;

[0013] Perform feature value comparison in the model library and identify the tissues in the first image information;

[0014] Establish the first three-dimensional model according to the identified tissues.

[0015] Optionally, the step of establishing the second three-dimensional model according to the second image information of the surgical area during surgery specifically includes:

[0016] Perform gray-scale normalization processing and binary processing on the second image information, and extract feature values;

[0017] Perform feature value comparison in the model library to identify the tissues in the second image information;

[0018] Establish the second three-dimensional model according to the identified tissues.

[0019] Optionally, the step of mapping the expected surgical operation position to the first three-dimensional model and obtaining the target surgical operation position on the first three-dimensional model specifically includes:

[0020] Register the first three-dimensional model and the second three-dimensional model, and establish a conversion relationship between the coordinate system of the first three-dimensional model and the coordinate system of the second three-dimensional model;

[0021] Obtain the coordinates of the expected surgical operation position in the coordinate system of the first three-dimensional model based on the coordinates of the expected surgical operation position in the coordinate system of the second three-dimensional model and the conversion relationship, as the target surgical operation position.

[0022] Optionally, the program further performs the following steps:

[0023] Generate a first intervention prompt message to prompt the planning of the expected surgical operation position.

[0024] Optionally, when planning the movement path, the program also uses a first preset condition as a limiting factor when generating the movement path, and the first preset condition includes at least one of the following:

[0025] The movement path is a loop passing through at least one of the target operation positions;

[0026] The movement path is at a predetermined distance from the tissue shown in the first three-dimensional model, and in the planar projection, the movement path is arranged along the surface of the tissue;

[0027] The movement path does not pass through the interior of the tissue shown in the first three-dimensional model.

[0028] Optionally, the program also performs the following steps:

[0029] Send the movement path to a display device for display.

[0030] Optionally, the program plans multiple movement paths; the program uses a second preset condition as a limiting factor when displaying the movement path, and the second preset condition includes any one of the length of the path, the length of the arrival time, and the difficulty of operation.

[0031] Optionally, the program also performs the following steps:

[0032] Generate a second intervention prompt message to prompt the confirmation of whether the movement path meets the requirements.

[0033] Optionally, when the movement path does not meet the requirements and the expected surgical operation position is updated, the program also performs the following steps:

[0034] Map the updated expected surgical operation position to the first three-dimensional model and obtain the updated target surgical operation position; and,

[0035] Update the movement path according to the starting position, the first three-dimensional model, and the updated target operation position.

[0036] Optionally, the program plans multiple movement paths, and the program also performs the following steps:

[0037] Generate a third intervention prompt message to prompt the selection of a movement path that meets the requirements.

[0038] Optionally, a variety of path generation algorithms are stored on the computer-readable storage medium, and the program further performs the following steps:

[0039] Generate a fourth intervention prompt message to prompt the selection of a path generation algorithm.

[0040] To achieve the above object, the present invention further provides an electronic device, including a processor and the computer-readable storage medium as described in any one of the preceding items, and the processor is configured to execute the program stored on the computer-readable storage medium.

[0041] To achieve the above object, the present invention further provides a path planning system, including:

[0042] An image acquisition device for acquiring first image information of the surgical area and second image information of a local area of the surgical area; and,

[0043] A control unit communicatively connected to the image acquisition device and configured to execute the program stored on the computer-readable storage medium as described in any one of the preceding items.

[0044] Optionally, the path planning device further includes a human-computer interaction unit communicatively connected to the control unit and including a display module for displaying the motion path.

[0045] Optionally, when the program is executed, the program further generates an intervention prompt message to prompt the execution of an intervention operation; the display module also displays the intervention prompt message, and the human-computer interaction unit further includes an input module for inputting an intervention instruction according to the intervention prompt message.

[0046] To achieve the above object, the present invention further provides a surgical robot system, including:

[0047] An image arm,

[0048] An image acquisition device disposed on the image arm and configured to acquire first image information of the surgical area and second image information of a local area of the surgical area; and,

[0049] A control unit communicatively connected to the image acquisition device and configured to execute the program stored on the computer-readable storage medium as described in any one of the preceding items.

[0050] Compared with the prior art, the computer-readable storage medium, electronic device, path planning and robot system of the present invention have the following advantages:

[0051] A program is stored on the aforementioned computer-readable storage medium, and when the program is executed, the following steps are performed: establishing a first three-dimensional model based on the first image information of the surgical area before surgery; establishing a second three-dimensional model based on the second image information of the surgical area during surgery, where the second three-dimensional model is used to plan the expected surgical operation position; mapping the expected surgical operation position to the first three-dimensional model to obtain the target surgical operation position on the first three-dimensional model; generating a movement path of the surgical instrument according to the starting position of the surgical instrument, the first three-dimensional model, and the target surgical operation position, so that when the surgical instrument moves along the movement path, it can move from the starting position to the target surgical operation position. The first image information is usually the image information of the entire range of the surgical area collected before surgery, and the second image information is the real-time image information of the local area of the surgical area during surgery. That is to say, the present invention combines the first three-dimensional model of the surgical area before surgery and the real-time three-dimensional model during surgery to plan the movement path of the surgical instrument, so that the movement path can better adapt to the actual situation during surgery, thereby improving the surgical efficiency and safety.

[0052] Further, the program also performs the following steps: generating a first intervention prompt message to prompt the planning of the expected surgical operation position. That is, the expected surgical operation position can be planned by the operator, taking into account the operator's need to adjust the plan according to the actual situation during surgery, and can also combine the operator's personal experience, with higher flexibility, better adapt to the actual situation during surgery, and is also more convenient for the operator to operate. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention. Among them:

[0054] Figure 1 is a schematic diagram of the application scenario of the surgical robot system provided by the present invention according to an embodiment;

[0055] Figure 2 is a schematic diagram of the first display device of the surgical robot system provided by the present invention according to an embodiment;

[0056] Figure 3 is a schematic diagram of the tool arm of the surgical robot system provided by the present invention according to an embodiment and the structure of the end surgical instrument connected to the tool arm;

[0057] Figure 4 is a flowchart of planning the movement path of the surgical instrument when the surgical robot system provided by the present invention performs surgery.

[0058] Figure 5Schematic diagram of the image acquisition device of the surgical robot system provided by the present invention according to an embodiment for acquiring first image information and second image information;

[0059] Figure 6 Block diagram of the control unit of the surgical robot system provided by the present invention according to an embodiment;

[0060] Figure 7 Schematic diagram of the connection relationship among the image acquisition device, the control unit, and the human - machine interaction unit of the surgical robot system provided by the present invention according to an embodiment;

[0061] Figure 8 Flowchart of the control unit of the surgical robot system provided by the present invention according to an embodiment for identifying tissues in the first image information or the second image information and then establishing a corresponding three - dimensional model;

[0062] Figure 9 Schematic diagram of the three - dimensional model established by the control unit of the surgical robot system provided by the present invention according to an embodiment, showing the model of the lesion in the illustration;

[0063] Figure 10 Schematic diagram of the process of obtaining the movement path of the surgical instrument when the surgical robot system provided by the present invention according to a specific embodiment performs a surgical operation. In the illustration, the control unit generates a first intervention prompt message, and the operator plans and determines the expected surgical operation position;

[0064] Figure 11 Flowchart of the control unit of the surgical robot system provided by the present invention according to an embodiment for planning a path according to the ant colony algorithm;

[0065] Figure 12 Schematic diagram of the content displayed by the display module of the surgical robot system provided by the present invention according to an embodiment, showing a first preset condition for the operator to select;

[0066] Figure 13 Schematic diagram of the process of obtaining the movement path of the surgical instrument when the surgical robot system provided by the present invention according to an embodiment performs a surgical operation. In the illustration, it shows the situation where the movement path planned by the control unit does not meet the requirements, and the operator re - plans the expected surgical operation position to update the movement path;

[0067] Figure 14 Schematic diagram of the content displayed by the display module of the surgical robot system provided by the present invention according to an embodiment, showing action selection items for the operator to select;

[0068] Figure 15FIG. 0 is a schematic diagram of the process of obtaining the movement path of a surgical instrument when the surgical robot system provided by an embodiment of the present invention performs a surgical operation. The figure shows that the control unit plans multiple movement paths, and the surgeon selects a movement path that meets the requirements, or all movement paths do not meet the requirements, and the surgeon re-plans the expected surgical operation position to update the movement path.

[0069] Figure 16 FIG. 4 is a schematic diagram of the content displayed by the display module of the surgical robot system provided by an embodiment of the present invention. The figure shows a second preset condition for the surgeon to select;

[0070] Figure 17 FIG. 8 is a schematic diagram of the content displayed by the display module of the surgical robot system provided by an embodiment of the present invention. The figure shows a path planning method for the surgeon to select. Detailed Embodiments

[0071] The following uses specific specific examples to illustrate the embodiments of the present invention. Those skilled in the art can easily understand the other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the diagrams provided in this embodiment only illustrate the basic concept of the present invention in a schematic manner. Therefore, only the components related to the present invention are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0072] In addition, each of the following embodiments of the description has one or more technical features. However, this does not mean that those who use the present invention must implement all the technical features in any one embodiment at the same time, or can only separately implement some or all of the technical features in different embodiments. In other words, on the premise that implementation is possible, those skilled in the art can, according to the disclosure of the present invention and depending on the design specifications or implementation requirements, selectively implement some or all of the technical features in any one embodiment, or selectively implement the combination of some or all of the technical features in multiple embodiments, thereby increasing the flexibility when implementing the present invention.

[0073] As used in this specification, the singular forms "a", "an", and "the" include plural referents, and the plural form "pluralities" includes more than two referents, unless the context clearly dictates otherwise. As used in this specification, the term "or" is generally used in the sense of including "and / or", unless the context clearly dictates otherwise, and the terms "mounted", "connected", and "coupled" should be construed broadly. For example, it may be a fixed connection, a detachable connection, or an integral connection. It may be a mechanical connection or an electrical connection. It may be directly connected or indirectly connected through an intermediate medium, and may be the internal communication of two components or the interaction relationship between two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0074] To make the objectives, advantages, and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. It should be noted that the drawings are all in very simplified forms and use non-precise scales, and are only used to facilitate and clearly assist in explaining the objectives of the embodiments of the present invention. The same or similar reference numerals in the drawings represent the same or similar components.

[0075] Figure 1 FIG. shows a schematic diagram of an application scenario of the surgical robot system provided by an embodiment of the present invention. Please refer to Figure 1 , the surgical robot system includes a control end and an execution end. The control end includes a doctor console and a doctor-side control device 10 provided on the doctor console. The doctor-side control device 10 includes an immersive display device 11. The execution end includes a patient-side control device (not shown in the figure), a surgical operation device 20, a first display device 30 (such as Figure 1 and Figure 2 shown) and other devices. Among them, the patient-side control device can be integrated with the surgical operation device 20, and a robotic arm is mounted on the surgical operation device 20. The robotic arm includes an image arm 21 and a tool arm 22. The tool arm 22 is used to mount a surgical instrument 40 (such as shown in FIGS. 1 and Figure 3 shown), and the surgical instrument 40 is used to insert into the surgical area of the patient and perform surgical operations. The image arm 21 is used to mount an image acquisition device 50, and the image acquisition device 50 is used to acquire image information of an area of interest or a device (such as the first image information of the surgical area and the second image information of a local area of the surgical area described hereinafter). The surgical robot system further includes a control unit 60 (please refer to Figure 6(the markings in), the control unit 60 is communicatively connected to the image arm 21, the tool arm 22, the image acquisition device 50, the first display device 30, and the immersive display device 11. The control unit 60 can be arranged at the patient-side control device, or at the doctor-side control device, or a part of it can be arranged at the patient-side control device and another part at the doctor-side control device, or it can be completely independent of the patient-side control device and the doctor-side control device. That is to say, the present invention does not limit the specific setting mode of the control unit 60, as long as it can perform relevant functions.

[0076] When performing a surgical operation using the surgical robot system, it is necessary to plan the movement path of the surgical instrument 40, and then drive the surgical instrument 40 to move along the movement path by controlling the movement of the tool arm 22 until the surgical instrument 40 reaches the surgical operation position. In this embodiment, please refer to Figure 4 , the method for planning the movement path includes the following steps:

[0077] Step S1: Establish a first three-dimensional model based on the first image information of the surgical area before the operation, and establish a second three-dimensional model based on the second image information of the surgical area during the operation.

[0078] Step S2: Plan the expected surgical operation position on the second three-dimensional model.

[0079] Step S3: Map the expected surgical operation position to the first three-dimensional model, and obtain the target surgical operation position on the first three-dimensional model.

[0080] And, Step S4: Plan the movement path of the surgical instrument 40 according to the starting position of the surgical instrument 40, the first three-dimensional model, and the target surgical operation position. In this way, when the surgical instrument 40 moves along the movement path, it can move from the starting position to the target surgical operation position.

[0081] Among them, the "surgical area" is determined according to the specific operation. For example, in laparoscopic surgery, the surgical area refers to the abdominal cavity, and in thoracoscopic surgery, the surgical area refers to the thoracic cavity. Hereinafter, laparoscopic surgery, that is, the surgical area is the abdominal cavity, will be taken as an example for description. The first image information is actually the image information of the entire range of the abdominal cavity. The second image information is generally the image information of a local area within the abdominal cavity. That is to say, the method for planning the movement path provided by the embodiments of the present invention combines the image information of the entire range of the surgical area before the operation and the real-time image information of the local area during the operation to plan the movement path of the surgical instrument 40, so that the movement path can better adapt to the actual situation during the operation, thereby improving the surgical efficiency and safety.

[0082] Optionally, both the first image information and the second image information are acquired by the image acquisition device 50. In an exemplary embodiment, as Figure 5 shown, the image acquisition device 50 is an endoscope. The image acquisition device 50 is inserted into the body to collect the first image information and the second image information. Those skilled in the art know that when performing a laparoscopic surgery, holes need to be made at a first hole position and a second hole position on the patient's abdomen. Generally, the endoscope first enters the abdominal cavity from the first hole position, and then the surgical instrument 40 enters the abdominal cavity from the second hole position. After the endoscope enters the abdominal cavity, it rotates within the maximum rotation angle allowed by it to obtain image information of the entire range within the abdominal cavity, and this image information can be used as the first image information. After the surgical instrument 40 enters the abdominal cavity, the operator drives the tool arm 21 to move under the guidance of the image information provided by the endoscope, and drives the surgical instrument 40 to move to the lesion (i.e., the position where the surgical operation needs to be performed). The field of view of the endoscope is limited. Therefore, during the process of moving the surgical instrument 40, the operator needs to adjust the position and pose (i.e., position and attitude) of the endoscope in real time according to the need, and obtain image information of different regions within the abdominal cavity to better guide the movement of the surgical instrument 40. In this embodiment, the image information obtained by the endoscope after each position and pose adjustment can be used as the second image information. That is to say, in the embodiments of the present invention, the "preoperative" refers to the time range before the surgical instrument 40 enters the patient's body, and the "intraoperative" includes the process of the surgical instrument 40 moving from the second hole position to the lesion and the time range after the surgical instrument 40 reaches the lesion.

[0083] As Figure 6 shown, in this embodiment, the control unit 60 may include an identification module 61 and a modeling module 62. As Figure 7 shown, the identification module 61 is communicatively connected to the image acquisition device 50, and the modeling module 62 is communicatively connected to the identification module 61. The step S1 may be executed by the control unit 60. The specific steps are as Figure 8 shown, including:

[0084] Step S1a: The identification module 61 first performs gray normalization processing and binarization processing on the image information by a conventional method, and then extracts eigenvalues from the processed image information through a tissue eigenvalue algorithm.

[0085] Step S1b: The identification module 61 also compares the eigenvalues obtained in step S1a with the eigenvalues pre-stored in the model library, and identifies the tissue in the image information. The model library stores models of various tissues or lesions in the form of eigenvalues plus names.

[0086] And, step S1c: The modeling module 62 establishes a three-dimensional model as shown in Figure 9 according to the tissue identified by the identification module 61. Those skilled in the art can understand that Figure 9 only schematically shows the established three-dimensional model, in which the three-dimensional model 1 of the lesion is shown.

[0087] That is, when the control unit 60 executes the steps S1a to S1c for the first image information, the first three-dimensional model is established, and when the control unit 60 executes the steps S1a to S1c for the second image information, the second three-dimensional model is established.

[0088] The second three-dimensional model is used to plan the expected surgical operation position. In the embodiment of the present invention, the expected surgical operation position can be planned in any suitable manner. In an alternative manner, the control unit 60 executes the steps of planning the expected surgical operation position. In another alternative manner, the operator plans the expected operation position. The advantage of doing so is that it can take into account the operator's need to adjust the plan according to the actual situation during the operation, and can also combine the operator's personal experience, which has higher flexibility, can better adapt to the actual situation during the operation, and is also more convenient for the operator to operate.

[0089] When the expected operation position is planned by the operator, please refer to Figure 10 , after the step S1 and before the step S2, the path planning method further includes a step S5: The control unit 60 generates a first intervention prompt message to prompt the operator to plan the expected surgical operation position. At the same time, the step S2 is specifically: the operator plans and inputs the expected surgical operation position on the second three-dimensional model. Therefore, please return to refer to Figure 7 , some components of the surgical robot system constitute a human-computer interaction unit. In an alternative implementation manner, the human-computer interaction unit includes a display module and a delivery module. The display module includes the first display device 30 and / or the immersive display device 11. The display module is used to display the first intervention prompt message, and the first intervention message is, for example, displayed as an option of "input the expected operation position". The input module is used to input the expected surgical operation position, which can be a mouse, a keyboard, a mechanical button, or a virtual button set on the display module, etc. In addition, the display module is also communicatively connected to the image acquisition device 50 (i.e., the endoscope) to display the image information collected by the image acquisition device 50.

[0090] Please return to refer to Figure 6, the control unit 60 further includes a conversion module 63, and the conversion module 63 is communicatively connected to the modeling module 62. The step S3 can be executed by the conversion module 63, as Figure 4 shown, the step S3 specifically includes step S3a: registering the first three-dimensional model and the second three-dimensional model to establish a mapping relationship between the coordinate system of the first three-dimensional model and the coordinate system of the second three-dimensional model. And step S3b: obtaining the coordinates of the expected surgical operation position in the coordinate system of the first three-dimensional model according to the coordinates of the expected surgical operation position in the coordinate system of the second three-dimensional model and the conversion relationship, and using the position where the coordinates are located as the target surgical operation position. Registering the first three-dimensional model and the second three-dimensional model to obtain the conversion relationship between the coordinate system of the first three-dimensional model and the coordinate system of the second three-dimensional model is a content that those skilled in the art can know, so it will not be introduced in detail here.

[0091] In the step S4, the starting position is set as needed, and it can be the position where the second hole is located. And, please refer to Figure 6 and Figure 7 , the control unit 60 further includes a path generation unit 64, the path generation unit is communicatively connected to the modeling module 62 and the conversion module 63, and is used to generate the motion path, that is, the step S4 can be executed by the control unit 60. At least one path generation algorithm can be pre-stored in the path generation unit 64 to generate the motion path, and optional path generation algorithms include the ant colony algorithm.

[0092] The principle of the ant colony algorithm is as follows: Let the number of ants in the entire ant colony be m, and the total number of target points be n. Here, a target point is a target operation position. The distance between target point i and target point j is d ij , at time t, the pheromone concentration on the path between target point i and target point j is τ ij (t). Initially, the pheromone concentrations on each path are the same, and are all set to τ ij (0)=τ0. Ant k determines the next target point to visit according to the pheromone concentration on the path between each target point, in order to represent the transfer probability that ant k walks from target point i to target point j at time t. The transfer probability formula is:

[0093]

[0094] where η ij (t) is the expected degree for ant k to transfer from target point i to target point j, α represents the proportion of τ, β represents the proportion of η, allowω k represents the set of target points to be visited by ant k, and initially, allowωk It includes all the targets except the starting target. When the ant traverses all the targets, allowω k is empty. After completing one cycle (i.e., traversing all the targets once), the path traveled by each ant is a solution.

[0095] Since while the ant releases pheromone, the pheromone on the paths between each target gradually disappears (i.e., gradually volatilizes), and ρ (0 < ρ < 1) represents the degree of pheromone volatilization. After all the ants complete one cycle, the pheromone concentration of each target needs to be updated, and the update formula is as follows:

[0096]

[0097] Among them, represents the pheromone concentration released by ant k on the path between target i and target j; Δτ ij represents the sum of the pheromone concentrations released by all the ants on the path between target i and target j. Preferably, when using the ant colony algorithm to plan the path and selecting the pheromone release model, it is based on the principle that the shorter the path, the higher the pheromone concentration released.

[0098] Based on the above principle, the process of adopting the ant colony algorithm is as Figure 11 shown, specifically including:

[0099] Step S4a: Calculate the initial path distances between each target and initialize parameters such as the pheromone concentration.

[0100] Step S4b: Equivalent the multi-objective path planning problem in three-dimensional space to the traveling salesman problem with a known path length, and randomly place the ants at different targets as the starting points, and determine the next target to visit according to the transfer probability formula until all the ants visit all the targets.

[0101] Step S4c: Calculate the path lengths passed by each ant, update the pheromone concentration on the paths of each target according to the update formula, and record the optimal solution in the current update times.

[0102] Step S4d: Judge the number of update times that have been carried out. If it has not reached the preset total number of updates, repeat Step S4a, Step S4b, and Step S4c. If it has reached the preset total number of updates, execute Step S4e.

[0103] Step S4e: Take the solution with the minimum total path length among the optimal solutions at each update as the final solution, that is, as the finally obtained motion path.

[0104] In an alternative embodiment, other algorithms such as genetic algorithms can also be used to generate the motion path. The genetic algorithm is an existing path generation algorithm, and those skilled in the art can know its specific execution steps, which will not be elaborated here.

[0105] Further, when performing path planning, the control unit 60 can also use the first preset condition as a limiting factor when generating the motion path. The first preset condition includes but is not limited to at least one of the following (1) to (3): (1) the motion path is a loop passing through at least one of the target operation positions; (2) the motion path is at a predetermined distance from the tissue shown in the first three-dimensional model, and in the planar projection, the surgical path is arranged along the surface of the tissue; (3) the motion path does not pass through the interior of the tissue shown in the first three-dimensional model. Among them, (2) and (3) can ensure that the surgical instrument 40 does not collide with the tissue when moving along the motion path, ensuring safety. In some cases, the control unit 60 has already preset the first preset condition as the limiting factor when generating the motion path. In other cases, the operator needs to select a suitable first preset condition as the limiting factor when generating the motion path. Here, before executing the step S4, the control unit 60 also executes step S6 (not shown in the figure): generating a fifth intervention prompt message to prompt the selection of the limiting factor when generating the motion path. As Figure 12 shown, this step can be implemented by displaying options of multiple first preset conditions on the display module for the operator to select. Then the operator executes step S7 (not shown in the figure): selecting at least one of the first preset conditions as the limiting factor when generating the motion path.

[0106] Optionally, as Figure 10 and Figure 13 shown, the motion path planning method further includes step S8, which is executed after step S4 and includes: sending the motion path to the display module for display, so that the operator can intuitively view the motion path.

[0107] In some embodiments, the control unit 60 can plan a motion path that meets the requirements at one time. However, in other embodiments, the motion path planned by the control unit 60 according to the current target operation position may not meet the requirements. Therefore, please refer back to Figure 10 and Figure 13 , after step S8, the planning method further includes step S9: confirming whether the motion path meets the requirements. Specifically, as Figure 10 and Figure 13As shown, the control unit 60 may first execute step S9a: generating a second intervention prompt message to prompt for confirmation whether the movement path meets the requirements. Here, the movement path meeting the requirements includes that the movement path is not blocked, so that the surgical instrument 40 can move along the movement path without touching tissues or lesions. When the operator confirms that the movement path meets the requirements, as Figure 10 and Figure 13 shown, the operator executes step S9b: inputting a confirmation instruction to end the path planning process. When the operator confirms that the movement path does not meet the requirements, the expected surgical operation position can be updated. That is to say, as Figure 13 shown, the planning method may further include step S10: the control unit 60 generates a sixth intervention prompt message to prompt for updating the expected surgical operation position. The step S9a and the step S10 can be implemented by the display module displaying operation selection items for the operator (as Figure 14 shown), Figure 14 The operation selection items shown in include confirming the movement path, inputting a new expected surgical operation position, etc. The option of "inputting a new expected surgical operation position" is the display form of the sixth intervention prompt message. When the operator executes the step S9b, the operator only needs to select the option of "confirming the movement path". When the operator selects the option of "inputting a new expected surgical operation position", it indicates that the operator determines that the movement path does not meet the requirements and the expected operation position needs to be updated. After the operator selects the option of "inputting a new expected surgical operation position", the operator will execute step S11: inputting a new expected surgical operation position. Then the movement path planning method will return to the step S3 and subsequent steps. That is, the control unit 60 maps the updated expected surgical operation position to the first three-dimensional model, obtains the updated target surgical operation position, and updates the movement path according to the starting position, the first three-dimensional model, and the updated target operation position.

[0108] Alternatively, in some other embodiments, the control unit 60 can plan multiple movement paths at one time. For example, multiple path planning methods are pre-stored in the control unit 60 to Figure 15Taking the example shown, the ant colony algorithm and the genetic algorithm are pre-stored in the control unit 60. When performing path planning, the control unit 60 can simultaneously use two path generation algorithm methods to obtain two different motion paths. In this case, step S9 may include step S9c and step S9d (that is, step S9 does not include step S9a and step S9d). Step S9c is: generating a third intervention prompt message to prompt the selection of a motion path that meets the requirements. Step S9d is: the operator inputs a selection instruction to select a motion path that meets the requirements. It can be understood that in this embodiment, the third intervention prompt message is displayed on the display module by displaying two "confirm motion path" options, and the two "confirm motion path" options are respectively set for the two motion paths. If the operator determines that both motion paths do not meet the requirements, the operator will click on the "input new expected surgical operation position" option. If the operator determines that at least one motion path meets the requirements, the operator executes step S9d by clicking on one "confirm motion path" option, and the path planning process ends.

[0109] In addition, when the control unit 60 plans multiple motion paths, the control unit 60 uses the second preset condition as a limiting factor when the display module displays the motion paths. The second preset condition includes but is not limited to any one of the length of the motion path, the length of the arrival time, and the difficulty of the motion. For example, when the length of the path is used as the limiting factor, the display module can display all the motion paths in ascending order of path length. When the length of the arrival time is used as the limiting factor, the display module can display all the motion paths in ascending order of arrival time. When the difficulty of the motion is used as the limiting factor, the display module displays all the motion paths in ascending order of motion difficulty.

[0110] In some cases, the second preset condition that is used as the limiting factor when displaying the motion path is preset in the control unit 60. In other cases, the operator needs to select a suitable second preset condition as the limiting factor when displaying the motion path. Therefore, the control unit can also execute step S12 (not shown in the figure): generating a seventh intervention prompt message to prompt the selection of the limiting factor when displaying the motion path. As Figure 16 shown, this step is implemented by the display module displaying options for all the second preset conditions for the operator to select. After that, the operator executes step S13 (not shown in the figure): clicking on a suitable second preset condition as the limiting factor when displaying the motion path. Step S12 and step S13 are executed before step S8.

[0111] It can be understood that when multiple path planning methods are pre-stored in the control unit 60, the path planning method may further include step S14 (not shown in the figure): generating a fourth intervention prompt message to prompt the selection of a path generation algorithm, so that the control unit 60 can generate the motion path by using only one path generation algorithm. In practice, as Figure 17 shown, this step is implemented by displaying options of the path generation algorithm on the display module for the operator to select. After that, the operator executes step S15: clicking on the option of the appropriate path generation algorithm to select the appropriate path generation algorithm. Step S14 may be executed before step S4, and the embodiment of the present invention does not particularly limit the execution order of step S14 and step S6.

[0112] Furthermore, an embodiment of the present invention further provides a computer-readable storage medium, on which a program is stored. When the program is executed, it executes the steps performed by the control unit 60 in the aforementioned motion path planning method.

[0113] Still further, an embodiment of the present invention further provides an electronic device, which includes a processor and the aforementioned computer-readable storage medium, and the processor is used to execute the program stored on the computer-readable storage medium.

[0114] Even further, an embodiment of the present invention further provides a path planning system, which includes an image acquisition device and a control unit. The image acquisition device is used to acquire first image information of the surgical area and second image information of a local area of the surgical area. The control unit is communicatively connected to the image acquisition device and is configured to execute the program stored on the aforementioned computer-readable storage medium.

[0115] Preferably, the path planning system further includes a human-computer interaction unit, which is communicatively connected to the control unit and includes a display module for displaying the motion path. More preferably, the display module is further used to display the aforementioned various intervention prompt messages, and the human-computer interaction unit further includes an input module for the operator to input an intervention instruction according to the intervention prompt message.

[0116] Although the present invention is disclosed as above, it is not limited thereto. Those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. A computer-readable storage medium, characterized in that, A program is stored thereon, and when the program is executed, the following steps are performed: Establish a first three-dimensional model based on first image information of the surgical area before surgery; Establish a second three-dimensional model based on second image information of the surgical area during surgery, where the second three-dimensional model is used to plan the expected surgical operation position; Map the expected surgical operation position to the first three-dimensional model and obtain a target surgical operation position on the first three-dimensional model; Generate a movement path of the surgical instrument based on the starting position of the surgical instrument, the first three-dimensional model, and the target surgical operation position, so that when the surgical instrument moves along the movement path, it can move from the starting position to the target surgical operation position; The second image information is obtained by an endoscope inserted into the surgical object through a first hole position on the patient's body surface; the pose of the endoscope is adjusted in real time during surgery, and the endoscope obtains one piece of the second image information after each pose adjustment. During surgery refers to the time range during and after the surgical instrument moves from a second hole position to the lesion; 2. The computer-readable storage medium according to claim 1, wherein The first image information is image information within the entire range of the surgical area; the second image information is image information of a local area of the surgical area; 3. The computer-readable storage medium according to claim 1, characterized in that, The step of establishing a first three-dimensional model based on first image information of the surgical area before surgery specifically includes: Perform gray normalization processing and binarization processing on the first image information, and extract eigenvalue; Perform eigenvalue comparison in the model library and identify the tissues in the first image information; Establish the first three-dimensional model according to the identified tissues; 4. The computer-readable storage medium according to any one of claims 1 to 3, characterized in that, The step of establishing a second three-dimensional model based on second image information of the surgical area during surgery specifically includes: Perform gray normalization processing and binarization processing on the second image information, and extract eigenvalue; Perform eigenvalue comparison in the model library to identify the tissues in the second image information; Establish the second three-dimensional model according to the identified tissues; 5. The computer-readable storage medium according to claim 1, characterized in that The step of mapping the expected surgical operation position to the first three-dimensional model and obtaining a target surgical operation position on the first three-dimensional model specifically includes: Register the first three-dimensional model and the second three-dimensional model, and establish a conversion relationship between the coordinate system of the first three-dimensional model and the coordinate system of the second three-dimensional model; Obtain the coordinates of the expected surgical operation position in the coordinate system of the first three-dimensional model according to the coordinates of the expected surgical operation position in the coordinate system of the second three-dimensional model and the conversion relationship, as the target surgical operation position; 6. The computer-readable storage medium according to claim 1, wherein The program also performs the following steps: Generate a first intervention prompt message to prompt the planning of the expected surgical operation position; 7. The computer-readable storage medium according to claim 1, wherein When planning the movement path, the program also uses a first preset condition as a limiting factor when generating the movement path. The first preset condition includes at least one of the following: The movement path is a loop passing through at least one target operation position; The movement path is at a predetermined distance from the tissues shown in the first three-dimensional model, and in the planar projection, the movement path is arranged along the surface of the tissues; The movement path does not pass through the interior of the tissue shown in the first three-dimensional model.

8. The computer-readable storage medium according to claim 1, wherein The program further performs the following steps: Send the movement path to a display device for display.

9. The computer-readable storage medium according to claim 7, wherein The program plans multiple movement paths; the program uses a second preset condition as a limiting factor when displaying the movement path, and the second preset condition includes any one of the length of the path, the length of the arrival time, and the difficulty of operation.

10. The computer-readable storage medium according to claim 1, wherein The program further performs the following steps: Generate a second intervention prompt message to prompt confirmation whether the movement path meets the requirements.

11. The computer-readable storage medium according to claim 9, wherein When the movement path does not meet the requirements and the expected surgical operation position is updated, the program further performs the following steps: Map the updated expected surgical operation position to the first three-dimensional model and obtain an updated target surgical operation position; and, Update the movement path according to the starting position, the first three-dimensional model, and the updated target operation position.

12. The computer-readable storage medium according to claim 1, wherein The program plans multiple movement paths, and the program further performs the following steps: Generate a third intervention prompt message to prompt selection of a movement path that meets the requirements.

13. The computer-readable storage medium according to claim 1, wherein A variety of path generation algorithms are stored on the computer-readable storage medium, and the program further performs the following steps: Generate a fourth intervention prompt message to prompt selection of a path generation algorithm.

14. An electronic device, characterized in that, Comprising a processor and a computer-readable storage medium as recited in any one of claims 1-12, the processor being configured to execute the program stored on the computer-readable storage medium.

15. A path planning system, characterized in that, Comprising: An image acquisition device for acquiring first image information of a surgical area and second image information of a local area of the surgical area; And, A control unit communicatively connected to the image acquisition device and configured to execute the program stored on a computer-readable storage medium as recited in any one of claims 1-13.

16. The path planning system according to claim 15, wherein The path planning system further includes a human-computer interaction unit communicatively connected to the control unit and including a display module for displaying the movement path.

17. The path planning system according to claim 16, wherein When the program is executed, the program further generates an intervention prompt message to prompt performance of an intervention operation; the display module also displays the intervention prompt message, and the human-computer interaction unit further includes an input module for inputting an intervention instruction according to the intervention prompt message.

18. A surgical robot system, characterized in that, Comprising: An image arm; An image acquisition device provided on the image arm and configured to acquire first image information of a surgical area and second image information of a local area of the surgical area; And, A control unit communicatively connected to the image acquisition device and configured to execute the program stored on a computer-readable storage medium as recited in any one of claims 1-13.

Images