Computer-readable storage medium, electronic device and surgical robot system

Abstract

The present invention relates to a computer-readable storage medium, an electronic device, and a surgical robot system. A program is stored on the computer-readable storage medium. When the program is executed, the following steps are performed: establishing a three-dimensional model based on first image information in a surgical object; planning a first path of a punching device according to a first hole position on the surface of the surgical object, a predetermined position in the surgical object, and the three-dimensional model, so that when the punching device moves along the first path, the punching end of the punching device penetrates the surface of the surgical object at the first hole position and reaches the predetermined position; and driving the punching device to move along the first path. When the computer-readable storage medium is applied to a surgical robot system and used for punching on the body of a surgical object, the degree of automation of punching can be improved, the dependence on the experience of doctors can be reduced, and the surgical safety can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical devices, and particularly relates to a computer-readable storage medium, an electronic device, and a surgical robot system. Background Art

[0002] The design concept of a surgical robot is to precisely perform complex surgical operations in a minimally invasive manner. Surgical robots have emerged in the face of various limitations in traditional surgical operations. Surgical robots break through the limitations of the human eye and can use stereoscopic imaging technology to present the internal organs of the human body more clearly to the operator. And for some narrow areas where a person's hand cannot reach, the surgical robot can still control the surgical instruments to complete movements, swings, clamps, and 360° rotations, and can avoid jitter, improve surgical accuracy, and further achieve the advantages of small incisions, less bleeding, fast postoperative recovery, and greatly shortening the postoperative hospital stay of the surgical subject. Therefore, surgical robots are deeply favored by the majority of doctors and patients and are widely used in various clinical surgeries.

[0003] Like traditional surgeries, before performing a surgery using a surgical robot, it is necessary to locate the lesion, determine the punching point according to the position of the lesion, and then punch at the punching point, and then carry out the surgical operation. The punching device used for punching is usually very sharp, and the operator usually needs to use a lot of force to pierce the surface of the surgical subject. Therefore, the punching operation is very dependent on the experience of the operator. An inexperienced operator is very likely to use too much force when punching, resulting in the punching device piercing the surface and then injuring the tissue, bringing unnecessary trauma to the surgical subject and affecting surgical safety. Summary of the Invention

[0004] The purpose of the present invention is to provide a computer storage medium, an electronic device, and a surgical robot system, aiming to automatically plan and control the tool arm to drive the punching device to complete the punching operation through the surgical robot system, reduce the dependence on the doctor's experience in the surgical punching operation, reduce the risk of the surgical punching operation, and improve surgical safety.

[0005] To achieve the above object, the present invention provides a computer-readable storage medium, on which a program is stored. When the program is executed, the following steps are performed:

[0006] Establish a three-dimensional model according to the first image information in the surgical subject's body;

[0007] Plan the first path of the punching device according to the first hole position on the surface of the surgical subject, the predetermined position in the surgical subject's body, and the three-dimensional model, so that when the punching device moves along the first path, the punching end of the punching device passes through the surface of the surgical subject at the first hole position and reaches the predetermined position.

[0008] Optionally, the first path includes a first global path. When the punching device moves along the first global path, the punching end can reach the predetermined position.

[0009] Optionally, the program executes:

[0010] When there is an obstacle on the first global path, plan a first local path. The first local path is planned outside the boundary of the obstacle, and both the starting point and the ending point of the first local path are on the first global path.

[0011] Optionally, the program executes the following steps to plan the first local path:

[0012] Establish an artificial potential field based on the three-dimensional model and the predetermined position, and plan the first local path according to the artificial potential field.

[0013] Optionally, there is a position q in the artificial potential field. The potential function of the position q in the artificial potential field is the sum of the attractive potential function U att (q) and the repulsive potential function U rep (q):

[0014] U(q) = U att (q) + U rep (q),

[0015]

[0016] In the formula, ζ is the attractive gain; d(q, q goal ) is the distance between the position q and the predetermined position; D(q) is the distance to the nearest obstacle from the position q; η is the repulsive gain; Q * is the force threshold of the obstacle. When the distance from the obstacle to the punching end is greater than Q * , the obstacle will not generate a repulsive force on the punching end.

[0017] Optionally, when planning the first path, the program also executes the following steps:

[0018] Perform dilation calculation on the three-dimensional model to expand the boundary of the tissue model in the three-dimensional model outward by a safety distance;

[0019] Plan the first path according to the dilated three-dimensional model.

[0020] Optionally, the maximum speed of the punching device when moving along the first path is V max1 , the acceleration is a1, the safety distance is d1, and the following relationship is satisfied:

[0021] d1 = V max1 2 / (2a1).

[0022] Optionally, after the punching end penetrates the body surface of the surgical object at the first hole position, the program further performs the following steps:

[0023] Obtain the punching state information according to the second image information of the punching end and the three-dimensional model, and generate guiding information.

[0024] Optionally, an image acquisition device is used to penetrate the body surface at the second hole position on the body surface of the surgical object and enter the body of the surgical object to obtain the second image information, and the program is further used to perform the following steps:

[0025] Plan the target pose of the image acquisition device in the body of the surgical object according to the first hole position and the three-dimensional model, so that when the image acquisition device is in the target pose, the first hole position is within the field of view of the image acquisition device;

[0026] Plan the motion scheme of the image acquisition device according to the initial pose, the three-dimensional model and the target pose of the image acquisition device in the body of the surgical object, and drive the image acquisition device to move according to the motion scheme and reach the target pose.

[0027] Optionally, the initial pose includes an initial position, and the target pose includes a target position; the motion scheme includes a second global path planned according to the initial position, the three-dimensional model and the target position, and the image acquisition device can reach the target position when moving along the second global path.

[0028] Optionally, when there is an obstacle on the second global path, the motion scheme further includes a second local path, the second local path is located outside the boundary of the obstacle, and both the starting point and the ending point of the second local path are on the second global path.

[0029] Optionally, the target pose further includes a target attitude, and the motion scheme further includes a rotation scheme planned according to the current attitude and the target attitude when the image acquisition device reaches the target position, and the image acquisition device can reach the target attitude when rotating according to the rotation scheme at the target position.

[0030] Optionally, the second image information is obtained by an image acquisition device, and when obtaining the second image information, the program further performs the following steps:

[0031] The pose of the image acquisition device is controlled by visual servo so that the punching end of the punching device is within the field of view of the image acquisition device.

[0032] Optionally, the punching state information includes at least one of the position information of the punching device, the speed information of the punching device, and the punching process information;

[0033] The program performs at least one of the following steps to obtain the punching state information:

[0034] Obtain the position information of the punching device according to the second image information and the three-dimensional model;

[0035] Obtain the speed information of the punching device according to the position change of the punching device;

[0036] Generate the punching process information according to the current position information of the punching device and the predetermined position;

[0037] Optionally, the guiding information includes collision reminder information; the program performs the following steps to obtain the guiding information:

[0038] Obtain the collision probability according to the position information of the punching end, the speed information of the punching end, and the three-dimensional model, and generate collision reminder information.

[0039] Optionally, the program performs the following steps to obtain the collision reminder information:

[0040] Obtain the target tissue closest to the punching end according to the position information of the punching end and the three-dimensional model, and calculate the distance between the punching end and the target tissue;

[0041] Calculate the collision occurrence time according to the speed of the punching end and the distance;

[0042] Judge whether the collision occurrence time is greater than the set time threshold, if not, determine that the collision probability is high, and generate the reminder information.

[0043] Optionally, the program performs the following steps to determine the first hole position on the body surface of the surgical object:

[0044] Establish a first physical sign image model according to the first body surface information and lesion information of the surgical object in the first state, and the first physical sign image model is used to plan the first pre-hole position;

[0045] Establish a second physical sign image model according to the second body surface information of the surgical object in the second state;

[0046] Register the second physical sign image model and the first physical sign image model to obtain a first target hole position corresponding to the first pre-hole position on the second physical sign image model, where the first target hole position is used to be guided to the surface of the surgical object to obtain the first hole position.

[0047] To achieve the above object, the present invention also provides an electronic device, including a processor and the computer-readable storage medium as described in any one of the previous items, and the processor is used to execute the program stored on the computer-readable storage medium.

[0048] To achieve the above object, the present invention also provides a surgical robot system, including:

[0049] A tool arm for connecting a punching device, where the punching device includes a punching end, and the punching end is used to penetrate the surface of the surgical object from the first hole position on the surface of the surgical object and reach a predetermined position inside the surgical object;

[0050] An image arm for connecting an image acquisition device, where the image acquisition device is used to acquire first image information inside the surgical object; and,

[0051] A control unit, communicatively connected to the tool arm, the image arm and the image acquisition device, and configured to implement the steps executed by the program as described in any one of the previous items.

[0052] Optionally, after the punching end penetrates the surface of the surgical object at the first hole position, the image acquisition device further acquires second image information of the punching end; the control unit is further configured to obtain punching state information and generate guiding information according to the second image information and the three-dimensional model;

[0053] The surgical robot system further includes a prompting device, communicatively connected to the control unit, and used to receive and display the punching state information and the guiding information.

[0054] Optionally, the surgical robot system includes a first imaging device and a second imaging device, both the first imaging device and the second imaging device are communicatively connected to the control unit, the first imaging device is used to acquire first body surface information and lesion information of the surgical object in a first state, and the second imaging device is used to acquire second body surface information of the surgical object in a second state; the control unit establishes a first physical sign image model according to the first body surface information and the lesion information, and establishes a second physical sign image model according to the second body surface information, and the first physical sign image model and the second physical sign image model are used to obtain the first hole position.

[0055] Optionally, the first imaging device includes any one of an MRI, an X-ray device, or an ultrasound; the second imaging device includes a binocular vision camera or a structured light camera.

[0056] Compared with the prior art, the computer-readable storage medium, the electronic device, and the surgical robot system of the present invention have the following advantages:

[0057] First, the aforementioned computer-readable storage stores a program that, when executed, performs the following steps: establishing a three-dimensional model based on the first image information in the surgical object's body, planning a first path for the punching device based on the first hole position on the surgical object's body surface, a predetermined position in the surgical object's body, and the three-dimensional model, so that when the punching device moves along the first path, the punching end of the punching device penetrates the surgical object's body surface at the first hole position and reaches the predetermined position; and driving the punching device to move along the first path. When the computer-readable storage medium is applied to a surgical robot system and the punching operation is performed using this surgical robot system, since the first path of the punching device is pre-planned, it is possible to avoid stabbing the target tissue during the punching process and causing unnecessary trauma to the surgical object, reduce the dependence on the doctor's experience, and improve the surgical safety.

[0058] Second, the surgical robot system also collects second image information of the punching end entering the surgical object's body through an image acquisition device, and obtains punching status information based on the second image information and the three-dimensional model to monitor the punching process in real time, ensure the smooth execution of the punching operation, and further improve safety.

[0059] Third, the surgical robot system also plans the target pose and its motion scheme of the image acquisition device, and drives the image acquisition device to move to the target pose according to the motion scheme, so that the image acquisition device can accurately collect the first image information, monitor the punching process, and avoid damaging the target tissue during the process of the image acquisition device moving to the target pose.

[0060] Fourth, when obtaining the first hole position, first plan a first pre-hole position on the first physical sign image model of the surgical object in the first state, register the second physical sign image model of the surgical object in the second state and the first physical sign image model to obtain a first target hole position on the second physical sign image model, and then guide the first target hole position to the body surface of the surgical object in the second state to obtain the first hole position, reducing the problem of inaccurate punching hole positions caused by changes in the surgical object's body position, state, etc. before the operation, further reducing the dependence on the doctor's experience, and improving the surgical safety. Description of the Drawings

[0061] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation to the present invention. Among them:

[0062] Figure 1 is a schematic diagram of an application scenario of a surgical robot system provided by the present invention according to an embodiment;

[0063] Figure 2 is a schematic diagram of a punching device used by the surgical robot system provided by the present invention according to an embodiment when performing a punching operation;

[0064] Figure 3 is a schematic structural diagram of an endoscope used by the surgical robot system provided by the present invention according to an embodiment when performing a punching operation;

[0065] Figure 4 is a schematic structural diagram of an endoscope used by the surgical robot system provided by the present invention according to another embodiment when performing a punching operation;

[0066] Figure 5 is a schematic diagram of the surgical robot system provided by the present invention according to an embodiment when performing a punching operation;

[0067] Figure 6 is a flowchart of the surgical robot system provided by the present invention according to an embodiment when performing a punching operation;

[0068] Figure 7 is a schematic diagram when a second imaging device provided by the present invention according to an embodiment acquires second body surface information of a surgical object;

[0069] Figure 8 is a schematic diagram of the imaging principle of a binocular vision camera provided by the present invention according to an embodiment;

[0070] Figure 9 is a flowchart of a control unit of the surgical robot system provided by the present invention according to an embodiment for acquiring second body surface information and establishing a second physical sign image model;

[0071] Figure 10 is a schematic diagram when establishing a mapping relationship among a control unit coordinate system, a surgical object body surface coordinate system, and a second imaging device coordinate system in the surgical robot system provided by the present invention according to an embodiment;

[0072] Figure 11 is a schematic diagram of a first hole position on the body surface of a surgical object provided by the present invention according to an embodiment;

[0073] Figure 12 is a schematic diagram of a second path planned by the surgical robot system provided by the present invention according to an embodiment inside a surgical object;

[0074] Figure 13 It is a flowchart of the endoscope moving to the target pose in the surgical robot system provided by the present invention according to an embodiment;

[0075] Figure 14 It is a schematic diagram when the control unit of the surgical robot system provided by the present invention according to an embodiment plans the second path, and there is a safety gap between the second path and the target tissue in the illustration;

[0076] Figure 15 It is a schematic diagram of the control unit of the surgical robot system provided by the present invention according to an embodiment planning the first local path;

[0077] Figure 16 It is a flowchart of the control unit of the surgical robot system provided by the present invention according to an embodiment obtaining the punching process information;

[0078] Figure 17 It is a schematic diagram of the control unit of the surgical robot system provided by the present invention according to an embodiment obtaining the collision prompt information;

[0079] Figure 18 It is a flowchart of the control unit of the surgical robot system provided by the present invention according to an embodiment obtaining the collision prompt information;

[0080] Figure 19 It is a schematic diagram of the principle of the control unit of the surgical robot system provided by the present invention according to an embodiment performing visual servo control on the endoscope. Detailed implementation manners

[0081] The following uses specific specific examples to illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in this embodiment only schematically illustrate the basic concept of the present invention. Therefore, only the components related to the present invention are shown in the drawings, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0082] In addition, each of the embodiments described below has one or more technical features. However, this does not mean that the inventor must implement all the technical features in any one embodiment simultaneously, or that only one or all of the technical features in different embodiments can be implemented separately. In other words, on the premise that implementation is possible, those skilled in the art can, according to the disclosure of the present invention and depending on design specifications or implementation requirements, selectively implement some or all of the technical features in any one embodiment, or selectively implement a combination of some or all of the technical features in multiple embodiments, thereby increasing the flexibility in implementing the present invention.

[0083] As used in this specification, the singular forms "a", "an", and "the" include plural referents, and the plural form "plural" includes more than two referents, unless the context clearly indicates otherwise. As used in this specification, the term "or" is generally used in the sense of including "and / or", unless the context clearly indicates otherwise, and the terms "mounted", "connected", and "coupled" should be understood broadly. For example, it can be a fixed connection, a detachable connection, or an integral connection. It can be a mechanical connection or an electrical connection. It can be directly connected or indirectly connected through an intermediate medium, and it can be the internal communication of two components or the interaction relationship between two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0084] To make the objectives, advantages, and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. It should be noted that the drawings are all in a very simplified form and use non-precise scales, only for the purpose of facilitating and clearly assisting in explaining the objectives of the embodiments of the present invention. The same or similar reference numerals in the drawings represent the same or similar components.

[0085] Figure 1 FIG. shows a schematic diagram of the application scenario of the surgical robot system provided according to an embodiment of the present invention; Figure 2 FIG. shows a schematic diagram of the punching device 10 used when the surgical robot system performs a punching operation; Figure 3 and Figure 4 FIG. shows a schematic structural diagram of the endoscope 20 used when the surgical robot system performs a punching operation. Figure 5 FIG. shows a schematic diagram when the control unit of the surgical robot system drives the tool arm 210 to move and further drives the punching device 10 to move to perform a punching operation.

[0086] Please refer to Figures 1 to 5, the surgical robot system includes a control end and an execution end. The control end includes a doctor's console and a doctor-end control device 100 disposed on the doctor's console. The execution end includes a patient-end control device, a surgical operation device 200, an image display device 300 and other devices. Among them, a tool arm 210 and an image arm 220 are disposed on the surgical operation device 200. The tool arm 210 is used to connect the punching device 10 and the surgical instrument. The punching device 10 has a punching end, such as a conical tip 11. The conical tip 11 is used to penetrate the skin and subcutaneous fat of the surgical object at the first hole position M1 on the surface of the surgical object and reach a predetermined position q inside the surgical object goal , and complete the punching operation. The surgical instrument is used to enter the surgical object's body from the first hole position M1 to perform a surgical operation. The image arm 220 is used to connect an image acquisition device. The image acquisition device is used to acquire image information of an area or device of interest (such as the first image information, the second image information, etc. mentioned later). The image acquisition device is, for example, the endoscope 20. It should be noted that in this article, the punching operation mentioned refers to punching at the first hole position on the surface of the surgical object by using the surgical robot system, so that the surgical instrument can enter the surgical object's body from the first hole position and perform a surgical operation.

[0087] The surgical robot system further includes a control unit. The control unit is communicatively connected to the tool arm 210, the image arm 220 and the endoscope 20. The control unit can be integrally disposed at the doctor-end control device, or integrally disposed at the patient-end control device, or a part of it is disposed at the doctor-end control device and another part is disposed at the patient-end control device, or at other positions. That is to say, the present invention does not limit the specific setting manner of the control unit, as long as it can perform related functions.

[0088] Before punching at the first hole position M1 on the surface of the surgical object by using the surgical robot system, the endoscope 20 enters the surgical object's body from the second hole position on the surface of the surgical object and acquires the first image information of the surgical object's body. The control unit is configured to establish a three-dimensional model according to the first image information, and according to the first hole position M1, the three-dimensional model and the predetermined position q goal plan the first path of the punching device 10, so that when the tool arm 210 drives the punching device 10 to move along the first path, the conical tip 11 can reach the predetermined position q goal and complete the punching operation. That is to say, the starting point of the first path is the first hole position M1 and the ending point is the predetermined position q goal . In the embodiment of the present invention, the second hole position and the first hole position can be determined simultaneously.

[0089] In this embodiment, the surgical robot system is used to punch a hole at the first hole position M1 on the body surface of the surgical object, and the first path of the punching device 10 is also pre-planned in advance, so that the punching device 10 moves along the first path and completes the punching operation, reducing the dependence on the personal experience of the operator, reducing the possibility of harm to the surgical object due to the lack of experience of the operator, improving the automation level of punching and the safety of the punching operation, effectively shortening the punching time, and reducing the fatigue degree of the operator.

[0090] Preferably, the first path includes a first global path L1 (as Figure 5 shown), when the punching device 10 moves along the first global path L1, the conical tip 11 can reach the predetermined position q goal . The first global path L1 plays an overall guiding role, but in a local section, it may pass through or be close to the target tissue, making this part of the target tissue an obstacle that hinders the movement of the punching device 10. At this time, if the punching device 10 moves completely along the first global path L1, it may collide with the target tissue and cause harm to the target tissue. In view of this, the first path also includes a first local path (not shown in the figure), and the first local path is actually a correction path. When the punching device 10 moves along the first local path, it can avoid colliding with the target tissue that constitutes the obstacle. Specifically, the first local path is located outside the obstacle, and both the starting point and the ending point of the first local path are located on the first global path L1, so that when the punching device 10 moves along the first local path, it can avoid the obstacle and return to the first global path after avoiding the obstacle. In this embodiment, the first global path L1 can be planned before punching starts, and the first local path is planned during the punching process.

[0091] Moreover, before punching, the control unit is further configured to determine the first hole position M1 and the second hole position on the body surface of the surgical object, further reducing the dependence on the doctor's experience. The determination methods of the first hole position M1 and the second hole position will be introduced in detail below.

[0092] In addition, the endoscope 20 also acquires second image information of the conical tip 11 entering the body of the surgical subject during the punching process. The control unit is further configured to obtain punching state information based on the second image information and the three-dimensional model, and generate guiding information to monitor the punching process in real time, stop punching when necessary, and allow the operator to intervene manually to adjust the punching state. In this embodiment, a marker 12 is provided on the conical tip 11 of the punching device 10 to enable the endoscope 20 to identify the conical tip 11 and acquire the second image information of the conical tip 11. The marker 12 can be, for example, a brightly colored reflector or a light emitter.

[0093] In addition, it should also be known that the surgical robot system punches at multiple first hole positions M1 on the surface of the surgical subject during the actual surgical process. Therefore, before each punching starts, the endoscope 20 needs to be adjusted so that the endoscope 20 is in a target pose corresponding to the corresponding first hole position M1. When the endoscope 20 is inside the surgical subject and in the target pose, the first hole position is within the field of view of the endoscope 20 to ensure that once the conical tip 11 of the punching device 10 enters the body of the surgical subject, the endoscope 20 can immediately acquire the second image information and monitor the punching operation in real time.

[0094] Thus, in a non-limiting embodiment, the process of using the surgical robot system to perform preoperative punching on the surface of the surgical subject can be as Figure 6 shown, including:

[0095] Step A10: Determine the first hole position M1 and the second hole position on the surface of the surgical subject;

[0096] Step A20: Punch at the second hole position, and insert an endoscope into the body of the surgical subject from the second hole position and place it in the initial pose;

[0097] Step A30: Place the endoscope in the target pose corresponding to the current first hole position M1. In this step, the endoscope 20 acquires first image information inside the surgical subject. The control unit plans the target pose of the endoscope 20, establishes a three-dimensional model based on the first image information, plans the movement scheme of the endoscope 20 according to the three-dimensional model, the initial pose, and the target pose, and drives the endoscope 20 to move along the movement scheme from the initial pose to the target pose..

[0098] Step A40: The control unit plans the first global path L1 of the punching device according to the three-dimensional model, the first hole position M1, and the predetermined position.

[0099] Step A50: The control unit drives the robotic arm to move, and further drives the robotic arm to drive the punching device to move along the first global path L1. During this process, if the punching device encounters an obstacle, the control unit also plans the first local path, and drives the punching device 10 to move along the first local path and temporarily deviate from the first global path L1 to avoid the obstacle, and returns to the first global path L1 after avoiding the obstacle until the punching operation is completed. At the same time, the control unit also obtains the punching status information according to the second image information collected by the endoscope 20 and the three-dimensional model to monitor the punching operation in real time.

[0100] Generally, the operator needs to punch holes at a plurality of first hole positions M1 on the body surface of the surgical object. The control unit can plan a corresponding first path for each first hole position M1 each time (as Figure 6 shown), or can complete the planning of the first paths of all the first hole positions M1 at one time. The present invention does not limit this. After the punching operation at one first hole position M1 is completed, the surgical robot system drives the endoscope 20 to return to the initial position, and then drives the endoscope 20 to move to the target pose corresponding to the next first hole position M1 to perform the punching operation on the next first hole position M1 until all the first hole positions M1 are punched. Finally, the endoscope 20 can retract into the trocar on the image arm 220 and perform corresponding operations according to subsequent surgical needs. It should be noted that Figure 6 the execution order of each step in the shown process is not fixed and can be changed according to the actual situation. For example, step A30 can be executed synchronously with step A40, or the target pose of the endoscope 20 can be planned during the execution of step A20, etc.

[0101] Next, the specific implementation manners of the above steps will be introduced in detail in this article.

[0102] Please continue to refer to Figure 6 , step A10 includes:

[0103] Step A11: Establish a first physical sign image model according to the first body surface information and lesion information of the surgical object in the first state.

[0104] Step A12: Plan the pre-hole positions on the first physical sign image model, and the pre-hole positions include a first pre-hole position and a second pre-hole position.

[0105] Step A13: Establish a second physical sign image model according to the second body surface information of the surgical object in the second state.

[0106] Step A14: Register the second physical sign image model and the first physical sign image model to obtain target holes corresponding to the pre-drilled holes on the second physical sign image model. The target holes include a first target hole and a second target hole.

[0107] Step A15: Use any suitable method to guide the target holes to the body surface of the surgical subject to obtain actual holes M (as Figure 11 shown), where the first target hole is guided to the body surface of the surgical subject to obtain the first hole M1, and the second target hole is guided to the body surface of the surgical subject to obtain the second hole. How to guide the target holes to the body surface of the surgical subject is well-known to those skilled in the art and will not be described in detail here.

[0108] In this embodiment, taking laparoscopic surgery as an example, the first state refers to the state before pneumoperitoneum of the surgical subject, and the second state is the state where pneumoperitoneum has been established for the surgical subject. The first body surface information and lesion information of the surgical subject in the first state are acquired by a first imaging device (not shown in the figure). The first imaging device includes but is not limited to MRI, CT, or other X-ray devices, or B-ultrasound, as long as it can perform three-dimensional scanning on the body surface and inside of the surgical subject simultaneously. The control unit performs three-dimensional reconstruction based on the first body surface information and lesion information acquired by the first imaging device to obtain the first physical sign image model, and matches the first physical sign image model with the parameters of the surgical robot system (mainly the parameters of the tool arm 210), and then plans the pre-drilled holes on the first physical sign image model by means of three-dimensional simulated drilling.

[0109] The second body surface information of the surgical subject in the second state is acquired by the second imaging device 30 (as Figure 7 shown). In one implementation, the second imaging device 30 is a binocular vision camera. At the same time, a plurality of feature points are provided on the body surface of the surgical subject. For example, a plurality of target pens 40 are provided on the body surface of the surgical subject to represent the feature points. Then the binocular vision camera recognizes the target pens 40 to obtain the image information of the target pens 40, and the image information of the target pens 40 is used to acquire the second body surface information. In this embodiment, there is no special limitation on the distribution of the feature points, and they can be reasonably set by the operator according to the actual situation.

[0110] The binocular vision camera includes a first camera 31 and a second camera 32 (as Figure 8 shown), and the image information of the target pen 40 includes the first sub-image information captured by the first camera 31 and the second sub-image information captured by the second camera 32. Figure 8The imaging principle of the binocular vision camera is shown. In the figure, f is the camera focal length, b is the baseline between the first camera and the second camera, and P(x, y, z) is the coordinate of any one of the target pens 40 being photographed. Then, f, b, and P(x, y, z) satisfy the following relationship:

[0111]

[0112] Thus, the control unit can, according to Figure 9 the method shown, obtain the second body surface information and establish the second physical sign image model, including:

[0113] Step S1: The control unit extracts the feature points from the first sub-image information and the second sub-image information of the target pen;

[0114] Step S2: The control unit pairs the feature points on the first sub-image information and the second sub-image information to form feature point pairs.

[0115] Step S3: The control unit locates the feature point pairs to the three-dimensional space positions in the binocular vision camera coordinate system according to the epipolar constraint. After locating all the feature point pairs, a point cloud model of the feature points is obtained as the second physical sign data.

[0116] Step S4: The control unit performs three-dimensional surface reconstruction according to the second body surface information to obtain the second physical sign image model.

[0117] In an alternative implementation, a reflective sphere can also be used to represent the feature points, and the second imaging device 30 can also be a 3D structured light camera or a laser sensor.

[0118] After that, the control unit can use the iterative closest point method (ICP) to register the second physical sign image model and the first physical sign image model to solve the coordinate transformation matrix between the second physical sign image model and the first physical sign image model, and based on this transformation matrix and the coordinates of the pre-drilled hole position on the first physical sign image model, obtain the coordinates of the target hole position on the second physical sign image model.

[0119] Next, the operator establishes the mapping relationship between the control unit coordinate system F1(X1, Y1, Z1) and the surgical object body surface coordinate system F2(X2, Y2, Z2), and indicates the target hole position on the surgical object body surface according to the mapping relationship to obtain the actual drilling point M (as Figure 11 shown). When establishing the mapping relationship, as Figure 10As shown, a mapping relationship is established between the coordinate system F3(X3, Y3, Z3) of the second imaging device 30 and the control unit coordinate system F1(X1, Y1, Z1) in the world coordinate system F0(X0, Y0, Z0) ( Figure 10 taking the surgical object end control device 400 of the surgical robot system as an example of the control unit). When collecting the second body surface information, a mapping relationship has been established between the body surface coordinate system F2(X2, Y2, Z2) of the surgical object and the coordinate system F3(X3, Y3, Z3) of the second imaging device. Thus, a mapping relationship can be obtained between the coordinate system F1(X1, Y1, Z1) of the control unit and the coordinate system F2(X2, Y2, Z2) of the surgical object body surface.

[0120] In view of the above introduction, during the process of determining the first hole position M1 and the second hole position on the surgical object body surface, a pre-hole position is established on the first physical sign image model of the surgical object by using the control unit, and then the first physical sign image model is registered with the second physical sign image model in the second state during actual hole drilling to obtain the target hole position on the second physical sign image model, and further obtain the actual hole position M (including the first hole position M1 and the second hole position), avoiding the problem that the actual hole position M is inaccurate due to different physical sign states of the surgical object, and laying a good foundation for subsequent hole drilling operations and surgical operations.

[0121] In step A20, the operator can use any suitable method to drill a hole at the second hole position, and insert the endoscope 20 into the surgical object from the second hole position and be in the initial pose. The initial pose includes an initial position and an initial attitude. The initial position and the initial attitude can be determined in advance by the operator or can be random. The present invention does not limit this.

[0122] In step A30, the target pose of the endoscope 20 includes a target position and a target attitude. The motion plan includes a second path. Similar to the first path, the second path preferably includes a second global path L2 and a second local path L3 (such as Figure 12As shown in the figure, the control unit is configured to plan the second global path L2 according to the initial position of the endoscope 20, the three-dimensional model, and the target position, so that when the endoscope 20 moves along the second global path L2, the endoscope 20 can reach the target position. When there are obstacles on the second global path L2 (i.e., target tissues that may hinder the movement of the endoscope 20 along the second global path L2), the control unit is configured to plan the second local path L3. The second local path L3 can be located outside the obstacle, and both the starting point and the ending point of the second local path L3 are located on the second global path L2. In this embodiment, the second global path L2 is planned by using any one of the Dijkstra algorithm, the A * (A-Star) algorithm, and the random forest algorithm before the endoscope 20 moves. The second local path L3 can be planned by using the dynamic window method according to the surrounding environment during the movement of the endoscope 20.

[0123] If the current posture of the endoscope 20 when it reaches the target position is not the target posture, the control unit is further configured to plan a rotation scheme according to the current posture and the target posture, and drive the endoscope 20 to rotate to the target posture according to the rotation scheme. The rotation scheme includes the rotation direction and rotation angle of the endoscope 20 around the second hole position.

[0124] Therefore, as Figure 13 shown, the specific steps of step A30 include:

[0125] Step A31: Plan the second global path L2 according to the three-dimensional model, the initial position of the endoscope, and the target position.

[0126] Step A32: Drive the image arm to move, and then drive the endoscope to move along the second global path L2. During this process, if the endoscope does not encounter an obstacle, the endoscope moves completely along the second global path L2 until it reaches the target position. If the endoscope encounters an obstacle during the movement along the second global path L2, the control unit also plans the second local path L3 and drives the endoscope to move along the second local path L3 to avoid the obstacle, and returns to the second global path L2 after avoiding the obstacle, realizing the automatic navigation control of the endoscope movement process until the endoscope reaches the target position, that is, the endoscope can automatically avoid obstacles and reach the target position.

[0127] Step A33: Plan the rotation scheme and drive the image arm to move to drive the endoscope to rotate to the target posture according to the rotation scheme.

[0128] Preferably, during the execution of step A30 by the control unit, the endoscope 20 continuously acquires the first image information, and the control unit updates the three-dimensional model at predetermined time intervals, thereby updating the second global path L2 and the second local path L3 to ensure that the endoscope 20 can avoid the obstacles in real time.

[0129] Those skilled in the art can understand that the image arm 220 includes at least one joint. When the control unit plans the second path, the control unit also performs time constraint on the second path to obtain the relationship between the pose (mainly the position) of the endoscope 20 and time. Then, the control unit performs inverse kinematic solution of the robot to obtain the acceleration, speed, and position of the joints on the image arm 220. Thus, the control unit can drive the joints on the image arm 220 to move to the corresponding positions according to the acceleration and the speed, so as to drive the endoscope 20 to move along the second path.

[0130] Further, when the control unit plans the second path (i.e., plans the second global path L2 and the second local path L3), the control unit is further configured to perform a dilation operation on the three-dimensional model, so that the tissue model S in the three-dimensional model expands outward by a safety distance to obtain a dilation boundary S1 (as Figure 14 shown). In this way, the control unit is configured to plan the second path according to the dilated three-dimensional model, so that there is a safety gap between the second path and the target tissue, further avoiding the endoscope 20 from colliding with the target tissue during movement and damaging the tissue, and improving the safety. Herein, the "outer side" refers to the side facing the outside of the tissue model.

[0131] Optionally, the maximum speed of the endoscope 20 during movement along the second path is V max2 , the acceleration is a2, the safety distance is d2, and the following relationship is satisfied: d2 = V max2 2 / (2a2).

[0132] In step A33, if the current pose of the endoscope 20 coincides with the target pose when the endoscope 20 reaches the target position, both the rotation speed and the rotation angle of the endoscope 20 in the rotation scheme are zero.

[0133] In step A40, the starting point of the first global path L1 can be the first hole position M1, and the ending point is the predetermined position q goal .

[0134] In the step A50, both the starting point and the ending point of the first local path can be on the first global path L1, so that after the punching device 10 moves along the first local path and avoids the obstacle, it can return to the first global path L1 again.

[0135] In this step, as Figure 15 shown, the control unit is configured to establish an artificial potential field according to the three-dimensional model and the predetermined position q goal and plan the first local path according to the artificial potential field. It should be understood that when establishing the artificial potential field, within a certain range around the predetermined position q goal the area should be within the artificial potential field, and the distance from any point in this area to the surface of the surgical object is not less than the distance from the predetermined position q goal to the first hole position M1.

[0136] There is a position q in the artificial potential field, and the potential function of the position q in the artificial potential field is the sum of the attractive potential function U att (q) and the repulsive potential function U rep (q):

[0137] U(q) = U att (q) + U rep (q),

[0138]

[0139] In the formula, ζ is the attractive gain; d(q, q goal ) is the distance between the position q and the predetermined position q goal ; D(q) is the distance to the nearest obstacle (the target tissue that may collide with the punching device) from the position q; η is the repulsive gain; Q * is the force threshold of the obstacle. When the distance from the obstacle to the conical tip 11 of the punching device 10 is greater than Q * , the obstacle will not generate a repulsive force on the conical tip 11.

[0140] The control unit can calculate the force generated by the artificial potential field on the position q when the conical tip 11 moves to the position q according to the potential function of the position q. This force acts on the conical tip 11 to make the conical tip 11 generate an acceleration component to avoid obstacles (such as Figure 15as shown by arrow F). Thus, the control unit can calculate the acting force on the conical tip 11 at any position on the first global path L1, and plan the first local path accordingly to locally correct the first global path L1 and avoid the obstacle. Those skilled in the art can understand that the position of the punching device 10 in the artificial potential field can be calculated by the method of robot kinematics.

[0141] Based on the three-dimensional model, the control unit performs scientific and reasonable calculations to locally correct the first global path L1, avoid the punching device 10 from colliding with the target tissue during the punching process, and improve safety.

[0142] Similar to the second path, please return for reference Figure 5 , when the control unit plans the first path (i.e., plans the first global path L1 and the first local path), the control unit performs dilation calculation on the three-dimensional model to make the tissue model in the three-dimensional model expand outward by a safety distance to obtain the dilated boundary S1. At the same time, the control unit plans the first path according to the dilated three-dimensional model so that there is a safety gap between the first path and the target tissue. In this embodiment, the maximum moving speed of the punching device 10 when moving along the first movement path is V max1 , the acceleration is a1, the safety distance is d1, and the following relationship is satisfied: d1 = V max1 2 / (2a1).

[0143] Similarly, when the control unit plans the first path (i.e., the first global path L1 and the first local path), it also performs time constraint on the first path to obtain the relationship between the position of the punching device 10 and time, and obtains the acceleration, speed and position of the joints on the tool arm 210 through inverse kinematic solution of the robot. Thus, the control unit can drive the joints on the tool arm 210 to move to the corresponding positions according to the acceleration and the speed, so as to drive the punching device 10 to move along the first path.

[0144] The punching state information includes the position of the punching device 10, punching process information, the moving speed of the punching device 10, and collision prompt information. Therefore, when acquiring the punching state information, the control unit is configured to acquire the position information of the punching device 10 according to the second image information and the three-dimensional model. According to the position information of the punching device 10 and the predetermined position q goalGenerate the punching process information. Obtain the speed information of the punching device 10 according to the position change of the punching device 10. Obtain the collision probability between the punching device 10 and the target tissue according to the position information of the punching device 10, the speed information of the punching device 10, and the three-dimensional model, and generate the collision prompt information.

[0145] Wherein, the control unit can obtain the position of the punching device 10 in the human body according to the second image information and the structural model of the punching device 10 pre-stored in the control unit. Alternatively, in an alternative implementation, markers are provided on all surfaces of the punching device 10 so that all structures of the punching device 10 entering the human body can be recognized by the endoscope 20, so that the control unit can directly judge its position according to the image information of the punching device 10.

[0146] And, as Figure 16 shown, the method for the control unit to obtain the punching process information includes:

[0147] Step S11: The control unit obtains the current punching depth z1 according to the current position of the punching device. The current punching depth z1 is the distance from the current position of the conical tip of the punching device to the first hole position M1.

[0148] Step S12: The control unit compares the current punching depth z1 with the expected punching depth z0 to obtain the ratio of the two as the punching process information. The expected punching depth z0 is the distance from the predetermined position q goal to the first hole position M1. And when the ratio of the current punching depth z1 to the expected punching depth z0 is 1, the control unit determines that the punching is completed.

[0149] As Figure 17 shown, when obtaining the collision prompt information, the control unit is configured to obtain the target tissue closest to the punching device 10 according to the position of the punching device 10 and the first three-dimensional model, and calculate the distance between the target tissue and the punching device 10 (mainly the conical tip 11). Then, calculate the collision occurrence time according to the moving speed of the punching device 10 and the distance, and judge whether the collision occurrence time is greater than a preset time threshold. If not, it is determined that the collision probability is high, and the collision prompt information is generated. If so, it is determined that the collision probability is low, and the collision prompt information is not generated or other information different from the prompt information is generated.

[0150] Therefore, as Figure 18 shown, the method for obtaining the collision prompt information includes the following steps:

[0151] Step S21: The control unit determines the target tissue closest to the punching device according to the position of the punching device and the first three-dimensional model.

[0152] Step S22: The control unit calculates the distance between the target tissue and the punching device.

[0153] Step S23: The control unit calculates the collision occurrence time t according to the speed of the conical tip and the distance. In this embodiment, the speed of the conical tip is v1, and the distance is D. Then the collision occurrence time t satisfies: t = D / v1.

[0154] Step S24: The control unit determines whether the time t is greater than the set time threshold t0. If not, it determines that the collision probability is high and generates the collision prompt information. If so, it determines that the collision probability is low and does not generate the collision prompt information or generates other information.

[0155] Further, the surgical robot system further includes a prompt device, which is used to communicate with the control unit to receive the punching status information and give a prompt. The prompt device can have various selections. For example, the prompt device may include a buzzer alarm to prompt the collision prompt information through buzzer alarm. The prompt device may also include a voice prompt device for broadcasting the collision prompt information and the punching process information. The prompt device may also include a display device for displaying the position of the punching device, the speed of the punching device, the collision prompt information, the punching process information, etc. through text and images. Further, the control unit can timely control the tool arm 210 to stop moving before the collision occurs to abort the punching operation, so that the operator can intervene to manually adjust the punching direction.

[0156] In this embodiment, during the process of the surgical robot system automatically performing punching, the punching process is also monitored in real time in combination with the second image information collected by the endoscope 20, so as to facilitate manual intervention at any time and further avoid damage to the target tissue caused by the punching operation.

[0157] In addition, when collecting the second image information, in order to ensure that the conical tip 11 of the punching device 10 is within the field of view of the endoscope 20, the control unit is further configured to use visual servo to control the pose of the endoscope 20 so that the conical tip 11 is always within the field of view of the endoscope 20, preferably located at the center of the field of view of the endoscope 20.

[0158] Visual servo control compares the image information obtained by real-time measurement with the given image information, and uses the obtained image error for feedback to form a closed-loop control, so that the controlled object is in the given pose. In this embodiment, the given image information is that the conical tip 11 is at the center of the field of view of the endoscope 20. Figure 19 The schematic diagram showing the principle of the control unit for performing visual servo control on the endoscope 20 is as Figure 19 shown. The control unit includes a visual servo controller 401 and an image arm joint controller 402. Joint sensors 221 are provided on the joints of the image arm 220. The process of visual servo control is as follows:

[0159] The endoscope 20 acquires the second image information as the actual image information and sends it to the servo controller 401.

[0160] The visual servo controller 401 determines whether the conical tip 11 that has entered the surgical object is within the field of view / center of the field of view of the endoscope 20 according to the actual image information. If not, the visual servo controller 401 extracts the actual pose of the endoscope 20 according to the error between the actual image information and the given image information, and obtains motion information such as the motion speed and motion direction of the endoscope 20 when moving from the actual pose to the given pose.

[0161] According to the motion information of the endoscope 20 and the robot inverse kinematics, the motion information of the joints of the image arm 220 is calculated and sent to the image arm joint controller 402.

[0162] The image arm joint controller 402 drives the relevant joints on the image arm 220 to move according to the calculated motion information of the joints of the image arm 220, and the joint sensors 221 feedback joint information in real time until the joints of the image arm 220 drive the endoscope 20 to move to the given pose.

[0163] During the movement process of the endoscope 20 (the movement process along the second path, the rotation process after reaching the target position, and the servo control process), the endoscope will generate a bending motion according to the pose of the endoscope 20. Here, the endoscope 20 used in this embodiment is a flexible endoscope. Please refer back to Figure 3 and Figure 4 , the endoscope 20 includes an image acquisition element 21 and a mirror arm 22. The mirror arm 22 includes a first rigid section 22a, a controllable bending section 22b, and a second rigid section 22c that are sequentially connected from the proximal end to the distal end. Among them, the controllable bending section 22b includes a bellows (as Figure 3 shown), or the controllable bending section 22b includes a snake bone (asFigure 4 As shown). The image acquisition element 21 is disposed on the second rigid segment 22c. The image acquisition element 21 may be a binocular vision camera. The imaging principle thereof, the process by which the control unit obtains the feature points of the target tissue in the second target area according to the third image information acquired by the binocular vision camera, and the process of establishing the three-dimensional model may refer to the foregoing introduction. It should be noted that when positioning the feature points of the tissue in the human body onto the coordinate system of the image acquisition element 21 of the endoscope 20, the position of the coordinate system of the image acquisition element 21 in the coordinate system of the control unit can be obtained according to the forward kinematics of the robot and the pre-calibrated coordinate system parameters of the image acquisition element 21.

[0164] In addition, the endoscope 20 further includes components such as a pull cord (not shown in the figure) and a light source (not shown in the figure). The pull cord is disposed in the pull cord hole (not shown in the figure) of the mirror arm 22, and the controllable bending section 22b is bent or straightened by tightening or loosening the pull cord. The specific setting thereof is a conventional technical means in the art and will not be described in detail herein. The light source is disposed on the second rigid segment 22c and is used to provide illumination for the image acquisition element 21.

[0165] Furthermore, an embodiment of the present invention further provides a computer-readable storage medium, on which a program is stored. When the program is executed, all operations performed by the foregoing control unit are executed.

[0166] Furthermore, an embodiment of the present invention further provides an electronic device. The electronic device includes a processor and the computer-readable storage medium, and the processor is used to execute the program stored on the computer-readable storage medium.

[0167] Still further, an embodiment of the present invention further provides a path planning method, which is at least used to plan the first movement path of the punching device, that is, it includes the step of planning the first movement path performed by the foregoing control unit. Moreover, the path planning method further includes the steps of planning the second global path and the second local path of the image acquisition device performed by the control unit.

[0168] Although the present invention is disclosed as above, it is not limited thereto. Those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these changes and modifications.

Claims

1. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed, the following steps are performed: Establish a three-dimensional model based on the first image information in the surgical object's body; Plan a first path of the punching device according to the first hole position on the surface of the surgical object, the predetermined position in the surgical object's body, and the three-dimensional model, so that when the punching device moves along the first path, the punching end of the punching device penetrates the surface of the surgical object at the first hole position and reaches the predetermined position; The first path includes a first global path. When the punching device moves along the first global path, the punching end can reach the predetermined position; The program executes: When there is an obstacle on the first global path, plan a first local path. The first local path is planned outside the boundary of the obstacle, and both the starting point and the ending point of the first local path are on the first global path.

2. The computer-readable storage medium according to claim 1, wherein The program performs the following steps to plan the first local path: Establish an artificial potential field according to the three-dimensional model and the predetermined position, and plan the first local path according to the artificial potential field.

3. The computer-readable storage medium according to claim 2, wherein In the artificial potential field, there is a position q, and the potential function of the position q in the artificial potential field is the sum of the attractive potential function U att (q) and the repulsive potential function U rep (q): U(q) = U att (q) + U rep (q), where ζ is the attraction gain; d(q, q goal ) is the distance between the position q and the predetermined position; D(q) is the distance to the nearest obstacle from the position q; η is the repulsion gain; Q * is the force threshold of the obstacle. When the distance from the obstacle to the end of the punching is greater than Q * , the obstacle will not generate a repulsive force on the end of the punching.

4. The computer-readable storage medium according to claim 1, wherein When planning the first path, the program also performs the following steps: Perform dilation calculation on the three-dimensional model so that the boundary of the tissue model in the three-dimensional model expands outward by a safety distance; Plan the first path according to the dilated three-dimensional model.

5. The computer-readable storage medium according to claim 4, wherein The maximum speed of the punching device when moving along the first path is V max1 , the acceleration is a1, the safety distance is d1, and the following relationship is satisfied: d1 = V max1 2 / (2a1).

6. The computer-readable storage medium according to claim 1, wherein After the punching end penetrates the surface of the surgical object at the first hole position, the program also performs the following steps: Obtain punching state information according to the second image information of the punching end and the three-dimensional model, and generate guiding information.

7. The computer-readable storage medium according to claim 6, wherein An image acquisition device is used to penetrate the surface at the second hole position on the surface of the surgical object and enter the surgical object's body to obtain the second image information. The program is also used to perform the following steps: Plan the target pose of the image acquisition device in the surgical object's body according to the first hole position and the three-dimensional model, so that when the image acquisition device is in the target pose, the first hole position is within the field of view of the image acquisition device; Plan a motion scheme of the image acquisition device according to the initial pose of the image acquisition device in the surgical object's body, the three-dimensional model, and the target pose, and drive the image acquisition device to move according to the motion scheme and reach the target pose.

8. The computer-readable storage medium according to claim 7, wherein The initial pose includes an initial position, and the target pose includes a target position; the motion scheme includes a second global path planned according to the initial position, the three-dimensional model, and the target position. When the image acquisition device moves along the second global path, it can reach the target position.

9. The computer-readable storage medium according to claim 8, wherein When there is an obstacle on the second global path, the motion scheme also includes a second local path. The second local path is located outside the boundary of the obstacle, and both the starting point and the ending point of the second local path are on the second global path.

10. The computer-readable storage medium according to claim 8 or 9, characterized in that The target pose further includes a target attitude, and the motion plan further includes a rotation plan planned according to the current attitude of the image acquisition device when it reaches the target position and the target attitude, so that the image acquisition device can reach the target attitude when rotating according to the rotation plan at the target position.

11. The computer-readable storage medium according to claim 6 or 7, wherein Obtain the second image information through the image acquisition device. When obtaining the second image information, the program further performs the following steps: Use visual servo to control the pose of the image acquisition device so that the punching end of the punching device is within the field of view of the image acquisition device.

12. The computer-readable storage medium according to claim 6, wherein The punching state information includes at least one of the position information of the punching device, the speed information of the punching device, and the punching process information; The program performs at least one of the following steps to obtain the punching state information: Obtain the position information of the punching device according to the second image information and the three-dimensional model; Obtain the speed information of the punching device according to the position change of the punching device; Generate the punching process information according to the current position information of the punching device and the predetermined position.

13. The computer-readable storage medium according to claim 12, wherein The guiding information includes collision reminder information; the program performs the following steps to obtain the guiding information: Obtain the collision probability according to the position information of the punching end, the speed information of the punching end, and the three-dimensional model, and generate a collision prompt information.

14. The computer-readable storage medium according to claim 13, wherein The program performs the following steps to obtain the collision reminder information: Obtain the target tissue closest to the punching end according to the position information of the punching end and the three-dimensional model, and calculate the distance between the punching end and the target tissue; Calculate the collision occurrence time according to the speed of the punching end and the distance; Judge whether the collision occurrence time is greater than a set time threshold. If not, determine that the collision probability is high and generate the prompt information.

15. The computer-readable storage medium according to claim 1, wherein The program performs the following steps to determine the first hole position on the body surface of the surgical object: Establish a first physical sign image model according to the first body surface information and lesion information of the surgical object in the first state, and the first physical sign image model is used to plan the first pre-hole position; Establish a second physical sign image model according to the second body surface information of the surgical object in the second state; Register the second physical sign image model and the first physical sign image model to obtain a first target hole position corresponding to the first pre-hole position on the second physical sign image model, and the first target hole position is used to be guided to the body surface of the surgical object to obtain the first hole position.

16. An electronic device, including a processor and a computer-readable storage medium according to any one of claims 1-15, and the processor is used to execute the program stored on the computer-readable storage medium.

17. A surgical robot system, characterized in that, Comprising: A tool arm for connecting a punching device, the punching device includes a punching end, and the punching end is used to penetrate the body surface of the surgical object from the first hole position on the body surface of the surgical object and reach a predetermined position inside the surgical object; An image arm for connecting an image acquisition device, and the image acquisition device is used to acquire first image information inside the surgical object; And, A control unit, communicatively connected to the tool arm, the image arm, and the image acquisition device, and configured to implement the steps performed by the program according to any one of claims 1-15.

18. The surgical robot system according to claim 17, wherein After the punching end penetrates the body surface of the surgical object at the first hole position, the image acquisition device further acquires second image information of the punching end; the control unit is further configured to obtain punching state information and generate guiding information according to the second image information and the three-dimensional model; The surgical robot system further includes a prompting device, communicatively connected to the control unit, and configured to receive and display the punching state information and the guiding information.

19. The surgical robot system according to claim 17, wherein The surgical robot system includes a first imaging device and a second imaging device, both the first imaging device and the second imaging device are communicatively connected to the control unit, the first imaging device is configured to acquire first body surface information and lesion information of a surgical object in a first state, and the second imaging device is configured to acquire second body surface information of the surgical object in a second state; the control unit establishes a first physical sign image model according to the first body surface information and the lesion information, and establishes a second physical sign image model according to the second body surface information, and the first physical sign image model and the second physical sign image model are used to obtain the first hole position.

20. The surgical robot system according to claim 19, wherein The first imaging device includes any one of an MRI, an X-ray device, or a B-ultrasound; the second imaging device includes a binocular vision camera or a structured light camera.

Images