Computer-readable storage medium, electronic device, and surgical robot system

Abstract

The present invention relates to a computer-readable storage medium, electronic device, and surgical robot system. The computer-readable storage medium stores a program that, when executed, performs the following steps: establishing a first three-dimensional model of a target area within a patient's body based on first image information of the target area; obtaining drilling status information based on second image information of a drilling tip of a drilling device that has penetrated the patient's body surface and the first three-dimensional model, and generating guidance information. When the computer-readable storage medium is applied to a surgical robot system, the drilling status can be monitored and guidance information generated, significantly reducing the operator's reliance on experience during the drilling operation and improving surgical safety.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a computer-readable storage medium, an electronic device, and a surgical robot system. Background Art

[0002] Surgical robots are designed to precisely perform complex surgical procedures using minimally invasive methods. Developed in response to the limitations of traditional surgical procedures, they transcend the limitations of the human eye, using stereoscopic imaging technology to present internal organs more clearly to the operator. Furthermore, even in confined areas where some people's hands cannot reach, surgical robots can still control the movement, swinging, clamping, and 360-degree rotation of surgical instruments, while avoiding vibration and improving surgical precision. This further enhances the advantages of smaller incisions, less bleeding, faster postoperative recovery, and significantly shortened postoperative hospital stays. Consequently, surgical robots are highly favored by doctors and patients, and are widely used in their respective clinical procedures.

[0003] As with traditional surgery, before using a surgical robot, the lesion must be located and the perforation point determined based on the lesion's location. The perforation is then performed at that point before the surgical procedure can begin. The perforation device used for perforation is typically very sharp, and the operator must apply considerable force to penetrate the patient's body surface. Therefore, the perforation operation is highly dependent on the operator's experience. Inexperienced operators can easily overexert themselves, causing the perforation device to penetrate the body surface and injure tissue, causing unnecessary trauma to the patient and compromising surgical safety. Summary of the Invention

[0004] The purpose of the present invention is to provide a computer-readable storage medium, an electronic device and a surgical robot system to guide the operator during the drilling process, improve surgical safety and reduce the experience requirements of the operator.

[0005] To achieve the above object, the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed, the following steps are performed:

[0006] establishing a first three-dimensional model of the target area in the patient's body according to first image information of the target area;

[0007] The punching state information is acquired based on the second image information of the punching end of the punching device passing through the patient's body surface and the first three-dimensional model, and the guidance information is generated.

[0008] Optionally, the first image information and the second image information are collected by an image acquisition device, and the image acquisition device is mounted on an image arm; the punching device passes through the patient's body surface at an actual punching point;

[0009] The program also performs the following steps:

[0010] determining a target posture of the image acquisition device according to the position of the actual punching point, so that when the image acquisition device is in the target posture, the actual punching point is within the field of view of the image acquisition device;

[0011] planning a motion plan of the image acquisition device according to at least one of the first three-dimensional model, the initial position of the image acquisition device, and the target position; and

[0012] The image arm is driven to move to drive the image acquisition device to move to the target posture according to the movement plan.

[0013] Optionally, the target posture includes a target position; the initial posture includes an initial position;

[0014] The program performs the following steps:

[0015] planning a motion path based on the first three-dimensional model, the initial position of the image acquisition device, and the target position;

[0016] The image acquisition device is driven to move from the initial position to the target position along the motion path.

[0017] Optionally, the program performs the following steps:

[0018] planning a global motion path according to the initial position and the target position, so that the image acquisition device can reach the target position when moving along the global motion path;

[0019] The image acquisition device is driven to move along the global motion path.

[0020] Optionally, when there is an obstacle on the global motion path, the program further performs the following steps:

[0021] Planning a local motion path, where the local motion path is set outside the boundary of the obstacle, and a starting point and an ending point of the local motion path are both on the global motion path;

[0022] The image acquisition device is driven to move along the local motion path so as to avoid the obstacle, and returns to the global motion path after avoiding the obstacle.

[0023] Optionally, before the image acquisition device reaches the target location, the image acquisition device collects the first image information in real time;

[0024] The program also performs the following steps:

[0025] Replanning the local motion path according to the current first image information at intervals of a first predetermined time; and / or,

[0026] The first three-dimensional model is updated according to the current first image information at intervals of a second predetermined time, and the global motion path is replanned according to the updated first three-dimensional model; the second predetermined time is greater than the first predetermined time.

[0027] Optionally, before the image acquisition device moves along the global motion path planned for the first time, the program further performs the following steps:

[0028] driving the image acquisition device to rotate to acquire third image information of an area surrounding the initial position;

[0029] establishing a second three-dimensional model of the area surrounding the initial position according to the third image information;

[0030] driving the image acquisition device to move a predetermined distance in a direction toward the outside of the body according to the second three-dimensional model; and

[0031] The image acquisition device is driven to move and return to the global motion path.

[0032] Optionally, when planning the local motion path, the program performs the following operations:

[0033] performing an expansion operation on the first three-dimensional model to expand the boundary of the obstacle outward by a safe distance to obtain an expanded boundary;

[0034] The local motion path is planned according to the first three-dimensional model after the dilation operation, and the local motion path is set outside the dilation boundary.

[0035] Optionally, the maximum speed of the image acquisition device during the movement according to the movement scheme is V max , acceleration is a, the safety distance is d, and the following formula is satisfied:

[0036] d=V max 2 / (2a).

[0037] Optionally, the target position further includes a target posture;

[0038] The program also performs the following steps:

[0039] planning a rotation scheme according to the current posture of the image acquisition device when it arrives at the target position and the target posture;

[0040] The image acquisition device is driven to rotate to the target posture according to the rotation scheme.

[0041] Optionally, the second image information is collected by an image acquisition device. When the image acquisition device collects the second image information, the program executes the following steps:

[0042] Visual servoing is used to control the position of the image acquisition device so that the punching end is within the field of view of the image acquisition device.

[0043] Optionally, the punching state information includes position information of the punching end and speed information of the punching end, and the guidance information includes at least one of punching progress information, collision warning information, and expected punching direction information;

[0044] The program performs the following steps to obtain the punching status information:

[0045] acquiring position information of the punching end in real time according to the second image information and the first three-dimensional model;

[0046] obtaining a speed of the punching end according to a change in the position of the punching end;

[0047] The program performs at least one of the following steps to obtain the guidance information:

[0048] generating the punching progress information according to the current position information of the punching end and the predetermined punching depth;

[0049] Acquire the collision probability according to the position information of the punching end, the speed information of the punching end, and the first three-dimensional model, and generate the collision warning information;

[0050] The expected punching direction information is acquired according to the position information of the punching end and the first three-dimensional model.

[0051] Optionally, the program performs the following steps to obtain the collision warning information:

[0052] Obtaining a target tissue closest to the punching device according to the position information of the punching end and the first three-dimensional model;

[0053] calculating the distance between the punching device and the target tissue;

[0054] Calculating a collision time based on the speed information of the punching end and the distance, and determining whether the collision time is greater than a set time threshold;

[0055] If not, it is determined that the collision probability is high, and the collision warning information is generated.

[0056] Optionally, the program performs the following steps to obtain the expected punching direction information:

[0057] obtaining a cutting plane of the target tissue;

[0058] Obtaining a direction vector of a punching end of the punching device;

[0059] The direction vector is projected onto the tangent plane to obtain the expected puncturing direction information.

[0060] Optionally, the punching device penetrates the patient's body surface at the actual punching point on the patient's body surface, and the program further performs the following steps:

[0061] establishing a first vital sign image model based on first body surface data and lesion data of a patient in a first state, wherein the first vital sign image model is used for planning pre-punching points;

[0062] establishing a second vital sign image model based on second body surface data of the patient in a second state;

[0063] Image registration is performed on the second vital signs image model and the first vital signs image model to convert the pre-punch points on the first vital signs image model into target punch points on the second vital signs image model, so that the target punch points represent the actual punch points on the patient's body.

[0064] To achieve the above objectives, the present invention further provides an electronic device, comprising a processor and a computer-readable storage medium as described in any of the preceding items.

[0065] To achieve the above objectives, the present invention further provides a surgical robot system, comprising:

[0066] A tool arm for mounting a punching device; the punching device includes a punching tip, the punching tip being configured to penetrate the patient's body surface at an actual punching point on the patient's body surface and enter the patient's body;

[0067] an imaging arm for mounting an image acquisition device, wherein the image acquisition device is inserted into a patient's body and acquires first image information of a target area in the patient's body and second image information of the punching end entering the patient's body; and

[0068] A control unit is communicatively connected to the image acquisition device and is configured to execute the program stored on the computer-readable storage medium as described in any of the preceding items.

[0069] Optionally, the surgical robot system also includes the image acquisition device; the image acquisition device includes a mirror arm and an image acquisition element; the mirror arm includes a first rigid segment, a controllable bending segment and a second rigid segment connected in sequence from the proximal end to the distal end, and the controllable bending segment includes a bellows or a snake bone; the image acquisition element is arranged on the second rigid segment, and the image acquisition element includes any one of a binocular camera, a laser sensor or a 3D structured light camera.

[0070] Optionally, the surgical robot system further includes a prompt unit, which is communicatively connected to the control unit and configured to receive the guidance information and provide guidance.

[0071] Optionally, the surgical robot system further includes the punching device, wherein a marker is provided on the punching end of the punching device, and the image acquisition device is used to identify the marker and acquire the second image information of the punching end.

[0072] Compared with the prior art, the computer-readable storage medium, electronic device, and surgical robot system of the present invention have the following advantages:

[0073] The aforementioned computer-readable storage medium stores a program that, when executed, performs the following steps: establishing a first three-dimensional model of a target area within a patient's body based on first image information of the target area; obtaining drilling status information based on second image information of the conical tip of a drilling device penetrating the patient's body surface and the first three-dimensional model, and generating guidance information. When the computer-readable storage medium is used in a surgical robotic system for drilling, the actual position of the drilling device within the target area within the patient's body can be displayed in real time, instructing the operator on how to proceed. This avoids unnecessary damage to the patient caused by tissue puncture during the drilling process, reduces surgical risk, and reduces reliance on the operator's experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0075] Figure 1 is a schematic diagram of surgical drilling guidance information provided according to one embodiment of the present invention when applied to a surgical robot system;

[0076] Figure 2 is a schematic diagram of a surgical robot system according to one embodiment of the present invention when drilling a hole on a patient's body surface;

[0077] Figure 3 is a schematic diagram of a punching device of a surgical robot system provided according to one embodiment of the present invention;

[0078] Figure 4is a flow chart of a guidance method for a surgical robot system provided according to one embodiment of the present invention;

[0079] Figure 5 is a schematic diagram of actual perforation points determined on a patient's body surface by a surgical robot system provided by one embodiment of the present invention;

[0080] Figure 6 This is a flow chart of determining an actual perforation point on a patient's body surface by a surgical robot system provided by one embodiment of the present invention;

[0081] Figure 7 is a schematic diagram of a surgical robot system according to one embodiment of the present invention acquiring first body surface information and lesion information of a patient using a first imaging device;

[0082] Figure 8 is a schematic diagram of a surgical robot system according to one embodiment of the present invention acquiring second body surface information of a patient using a second imaging device;

[0083] Figure 9 This is a schematic diagram of a surgical robot system according to one embodiment of the present invention establishing a mapping relationship between a target on a patient's body surface and a control unit;

[0084] Figure 10 1 is a schematic diagram of the working principle of a binocular camera of an endoscope of a surgical robot system provided in accordance with one embodiment of the present invention;

[0085] Figure 11 This is a flow chart of a control unit for surgical drilling guidance information according to an embodiment of the present invention establishing a first three-dimensional model based on first image information;

[0086] Figure 12 This is a flow chart of a control unit of a surgical robot system according to one embodiment of the present invention planning a plan and driving an endoscope to move along the plan;

[0087] Figure 13 is a schematic diagram of a control unit of a surgical robot system according to one embodiment of the present invention planning a motion path of an endoscope;

[0088] Figure 14 is a schematic diagram of a control unit of a surgical robot system according to an embodiment of the present invention planning a motion plan for an endoscope, wherein a safe distance is maintained between the local motion path and obstacles;

[0089] Figure 15 is a schematic diagram of an endoscope of a surgical robot system provided by one embodiment of the present invention being retracted toward the outside of the body before moving along a motion plan;

[0090] Figure 16 is a schematic structural diagram of an endoscope of a surgical robot system provided according to one embodiment of the present invention;

[0091] Figure 17 is a schematic structural diagram of an endoscope of a surgical robot system provided by the present invention according to an alternative embodiment;

[0092] Figure 18 is a schematic diagram showing a principle of visual servo control of an endoscope by a control unit of a surgical robot system provided in accordance with one embodiment of the present invention;

[0093] Figure 19 This is a flow chart of a control unit of a surgical robot system according to an embodiment of the present invention obtaining a drilling process;

[0094] Figure 20 This is a flow chart of a control unit of a surgical robot system according to one embodiment of the present invention obtaining a collision reminder and a drilling direction;

[0095] Figure 21 is a schematic diagram of a control unit of a surgical robot system according to an embodiment of the present invention acquiring a cutting plane of a target tissue;

[0096] Figure 22 This is a schematic diagram of a control unit of a surgical robot system according to one embodiment of the present invention obtaining an expected drilling direction. DETAILED DESCRIPTION

[0097] The following describes the embodiments of the present invention through specific specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in this embodiment only illustrate the basic concept of the present invention in a schematic manner, and the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components during actual implementation. During actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0098] In addition, each embodiment described below has one or more technical features. However, this does not mean that users of the present invention must implement all technical features in any embodiment at the same time, or that they can only implement some or all technical features in different embodiments separately. In other words, as long as implementation is possible, those skilled in the art can, based on the disclosure of the present invention and depending on design specifications or implementation requirements, selectively implement some or all technical features in any embodiment, or selectively implement a combination of some or all technical features in multiple embodiments, thereby increasing the flexibility of the implementation of the present invention.

[0099] As used in this specification, the singular forms "a", "an", and "the" include plural objects, and the plural form "a plurality" includes more than two objects, unless the content clearly indicates otherwise. As used in this specification, the term "or" is generally used in a sense including "and / or", unless the content clearly indicates otherwise, and the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection. It can be a mechanical connection or an electrical connection. It can be directly connected or indirectly connected through an intermediate medium, and it can be a connection between the internal parts of two elements or an interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0100] To make the objects, advantages, and features of the present invention more apparent, the present invention is further described below in detail with reference to the accompanying drawings. It should be noted that the drawings are greatly simplified and not to exact scale, and are intended solely to facilitate and clearly illustrate the embodiments of the present invention. The same or similar reference numerals in the drawings represent the same or similar components.

[0101] Figure 1 The schematic diagram of the application scenario of the surgical robot system of the present invention is shown. Figure 1 The surgical robot system includes a control end and an execution end. The control end includes a doctor's console and a doctor's end control device 10 provided on the doctor's console. The execution end includes a patient-end control device, a surgical operation device 20, an image display device 30 and other equipment. Among them, an image arm 300 and a tool arm 400 are mounted on the surgical operation device 20. The tool arm 400 is used to mount a punching device 100, and the punching device 100 is used to punch at the actual punching point M (such as Figure 2The image arm 300 is used to mount an image acquisition device, and the image acquisition device is used to acquire image information of the area or device of interest (such as the first image information, the second image information, etc. described later). The image acquisition device is, for example, an endoscope 200. In addition, the surgical robot system further includes a control unit, which is communicatively connected with the image arm 300, the tool arm 400 and the endoscope 200. The control unit may be arranged at the patient-end control device, or at the doctor-end control device, or a part may be arranged at the patient-end control device and the other part may be arranged at the doctor-end control device. That is to say, the present invention does not limit the specific setting method of the control unit, as long as it can perform the relevant functions. Figure 2 FIG2 shows a schematic diagram of a surgical robot system provided by an embodiment of the present invention when performing a drilling operation. Figure 3 FIG. 1 shows a schematic structural diagram of a punching device. Figure 2 and Figure 3 As shown, the punching device 100 includes a punching end, such as a tapered tip 110, which is used to penetrate the patient's body surface at the actual punching point M to perform the punching operation. The endoscope 200 is used to be inserted into the patient's body and collect first image information of the target area in the patient's body, and to collect second image information of the tapered tip 110 in real time after the tapered tip 110 penetrates the patient's body surface (i.e., the tapered tip enters the patient's body). The control unit is configured to establish a first three-dimensional model of the target area based on the first image information; and after the tapered tip 110 penetrates the patient's body surface, obtain punching status information based on the second image information and the first three-dimensional model, and generate guidance information to guide the punching operation.

[0102] The control unit establishes a three-dimensional model of the target area based on the first image information captured by the endoscope 200, thereby determining the positions of various tissues within the target area. Furthermore, the endoscope 200 monitors the position and velocity of the conical tip 110 of the punching device 100 by capturing in real time second image information of the conical tip 110 of the punching device 100 after it has penetrated the patient's body surface. Based on the positions of various tissues and the position and velocity of the conical tip 110, the operator can obtain real-time information about the punching status and guidance information, and adjust the punching direction and other operations based on the guidance information to avoid tissue puncture, thereby reducing the operator's reliance on experience, improving surgical safety, shortening surgical time, and reducing operator fatigue. In this embodiment, the "target area" is determined based on the specific procedure. For example, in laparoscopic surgery, the target area is the abdominal cavity, while in thoracoscopic surgery, the target area is the thoracic cavity. The following description uses the abdominal cavity as the target area in laparoscopic surgery as an example. Furthermore, a marker 111 may be provided on the surface of the conical tip 110. The marker 111 can be identified by the endoscope 200 so that the endoscope 200 can obtain the second image information of the conical tip 110. The marker 111 may be, for example, a reflective material or a luminous body coated on the surface of the conical tip 110. Furthermore, those skilled in the art will appreciate that the actual perforation point M refers to the point at which the surgical instrument penetrates the body.

[0103] The endoscope 200 is inserted into the patient's abdominal cavity before punching (the punching here is punching at the actual punching point M), for example, it is pre-inserted into the patient's abdominal cavity according to the doctor's experience and is in a predetermined initial position. The endoscope 200 includes an image acquisition element 210 (such as Figure 16 and Figure 17As shown in FIG2 , when the endoscope 200 is in the initial position, the image acquisition element 210 can acquire the first image information, in which the image of the target tissue in the abdominal cavity is displayed. When the endoscope 200 is in the initial position, the image acquisition element 210 may completely deviate from the actual punching point M, causing the endoscope 200 to be unable to timely acquire the second image information after the conical tip 110 penetrates the patient's body surface. Therefore, the control unit is further configured to determine the target position of the endoscope 200 based on the position of the actual punching point M, so that when the endoscope 200 is in the target position, the actual punching point M is within the field of view of the endoscope 200 (that is, the actual punching point is within the field of view of the image acquisition element 210). In this way, as long as the conical tip 110 penetrates the patient's body surface, the endoscope 200 can immediately acquire the second image information. Furthermore, the control unit is further configured to plan a motion plan for the endoscope 200 based on the first three-dimensional model, the initial position of the endoscope 200, and the target position, and drive the endoscope 200 to move to the target position according to the motion plan. As previously mentioned, the endoscope 200 is mounted on the imaging arm 300, so the control unit can drive the endoscope 200 to move by controlling the movement of the imaging arm 300.

[0104] Furthermore, before performing the puncture operation, it is also necessary to determine the position of the actual puncture point M on the patient's body surface. In this embodiment, the actual puncture point M can be determined by the control unit.

[0105] Thus, in an exemplary embodiment, the method for performing drilling guidance during surgical drilling using the surgical robot system can be as follows: Figure 4 As shown, the following steps are included:

[0106] Step S10: determining the actual puncture point on the patient's body surface, for example, the doctor determines the actual puncture point manually or automatically using a robot.

[0107] Step S20: inserting the endoscope into the patient's abdominal cavity, for example, selecting a suitable point for inserting the endoscope based on the doctor's experience.

[0108] Step S30: The endoscope collects first image information of the abdominal cavity (ie, the target area) and sends the first image information to the control unit.

[0109] Step S40: The control unit establishes a first three-dimensional model of the abdominal cavity according to the first image information.

[0110] Step S50: The control unit determines the target posture of the endoscope according to the actual punching point.

[0111] Step S60: The control unit plans a motion plan of the endoscope based on the first three-dimensional model, the initial position of the endoscope and the target position of the endoscope, and drives the image arm to move to drive the endoscope to move to the target position according to the motion plan.

[0112] Step S70: The control unit drives the tool arm to move to start punching using the punching device, and after the conical tip of the punching device penetrates the patient's body surface and enters the abdominal cavity, the endoscope collects second image information of the conical tip and sends it to the control unit.

[0113] Step S80: The control unit obtains punching status information based on the second image information and the first three-dimensional model, and generates guidance information to guide the punching until the hole is formed (i.e., the punching is completed). The operator adjusts the punching operation based on the punching status information and the guidance information to avoid puncturing tissue.

[0114] After a punching operation is completed, the operator can return the endoscope 200 to its initial position. This process can be performed by any suitable method, such as retracting the endoscope 200 into the punching card and then moving the endoscope 200 back to the initial position. Steps S50 to S80 are then repeated until all punching operations are completed.

[0115] Next, this article will introduce in detail with the accompanying drawings Figure 4 The implementation of each step of the method shown.

[0116] Figure 5 A schematic diagram showing the actual puncture point M on the patient's body surface is shown. Figure 6 A flow chart for determining the actual puncture point M on the patient's body surface is shown.

[0117] Please refer to Figure 5 and Figure 6 The method for determining the actual puncture point on the patient's body surface comprises the following steps:

[0118] Step S11: The control unit obtains a first vital sign image model. The first vital sign image model is established based on first body surface data and lesion data of a patient in a first state. In this embodiment, the control unit may directly establish the first vital sign image model. In an alternative embodiment, the first vital sign image model may be established by an external device, after which the control unit may receive the first vital sign image model via data transmission, or the operator may manually input relevant data of the first vital sign image model into the control unit.

[0119] Step S12: Planning pre-punch points on the first vital sign image model. In this embodiment, the pre-punch points can be determined by the control device through a three-dimensional simulation of the patient's vital signs, combined with a model of the robotic arm to simulate punching, while comprehensively considering the robotic arm's workspace and collision safety. Alternatively, in an alternative embodiment, the pre-punch points can be determined by the operator based on experience, and then simulated punching can be performed by the control unit to verify the appropriate location of the pre-punch points.

[0120] Step S13: the control unit establishes a second vital sign image model according to the second vital sign data of the patient in the second state.

[0121] Step S14: the control unit performs image registration on the second vital sign image model and the first vital sign image model to convert the pre-punching points on the first vital sign image model into target punching points on the second vital sign image model.

[0122] Step S15: marking the target perforation point on the second vital sign image model on the patient's body according to the mapping relationship between the second vital sign image model and the patient, so as to obtain the actual perforation point.

[0123] In this embodiment, Figure 7 As shown, the first body surface data and lesion data are acquired by a first imaging device 600, which may include MRI, CT or other X-ray devices, as long as it can simultaneously scan the patient's body surface features and lesion features. Figure 8 As shown, the second body surface data is collected by the second imaging device 500, and the second imaging device includes but is not limited to a 3D visual system. When the patient is in the first state and the second state respectively, there are differences in the patient's body position. Generally, the first state refers to the state of the patient in the diagnosis stage, and the second state refers to the state of the patient in preoperative preparation. In laparoscopic surgery, the first state refers to the state of the patient before pneumoperitoneum, and the second state is the state of the patient after pneumoperitoneum is established. In other surgeries or other environments, the difference between the first state and the second state may also be due to different states caused by reasons such as the patient holding his breath, being full, or having a bowel movement. In addition, those skilled in the art will understand that the order of step S12 and step S13 may also be interchangeable.

[0124] In practice, the first imaging device 600, the second imaging device 500, the control unit and the patient are in different coordinate systems, but those skilled in the art can use conventional methods to establish mapping relationships between different coordinate systems. In a specific embodiment, Figures 7 to 9As shown, when the second imaging device collects the second body surface data, a plurality of targets 1 are distributed on the patient's body surface. The targets 1 can be target pens or reflective balls. The positions of the plurality of targets 1 are calibrated by the operator, and a first coordinate system F1 (i.e., the patient coordinate system) is established based on the positions of the plurality of targets 1. The second imaging device 500 is in the second coordinate system F2. The second imaging device 500 obtains the coordinates of the target 1 as the second body surface data, thereby knowing the mapping relationship between the second coordinate system F2 and the first coordinate system F1. The first imaging device 600 is in the third coordinate system F3. In step S14, the mapping relationship between the second coordinate system F2 and the third coordinate system F3 can be obtained through image registration, and then the position of the target punching position in the first coordinate system F1 can be obtained. The imaging arm 300 is in the fourth coordinate system F4. The mapping relationship between the fourth coordinate system F4 and the first coordinate system F1 can be directly obtained in the world coordinate system F0. Thus, a mapping relationship between various coordinate systems can be established to realize coordinate conversion between different coordinate systems, thereby obtaining the actual punching point M on the patient's body surface (e.g. Figure 5 shown).

[0125] Next, the operator inserts the endoscope 200 into the patient's abdominal cavity and uses the image acquisition element 210 to capture the first image information. It should be noted that the endoscope 200 is inserted into the patient's abdominal cavity from an endoscopic point on the patient's body surface. The endoscopic point and the actual perforation point can be determined simultaneously, that is, the endoscopic point is planned simultaneously with the actual perforation point. Thus, step S12 also plans a pre-endoscopic point, step S14 also includes converting the pre-endoscopic point into a target endoscopic point, and step S15 also includes indicating the pre-endoscopic point on the patient's body to obtain the actual endoscopic point.

[0126] Optionally, the image acquisition element 210 is a binocular camera. The binocular camera includes a first camera 211 and a second camera 212 (eg Figure 10 As shown in FIG, the first image information includes the first sub-image information captured by the first camera 211 and the second sub-image information captured by the second camera 212. Therefore, the step S30 includes the first camera 211 capturing the first sub-image information and the second camera 212 capturing the second sub-image information. The imaging principle of the binocular camera is as follows: Figure 10 As shown in the figure, f is the focal length of the camera, b is the baseline of the first camera and the second camera, and P(x, y, z) is the coordinate of the spatial point being photographed. Then, f, b and P(x, y, z) satisfy the following relationship:

[0127]

[0128] Those skilled in the art will appreciate that, in alternative embodiments, the image acquisition element 210 may also be a laser scanner or a 3D structured light camera.

[0129] Therefore, if Figure 11 As shown, the step S40 may include:

[0130] Step S41: the control unit extracts feature points from the first sub-image information and the second sub-image information.

[0131] Step S42: the control unit pairs the feature points on the first sub-image information and the second sub-image information to form a feature point pair.

[0132] Step S43: The control unit locates the feature point pairs in three-dimensional space within the camera coordinate system based on the epipolar constraints. Since the endoscope 200 is driven by the imaging arm 300, the position of the camera coordinate system within the control unit's coordinate system can be calculated based on the forward kinematics of the imaging arm 300 and pre-calibrated camera coordinate system parameters. After all feature points are located, a point cloud model of the feature points is generated.

[0133] Step S44: the control unit establishes the first three-dimensional model according to the point cloud model.

[0134] Next, the control unit executes step S50 and step S60.

[0135] In step S50, the target position determined by the control unit includes a target position and a target posture. The target position is close to the actual punching point, and when the endoscope 200 is located at the target position and in the target posture, the image acquisition element 210 of the endoscope 200 is set toward the actual punching point M. In this way, the actual punching point M is within the field of view of the endoscope 200, so that when the conical tip 110 of the punching device 100 penetrates the patient's body surface, it can be immediately recognized by the endoscope 200. It should be understood that the image acquisition element 210 can be arranged directly opposite the actual punching point M or slightly offset, as long as the actual punching point M is within the field of view of the endoscope 200.

[0136] In step S60, the motion scheme may include a movement scheme along the motion path and a rotation scheme at the target position. Figure 12 As shown, the step S60 includes:

[0137] Step S61: The control unit plans the motion path based on the first three-dimensional model, the initial position of the endoscope (i.e., the position when the endoscope is in the initial posture) and the target position of the endoscope; and drives the endoscope to move along the motion path to the target position.

[0138] Step S62: The control unit plans the rotation plan based on the current posture of the endoscope when it reaches the target position and the target posture of the endoscope. When the endoscope rotates according to the rotation plan, the endoscope can move to the target posture. The rotation plan includes a rotation direction and a rotation angle.

[0139] Step S63: The control unit drives the endoscope to move to the target posture according to the rotation scheme.

[0140] It is understandable that after step S61 is completed, if the current posture of the endoscope 200 coincides with the target posture, then there is no need to plan the rotation scheme, that is, step S62 and step S63 can be omitted.

[0141] For further information, please refer to Figure 12 , and combined with Figure 13 , the motion path may include a global motion path L1, and step S61 includes:

[0142] Step S611: the control unit plans the global motion path L1 according to the initial position and the target position, so that the endoscope can reach the target position when moving along the global motion path L1.

[0143] Step S612: The control unit drives the endoscope to move along the global motion path L1.

[0144] In this embodiment, the image arm 300 is used to drive the movement of the endoscope 200. The image arm 300 includes at least one joint. Therefore, after planning the global motion path L1 of the endoscope, a time constraint can be imposed on the global motion path L1 to obtain the temporal relationship between the position of the endoscope 200 and time. The acceleration, velocity, and position of the joints on the image arm 300 can then be calculated using inverse kinematics. The control unit can then drive the joints on the image arm 300 to move according to the acceleration, velocity, and position, thereby driving the endoscope 200 to move along the global motion path L1. The global motion path L1 can originate from the initial position and terminate at the target position; alternatively, the global motion path L1 can originate from the endoscope point and terminate at the target position. This is not limited in this embodiment, as long as the endoscope 200 can reach the target position while moving along the global motion path L1.

[0145] As 1 Figure 12 As shown, in some cases, the global motion path L1 avoids all human tissues in the abdominal cavity. At this time, the control unit may continue to execute step S612 until the endoscope 200 moves along the global motion path L1 to the target area.

[0146] However, in other cases, the global motion path L1 passes through part of human tissue. That is, when the endoscope 200 moves along the global motion path L1, the part of the tissue will constitute an obstacle that hinders the endoscope 200 from moving along the global motion path L1. In this case, please continue to refer to Figure 12 , the motion scheme also includes a local motion path L2, then step S61 further includes:

[0147] Step S613: the control unit plans a local motion path L2, where the local motion path L2 is set outside the boundary of the obstacle S, and both the starting point and the ending point of the local motion path L2 are on the global motion path L1.

[0148] Step S614: the control unit drives the endoscope to move along the local motion path L2 so that the endoscope bypasses the obstacle S and returns to the global motion path L1 after avoiding the obstacle S.

[0149] Afterwards, the control unit returns to execute step S612 and continues to drive the image arm 300 to move the endoscope 200 along the global motion path L1 (i.e., the control unit continues to execute step S612). When there are multiple obstacles S on the global motion path L1, the control unit repeats steps S612 to S614 until the endoscope 200 reaches the target position. In this embodiment, the control unit can use methods such as Dijkstra algorithm, A* algorithm, random forest algorithm, artificial potential field method, etc. to obtain the local motion path L2.

[0150] Preferably, please continue to Figure 12 , and combined with Figure 14 When planning the local motion path L2, the control unit further performs the following steps:

[0151] Step S6131: performing an expansion operation on the first three-dimensional model to expand the boundary of the obstacle S outward by a safe distance to obtain an expanded boundary S1;

[0152] Step S6132: planning the local motion path according to the first three-dimensional model after the dilation operation, so that the local motion path L2 is set outside the dilation boundary S1.

[0153] Optionally, during the movement of the endoscope 200 according to the movement scheme, the maximum movement speed of the endoscope 200 is V max , the acceleration is a, the safety distance d can satisfy the following formula: d = V max 2 / (2a). It can be understood that the maximum movement speed V max , and the acceleration a are all artificially set values. In addition, the "outside" mentioned here refers to the side facing the outside of the obstacle.

[0154] The advantage of this arrangement is that the local motion path L2 is spaced at least a safe distance d from the tissue serving as the obstacle S, thereby preventing the endoscope 200 from colliding with the obstacle S during movement and damaging the tissue, thereby improving safety.

[0155] Furthermore, before the endoscope 200 moves to the target position, the endoscope 200 collects the first image information in real time, and the control unit is configured to re-plan the local motion path L2 based on the current first image information at every first predetermined time interval. Specifically, the local motion path L2 is updated based on the information around the obstacle S on the current first image information. Furthermore, the control unit updates the first three-dimensional model based on the current first image information at every second predetermined time interval, and re-plans the global motion path L1 based on the updated first three-dimensional model. This ensures that the motion path adapts to changes in the patient's condition and improves the movement safety of the endoscope 200. Since the first image information required to construct the first three-dimensional model is large and the time is long, the second predetermined time is greater than the first predetermined time.

[0156] Furthermore, after the first planning of the global motion path L1 is completed and before the endoscope 200 moves along the global motion path L1, the control unit is further configured to drive the endoscope 200 to rotate (eg Figure 15 The control unit drives the endoscope 200 to move a predetermined distance (such as the distance between the endoscope 200 and the target) in a direction toward the outside of the body according to the second three-dimensional model. Figure 15 (indicated by arrow c in the middle). The "predetermined distance" is determined based on actual conditions, specifically, after the endoscope 200 has moved the predetermined distance, no tissue prevents it from returning to the global motion path L1. The control unit then drives the endoscope 200 to move and return to the global motion path L1, and then drives the endoscope 200 to move along the global motion path L1. This operation can avoid the problem of the endoscope 200 being unable to move due to an unfavorable starting position.

[0157] During the movement of the endoscope 200, the endoscope 200 may need to generate bending motion according to the posture of the endoscope 200. Therefore, the endoscope 200 used in this embodiment is a flexible endoscope. Figure 16 and Figure 17 As shown, the endoscope 200 includes the image acquisition element 210 and a mirror arm, wherein the mirror arm includes a first rigid section 221, a controllable bending section 222 and a second rigid section 223 connected in sequence from the proximal end to the distal end. The controllable bending section 222 includes a bellows (such as Figure 15 As shown), or the controllable bending section 222 includes a snake bone (as shown Figure 16 The image capture element 210 is disposed on the second rigid section 223. Furthermore, the endoscope 200 further includes components such as a drawstring (not shown) and a light source (not shown). The drawstring is disposed in a drawstring hole in the endoscope arm, and the controllable bending section 222 is bent or straightened by tightening or loosening the drawstring. The specific configuration of the drawstring is conventional in the art and will not be described in detail here. The light source is disposed on the second rigid section 223 to provide illumination for the image capture element 210.

[0158] As previously described, by motion planning and driving the endoscope 200 to the target position before drilling, the image acquisition element 210 of the endoscope 200 can be positioned toward the actual drilling point M, allowing the endoscope 200 to promptly capture the second image information. However, during the drilling process, after the conical tip 110 of the drilling device 100 penetrates the patient's body surface, as the drilling process progresses, the drilling device 100 enters an increasing portion of the abdominal cavity. If the endoscope 200 remains in the target position, the conical tip 110 may move outside the field of view of the endoscope 200. To address this issue, in step S70, the control unit is further configured to use visual servoing to control the position of the endoscope 200, thereby controlling the movement of the endoscope so that the conical tip 110 is within the field of view of the endoscope 200, preferably at the center of the field of view, that is, so that the field of view of the endoscope 200 automatically captures the conical tip 110.

[0159] Visual servo control compares real-time measured image information with given image information and uses the resulting image error for feedback to form a closed-loop control loop, ensuring the controlled object maintains a given posture. In this embodiment, the given image information is the center of the endoscope's 200 field of view, with the conical tip 110 positioned at the center. Figure 18 FIG. 4 shows a schematic diagram showing the principle of the control unit performing visual servo control on the endoscope 200, as shown in FIG. Figure 18 As shown, the control unit includes a visual servo controller 601 and an image arm joint controller 602, and a joint sensor 301 is provided on the joint of the image arm 300. The process of visual servo control is as follows:

[0160] The endoscope 200 collects the second image information as actual image information and sends it to the servo controller.

[0161] The visual servo controller determines whether the conical tip 110 that has entered the abdominal cavity is in the center of the field of view of the endoscope 200 based on the actual image information. If not, the visual servo controller extracts the actual posture of the endoscope 200 based on the error between the actual image information and the given image information, and obtains motion information such as the motion speed and motion direction of the endoscope 200 when it moves from the actual posture to the given posture.

[0162] The motion information of the joints of the image arm 300 is calculated based on the motion information of the endoscope 200 and the inverse kinematics of the image arm, and is sent to the image arm joint controller 602 .

[0163] The image arm joint controller 602 drives the relevant joints on the image arm 300 to move, and the joint sensor 301 feeds back joint information in real time until the joints of the image arm 300 drive the endoscope 200 to move to the specified posture.

[0164] In step S80, the drilling status information acquired by the control unit includes the position of the tapered tip 110 and the speed of the tapered tip 110. The guidance information includes drilling progress, collision reminder, and expected drilling direction.

[0165] The control unit is configured to obtain a position of the conical tip 110 based on the second image information and the first three-dimensional model. Obtain a speed of the conical tip 110 based on a change in the position of the conical tip 110. Obtain the drilling progress based on the position of the conical tip 110 and a predetermined drilling depth. Obtain a collision probability based on the position of the conical tip 110, the speed of the conical tip 110, and the first three-dimensional model, and generate a collision alert. Obtain the expected drilling direction based on the position of the conical tip 110 and the first three-dimensional model.

[0166] Among them, such as Figure 19 As shown, the method for obtaining the punching process includes:

[0167] Step S81: the control unit obtains the current drilling depth z1 according to the position of the conical tip.

[0168] Step S82: The control unit compares the current punching depth z1 with the expected punching depth z0 to obtain a ratio between the two as the punching progress.

[0169] like Figure 20 As shown, the method for obtaining the collision reminder includes the following steps:

[0170] Step S83: The control unit obtains the target tissue closest to the punching device according to the position of the conical tip and the first three-dimensional model.

[0171] Step S84: the control unit calculates the distance between the target tissue and the punching device.

[0172] Step S85: The control unit calculates the collision occurrence time t based on the speed of the conical tip and the distance. In this embodiment, the speed of the conical tip is v1 and the distance is D, then the collision occurrence time t satisfies: t=D / v1.

[0173] Step S86: The control unit determines whether the time t is greater than the set time threshold t0. If not, it determines that the collision probability is large and generates the collision reminder. If so, it determines that the collision probability is small and does not generate the collision reminder.

[0174] Generally, the control unit generates the expected punching direction only after determining that the collision probability is high. If the collision probability is low, there is no need to generate the expected punching direction. Figure 20 , and combined with Figure 21 and Figure 22 The method in which the control unit generates the expected punching direction is as follows:

[0175] Step S87: The control unit obtains the cutting plane Q of the target tissue and the direction vector of the conical tip 110

[0176] Step S88: The control unit converts the direction vector Projected onto the tangential plane Q of the target tissue to obtain the expected punching direction vector

[0177] Furthermore, the surgical robot system also includes a prompting device, which is used to communicate with the control unit to receive the guidance information and provide prompts through the prompting device. Optionally, the prompting device can have multiple options. For example, the prompting device may include a buzzer alarm to prompt the collision reminder through a buzzer alarm. The prompting device may also include a voice prompting device for broadcasting the collision reminder, the punching progress and the expected punching direction. The prompting device may also include a display device for displaying the collision reminder, the punching progress and the expected punching direction in a visual manner such as text and images. This embodiment is not limited to this.

[0178] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon, and when the program is executed, the corresponding steps executed by the control unit as described above are executed.

[0179] Furthermore, an embodiment of the present invention further provides an electronic device, comprising a processor and the aforementioned computer-readable storage medium, wherein the processor is configured to execute a program stored on the computer-readable storage medium.

[0180] Furthermore, an embodiment of the present invention also provides a punching guidance method, which includes the steps performed by the aforementioned program.

[0181] While the present invention is disclosed above, it is not limited thereto. Those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to encompass such modifications and variations as long as they fall within the scope of the claims and their equivalents.

Claims

1. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed, the following steps are performed: establishing a first three-dimensional model of the target area in the patient's body according to first image information of the target area; Acquire drilling state information based on second image information of the drilling end of the drilling device passing through the patient's body surface and the first three-dimensional model, and generate guidance information; The punching status information includes the position information and speed information of the punching end; the guidance information includes collision warning information; The program performs the following steps to obtain the collision warning information: Obtaining a target tissue closest to the punching device according to the position information of the punching end and the first three-dimensional model; calculating the distance between the punching device and the target tissue; Calculating a collision time based on the speed information of the punching end and the distance, and determining whether the collision time is greater than a set time threshold; If not, it is determined that the collision probability is high, and the collision warning information is generated.

2. The computer-readable storage medium according to claim 1, wherein The first image information and the second image information are collected by an image acquisition device connected to an image arm; the punching device passes through the patient's body surface at an actual punching point; The program also performs the following steps: determining a target posture of the image acquisition device according to the position of the actual punching point, so that when the image acquisition device is in the target posture, the actual punching point is within the field of view of the image acquisition device; planning a motion plan of the image acquisition device according to at least one of the first three-dimensional model, an initial pose of the image acquisition device, and the target pose; as well as, The image arm is driven to move to drive the image acquisition device to move to the target posture according to the movement plan.

3. The computer-readable storage medium according to claim 2, wherein: The target posture includes a target position; the initial posture includes an initial position; The program performs the following steps: planning a motion path based on the first three-dimensional model, the initial position of the image acquisition device, and the target position; The image acquisition device is driven to move from the initial position to the target position along the motion path.

4. The computer-readable storage medium according to claim 3, wherein: The program performs the following steps: planning a global motion path according to the initial position and the target position, so that the image acquisition device can reach the target position when moving along the global motion path; The image acquisition device is driven to move along the global motion path.

5. The computer-readable storage medium according to claim 4, wherein: When there is an obstacle on the global motion path, the program further performs the following steps: Planning a local motion path, where the local motion path is set outside the boundary of the obstacle, and a starting point and an ending point of the local motion path are both on the global motion path; The image acquisition device is driven to move along the local motion path so as to avoid the obstacle, and returns to the global motion path after avoiding the obstacle.

6. The computer-readable storage medium according to claim 5, wherein: Before the image acquisition device arrives at the target location, the image acquisition device collects the first image information in real time; The program also performs the following steps: Replanning the local motion path according to the current first image information at intervals of a first predetermined time; and / or, The first three-dimensional model is updated according to the current first image information at intervals of a second predetermined time, and the global motion path is replanned according to the updated first three-dimensional model; the second predetermined time is greater than the first predetermined time.

7. The computer-readable storage medium according to claim 6, wherein: Before the image acquisition device moves along the global motion path planned for the first time, the program further performs the following steps: driving the image acquisition device to rotate to acquire third image information of an area surrounding the initial position; establishing a second three-dimensional model of the area surrounding the initial position according to the third image information; driving the image acquisition device to move a predetermined distance in a direction toward the outside of the body according to the second three-dimensional model; and The image acquisition device is driven to move and return to the global motion path.

8. The computer-readable storage medium according to any one of claims 5 to 7, wherein: When planning the local motion path, the program performs the following operations: performing an expansion operation on the first three-dimensional model to expand the boundary of the obstacle outward by a safe distance to obtain an expanded boundary; The local motion path is planned according to the first three-dimensional model after the dilation operation, and the local motion path is set outside the dilation boundary.

9. The computer-readable storage medium according to claim 8, wherein: The maximum speed of the image acquisition device during the movement according to the movement scheme is V max , acceleration is a, the safety distance is d, and the following formula is satisfied: d=V max 2 / (2a)。 10. The computer-readable storage medium according to claim 3, wherein The target position also includes a target posture; The program also performs the following steps: planning a rotation scheme according to the current posture of the image acquisition device when it arrives at the target position and the target posture; The image acquisition device is driven to rotate to the target posture according to the rotation scheme.

11. The computer-readable storage medium according to claim 1 or 2, wherein: The second image information is collected by an image acquisition device. When the image acquisition device collects the second image information, the program executes the following steps: Visual servoing is used to control the position of the image acquisition device so that the punching end is within the field of view of the image acquisition device.

12. The computer-readable storage medium according to claim 1, wherein The guidance information further includes at least one of punching progress information and expected punching direction information; The program performs the following steps to obtain the punching status information: acquiring position information of the punching end in real time according to the second image information and the first three-dimensional model; obtaining a speed of the punching end according to a change in the position of the punching end; The program performs at least one of the following steps: generating the punching progress information according to the current position information of the punching end and the predetermined punching depth; The expected punching direction information is acquired according to the position information of the punching end and the first three-dimensional model.

13. The computer-readable storage medium according to claim 12, wherein: The program performs the following steps to obtain the expected punching direction information: obtaining a cutting plane of the target tissue; Obtaining a direction vector of a punching end of the punching device; The direction vector is projected onto the tangent plane to obtain the expected puncturing direction information.

14. The computer-readable storage medium according to claim 1, wherein The punching device penetrates the patient's body surface at the actual punching point on the patient's body surface, and the program further performs the following steps: establishing a first vital sign image model based on first body surface data and lesion data of a patient in a first state, wherein the first vital sign image model is used for planning pre-punching points; establishing a second vital sign image model based on second body surface data of the patient in a second state; Image registration is performed on the second vital signs image model and the first vital signs image model to convert the pre-punch points on the first vital signs image model into target punch points on the second vital signs image model, so that the target punch points represent the actual punch points on the patient's body.

15. An electronic device, characterized in that: The method comprises a processor and a computer readable storage medium for executing the program according to any one of claims 1 to 14.

16. A surgical robot system, characterized in that: include: a tool arm for connecting a punching device; The punching device includes a punching tip, which is used to penetrate the patient's body surface at an actual punching point on the patient's body surface and enter the patient's body; an imaging arm, configured to connect to an image acquisition device, wherein the image acquisition device is configured to be inserted into a patient's body and acquire first image information of a target area in the patient's body and second image information of the piercing end entering the patient's body; as well as, A control unit is communicatively connected to the image acquisition device and is configured to implement the steps executed by the program according to any one of claims 1 to 14.

17. The surgical robot system according to claim 16, wherein: The surgical robot system also includes the image acquisition device; the image acquisition device includes a mirror arm and an image acquisition element; the mirror arm includes a first rigid section, a controllable bending section and a second rigid section connected in sequence from the proximal end to the distal end, and the controllable bending section includes a bellows or a snake bone; the image acquisition element is arranged on the second rigid section, and the image acquisition element includes any one of a binocular camera, a laser sensor or a 3D structured light camera.

18. The surgical robot system according to claim 16, wherein: The surgical robot system further includes a prompting unit, which is communicatively connected to the control unit and configured to receive the guidance information and provide guidance.

19. The surgical robot system according to claim 16, wherein: The surgical robot system further includes the punching device, wherein a marker is provided on the punching end of the punching device, and the image acquisition device is used to identify the marker and acquire the second image information of the punching end.

Images