Computer-readable storage medium, electronic device, position calibration, and robot system

Abstract

The present invention relates to a computer-readable storage medium, an electronic device, a position calibration, and a robotic system. The computer-readable storage medium stores a program that, when executed, performs the following steps: establishing a first physical sign image model based on first body surface information and lesion information of a surgical object in a first state for planning a pre-hole position; establishing a second physical sign image model based on second body surface information of the surgical object in a second state; registering the second physical sign image model with the first physical sign image model to obtain a target hole position on the second physical sign image model; obtaining position information of a calibration end of a positioning device that moves on the surface of the surgical object, the calibration end being in contact with the surface of the surgical object; determining whether the position information of the calibration end matches the target hole position, and if so, determining the position where the surface of the surgical object and the calibration end contact each other as the actual hole position. Using this method, the actual hole position can be quickly and accurately obtained, which is beneficial for shortening preoperative preparation time.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a computer-readable storage medium, electronic equipment, position calibration, and robot system. Background Art

[0002] Surgical robots are designed to precisely perform complex surgical procedures using minimally invasive methods. Developed in response to the limitations of traditional surgical procedures, surgical robots transcend the limitations of the human eye, utilizing stereoscopic imaging technology to present internal organs more clearly to the operator. Furthermore, even in confined areas where some people's hands cannot reach, surgical robots can still control the movement, swinging, clamping, and 360-degree rotation of surgical instruments, while avoiding vibration and improving surgical precision. This further enhances the advantages of smaller incisions, less bleeding, faster postoperative recovery, and significantly shortened postoperative hospital stays. Consequently, surgical robots are highly favored by doctors and patients, and are widely used in their respective clinical procedures.

[0003] Like traditional surgery, before using a surgical robot for surgery, the lesion needs to be located, and the punch point needs to be determined based on the location of the lesion. The robotic arm on the surgical robot is then guided to the punch point to carry out the surgical operation.

[0004] The existing preoperative drilling environment has the following problems: 1) Due to the varying locations of lesions and physical characteristics of surgical patients, a fixed drilling location is not always the optimal one. Using a fixed drilling location can easily cause tool arm interference or positioning limitations during subsequent surgeries, hindering surgical operations. 2) After drilling locations are planned for different surgical patients, the patient's position often changes during the actual surgery due to various reasons (e.g., holding their breath, being full, or inducing pneumoperitoneum), making it impossible to quickly and accurately locate the planned drilling location. Summary of the Invention

[0005] The purpose of the present invention is to provide a computer-readable storage medium, electronic equipment, position calibration and robotic system that can quickly and accurately determine the drilling positions on the surface of the surgical subject according to the individual differences of the surgical subject, reduce preoperative preparation time and improve drilling accuracy.

[0006] To achieve the above object, the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed, performs the following steps:

[0007] Establishing a first physical sign image model based on first body surface information and lesion information of a surgical object in a first state; the first physical sign image model is used to plan a pre-drilling position;

[0008] establishing a second physical sign image model according to second body surface information of the surgical object in a second state;

[0009] Performing image registration on the second vital sign image model and the first vital sign image model to convert the pre-hole position on the first vital sign image model into the target hole position on the second vital sign image model;

[0010] Acquiring position information of a calibration end of a positioning device that moves on the surface of a surgical subject, wherein the calibration end is in contact with the surface of the surgical subject;

[0011] It is determined whether the position information of the calibration end matches the target hole position. If so, the position where the surgical object's body surface contacts the calibration end is determined to be the actual hole position.

[0012] Optionally, the second vital sign image model is located in a second coordinate system; and the program is configured to execute the following steps to obtain the actual hole position:

[0013] Acquire position information of the calibration end in the second coordinate system according to the first image information of the calibration end;

[0014] Determine whether the position information of the calibration end in the second coordinate system matches the target hole position.

[0015] Optionally, the second vital sign image model is located in a second coordinate system; the positioning device is provided with p auxiliary calibration ends, p is an integer greater than or equal to 3, and each of the auxiliary calibration ends has a predetermined positional relationship with the calibration end;

[0016] The program performs the following steps to obtain the actual hole position:

[0017] Acquire positions of at least three of the auxiliary calibration ends in the second coordinate system according to the second image information of the positioning device, and acquire the position of the calibration end in the second coordinate system according to the positions of the at least three auxiliary calibration ends in the second coordinate system and the predetermined positional relationship;

[0018] Determine whether the position information of the calibration end in the second coordinate system matches the target hole position.

[0019] Optionally, the first vital sign image model is located in a first coordinate system, and the second vital sign image model is located in a second coordinate system;

[0020] The program performs the following steps to obtain the target hole position:

[0021] Extracting first feature points on the first vital sign image model and extracting second feature points on the second vital sign image model;

[0022] Performing a maximum similarity calculation on the first feature point and the second feature point, and obtaining a transformation matrix between the first coordinate system and the second coordinate system;

[0023] Converting the pre-hole position on the first vital sign image model into a pre-target hole position in the second coordinate system according to the conversion matrix;

[0024] Determine whether the pre-target hole position is on the surface of the second vital signs image model; if so, determine that the pre-target hole position is the target hole position; if not, project the pre-target hole position along a first direction onto the surface of the second vital signs image model to obtain the target hole position; the first direction is the direction of a line connecting the pre-target hole position and the point on the surface of the second vital signs image model with the shortest distance to the pre-target hole position, and the first direction points to the surface of the second vital signs image model.

[0025] Optionally, the program performs maximum similarity calculation on the first feature point and the second feature point in a manner of minimizing image grayscale values.

[0026] Optionally, when the program determines that the position of the calibration end matches the target hole position, the program further performs the following steps: generating a first prompt message to indicate that the match is successful; and / or

[0027] When the program determines that the position of the calibration end does not match the target hole position and the distance between the position of the calibration end and the target hole position is within a predetermined range, the program further performs the following steps: generating a second prompt information, the second prompt information including the expected movement direction of the positioning device.

[0028] To achieve the above objectives, the present invention further provides an electronic device, comprising a processor and a computer-readable storage medium as described in any of the preceding items, wherein the processor is configured to execute a program stored on the computer-readable storage medium.

[0029] To achieve the above-mentioned purpose, the present invention also provides a position calibration system, including a control unit and a positioning device, wherein the positioning device is provided with a calibration end, and the position information of the calibration end is used to calibrate the hole position on the body surface of the surgical object, and the control unit is used to implement the steps performed by the program described in any of the previous items.

[0030] Optionally, the control unit is further configured to generate prompt information to prompt whether the position information of the calibration end of the positioning device matches or does not match the target hole position.

[0031] Optionally, the system further includes a prompt unit, which is communicatively connected to the control unit and is configured to display the prompt information.

[0032] Optionally, the position calibration system includes a second imaging device, which is communicatively connected to the control unit; the positioning device moves within the field of view of the second imaging device and a marker is provided on the calibration end, and the second imaging device identifies the marker to obtain first image information of the calibration end; the control unit obtains the position information of the calibration end based on the first image information.

[0033] Optionally, the positioning device includes P auxiliary calibration ends, P is an integer greater than or equal to 3, and each of the auxiliary calibration ends has a predetermined positional relationship with the calibration end; a marker is provided on the auxiliary calibration end, and the second imaging device recognizes the marker to obtain second image information of the auxiliary calibration end; the control unit obtains the position of the calibration end according to the second image information.

[0034] Optionally, the positioning device includes an auxiliary calibration part and a contact part; the auxiliary calibration end is provided on the auxiliary calibration part; the contact part is connected to the auxiliary calibration part, and the end of the contact part away from the auxiliary calibration part is formed as the calibration end and is used to contact the surface of the surgical object during the position calibration process.

[0035] Optionally, the auxiliary calibration portion comprises a cubic structure, and includes a first face and five second faces; the contact portion is a rod-shaped structure, and one end of the contact portion is connected to the center point of the first face, and the other end forms the calibration end;

[0036] The auxiliary marking ends are formed on at least three of the second surfaces; or, the auxiliary marking ends are formed at at least three vertices of the auxiliary marking portion.

[0037] Optionally, the positioning device includes a first rod, a second rod and a third rod that are perpendicular to each other and intersect at one point; wherein the two ends of the first rod, the two ends of the second rod and one end of the third rod form the auxiliary calibration ends, and the other end of the third rod forms the calibration end.

[0038] Optionally, the position calibration system also includes a first imaging device and a second imaging device, and the first imaging device and the second imaging device are both communicatively connected to the control unit; the first imaging device is used to obtain first body surface information and lesion information of the surgical object in a first state, and the second imaging device is used to obtain second body surface information of the surgical object in a second state.

[0039] Optionally, the first imaging device includes any one of an MRI machine, an X-ray device or a B-ultrasound device; the second imaging device includes a binocular vision camera or a structured light camera.

[0040] To achieve the above-mentioned objectives, the present invention also provides a surgical robot system, which includes a position calibration system and a surgical execution system as described in any of the previous items; the surgical execution system includes a robotic arm, which is used to connect surgical tools, and the position calibration system is used to determine the actual hole position on the surface of the surgical object, so that the robotic arm can control the surgical tool to enter the body of the surgical object at the actual hole position.

[0041] Compared with the prior art, the computer-readable storage medium, electronic device, position calibration, and robot system of the present invention have the following advantages:

[0042] First, a program is stored on the aforementioned computer-readable storage medium, and when the program is executed, the following steps are performed: a first physical sign image model is established based on first body surface information and lesion information of a surgical object in a first state; the first physical sign image model is used to plan a pre-hole position; a second physical sign image model is established based on second body surface data of the surgical object in a second state; image registration is performed on the second physical sign image model and the first physical sign image model to convert the pre-hole position on the first physical sign image model into a target hole position on the second physical sign image model; the position of a calibration end of a positioning device that moves on the surface of the surgical object is obtained, and the calibration end is in contact with the surface of the surgical object; it is determined whether the position of the calibration end matches the target hole position, and if so, the position where the surface of the surgical object contacts the calibration end is determined to be the actual hole position. When the computer-readable storage medium is applied to a position calibration system and the position calibration system is used in laparoscopic surgery, thoracoscopic surgery or other surgical operations that require drilling holes in the body of the surgical subject, the actual hole position on the surface of the surgical subject can be quickly and accurately determined before the operation, thereby reducing preoperative preparation time and increasing the operating space during subsequent operations.

[0043] Second, the position calibration system also includes a positioning device, which can be manually moved along the surface of the surgical object within the field of view of a second imaging device, and the second image information of the positioning device is collected by the second imaging device. The second image information is used to obtain the position of the calibration end of the positioning device, so that the positioning device does not need to directly interact with the control unit for information, so that the positioning device can be set as a passive device for easy maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0045] Figure 1 1 is a schematic diagram of a scenario in which the position calibration system provided by an embodiment of the present invention determines the actual hole position on the surface of a surgical object;

[0046] Figure 2 This is a flow chart of determining the actual hole position on the body surface of the surgical subject in an embodiment of the present invention;

[0047] Figure 3 yes Figure 2 A specific flow chart of step S6 in the flow chart shown;

[0048] Figure 4 This is a schematic diagram of the first imaging device of the position calibration system provided in an embodiment of the present invention acquiring first body surface information and lesion information of a surgical object;

[0049] Figure 5 1 is a schematic diagram of a first vital sign image model established by a control unit of a position calibration system provided by an embodiment of the present invention, wherein a) the lesion is not shown, and b) the lesion is shown;

[0050] Figure 6 Schematic diagram of planning pre-hole positions on a first physical sign image model by an operator in an embodiment of the present invention, wherein a) is a three-dimensional schematic diagram and b) is a planar schematic diagram;

[0051] Figure 7 This is a schematic diagram of the second imaging device of the position calibration system provided by an embodiment of the present invention acquiring second body surface information;

[0052] Figure 8 is a schematic diagram of a second vital sign image model established by a control unit of a position calibration system provided by an embodiment of the present invention;

[0053] Figure 9 This is a schematic diagram of a control unit of a position calibration system provided by an embodiment of the present invention performing image registration on a first vital sign image model and a second vital sign image model;

[0054] Figure 10 1 is a schematic diagram of a control unit of a position calibration system provided by an embodiment of the present invention performing image registration on a first vital sign image model and a second vital sign image model, wherein only the first feature point and the second feature point are shown;

[0055] Figure 11 Schematic diagram of a process in which a control unit of a position calibration system provided in an embodiment of the present invention obtains a target hole position of a second vital sign image model, wherein a) is a schematic diagram of a pre-target hole position on a first vital sign image model, and b) is a schematic diagram of a pre-target hole position on a second vital sign image model; c) is a schematic diagram of projecting the pre-target hole position of the second vital sign image model onto the surface of a surgical subject in the second vital sign image model; and d) is a schematic diagram of obtaining the target hole position of the second vital sign image model.

[0056] Figure 12This is a schematic diagram of the second imaging device of the position calibration system provided by an embodiment of the present invention identifying the position of the calibration end of the positioning device on the surface of the surgical object;

[0057] Figure 13 1 is a schematic structural diagram of a positioning device of a position calibration system provided by an embodiment of the present invention;

[0058] Figure 14 is a structural schematic diagram of a positioning device of a position calibration system provided by another embodiment of the present invention;

[0059] Figure 15 is a structural diagram of a positioning device of a position calibration system provided in yet another embodiment of the present invention;

[0060] Figure 16 It is a schematic diagram of an application scenario of the surgical robot system provided by an embodiment of the present invention.

[0061] In the diagram:

[0062] 1-first sign image model, 10-pre-hole position, 2-second sign image model, 20-target hole position, 30-pre-target hole position, 40-surgical tool;

[0063] 100-first imaging device, 200-second imaging device, 300-positioning device, 301-calibration end, 302-auxiliary calibration end, 310-auxiliary calibration part, 320-contact part, 330-first rod, 340-second rod, 350-third rod, 400-display device, 500-surgical trolley, 600-surgical execution system, 610-robotic arm. DETAILED DESCRIPTION

[0064] The following describes the embodiments of the present invention through specific specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in this embodiment only illustrate the basic concept of the present invention in a schematic manner, and the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components during actual implementation. During actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0065] In addition, each embodiment described below has one or more technical features. However, this does not mean that users of the present invention must implement all technical features in any embodiment at the same time, or that they can only implement some or all technical features in different embodiments separately. In other words, as long as implementation is possible, those skilled in the art can, based on the disclosure of the present invention and depending on design specifications or implementation requirements, selectively implement some or all technical features in any embodiment, or selectively implement a combination of some or all technical features in multiple embodiments, thereby increasing the flexibility of the implementation of the present invention.

[0066] As used in this specification, the singular forms "a", "an", and "the" include plural objects, unless the content clearly indicates otherwise. As used in this specification, the term "or" is generally used in the sense of including "and / or", unless the content clearly indicates otherwise, and the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection. It can be a mechanical connection or an electrical connection. It can be directly connected or indirectly connected through an intermediate medium. It can be internal communication between two elements or an interactive relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0067] To make the objects, advantages, and features of the present invention more apparent, the present invention is further described below in detail with reference to the accompanying drawings. It should be noted that the drawings are greatly simplified and not to exact scale, and are intended solely to facilitate and clearly illustrate the embodiments of the present invention. The same or similar reference numerals in the drawings represent the same or similar components.

[0068] The object of the present invention is to provide a position calibration system, such as Figure 1 、 Figure 5 、 Figure 6 and Figure 11As shown, the position calibration system includes a control unit, which is configured to establish a first physical sign image model 1 based on first body surface information and lesion information of a surgical object in a first state, wherein the first physical sign image model 1 is used to plan a pre-hole position; establish a second physical sign image model 2 based on second body surface information of the surgical object in a second state; align the second physical sign image model 2 with the first physical sign image model 1 to convert the pre-hole position 10 on the first physical sign image model 1 into a target hole position 20 on the second physical sign image model 2; obtain the position of a calibration end 301 of a positioning device 300 that moves on the surface of the surgical object, wherein the calibration end 301 is in contact with the surface of the surgical object; determine whether the position of the calibration end 301 matches the target hole position 20, and if so, determine that the position where the surface of the surgical object contacts the calibration end 301 is the actual hole position.

[0069] The position calibration system further includes a first imaging device 100 (such as Figure 1 and Figure 4 As shown) and the second imaging device 200 (as shown Figure 1 and Figure 7 As shown). Under normal circumstances, before performing a surgical operation on a surgical object, it is necessary to obtain the first body surface information and lesion information of the surgical object through the first imaging device 100, and construct a physical sign model including a lesion model for planning the pre-hole position during surgery. Then, according to the pre-hole position, a site corresponding to the pre-hole position is found on the surface of the surgical object to perform drilling. In this embodiment, the first physical sign image model 1 is established based on the first body surface information and the lesion information, that is, the first physical sign image model 1 reflects the positional relationship between the surface of the surgical object and the lesion, so medical staff can plan the pre-hole position 10 on the first physical sign image model. The second physical sign image model reflects the surface characteristics of the surgical object during actual drilling, so when the control unit determines that the position of the calibration end 301 matches the target hole position on the second physical sign image model, it can be determined that the site where the surface of the surgical object contacts the calibration end 301 is the actual hole position.

[0070] In practice, when planning the pre-hole position 10, the position of the surgical subject and the position of the surgical subject during the actual drilling are deviated due to various reasons. For example, before laparoscopic surgery, the first body surface information and the lesion information of the surgical subject are obtained when the pneumoperitoneum operation is not performed (that is, the surgical subject is in the first state) and the first vital sign image model is constructed based on this, and the second body surface data of the surgical subject after pneumoperitoneum (the surgical subject is in the second state) are obtained using the second imaging device 200 to construct the second vital sign image model. At this time, the position of the surgical subject is distorted due to pneumoperitoneum, resulting in the pre-hole position 10 constructed on the first vital sign image model not necessarily being applicable to the vital signs of the surgical subject after pneumoperitoneum. Therefore, this embodiment uses two vital sign image models to optimize the pre-hole position to obtain the target hole position, and the control unit identifies the position of the calibration end 301 of the positioning device 300 to quickly and accurately obtain the actual hole position. Those skilled in the art will understand that before other types of surgery, the surgical subject may also change his position due to reasons such as flatulence, defecation, or position movement. For the sake of convenience, laparoscopic surgery will be used as an example for description.

[0071] Please refer to Figures 2 to 12 In a non-limiting embodiment, a method for obtaining actual hole positions on the surface of a surgical subject using the position calibration system includes the following steps:

[0072] Step S1: using the first imaging device 100 to obtain first body surface information and lesion information of the surgical subject before pneumoperitoneum.

[0073] Step S2: The control unit constructs the first vital sign image model 1 according to the first body surface information and the lesion information.

[0074] Step S3: planning the pre-hole position 10 on the first vital sign image model S1.

[0075] Step S4: using the second imaging device 200 to obtain second body surface information of the surgical object after pneumoperitoneum.

[0076] Step S5: the control unit constructs the second vital sign image model 2 according to the second body surface information.

[0077] Step S6 : the control unit performs image registration on the second vital sign image model 2 and the first vital sign image model 1 to obtain the target hole position 20 on the second vital sign image model 2 .

[0078] Step S7: The positioning device 300 is driven to move. During the movement, the calibration end 301 always keeps in contact with the surface of the surgical object. At the same time, the control unit obtains the position of the calibration end 301 in real time and determines whether the position of the calibration end 301 matches the target hole position 20. If so, the site where the surgical object's body surface contacts the calibration end 301 is determined to be the actual hole position. If not, the positioning device is continued to be driven to move.

[0079] In other embodiments, step S4 may be performed before step S2, and step S5 may be performed simultaneously with step S2, which is not limited in the present invention.

[0080] In addition, this embodiment does not particularly limit the specific types of the first imaging device 100 and the second imaging device 200. The first imaging device 100 can be an MRI machine, a CT machine, or other X-ray device, or a B-ultrasound machine, as long as it can simultaneously collect body surface information and lesion information of the surgical subject. The second imaging device 200 can be a 3D vision system or other device that can collect body surface data of the surgical subject and does not interfere with the movement of the positioning device 300. The second imaging device 200 includes, but is not limited to, a binocular vision camera or a structured light camera.

[0081] Please refer to Figure 5 and Figure 8 , the control unit is in the first coordinate system F L The first vital sign image model 1 is established in the second coordinate system F cam The second vital sign image model 2 is established.

[0082] Please refer to Figure 3 In the step S6, the control unit first executes step S61: performing a maximum similarity calculation on the first vital sign image model 1 and the second vital sign image model 2 to obtain the first coordinate system F L and the second coordinate system F cam The transformation matrix Specifically, if Figure 9 and Figure 10 As shown, the control unit extracts a plurality of first feature points X on the first physical sign image model 1 and a plurality of second feature points Y on the second physical sign model, and performs a maximum similarity calculation on all the first feature points X and all the second feature points Y to obtain the transformation matrix The control unit performs maximum similarity calculation on the first feature point X and the second feature point Y by minimizing the sum of square errors of the image grayscale values. The formula is as follows:

[0083]

[0084] Wherein, Image1 represents the first vital sign image model, Image2 represents the second vital sign image model, m represents the number of the first feature points X, n represents the number of the second feature points Y, i is a positive integer less than or equal to n, j is a positive integer less than or equal to m, ΩImage1(X i )-ΩImage2(T j (Y i )) represents the error in grayscale value between the i-th first feature point X and the j-th second feature point Y.

[0085] Using the above formula, each of the first feature points X traverses all of the second feature points Y to complete the maximum similarity calculation. It is well known to those skilled in the art that image registration technology is an existing mature image processing method, which has multiple implementation methods. As long as the first coordinate system F can be obtained, L With the second coordinate system F cam The transformation matrix between That's it. In addition, when performing image registration, it is only necessary to perform maximum similarity calculation on each of the first feature points X and each of the second feature points Y. Therefore, this embodiment has no requirements for the positioning of the surgical object after pneumoperitoneum and the positioning of the operating trolley 500 (used to carry the surgical object after pneumoperitoneum), as long as the surgical object is always within the field of view of the second imaging device 200. Moreover, when the positioning of the surgical object or the positioning of the operating trolley 500 changes, the second imaging device 200 obtains the body surface information after the positioning change (i.e., the new second state) in real time as new second body surface information, and the control unit constructs a new second vital sign image model based on the new second body surface information, and performs image registration on the new second vital sign image model and the first vital sign image model.

[0086] Then, if Figure 11 As shown, the control unit executes step S62: according to the conversion matrix The pre-drilled position 10 on the first vital sign image model 1 is converted into the second coordinate system F cam The coordinates of the pre-target hole position 10 and the coordinates of the pre-target hole position 30 satisfy the following relationship:

[0087]

[0088] in, Indicates that the pre-hole position 10 is in the first coordinate system F L The coordinates in Indicates that the pre-target hole position 30 is in the second coordinate system F camThe coordinates in .

[0089] Since the body surface morphology of the surgical object changes due to reasons such as abdominal deformation and body position change before and after pneumoperitoneum, the control unit further performs step S63: judging whether the pre-target hole position 30 is on the body surface of the second vital sign image model 2, if so, determining that the pre-target hole position 30 is the target hole position 20, if not, moving the pre-target hole position 30 along the first direction Projected onto the surface of the second vital sign image model 2 to obtain the target hole position 20, the first direction It refers to the direction of the line between the pre-target hole position 20 and the point where the distance between the body surface of the second vital sign image model 2 and the pre-target hole position 20 is the smallest. At this time, the coordinates of the target hole position 20 and the coordinates of the pre-target hole position 30 satisfy the following relationship:

[0090]

[0091] in, Indicates that the target hole position 20 is in the second coordinate system F cam The coordinates inside.

[0092] Next, the positioning device 300 is driven to move, and at the same time, the control unit obtains the position of the calibration end 301 on the positioning device 300 in real time, and determines whether the position of the calibration end 301 matches the target hole position 20. If so, the point where the surgical object's body surface contacts the calibration end 301 is determined to be the actual hole position. Furthermore, the control unit obtains the position of the calibration end 301 in the second coordinate system F in real time. cam and judge the position of the calibration end 301 in the second coordinate system F cam Whether the position inside coincides with the target hole position, if so, the two match, if not, the two do not match.

[0093] Alternatively, as Figure 12 As shown, the positioning device 300 moves within the field of view of the second imaging device 200, so that the second imaging device 200 can collect image information of at least part of the structure of the positioning device 300 (for example, the first image information of the calibration end 301 of the positioning device 300 or the second image information of the auxiliary calibration end 302 of the positioning device 300 mentioned later), and send the image information to the control unit, and the control unit obtains the position of the calibration end 301 in the second coordinate system F according to the image information. camIn this way, the positioning device 300 does not directly exchange data with the control unit, so that the positioning device 300 can be designed as a passive device, which is convenient for maintenance. Of course, in alternative embodiments, the positioning device can also directly exchange data with the control unit.

[0094] In some embodiments, a marker is provided on the calibration end 301, and the second imaging device 200 can identify the marker and obtain the first image information of the calibration end 301. In this way, the image information includes the first image information of the calibration end 301, so that the control unit 200 can directly obtain the calibration end 301 in the second coordinate system F cam It is well known to those skilled in the art that the information of the calibration end 301 is collected by the second imaging device 200, and then the position of the calibration end 301 in the second coordinate system F is obtained. cam The position within is common knowledge and will not be described in detail here.

[0095] In other embodiments, the positioning device 300 is further provided with p auxiliary calibration ends 302, where p is an integer greater than or equal to 3, and each of the auxiliary calibration ends 302 has a predetermined positional relationship with the calibration end 301. This predetermined positional relationship will be described in detail later in conjunction with the detailed structure of the positioning device 300. In this way, if the image information of the positioning device 300 captured by the second imaging device 200 does not include the first image information of the calibration end 301 due to obstruction by other equipment, but includes the second image information of at least three of the auxiliary calibration ends 302, the control unit can also obtain the position of the calibration end 301 in the second coordinate system F according to the second image information of the at least three auxiliary calibration ends 302. cam Specifically, the control unit may first obtain the positions of at least three of the auxiliary calibration ends 302 in the second coordinate system F according to the second image information. cam Then, the position of the calibration end 301 in the second coordinate system F is calculated based on the predetermined position relationship. cam Similarly, according to the second image information of the auxiliary calibration end 302 collected by the second imaging device 200, the position of the auxiliary calibration end 302 in the second coordinate system F is obtained. cam The location within is also common knowledge.

[0096] The embodiment of the present invention does not limit the structure of the positioning device 300, and there are many options. For example, in one embodiment, Figure 13As shown, the positioning device 300 includes an auxiliary calibration part 310 and a contact part 320, wherein the auxiliary calibration end 302 is provided on the auxiliary calibration part 310, the contact part 320 is connected to the auxiliary calibration part 310, and the calibration end 301 is provided at one end of the contact part 320 away from the auxiliary calibration part 310. In more detail, the auxiliary calibration part 310 may be a cubic structure, the contact part 320 is a rod-shaped structure, and one end of the rod-shaped structure is connected to the center point of one face of the cubic structure. In this embodiment, the calibration end 301 is provided on the other end of the rod-shaped structure, and each vertex of the cubic structure is provided with an auxiliary calibration end 302 (that is, a total of eight auxiliary calibration ends 302 are provided). Alternatively, the cubic structure includes a first face and five second faces, wherein the contact part is connected to the center point of the first face. The auxiliary calibration ends 302 are provided on at least three of the second faces (such as Figure 14 As shown), preferably, each of the second surfaces is provided with at least one auxiliary calibration end 302. For example, in another embodiment, as Figure 15 As shown, the positioning device 300 includes a first rod 330, a second rod 340, and a third rod 350, which are mutually perpendicular and intersect at a point. Any two rods form a "cross" structure. The auxiliary calibration end 302 is provided at each end of the first rod 330, each end of the second rod 340, and one end of the third rod 350. The calibration end 301 is formed at the other end of the third rod 350. It is understood that markers are provided at the calibration end 301 and the auxiliary calibration end 302, and the second imaging device 200 can identify the markers to obtain corresponding image information.

[0097] Then, Figure 13 The structure of the positioning device 300 shown is taken as an example to introduce the predetermined position relationship.

[0098] The third coordinate system F is established by taking the extension directions of the three mutually perpendicular sides of the cube structure as the three axes of the coordinate system and the intersection point of the three sides as the origin. cali Assuming that the lengths of the three sides are l1, l2 and l3 respectively, and the length of the rod-shaped structure is l4, then the eight auxiliary calibration ends 302 are in the third coordinate system F cali The coordinates in the third coordinate system F are (0,0,0), (l1,0,0), (l1,0,l3), (0,0,l3), (0,l2,0), (0,l2,l3), (l1,l2,0) and (l1,l2,l3). The calibration end 301 is in the third coordinate system F cali The coordinates inside are In the third coordinate system F caliAccording to the coordinates of each auxiliary calibration end 302 and the calibration end 301, the predetermined position relationship can be obtained.

[0099] In other forms of the positioning device 300, the predetermined positional relationship between each auxiliary calibration end 302 and the calibration end 301 can also be determined by the same method, which will not be described in detail here.

[0100] In a further improved solution, the control unit is configured to generate a first prompt message when it is determined that the position of the calibration end 301 matches the target hole position 20, indicating that the positioning is successful, and to generate a second prompt message when it is determined that the calibration end 301 does not match the target hole position 20 and the distance between the position of the calibration end 301 and the target hole position 20 is within a predetermined range, indicating that the positioning device 300 continues to be moved. At the same time, the surgical robot system also includes a prompt device, which is communicatively connected to the control unit and is configured to receive the first prompt message and the second prompt message and provide prompts.

[0101] In some embodiments, the prompt device can be a buzzer alarm. For example, when the calibration end 301 matches the target hole position 20, the buzzer alarm emits a first frequency and / or first tone alarm. When the distance between the calibration end 301 and the target hole position 20 is within a predetermined range, for example, when the distance between the calibration end 301 and the target hole position 20 is less than 10 mm, the buzzer alarm emits a second frequency and / or second tone alarm. In another embodiment, the prompt device can be an indicator light of different colors. For example, when the position of the calibration end 301 matches the target hole position 20, the green indicator light of the prompt device is on. When the position of the calibration end 301 does not match the target hole position 20 and the distance between the calibration end 301 and the target hole position 20 is within a predetermined range, the red indicator light of the prompt device is on. In still other embodiments, the prompt device can be a voice prompt device. In still other embodiments, the prompt device can be a display device 400, which can display the first prompt information and / or the second prompt information in text form. In particular, when the prompt device is a voice prompt device or a display device 400, the second prompt information may further include an expected moving direction of the prompt positioning device 300, which is conducive to quickly completing the positioning.

[0102] In this embodiment, the positioning device 300 can be configured as a handheld operating device, which is handheld by medical staff to drive it to move. Therefore, after the prompt device indicates that the positioning is successful, the medical staff can manually mark the site where the surface of the surgical object contacts the calibration end 301, as the actual hole position in subsequent operations. Then the next punching position is positioned. However, in other embodiments, the positioning device 300 can also be clamped by a manipulator and driven to move, and the manipulator can be connected to the control unit for communication. When the control unit determines that the position of the calibration end 301 matches the target hole position, the control unit can send a signal to the manipulator to control the manipulator to stop moving until the medical staff completes the marking and then re-controls the manipulator to move.

[0103] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed, executes the steps executed by the control unit of the surgical robot system as described above.

[0104] Furthermore, an embodiment of the present invention also provides an electronic device, comprising a processor and the computer-readable storage medium as described above, wherein the processor is configured to execute a program stored on the computer-readable storage medium.

[0105] Furthermore, the present invention also provides a surgical robot system, please refer to Figure 16 The surgical robot system includes the aforementioned position calibration system and surgical execution system 600; the surgical execution system 600 includes a robotic arm 610, which is used to connect to the surgical tool 40. The position calibration system is used to determine the actual hole position on the surface of the surgical subject, and the robotic arm 610 is used to control the surgical tool 40 to enter the surgical subject's body at the actual hole position to perform the surgical operation.

[0106] It is understandable that the surgical robot system may be a master-slave mapping robot system or a non-master-slave mapping robot system, and the embodiment of the present invention does not limit this.

[0107] Furthermore, an embodiment of the present invention also provides a hole position planning method, including the steps executed by the aforementioned control unit when planning the target hole position, and the steps executed when directing the target hole position to the surface of the surgical object.

[0108] While the present invention is disclosed above, it is not limited thereto. Those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, provided such modifications and variations fall within the scope of the claims and their equivalents, the present invention is intended to encompass such modifications and variations.

Claims

1. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed, the following steps are performed: Establishing a first vital sign image model in a first coordinate system based on first body surface information and lesion information of a surgical object in a first state acquired by a first imaging device; the first vital sign image model is used to plan a pre-drilling position; establishing a second vital sign image model in a second coordinate system according to second body surface information of the surgical object in a second state acquired by a second imaging device; Performing image registration on the second vital sign image model and the first vital sign image model to convert the pre-hole position on the first vital sign image model into the target hole position on the second vital sign image model; acquiring position information of a calibration end of a positioning device moving on the surface of a surgical subject in the second coordinate system based on image information of at least a portion of the structure of the positioning device acquired by the second imaging device, the calibration end being in contact with the surface of the surgical subject; determining whether the position information of the calibration end in the second coordinate system coincides with the target hole position, and if so, determining that the position where the surgical subject's body surface contacts the calibration end is the actual hole position; The program executes the following steps to obtain the target hole position: Obtaining a transformation matrix between the first coordinate system and the second coordinate system; Converting the pre-hole position on the first vital sign image model into a pre-target hole position in the second coordinate system according to the conversion matrix; Determine whether the pre-target hole position is on the surface of the second vital signs image model; if so, determine that the pre-target hole position is the target hole position; if not, project the pre-target hole position along a first direction onto the surface of the second vital signs image model to obtain the target hole position; the first direction is the direction of a line connecting the pre-target hole position and the point on the surface of the second vital signs image model with the shortest distance to the pre-target hole position, and the first direction points to the surface of the second vital signs image model.

2. The computer-readable storage medium according to claim 1, wherein The second vital sign image model is located in a second coordinate system; the program is used to perform the following steps to obtain the actual hole position: Acquire position information of the calibration end in the second coordinate system according to the first image information of the calibration end; Determine whether the position information of the calibration end in the second coordinate system matches the target hole position.

3. The computer-readable storage medium according to claim 1, wherein The second vital sign image model is located in a second coordinate system; the positioning device is provided with p auxiliary calibration ends, where p is an integer greater than or equal to 3, and each of the auxiliary calibration ends has a predetermined positional relationship with the calibration end; The program performs the following steps to obtain the actual hole position: Acquire positions of at least three of the auxiliary calibration ends in the second coordinate system according to the second image information of the positioning device, and acquire the position of the calibration end in the second coordinate system according to the positions of the at least three auxiliary calibration ends in the second coordinate system and the predetermined positional relationship; Determine whether the position information of the calibration end in the second coordinate system matches the target hole position.

4. The computer-readable storage medium according to claim 1, wherein The program performs the following steps to obtain the transformation matrix: Extracting first feature points on the first vital sign image model and extracting second feature points on the second vital sign image model; A maximum similarity calculation is performed on the first feature point and the second feature point, and a transformation matrix between the first coordinate system and the second coordinate system is obtained.

5. The computer-readable storage medium according to claim 4, wherein: The program performs a maximum similarity calculation on the first feature point and the second feature point in a manner of minimizing the image grayscale value.

6. The computer-readable storage medium according to claim 1, wherein When the program determines that the position of the calibration end matches the target hole position, the program further performs the following steps: generating a first prompt message to indicate that the match is successful; and / or When the program determines that the position of the calibration end does not match the target hole position and the distance between the position of the calibration end and the target hole position is within a predetermined range, the program further performs the following steps: generating a second prompt information, the second prompt information including the expected movement direction of the positioning device. 7 . An electronic device comprising a processor and the computer-readable storage medium according to claim 1 , wherein the processor is configured to execute a program stored on the computer-readable storage medium.

8. A position calibration system, characterized in that: It includes a control unit and a positioning device, the positioning device is provided with a calibration end, the position information of the calibration end is used to calibrate the hole position on the surface of the surgical object, and the control unit is used to implement the steps performed by the program as described in any one of claims 1-6.

9. The position calibration system according to claim 8, characterized in that: The control unit is further configured to generate prompt information to indicate whether the position information of the calibration end of the positioning device matches or does not match the target hole position.

10. The position calibration system according to claim 9, characterized in that: The system further comprises a prompt unit, which is communicatively connected with the control unit and is configured to display the prompt information.

11. The position calibration system according to claim 8, characterized in that: The position calibration system includes a second imaging device, which is communicatively connected to the control unit; the positioning device moves within the field of view of the second imaging device and a marker is provided on the calibration end, and the second imaging device recognizes the marker to obtain first image information of the calibration end; the control unit obtains the position information of the calibration end based on the first image information.

12. The position calibration system according to claim 11, characterized in that: The positioning device includes P auxiliary calibration ends, where P is an integer greater than or equal to 3, and each of the auxiliary calibration ends has a predetermined positional relationship with the calibration end; a marker is provided on the auxiliary calibration end, and the second imaging device recognizes the marker to obtain second image information of the auxiliary calibration end; the control unit obtains the position of the calibration end based on the second image information.

13. The position calibration system according to claim 12, characterized in that: The positioning device includes an auxiliary calibration part and a contact part; the auxiliary calibration end is provided on the auxiliary calibration part; the contact part is connected to the auxiliary calibration part, and the end of the contact part away from the auxiliary calibration part is formed as the calibration end and is used to contact the surface of the surgical object during the position calibration process.

14. The position calibration system according to claim 13, characterized in that: The auxiliary calibration portion comprises a cubic structure, and includes a first face and five second faces; the contact portion is a rod-shaped structure, and one end of the contact portion is connected to the center point of the first face, and the other end forms the calibration end; The auxiliary marking ends are formed on at least three of the second surfaces; or, the auxiliary marking ends are formed at at least three vertices of the auxiliary marking portion.

15. The position calibration system according to claim 12, characterized in that: The positioning device includes a first rod, a second rod and a third rod that are perpendicular to each other and intersect at a point; wherein the two ends of the first rod, the two ends of the second rod and one end of the third rod form the auxiliary calibration ends, and the other end of the third rod forms the calibration end.

16. The position calibration system according to claim 8, characterized in that: The position calibration system also includes a first imaging device and a second imaging device, both of which are communicatively connected to the control unit; the first imaging device is used to obtain first body surface information and lesion information of the surgical object in a first state, and the second imaging device is used to obtain second body surface information of the surgical object in a second state.

17. The position calibration system according to claim 16, characterized in that: The first imaging device includes any one of an MRI machine, an X-ray device or a B-ultrasound device; the second imaging device includes a binocular vision camera or a structured light camera.

18. A surgical robot system, characterized in that: The surgical robot system includes a position calibration system and a surgical execution system as described in any one of claims 8 to 17; the surgical execution system includes a robotic arm, which is used to connect surgical tools, and the position calibration system is used to determine the actual hole position on the surface of the surgical object, so that the robotic arm can control the surgical tool to enter the body of the surgical object at the actual hole position.

Images