Surgical instrument guidance method, surgical robot and medium

Abstract

The present invention provides a surgical instrument guidance method, a surgical robot, and a storage medium. The method includes: obtaining current posture information and target posture information of the surgical instrument to be guided, wherein the target posture is within the current field of view of the endoscope; planning the movement path of the surgical instrument to be guided based on the intraoperative image captured by the endoscope and the current posture information and target posture information of the surgical instrument to be guided, so that the surgical instrument to be guided can move to the target posture without collision; and controlling the surgical instrument to be guided to perform corresponding movement based on the movement path of the surgical instrument. The surgical robot includes a controller and a robotic arm, and the controller is configured to implement the surgical instrument guidance method. The present invention can not only effectively guide a surgical instrument located outside the field of view of the endoscope to move into the field of view of the endoscope, but also ensure that the surgical instrument will not collide during movement.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical robots, and in particular to a surgical instrument guiding method, a surgical robot and a storage medium. Background Art

[0002] Minimally invasive surgery refers to a surgical method that uses modern medical devices such as laparoscopes, thoracoscopes and related equipment to perform surgery inside the human body cavity. Compared with traditional surgical methods, minimally invasive surgery has the advantages of less trauma, less pain and faster recovery.

[0003] With technological advancements, minimally invasive surgical robotics have matured and are now widely used. A surgical robot consists of a doctor's console and a patient cart. The cart includes at least one robotic arm, with surgical instruments and / or an endoscope mounted on its end. The doctor's console includes a master control arm and a display. The doctor manipulates the master control arm within the endoscope's field of view, displayed on the display, to control the movement of the robotic arm, endoscope, and surgical instruments.

[0004] During some operations, it is inevitable that the surgical instrument will not be within the field of view of the endoscope. In this case, the surgical instrument outside the field of view of the endoscope needs to be moved into the field of view of the endoscope. In the prior art, this movement operation is mostly completed by manual adjustment by the doctor, or the movement path is planned according to the position of the endoscope and the position of the surgical instrument outside the field of view. Since the doctor cannot quickly and accurately identify the adjustment direction when manually adjusting, multiple degrees of freedom attempts are required, and there are potential unknown risks in the process of trying. The movement path planned only according to the position of the endoscope and the position of the surgical instrument outside the field of view may hit human organs and tissues or other surgical instruments, which will also bring certain surgical risks.

[0005] It should be noted that the information disclosed in the background technology section of the invention is only intended to deepen the understanding of the general background technology of the invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art. Summary of the Invention

[0006] The purpose of the present invention is to provide a surgical instrument guiding method, a surgical robot and a storage medium, which can not only effectively guide the surgical instrument located outside the field of view of the endoscope to move into the field of view of the endoscope, but also ensure that the surgical instrument will not collide during the movement.

[0007] To achieve the above-mentioned object, the present invention provides a surgical instrument guiding method, which is applied to a surgical robot. The surgical instrument guiding method comprises:

[0008] Acquiring current posture information and target posture information of the surgical instrument to be guided, wherein the target posture is located within the current field of view of the endoscope and the current posture is located outside the current field of view of the endoscope;

[0009] Planning a motion path of the surgical instrument to be guided based on the intraoperative image captured by the endoscope and the current posture information and target posture information of the surgical instrument to be guided, so that the surgical instrument to be guided can move to the target posture without collision;

[0010] According to the movement path of the surgical instrument to be guided, the surgical instrument to be guided is controlled to perform corresponding movement.

[0011] Optionally, the surgical instrument guiding method further includes: determining a surgical instrument located outside the current field of view of the endoscope, and using the surgical instrument located outside the current field of view of the endoscope as the surgical instrument to be guided;

[0012] Wherein, determining the surgical instrument located outside the current field of view of the endoscope includes:

[0013] Acquiring current posture information of each installed surgical instrument, current posture information of the endoscope, and current camera parameter information of the endoscope;

[0014] Acquiring position range information corresponding to the current field of view of the endoscope according to the current position information of the endoscope and the current camera parameter information;

[0015] For each of the surgical instruments, determining whether the current posture of the surgical instrument is within the posture range corresponding to the current field of view of the endoscope; if not, determining that the surgical instrument is outside the current field of view of the endoscope; and / or

[0016] Acquiring a current frame of intraoperative image collected by the endoscope;

[0017] Performing surgical instrument recognition on the intraoperative image of the current frame to identify the surgical instrument located within the current field of view of the endoscope;

[0018] The surgical instrument located within the current field of view of the endoscope is compared with installed surgical instruments to determine the surgical instrument located outside the current field of view of the endoscope.

[0019] Optionally, planning a motion path of the surgical instrument to be guided based on the intraoperative image collected by the endoscope and the current posture information and target posture information of the surgical instrument to be guided includes:

[0020] Acquiring historical intraoperative images collected by the endoscope;

[0021] determining whether a motion path of the surgical instrument to be guided can be planned based on the historical intraoperative images;

[0022] If so, the motion path of the surgical instrument to be guided is planned according to the historical intraoperative images and the current posture information and target posture information of the surgical instrument to be guided.

[0023] Optionally, the determining whether a motion path of the surgical instrument to be guided can be planned based on the historical intraoperative images includes:

[0024] determining, based on the historical intraoperative images, whether the historical intraoperative images contain image information showing that the surgical instrument to be guided has moved out of the field of view of the endoscope;

[0025] If so, determining whether a collision occurs in the process of the surgical instrument to be guided moving out of the field of view of the endoscope based on image information of the surgical instrument to be guided contained in the historical intraoperative image, and if not, determining that a motion path of the surgical instrument to be guided can be planned based on the historical intraoperative image;

[0026] If not, determine whether the historical intraoperative image contains all image information of the process of the surgical instrument to be guided moving from its current position to its target position. If so, determine that the movement path of the surgical instrument to be guided can be planned based on the historical intraoperative image.

[0027] Optionally, if the historical intraoperative image contains image information of the surgical instrument to be guided moving out of the field of view of the endoscope, planning the motion path of the surgical instrument to be guided based on the historical intraoperative image and the current posture information and target posture information of the surgical instrument to be guided includes:

[0028] acquiring, based on image information of the surgical instrument to be guided moving out of the field of view of the endoscope contained in the historical intraoperative image, information on a movement path of the surgical instrument to be guided out of the field of view of the endoscope;

[0029] The motion path of the surgical instrument to be guided is planned according to the removal path information and the current posture information and target posture information of the surgical instrument to be guided.

[0030] Optionally, if the historical intraoperative image contains all image information of the surgical instrument to be guided during the process of moving from its current posture to its target posture, planning the motion path of the surgical instrument to be guided based on the historical intraoperative image and the current posture information and target posture information of the surgical instrument to be guided includes:

[0031] Acquiring position information of various obstacles in an in-vivo environment based on the historical intraoperative images;

[0032] Planning a motion path of the surgical instrument to be guided based on the posture information of the obstacle and the current posture information and target posture information of the surgical instrument to be guided; or

[0033] constructing a three-dimensional model of the in vivo environment based on the historical intraoperative images;

[0034] The motion path of the surgical instrument to be guided is planned according to the three-dimensional model of the internal body environment and the current posture information and target posture information of the surgical instrument to be guided.

[0035] Optionally, if the motion path of the surgical instrument to be guided cannot be planned based on the historical intraoperative images, the motion path of the surgical instrument to be guided is planned by the following steps:

[0036] Taking the current field of view of the endoscope as the initial field of view, adjusting the field of view of the endoscope until the surgical instrument to be guided is within the field of view of the endoscope, and acquiring real-time intraoperative images collected by the endoscope in each field of view;

[0037] Planning a motion path of the surgical instrument to be guided based on the real-time intraoperative image and the current posture information and target posture information of the surgical instrument to be guided;

[0038] Before controlling the surgical instrument to be guided to perform corresponding movement according to the movement path of the surgical instrument to be guided, the surgical instrument guiding method further includes:

[0039] The field of view of the endoscope is adjusted back to the initial field of view.

[0040] Optionally, adjusting the field of view of the endoscope includes:

[0041] The posture of the endoscope and / or the camera parameters of the endoscope are adjusted to adjust the field of view of the endoscope.

[0042] Optionally, planning a motion path of the surgical instrument to be guided based on the real-time intraoperative image and the current posture information and target posture information of the surgical instrument to be guided includes:

[0043] Acquiring position information of various obstacles in the in-vivo environment based on the real-time intraoperative image;

[0044] Planning a motion path of the surgical instrument to be guided based on the posture information of the obstacle and the current posture information and target posture information of the surgical instrument to be guided; or

[0045] constructing a three-dimensional model of the in vivo environment based on the real-time intraoperative image;

[0046] The motion path of the surgical instrument to be guided is planned according to the three-dimensional model of the internal body environment and the current posture information and target posture information of the surgical instrument to be guided.

[0047] Optionally, in the process of controlling the surgical instrument to be guided to perform corresponding movements, the surgical instrument guiding method further includes:

[0048] Acquiring real-time position information of the surgical instrument to be guided, and calculating the real-time distance between the surgical instrument to be guided and the obstacle based on the real-time position information of the surgical instrument to be guided;

[0049] Determine whether the real-time distance is less than a preset distance threshold. If so, send an alarm message. When the surgical instrument to be guided collides with the obstacle, lock the surgical instrument to be guided, send an alarm message, and replan the movement path of the surgical instrument to be guided.

[0050] Optionally, in the process of controlling the surgical instrument to be guided to perform corresponding movements, the surgical instrument guiding method further includes:

[0051] The real-time posture information of the surgical instrument to be guided is obtained, and the motion path of the surgical instrument to be guided is updated in real time according to the real-time posture information of the surgical instrument to be guided and its target posture information.

[0052] Optionally, in the process of controlling the surgical instrument to be guided to perform corresponding movements, the surgical instrument guiding method further includes:

[0053] Acquiring real-time posture information of the surgical instrument to be guided, and calculating the real-time posture deviation of the surgical instrument to be guided based on the real-time posture information of the surgical instrument to be guided and its target posture information;

[0054] Determine whether the real-time posture deviation is within a preset error range, and if so, stop the movement of the surgical instrument to be guided.

[0055] Optionally, the surgical instrument guiding method further includes:

[0056] Displaying the motion path of the surgical instrument to be guided; and / or

[0057] After the surgical instrument to be guided moves into the current field of view of the endoscope, the target position and the movement direction of the surgical instrument to be guided are displayed.

[0058] Optionally, controlling the surgical instrument to be guided to perform corresponding movement according to the movement path of the surgical instrument to be guided includes:

[0059] Manually controlling the surgical instrument to be guided to move accordingly according to the movement path of the surgical instrument to be guided; or

[0060] According to the movement path of the surgical instrument to be guided, the surgical instrument to be guided is automatically controlled to perform corresponding movement.

[0061] Optionally, manually controlling the surgical instrument to be guided to perform corresponding movement according to the movement path of the surgical instrument to be guided includes:

[0062] According to the movement path of the surgical instrument to be guided, a guiding force with the same direction as the movement direction of the surgical instrument to be guided is applied to the robotic arm where the surgical instrument to be guided is located, so as to guide the operator to manually control the surgical instrument to be guided to perform corresponding movement according to the movement path.

[0063] To achieve the above-mentioned objectives, the present invention also provides a surgical robot, comprising a controller and at least one robotic arm, wherein surgical instruments and an endoscope are installed at the end of at least one of the robotic arms, the controller is coupled to the robotic arm, and the controller is configured to implement any of the surgical instrument guidance methods described above.

[0064] To achieve the above-mentioned object, the present invention further provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, the surgical instrument guidance method described above is implemented.

[0065] Compared with the prior art, the surgical instrument guidance method, surgical robot, and storage medium provided by the present invention have the following advantages:

[0066] The surgical instrument guidance method provided by the present invention first obtains the current posture information and target posture information of the surgical instrument to be guided, wherein the target posture is located within the current field of view of the endoscope and the current posture is located outside the current field of view of the endoscope; then, based on the intraoperative image captured by the endoscope and the current posture information and target posture information of the surgical instrument to be guided, the movement path of the surgical instrument to be guided is planned so that the surgical instrument to be guided can move to the target posture without collision; finally, based on the movement path of the surgical instrument to be guided, the surgical instrument to be guided is controlled to perform corresponding movement. It can be seen that the surgical instrument guidance method provided by the present invention can not only effectively guide the surgical instrument located outside the field of view of the endoscope to move into the field of view of the endoscope, but also ensure that the surgical instrument will not collide during the movement process, thereby effectively reducing unknown risks during the operation and improving the safety of the operation.

[0067] Since the surgical robot and storage medium provided by the present invention belong to the same inventive concept as the surgical instrument guidance method provided by the present invention, the surgical robot and storage medium provided by the present invention have all the advantages of the surgical instrument guidance method provided by the present invention, so the beneficial effects of the surgical robot and storage medium provided by the present invention will not be described one by one. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A schematic diagram of an application scenario of a surgical robot provided by one embodiment of the present invention;

[0069] Figure 2 A schematic structural diagram of a patient trolley provided in one embodiment of the present invention;

[0070] Figure 3 A schematic diagram of the structure of a doctor's console provided in one embodiment of the present invention;

[0071] Figure 4 A schematic flow chart of a surgical instrument guidance method according to one embodiment of the present invention

[0072] Figure 5 A schematic diagram of a process for determining surgical instruments located outside the current field of view of an endoscope according to one embodiment of the present invention;

[0073] Figure 6 A schematic diagram of an endoscope collecting intraoperative images provided in a specific example of the present invention;

[0074] Figure 7 A schematic diagram of a process for accurately determining a surgical instrument located outside the current field of view of an endoscope, provided in accordance with another embodiment of the present invention;

[0075] Figure 8A schematic diagram of a specific process for planning the motion path of a surgical instrument to be guided provided by one embodiment of the present invention;

[0076] Figure 9 A schematic diagram of a specific process for determining whether the motion path of the surgical instrument to be guided can be planned based on historical intraoperative images, provided in one embodiment of the present invention;

[0077] Figure 10 A schematic diagram of a method for adjusting the field of view of an endoscope according to an embodiment of the present invention;

[0078] Figure 11 A schematic diagram of a three-dimensional model of an in-vivo environment constructed based on real-time intraoperative images provided in a specific example of the present invention;

[0079] Figure 12 A schematic diagram of a protection strategy for a surgical instrument to be guided during movement provided by one embodiment of the present invention;

[0080] Figure 13 A schematic diagram of the control flow of a surgical instrument to be guided during movement according to an embodiment of the present invention;

[0081] Figure 14 A schematic diagram showing the motion path of a surgical instrument to be guided on an image interface according to an embodiment of the present invention;

[0082] Figure 15 A schematic diagram of the display of an image interface after a surgical instrument to be guided enters the field of view of an endoscope provided in one embodiment of the present invention;

[0083] Figure 16 A schematic diagram of a process for updating a motion path according to an embodiment of the present invention;

[0084] Figure 17 A schematic block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0085] The following is a further detailed description of the surgical instrument guidance method, surgical robot, electronic device and storage medium proposed in the present invention in conjunction with the accompanying drawings and specific embodiments. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In order to make the purposes, features and advantages of the present invention more obvious and easy to understand, please refer to the drawings. It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions for the implementation of the present invention. Any modification of the structure, change in the proportional relationship or adjustment of the size should still fall within the scope of the technical content disclosed in the present invention when the effects and purposes that can be achieved are the same or similar to those that can be produced by the present invention.

[0086] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0087] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "axial", "radial", "circumferential" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0088] In the description of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "connected," and "fixed" should be understood in a broad sense. For example, they may refer to fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; and internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention in specific circumstances.

[0089] In addition, in the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0090] The core idea of ​​the present invention is to provide a surgical instrument guidance method, surgical robot, electronic device and storage medium, which can not only effectively guide the surgical instrument located outside the field of view of the endoscope to move into the field of view of the endoscope, but also ensure that the surgical instrument will not collide during the movement.

[0091] It should be noted that the surgical instrument guidance method provided by the present invention can be applied to the electronic device of the embodiment of the present invention, wherein the electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a hardware device with various operating systems such as a mobile phone and a tablet computer. The electronic device provided by the present invention can be applied to the surgical robot provided by the present invention. In addition, it should be noted that, as those skilled in the art can understand, the posture information of the endoscope, the posture information of the surgical instrument, the posture information of the obstacle, etc. involved in the present invention are all described based on the same reference coordinate system, and the reference coordinate system includes but is not limited to the base coordinate system of the surgical robot, and can also be other coordinate systems that can be used as references obtained by converting the base coordinate system, such as the coordinate system of the doctor's console, etc.

[0092] Example 1

[0093] To realize the above idea, the present invention provides a surgical instrument guiding method, which is applied to a surgical robot. Figure 1, which schematically shows an application scenario diagram of the surgical robot provided by an embodiment of this invention. Figure 1 As shown, the surgical robot includes a doctor console 100 , a patient trolley 200 , and an imaging trolley 300 that are communicatively connected.

[0094] Please continue to refer to Figure 2 , which schematically shows the structure of the patient trolley provided by one embodiment of this embodiment. Figure 2 As shown, the patient cart 200 includes a base 210 and at least one robotic arm 220 mounted on the base 210. A surgical instrument 400 is mounted on the distal end of at least one of the robotic arms 220, and an endoscope 500 is mounted on the distal end of at least one of the robotic arms 220. It should be noted that, as those skilled in the art will appreciate, when only one robotic arm 220 is provided on the base 210, the surgical instrument 400 and the endoscope can be mounted on the same robotic arm 220; when multiple robotic arms 220 are provided on the base 210, the surgical instrument 400 and the endoscope can be mounted on different robotic arms 220. It should be noted that, as those skilled in the art will appreciate, this embodiment does not specifically limit the types of the endoscope and surgical instrument. For example, the endoscope can be a detection instrument for collecting surgical environment information (including but not limited to tissue, organ, and vascular information, and status information of surgical instruments and consumables). The surgical instrument can be a surgical tool such as scissors, needle holders, grasping forceps, electrocautery, or electrocoagulation forceps. The endoscope 160 may be a 3D endoscope or a 2D endoscope. Preferably, the endoscope is a 2D endoscope.

[0095] Specifically, the surgical instrument 400 and the endoscope 500 can be inserted into the patient's body through the puncture hole on the patient's surface. The endoscope 500 can collect intraoperative images, specifically including obtaining surgical scene image information such as human tissues and organs, surgical instruments 400, blood vessels, and body fluids. The collected intraoperative images can be transmitted to the first display unit 310 of the image trolley 300 (for ease of distinction, the display component on the image trolley 300 is represented by the first display unit 310, and the display component on the doctor's console 100 is represented by the second display unit 120) for display.

[0096] Please continue to refer to Figure 3 , which schematically shows the structure of the doctor console provided in one embodiment of this embodiment. Figure 3As shown, the doctor's console 100 includes at least one master control arm 110. During the operation, the main operating doctor sitting at the doctor's console 100 can control the movement of the surgical instruments 400 and the endoscope 500 located on the robotic arm 220 by manipulating the master control arm 110 to complete various operations, thereby achieving the purpose of performing surgery on the patient. During the actual operation, the operator observes the intraoperative images transmitted back through the second display unit 120 on the doctor's console 100 and controls the movement of the surgical instruments 400 and the endoscope 500 located on the robotic arm 220 by manipulating the master control arm 110. During the operation, if the surgical instruments are outside the field of view of the endoscope, the main operating doctor cannot continue the operation. In order to ensure the normal progress and safety of the operation, the surgical instruments need to be adjusted to be within the field of view of the endoscope.

[0097] Please continue to refer to Figure 1 ,like Figure 1 As shown, in an exemplary embodiment, the surgical robot further includes a tool trolley 610 for storing surgical instruments 400 and an auxiliary trolley 620 (including a ventilator and an anesthesia machine) for use during surgery. It should be noted that, as those skilled in the art will understand, those skilled in the art can select and configure these auxiliary trolleys 620 based on existing technology, so they will not be described in detail here. In addition, it should be noted that more information about the working principles of the surgical robot can be found in the existing technology and will not be elaborated here.

[0098] Please continue to refer to Figure 4 , which schematically shows a flow chart of a surgical instrument guiding method provided by one embodiment of the present invention. Figure 4 As shown, the surgical instrument guiding method includes the following steps:

[0099] Step S100 , obtaining current posture information and target posture information of the surgical instrument to be guided, wherein the target posture is located within the current field of view of the endoscope 500 , and the current posture is located outside the current field of view of the endoscope 500 .

[0100] Step S200 : planning a motion path of the surgical instrument to be guided based on the intraoperative image captured by the endoscope 500 and the current posture information and target posture information of the surgical instrument to be guided.

[0101] Step S300: Control the surgical instrument to be guided to move accordingly according to the movement path of the surgical instrument to be guided.

[0102] It should be noted that, as those skilled in the art will appreciate, the planned motion path of the surgical instrument to be guided should satisfy the following requirements: when the surgical instrument to be guided moves along the motion path, it can move to the target position without collision. Thus, the surgical instrument guidance method provided by the present invention can not only effectively guide a surgical instrument 400 located outside the field of view of the endoscope 500 to move within the field of view of the endoscope 500, but also ensure that the surgical instrument 400 does not collide during movement, thereby effectively reducing unknown risks during surgery and improving safety during surgery.

[0103] In an exemplary embodiment, before performing step S100, the surgical instrument guidance method further includes:

[0104] The surgical instrument 400 located outside the current field of view of the endoscope 500 is determined, and the surgical instrument 400 located outside the current field of view of the endoscope 500 is used as the surgical instrument to be guided.

[0105] Specifically, please refer to Figure 5 , which schematically shows a flow chart of determining the surgical instrument 400 located outside the current field of view of the endoscope 500 provided by the first embodiment of this embodiment. Figure 5 As shown, in this embodiment, determining the surgical instrument 400 that is outside the current field of view of the endoscope 500 includes:

[0106] Acquire current posture information of each installed surgical instrument 400, current posture information of the endoscope 500, and current camera parameter information of the endoscope 500;

[0107] Acquire the posture range information corresponding to the current field of view of the endoscope 500 according to the current posture information of the endoscope 500 and the current camera parameter information;

[0108] For each surgical instrument 400 , determine whether the current posture of the surgical instrument 400 is within the posture range corresponding to the current field of view of the endoscope 500 . If not, determine that the surgical instrument 400 is outside the current field of view of the endoscope 500 .

[0109] Specifically, for each installed surgical instrument 400, the current position information of the surgical instrument 400 (the current position information in the reference coordinate system, such as the current position information of the base coordinate system of the surgical robot) is obtained based on the current position information of each joint on the robotic arm where the surgical instrument 400 is located and combined with the positive kinematics model of the robotic arm. Similarly, the current position information of the endoscope 500 (the current position information in the reference coordinate system, such as the current position information of the base coordinate system of the surgical robot) can be obtained based on the current position information of each joint on the robotic arm where the endoscope 500 is located and combined with the positive kinematics model of the robotic arm. It should be noted that, as can be understood by those skilled in the art, the position information of each joint on the robotic arm can be measured by encoders installed on each joint.

[0110] As can be understood by those skilled in the art, the field of view of the endoscope 500 is an area that actually has boundaries, so the field of view of the endoscope 500 can be converted into a posture range under the reference coordinate system (such as the base coordinate system of the surgical robot). Therefore, after calculating the posture range corresponding to the current field of view of the endoscope 500 based on the current posture information of the endoscope 500 and the current camera parameter information, for each of the surgical instruments 400, if the current posture of the surgical instrument 400 is within the posture range defined by the current field of view of the endoscope 500, it is determined that the surgical instrument 400 is within the current field of view of the endoscope 500; otherwise, it is determined that the surgical instrument 400 is outside the current field of view of the endoscope 500, that is, the surgical instrument 400 is determined to be a surgical instrument to be guided. Please refer to Figure 6 , which schematically shows a schematic diagram of the endoscope 500 collecting intraoperative images provided in a specific example of this embodiment. Figure 6 As shown, based on the posture information (including position information and directional angle information) of the endoscope 500 and the camera parameter information (including visual angle information), the posture range corresponding to the field of view of the endoscope 500 can be determined. Since the posture of the surgical instrument 400A in the figure is within the posture range limited by the field of view of the endoscope 500, it can be determined that the surgical instrument 400A is within the field of view of the endoscope 500; since the posture of the surgical instrument 400B in the figure is outside the posture range limited by the field of view of the endoscope 500, it can be determined that the surgical instrument 400B is outside the field of view of the endoscope 500, that is, the surgical instrument 400B is a surgical instrument to be guided.

[0111] Please continue to refer to Figure 7 , which schematically shows a flow chart of another embodiment of the present invention for accurately determining the surgical instrument 400 located outside the current field of view of the endoscope 500. Figure 7As shown, in this embodiment, determining the surgical instrument 400 that is outside the current field of view of the endoscope 500 includes:

[0112] Acquire a current frame of intraoperative image captured by the endoscope 500;

[0113] Identify the surgical instrument 400 on the intraoperative image of the current frame to identify the surgical instrument 400 located within the current field of view of the endoscope 500;

[0114] The surgical instrument 400 located within the current field of view of the endoscope 500 is compared with the installed surgical instruments 400 to determine the surgical instrument 400 located outside the current field of view of the endoscope 500 .

[0115] Specifically, the intraoperative image of the current frame can be recognized by using a pre-trained neural network model to identify the category of the surgical instrument 400 located within the current field of view of the endoscope 500. The category of the surgical instrument 400 located within the current field of view of the endoscope 500 can then be compared with the category of the installed surgical instruments 400 to determine which categories of installed surgical instruments 400 do not appear within the current field of view of the endoscope 500. These surgical instruments 400 that do not appear within the current field of view of the endoscope 500 are the surgical instruments 400 located outside the current field of view of the endoscope 500, i.e., the surgical instruments to be guided. It should be noted that, as can be understood by those skilled in the art, the intraoperative image of the current frame refers to the intraoperative image captured by the endoscope 500 in the current field of view.

[0116] Please continue to refer to Figure 8 , which schematically shows a specific flow chart of planning the motion path of the surgical instrument to be guided provided by an embodiment of this embodiment. Figure 8 As shown, in an exemplary embodiment, planning the motion path of the surgical instrument to be guided based on the intraoperative image captured by the endoscope 500 and the current posture information and target posture information of the surgical instrument to be guided includes:

[0117] Acquiring historical intraoperative images collected by the endoscope 500;

[0118] determining whether a motion path of the surgical instrument to be guided can be planned based on the historical intraoperative images;

[0119] If so, the motion path of the surgical instrument to be guided is planned according to the historical intraoperative images and the current posture information and target posture information of the surgical instrument to be guided.

[0120] Specifically, the historical intraoperative images include all intraoperative images captured by the endoscope 500 from the start of the surgery to the current moment. Thus, by first determining whether the motion path of the surgical instrument to be guided can be planned based on the historical intraoperative images and then planning the motion path of the surgical instrument to be guided, the efficiency and accuracy of planning the motion path of the surgical instrument to be guided can be further improved, thereby further ensuring the reliability of the planned motion path of the surgical instrument to be guided.

[0121] Please continue to refer to Figure 9 , which schematically shows a specific flow chart of determining whether the motion path of the surgical instrument to be guided can be planned based on historical intraoperative images provided by an embodiment of this invention. Figure 9 As shown, in an exemplary embodiment, the determining whether the motion path of the surgical instrument to be guided can be planned based on the historical intraoperative image includes:

[0122] determining, based on the historical intraoperative images, whether the historical intraoperative images contain image information showing that the surgical instrument to be guided has moved out of the field of view of the endoscope 500;

[0123] If so, determining whether a collision occurs during the process of the surgical instrument to be guided moving out of the field of view of the endoscope 500 based on image information of the surgical instrument to be guided contained in the historical intraoperative image, and if not, determining that a motion path of the surgical instrument to be guided can be planned based on the historical intraoperative image;

[0124] If not, determine whether the historical intraoperative image contains all image information of the process of the surgical instrument to be guided moving from its current position to its target position. If so, determine that the movement path of the surgical instrument to be guided can be planned based on the historical intraoperative image.

[0125] Specifically, the surgical instrument 400 can be identified in the intraoperative images at each acquisition moment in the historical intraoperative images in order of acquisition time (for example, a pre-trained neural network model can be used to identify the surgical instrument 400). If the recognition result is that the surgical instrument to be guided can be identified in the intraoperative images at some acquisition moments, but is not identified in the intraoperative images at subsequent acquisition moments, it indicates that the surgical instrument to be guided has moved out of the field of view of the endoscope 500, that is, it can be determined that the historical intraoperative images contain image information of the surgical instrument to be guided moving out of the field of view of the endoscope 500. Therefore, based on the image information of the surgical instrument to be guided moving out of the field of view of the endoscope 500 contained in the historical intraoperative image, the position information of each obstacle 700 in the internal body environment within the field of view of the endoscope 500 is obtained, and compared with the position information of the surgical instrument to be guided in the process of moving out of the field of view of the endoscope 500, so as to determine whether the surgical instrument to be guided collides with the obstacle 700 in the internal body environment in the process of moving out of the field of view of the endoscope 500. If the judgment result is no, it is determined that the movement path of the surgical instrument to be guided can be planned based on the historical intraoperative image; otherwise, it is determined that the movement path of the surgical instrument to be guided cannot be planned based on the historical intraoperative image. If the judgment result is that the historical intraoperative image does not contain image information of the surgical instrument to be guided moving out of the field of view of the endoscope 500, then further combined with the historical motion data of the surgical instrument to be guided, it is judged whether the historical intraoperative image contains all image information of the process of the surgical instrument to be guided moving from its current posture to its target posture (that is, whether the historical motion data of the surgical instrument to be guided contains the current posture and the target posture. If included, it is determined that the historical intraoperative image contains all image information of the process of the surgical instrument to be guided moving from its current posture to its target posture. Otherwise, it is determined that the historical intraoperative image does not contain all image information of the process of the surgical instrument to be guided moving from its current posture to its target posture). If the judgment result is yes, it is determined that the motion path of the surgical instrument to be guided can be planned based on the historical intraoperative image. Otherwise, it is determined that the motion path of the surgical instrument to be guided cannot be planned based on the historical intraoperative image.

[0126] In an exemplary embodiment, if the historical intraoperative image contains image information of the surgical instrument to be guided moving out of the field of view of the endoscope 500 and the surgical instrument to be guided does not collide during the process of moving out of the field of view of the endoscope 500, then planning the motion path of the surgical instrument to be guided based on the historical intraoperative image and the current posture information and target posture information of the surgical instrument to be guided includes:

[0127] Acquiring, based on image information of the surgical instrument to be guided moving out of the field of view of the endoscope 500 contained in the historical intraoperative image, information on a path along which the surgical instrument to be guided moves out of the field of view of the endoscope 500;

[0128] The motion path of the surgical instrument to be guided is planned according to the removal path information and the current posture information and target posture information of the surgical instrument to be guided.

[0129] Specifically, the surgical instrument to be guided can be identified by performing the intraoperative images at each acquisition moment in the process of the surgical instrument to be guided moving out of the field of view of the endoscope 500 contained in the historical intraoperative images, so as to obtain the movement path information of the surgical instrument to be guided in the process of moving out of the field of view of the endoscope 500 (determined by the various posture information of the surgical instrument to be guided in the process of moving out of the field of view of the endoscope 500), so as to plan the movement path of the surgical instrument to be guided based on the movement path information and the current posture information and target posture information of the surgical instrument to be guided, wherein the movement path of the surgical instrument to be guided includes the movement path of the surgical instrument to be guided, so that the surgical instrument to be guided can return along the original movement path until it reaches the target posture.

[0130] In an exemplary embodiment, if the historical intraoperative image contains all image information of the surgical instrument to be guided during the process of moving from its current posture to its target posture, then planning the motion path of the surgical instrument to be guided based on the historical intraoperative image and the current posture information and target posture information of the surgical instrument to be guided includes:

[0131] Acquiring position information of each obstacle 700 in the in-vivo environment based on the historical intraoperative images;

[0132] Planning a motion path of the surgical instrument to be guided based on the posture information of the obstacle 700 and the current posture information and target posture information of the surgical instrument to be guided; or

[0133] constructing a three-dimensional model of the in vivo environment based on the historical intraoperative images;

[0134] The motion path of the surgical instrument to be guided is planned according to the three-dimensional model of the internal body environment and the current posture information and target posture information of the surgical instrument to be guided.

[0135] Specifically, the intraoperative images at each acquisition moment in the historical intraoperative images can be sequentially identified for obstacles 700 in the order of acquisition time to identify each obstacle 700 in the in vivo environment (including other surgical instruments 400 and various organs and tissues of the human body), thereby obtaining the position information of each obstacle 700 in the in vivo environment (position information in the reference coordinate system), and then, based on the current position information of the surgical instrument to be guided, the target position information, and the position information of each obstacle 700, the movement path of the surgical instrument to be guided can be planned so that the surgical instrument to be guided will not collide with the obstacle 700 during the process of moving from the current position to the target position. It should be noted that, as can be understood by those skilled in the art, after obtaining the position information of each obstacle 700 in the in vivo environment, three-dimensional modeling of the in vivo environment can be performed based on the position information of each obstacle 700 to construct a three-dimensional model of the in vivo environment. Furthermore, based on the three-dimensional model of the in-vivo environment, a motion path can also be planned so that the surgical instrument to be guided can move from the current position to the target position without collision.

[0136] In addition, it should be noted that, as those skilled in the art will understand, when there are multiple motion paths that do not collide, an optimization selection can be made in combination with constraints, for example, the shortest path and / or the smoothest path can be combined to select the best motion path.

[0137] Furthermore, when the endoscope 500 is a 3D endoscope, since the 3D endoscope has two lenses of the same specifications and the two lenses are distributed at a certain interval, the conditions of the binocular positioning principle are met, and thus two intraoperative images can be simultaneously collected through the binocular lens of the endoscope 500. By respectively identifying the two intraoperative images, the feature points or contours of each obstacle 700 can be respectively identified in the two intraoperative images (the feature points or contours can be extracted using grayscale changes and edge image detection algorithms). After identifying the feature points or contours of each obstacle 700, the two intraoperative images can be aligned first, and then according to the alignment results, for each obstacle 700, its corresponding feature points in the two images are obtained. Then, based on the basic principle of binocular positioning and the camera parameters of the endoscope 500 when collecting the two intraoperative images, the three-dimensional coordinates of the feature points of the obstacle 700 in the coordinate system of the endoscope 500 can be obtained. Specifically, assuming that the distance between the optical axis centers of the binocular lenses is L, and the distance between the binocular lenses and the imaging plane is f, when the binocular lenses simultaneously observe the feature point P, the horizontal coordinate observed by the "left eye" is xl , the horizontal coordinate observed by the “right eye” is x r , define “parallax” D = x l -x r ; And because the imaging plane of the binocular lens is the same baseline plane, the observed longitudinal coordinates are equal, both are y, then according to the triangle similarity principle, the following relationship exists:

[0138]

[0139] Where (x l ,y) represents the position coordinate of the feature point P in the intraoperative image taken by the lens on the left, (x r ,y) represents the position coordinate of the feature point P in the intraoperative image taken by the lens on the right, (x c ,y c ,z c ) represents the actual position of the feature point P in the coordinate system of the endoscope 500.

[0140] Thus, for each feature point, the actual position information of the feature point in the coordinate system of the endoscope 500 can be obtained using the above formula. It should be noted that, as will be understood by those skilled in the art, the pose information of the endoscope 500 in the base coordinate system of the surgical robot can be obtained based on the poses of the various joints of the robotic arm 220 on which the endoscope 500 is mounted, thereby obtaining a rotation matrix used to characterize the mapping relationship between the coordinate system of the endoscope 500 and the base coordinate system of the surgical robot. Thus, for each feature point, the actual position information of the feature point in the reference coordinate system can be obtained based on the actual pose information of the feature point in the coordinate system of the endoscope 500, the mapping relationship between the coordinate system of the endoscope 500 and the base coordinate system, and the mapping relationship between the base coordinate system and the reference coordinate system. For each obstacle 700, the pose information of the obstacle 700 in the reference coordinate system can be obtained based on the actual position information of multiple feature points on the obstacle 700 in the reference coordinate system. In addition, it should be noted that, as those skilled in the art can understand, if the obstacle 700 is a surgical machine other than the surgical instrument to be guided, the posture information of the surgical instrument 400 in the reference coordinate system can be directly obtained based on the posture information of the surgical instrument 400 in the base coordinate system of the surgical robot and the mapping relationship between the base coordinate system and the reference coordinate system.

[0141] Please continue to refer to Figure 8 ,like Figure 8As shown, in an exemplary embodiment, if the motion path of the surgical instrument to be guided cannot be planned based on the historical intraoperative images, the motion path of the surgical instrument to be guided is planned by the following steps:

[0142] Taking the current field of view of the endoscope 500 as the initial field of view, adjusting the field of view of the endoscope 500 until the surgical instrument to be guided is within the field of view of the endoscope 500, and acquiring real-time intraoperative images collected by the endoscope 500 in each field of view;

[0143] Planning a motion path of the surgical instrument to be guided based on the real-time intraoperative image and the current posture information and target posture information of the surgical instrument to be guided;

[0144] Before controlling the surgical instrument to be guided to perform corresponding movement according to the movement path of the surgical instrument to be guided, the surgical instrument guiding method further includes:

[0145] The field of view of the endoscope 500 is adjusted back to the initial field of view.

[0146] Specifically, the position of the endoscope 500 and / or the camera parameters of the endoscope 500 can be adjusted to adjust the field of view of the endoscope 500. For more details, please refer to Figure 10 , which schematically shows the principle diagram of adjusting the field of view of the endoscope 500 provided in one embodiment of this embodiment. Figure 10 As shown, in this embodiment, the field of view of the endoscope 500 is adjusted by simultaneously adjusting the position and posture (direction angle) of the endoscope 500, so that the surgical instrument to be guided (i.e., the surgical instrument 400 located outside the initial field of view of the endoscope 500) is located within the field of view of the endoscope 500. It should be noted that, as will be understood by those skilled in the art, in other embodiments, the field of view of the endoscope 500 may be adjusted only by adjusting the position of the endoscope 500; the field of view of the endoscope 500 may be adjusted only by adjusting the direction angle (direction angle) of the endoscope 500; the field of view of the endoscope 500 may be adjusted only by adjusting the camera parameters (viewing angle) of the endoscope 500; or the field of view of the endoscope 500 may be adjusted by adjusting both the posture of the endoscope 500 and the camera parameters of the endoscope 500 at the same time.

[0147] During the process of adjusting the field of view of the endoscope 500, the intraoperative image captured by the endoscope 500 (i.e., the real-time intraoperative image) is obtained in real time, so that the guided surgical instrument can be planned to move from the current position to the target position without collision based on the real-time intraoperative image.

[0148] Furthermore, planning a motion path of the surgical instrument to be guided based on the real-time intraoperative image and the current posture information and target posture information of the surgical instrument to be guided includes:

[0149] Acquiring position information of each obstacle 700 in the in vivo environment based on the real-time intraoperative image;

[0150] Planning a motion path of the surgical instrument to be guided based on the posture information of the obstacle 700 and the current posture information and target posture information of the surgical instrument to be guided; or

[0151] constructing a three-dimensional model of the in vivo environment based on the real-time intraoperative image;

[0152] The motion path of the surgical instrument to be guided is planned according to the three-dimensional model of the internal body environment and the current posture information and target posture information of the surgical instrument to be guided.

[0153] It should be noted that, as will be understood by those skilled in the art, for details on how to obtain the positional information of each obstacle 700 in the in-vivo environment based on the real-time intraoperative images, reference can be made to the above description regarding obtaining the positional information of each obstacle 700 in the in-vivo environment based on the historical intraoperative images, and will not be further elaborated upon here. Similarly, for details on how to construct a three-dimensional model of the in-vivo environment based on the real-time intraoperative images, reference can be made to the above description regarding constructing a three-dimensional model of the in-vivo environment based on the historical intraoperative images, and will not be further elaborated upon here.

[0154] Please continue to refer to Figure 11 , which schematically shows a schematic diagram of a three-dimensional model of the internal environment constructed based on real-time intraoperative images provided in a specific example of this embodiment. In the figure, the surgical instrument 400B located outside the initial field of view of the endoscope 500 is the surgical instrument to be guided. Figure 11 As shown, by adjusting the field of view of the endoscope 500 so that the surgical instrument to be guided that is located outside the initial field of view (current field of view) of the endoscope 500 is located within the field of view of the endoscope 500, and in the process of adjusting the field of view, intraoperative images are collected in real time to obtain real-time intraoperative images, so that a three-dimensional model of the in vivo environment can be constructed based on the real-time intraoperative images. According to the three-dimensional model of the in vivo environment, the position information of each obstacle 700 in the in vivo environment (the surgical instrument 400A and the surgical instrument 400C located in the initial field of view of the endoscope 500 are also regarded as obstacles) can be clearly known, so that according to the three-dimensional model of the in vivo environment, the motion path of the surgical instrument to be guided (that is, the surgical instrument 400B located outside the initial field of view of the endoscope 500) can be determined to move from the current position to the target position without collision.

[0155] Please continue to refer to Figure 12 , which schematically shows a protection strategy diagram of the guided surgical instrument 400 during movement provided in an embodiment of this embodiment. Figure 12 As shown, in an exemplary embodiment, in the process of controlling the surgical instrument to be guided to perform corresponding movements, the surgical instrument guiding method further includes:

[0156] Acquiring real-time position information of the surgical instrument to be guided, and calculating the real-time distance between the surgical instrument to be guided and the obstacle 700 according to the real-time position information of the surgical instrument to be guided;

[0157] Determine whether the real-time distance is less than a preset distance threshold. If so, send an alarm message. When the surgical instrument to be guided collides with the obstacle 700, lock the surgical instrument to be guided, send an alarm message, and replan the movement path of the surgical instrument to be guided.

[0158] Thus, during the movement of the surgical instrument to be guided, whether the distance between the surgical instrument to be guided and each obstacle 700 is less than a preset distance threshold is determined in real time. If the distance between the surgical instrument to be guided and the obstacle 700 is less than the preset distance threshold, an alarm message is sent to serve as a warning, prompting the operator to promptly adjust the movement path of the surgical instrument to be guided to prevent the surgical instrument from colliding with the obstacle 700. If the surgical instrument to be guided collides with the obstacle 700 during movement, the surgical instrument to be guided is locked, preventing it from continuing to move and outputting energy. At the same time, an alarm message is sent to prompt the operator to replan the movement path of the surgical instrument to be guided, so that the surgical instrument can continue to move according to the replanned movement path.

[0159] In an exemplary embodiment, controlling the surgical instrument to be guided to perform corresponding movement according to the movement path of the surgical instrument to be guided includes:

[0160] Manually controlling the surgical instrument to be guided to move accordingly according to the movement path of the surgical instrument to be guided; or

[0161] According to the movement path of the surgical instrument to be guided, the surgical instrument to be guided is automatically controlled to perform corresponding movement.

[0162] Please continue to refer to Figure 13 , which schematically shows a control flow diagram of the guided surgical instrument during movement provided by an embodiment of this invention. Figure 13As shown, when the manual adjustment mode is selected, the operator can manually move the surgical instrument to be guided according to the movement path of the surgical instrument to be guided; when the automatic adjustment mode is selected, the system can automatically move the surgical instrument to be guided according to the movement path of the surgical instrument to be guided.

[0163] Please continue to refer to Figure 13 ,like Figure 13 As shown, in an exemplary embodiment, manually controlling the surgical instrument to be guided to move accordingly according to the movement path of the surgical instrument to be guided includes:

[0164] According to the movement path of the surgical instrument to be guided, a guiding force with the same direction as the movement direction of the surgical instrument to be guided is applied to the robotic arm where the surgical instrument to be guided is located, so as to guide the operator to manually control the surgical instrument to be guided to perform corresponding movement according to the movement path.

[0165] Therefore, when the manual adjustment mode is selected, a guiding force in the same direction as the movement direction of the surgical instrument to be guided is applied to the robotic arm where the surgical instrument to be guided is located according to the movement path of the surgical instrument to be guided, so that the operator can be guided to adjust the posture of the surgical instrument to be guided in the correct direction, so that the operator can smoothly adjust the surgical instrument to be guided from the current posture to the target posture according to the movement path of the surgical instrument to be guided.

[0166] In an exemplary embodiment, during the process of controlling the surgical instrument to be guided to perform corresponding movements, the surgical instrument guiding method further includes:

[0167] Acquiring real-time posture information of the surgical instrument to be guided, and calculating the real-time posture deviation of the surgical instrument to be guided based on the real-time posture information of the surgical instrument to be guided and its target posture information;

[0168] Determine whether the real-time posture deviation is within a preset error range, and if so, stop the movement of the surgical instrument to be guided.

[0169] Please continue to refer to Figure 13 ,like Figure 13 As shown, in both manual adjustment mode and automatic adjustment mode, it is determined in real time whether the deviation between the real-time posture of the surgical instrument to be guided and the target posture of the surgical instrument to be guided (i.e., the real-time posture deviation) is within a preset error range (i.e., whether the absolute value of the real-time posture deviation is less than the acceptable maximum error Δ). If the judgment result is yes, it indicates that the surgical instrument to be guided has moved to the target posture, and the movement of the surgical instrument to be guided can be stopped at this time.

[0170] In an exemplary embodiment, the surgical instrument guidance method further includes:

[0171] Displaying the motion path of the surgical instrument to be guided; and / or

[0172] After the surgical instrument to be guided moves into the current field of view of the endoscope 500 , the target position and the movement direction of the surgical instrument to be guided are displayed.

[0173] Please continue to refer to Figure 14 and Figure 15 ,in, Figure 14 A schematic diagram of displaying the motion path of the surgical instrument to be guided on the image interface provided in an embodiment of the present invention is schematically provided; Figure 15 The following schematic diagram shows the display of the image interface after the surgical instrument to be guided enters the field of view of the endoscope 500 provided in one embodiment of the present invention. Figure 14 and Figure 15 As shown, the image interface will display the adjustment method selected by the operator. For example, when the operator selects the manual adjustment mode, the icon corresponding to the manual adjustment mode will be displayed in a striking manner. When the operator selects the automatic adjustment mode, the image corresponding to the automatic adjustment mode will be displayed in a striking manner. At the same time, the image interface will also display the movement path of the surgical instrument to be guided, so that the operator can more intuitively know the movement path of the surgical instrument to be guided. After the surgical instrument to be guided moves into the field of view of the endoscope 500, the target position and movement direction of the surgical instrument to be guided are indicated with a striking mark, so that the operator can more intuitively know the next movement path of the surgical instrument to be guided. In particular, when the operator selects the manual adjustment mode, the operator can be better guided to move the surgical instrument to be guided in the correct direction of movement.

[0174] In an exemplary embodiment, during the process of controlling the surgical instrument to be guided to perform corresponding movements, the surgical instrument guiding method further includes:

[0175] The real-time posture information of the surgical instrument to be guided is obtained, and the motion path of the surgical instrument to be guided is updated in real time according to the real-time posture information of the surgical instrument to be guided and its target posture information.

[0176] Please continue to refer to Figure 16 , which schematically shows a flow chart of updating the motion path provided by an implementation method of this embodiment. Figure 16As shown, after planning the motion path (initial motion path) of the surgical instrument to be guided, the operator can select an adjustment mode. After selecting the adjustment mode, the surgical instrument to be guided can be moved (if the manual adjustment mode is selected, the surgical instrument to be guided is moved manually; if the automatic adjustment mode is selected, the surgical instrument to be guided is moved automatically). In the process of moving the surgical instrument to be guided, the posture information of the surgical instrument to be guided is obtained in real time (i.e., the real-time posture information of the surgical instrument to be guided is obtained), and the motion path of the surgical instrument to be guided is updated in real time based on the real-time posture information of the surgical instrument to be guided and its target posture information. The updated motion path will be synchronously displayed on the image interface. Therefore, by updating the motion path of the surgical instrument to be guided in real time based on the real-time posture information of the surgical instrument to be guided, it can be further ensured that the surgical instrument to be guided can move to its target posture without collision, further reducing unknown risks and improving safety during surgery.

[0177] Example 2

[0178] Corresponding to the above-mentioned surgical instrument guiding method, the present invention also provides a surgical robot, please refer to Figure 1 .like Figure 1 As shown, in addition to the doctor's console 100, patient trolley 200, and imaging trolley 300 described above, the surgical robot also includes a controller 800. The doctor's console 100, patient trolley 200, and imaging trolley 300 are all in communication with the controller 800. The controller 800 is configured to implement the surgical instrument guidance method described above. Since the controller 800 in the surgical robot provided in this embodiment is capable of implementing the surgical instrument guidance method described above, the surgical robot provided in this embodiment can not only effectively guide the surgical instrument 400 located outside the field of view of the endoscope 500 to move into the field of view of the endoscope 500, but also ensure that the surgical instrument 400 does not collide during movement, thereby effectively reducing unknown risks during surgery and improving safety during surgery.

[0179] It should be noted that, as those skilled in the art will appreciate, the controller 800 can be provided in combination with any one or more devices of the surgical robot described above. For example, the controller 800 can be provided at the doctor's console 100, or at the patient cart 200, or at the imaging cart 300, etc. In some other embodiments, the controller 800 can also be provided separately. In addition, it should be noted that, as those skilled in the art will appreciate, the controller 800 can be a specific hardware or software unit, or a combination of hardware and software. The present invention does not limit the specific configuration of the controller 800.

[0180] Example 3

[0181] Based on the same inventive concept, the present invention also provides an electronic device, please refer to Figure 17 , schematically shows a block diagram of an electronic device provided by an embodiment of the present invention. Figure 17 As shown, the electronic device includes a processor 910 and a memory 930. The memory 930 stores a computer program. When the computer program is executed by the processor 910, it implements the surgical instrument guidance method described above. Since the electronic device provided in this embodiment and the surgical instrument guidance method provided in Example 1 are based on the same inventive concept, the electronic device provided in this embodiment has all the advantages of the surgical instrument guidance method provided in Example 1, and therefore the advantages of the electronic device provided in this embodiment will not be detailed one by one.

[0182] like Figure 17 As shown, the electronic device further includes a communication interface 920 and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. The communication bus 940 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 940 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or one type of bus. The communication interface 920 is used for communication between the above-mentioned electronic device and other devices.

[0183] The processor 910 referred to in the present invention may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The processor 910 is the control center of the electronic device, connecting various parts of the entire electronic device using various interfaces and lines.

[0184] The memory 930 may be used to store the computer program. The processor 910 implements various functions of the electronic device by running or executing the computer program stored in the memory 930 and calling the data stored in the memory 930.

[0185] The memory 930 may include nonvolatile and / or volatile memory. Nonvolatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0186] Example 4

[0187] The present invention also provides a readable storage medium having a computer program stored therein. When executed by a processor, the computer program can implement the surgical instrument guidance method described above. Because the readable storage medium provided in this embodiment and the surgical instrument guidance method provided in Example 1 are based on the same inventive concept, the readable storage medium provided in this embodiment possesses all the advantages of the surgical instrument guidance method provided in Example 1. Therefore, the advantages of the readable storage medium provided in this embodiment will not be detailed here.

[0188] The readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer hard disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this article, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0189] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0190] In summary, compared with the prior art, the surgical instrument guidance method, surgical robot, electronic device, and storage medium provided by the present invention have the following advantages:

[0191] The surgical instrument guidance method provided by the present invention first obtains the current posture information and target posture information of the surgical instrument to be guided, wherein the target posture is within the current field of view of the endoscope and the current posture is outside the current field of view of the endoscope; then, based on the intraoperative image captured by the endoscope and the current posture information and target posture information of the surgical instrument to be guided, the movement path of the surgical instrument to be guided is planned so that the surgical instrument to be guided can move to the target posture without collision; finally, based on the movement path of the surgical instrument to be guided, the surgical instrument to be guided is controlled to perform corresponding movement. It can be seen that the surgical instrument guidance method provided by the present invention can not only effectively guide the surgical instrument located outside the field of view of the endoscope to move into the field of view of the endoscope, but also ensure that the surgical instrument will not collide during the movement process, thereby effectively reducing unknown risks during the operation and improving the safety of the operation.

[0192] Since the surgical robot, electronic device and storage medium provided by the present invention belong to the same inventive concept as the surgical instrument guidance method provided by the present invention, the surgical robot, electronic device and storage medium provided by the present invention have all the advantages of the surgical instrument guidance method provided by the present invention, so the beneficial effects of the surgical robot, electronic device and storage medium provided by the present invention will not be described one by one.

[0193] It should be noted that, as will be appreciated by those skilled in the art, the computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0194] It should be noted that the devices and methods disclosed in the embodiments of this document may also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the devices, methods, and computer program products according to the various embodiments of this document. In this regard, each box in the flowchart or block diagram may represent a module, program, or portion of code, wherein the module, program segment, or portion of code contains one or more executable instructions for implementing a specified logical function, and the module, program segment, or portion of code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified function or action, or may be implemented by a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of this document may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0195] The above description is merely a description of preferred embodiments of the present invention and does not limit the scope of the present invention. Any changes or modifications made by persons skilled in the art based on the above disclosure are within the scope of protection of the present invention. Obviously, various modifications and variations may be made by persons skilled in the art without departing from the spirit and scope of the present invention. Thus, if such modifications and variations fall within the scope of the present invention and its equivalents, the present invention is intended to include such modifications and variations.

Claims

1. A readable storage medium, applied to a surgical robot, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by the processor, the following steps are implemented: Acquiring current posture information and target posture information of the surgical instrument to be guided, wherein the target posture is located within the current field of view of the endoscope and the current posture is located outside the current field of view of the endoscope; Planning a motion path of the surgical instrument to be guided based on the intraoperative image captured by the endoscope and the current posture information and target posture information of the surgical instrument to be guided, so that the surgical instrument to be guided can move to the target posture without collision; Controlling the surgical instrument to be guided to move accordingly according to the movement path of the surgical instrument to be guided; The step of planning a motion path of the surgical instrument to be guided based on the intraoperative image collected by the endoscope and the current posture information and target posture information of the surgical instrument to be guided comprises: Acquiring historical intraoperative images collected by the endoscope; determining whether a motion path of the surgical instrument to be guided can be planned based on the historical intraoperative images; If not, the motion path of the surgical instrument to be guided is planned by the following steps: Taking the current field of view of the endoscope as the initial field of view, adjusting the field of view of the endoscope until the surgical instrument to be guided is within the field of view of the endoscope, and acquiring real-time intraoperative images collected by the endoscope in each field of view; Planning a motion path of the surgical instrument to be guided based on the real-time intraoperative image and the current posture information and target posture information of the surgical instrument to be guided; If yes, planning a motion path of the surgical instrument to be guided based on the historical intraoperative image and the current posture information and target posture information of the surgical instrument to be guided; Before controlling the surgical instrument to be guided to move accordingly according to the movement path of the surgical instrument to be guided, the following steps are further performed: The field of view of the endoscope is adjusted back to the initial field of view.

2. The readable storage medium according to claim 1, wherein When the computer program is executed by the processor, the following steps are further implemented: determining a surgical instrument located outside the current field of view of the endoscope, and using the surgical instrument located outside the current field of view of the endoscope as the surgical instrument to be guided; Wherein, determining the surgical instrument located outside the current field of view of the endoscope includes: Acquiring current posture information of each installed surgical instrument, current posture information of the endoscope, and current camera parameter information of the endoscope; Acquiring position range information corresponding to the current field of view of the endoscope according to the current position information of the endoscope and the current camera parameter information; For each of the surgical instruments, determining whether the current posture of the surgical instrument is within the posture range corresponding to the current field of view of the endoscope; if not, determining that the surgical instrument is outside the current field of view of the endoscope; and / or Acquiring a current frame of intraoperative image collected by the endoscope; Performing surgical instrument recognition on the intraoperative image of the current frame to identify the surgical instrument located within the current field of view of the endoscope; The surgical instrument located within the current field of view of the endoscope is compared with installed surgical instruments to determine the surgical instrument located outside the current field of view of the endoscope.

3. The readable storage medium according to claim 1, wherein The determining whether the motion path of the surgical instrument to be guided can be planned based on the historical intraoperative image includes: determining, based on the historical intraoperative images, whether the historical intraoperative images contain image information showing that the surgical instrument to be guided has moved out of the field of view of the endoscope; If so, determining whether a collision occurs in the process of the surgical instrument to be guided moving out of the field of view of the endoscope based on image information of the surgical instrument to be guided contained in the historical intraoperative image, and if not, determining that a motion path of the surgical instrument to be guided can be planned based on the historical intraoperative image; If not, determine whether the historical intraoperative image contains all image information of the process of the surgical instrument to be guided moving from its current position to its target position. If so, determine that the movement path of the surgical instrument to be guided can be planned based on the historical intraoperative image.

4. The readable storage medium according to claim 3, wherein: If the historical intraoperative image contains image information of the surgical instrument to be guided moving out of the field of view of the endoscope, planning a motion path of the surgical instrument to be guided based on the historical intraoperative image and the current posture information and target posture information of the surgical instrument to be guided includes: acquiring, based on image information of the surgical instrument to be guided moving out of the field of view of the endoscope contained in the historical intraoperative image, information on a movement path of the surgical instrument to be guided out of the field of view of the endoscope; The motion path of the surgical instrument to be guided is planned according to the removal path information and the current posture information and target posture information of the surgical instrument to be guided.

5. The readable storage medium according to claim 3, wherein: If the historical intraoperative image contains all image information of the surgical instrument to be guided during the process of moving from its current posture to its target posture, planning the motion path of the surgical instrument to be guided based on the historical intraoperative image and the current posture information and target posture information of the surgical instrument to be guided includes: Acquiring position information of various obstacles in an in-vivo environment based on the historical intraoperative images; Planning a motion path of the surgical instrument to be guided based on the posture information of the obstacle and the current posture information and target posture information of the surgical instrument to be guided; or constructing a three-dimensional model of the in vivo environment based on the historical intraoperative images; The motion path of the surgical instrument to be guided is planned according to the three-dimensional model of the internal body environment and the current posture information and target posture information of the surgical instrument to be guided.

6. The readable storage medium according to claim 1, wherein The adjusting the field of view of the endoscope includes: The posture of the endoscope and / or the camera parameters of the endoscope are adjusted to adjust the field of view of the endoscope.

7. The readable storage medium according to claim 1, wherein: Planning a motion path of the surgical instrument to be guided based on the real-time intraoperative image and the current posture information and target posture information of the surgical instrument to be guided includes: Acquiring position information of various obstacles in the in-vivo environment based on the real-time intraoperative image; Planning a motion path of the surgical instrument to be guided based on the posture information of the obstacle and the current posture information and target posture information of the surgical instrument to be guided; or constructing a three-dimensional model of the in vivo environment based on the real-time intraoperative image; The motion path of the surgical instrument to be guided is planned according to the three-dimensional model of the internal body environment and the current posture information and target posture information of the surgical instrument to be guided.

8. The readable storage medium according to claim 5 or claim 7, wherein: In the process of controlling the surgical instrument to be guided to perform corresponding movements, the following steps are also performed: Acquiring real-time position information of the surgical instrument to be guided, and calculating the real-time distance between the surgical instrument to be guided and the obstacle based on the real-time position information of the surgical instrument to be guided; Determine whether the real-time distance is less than a preset distance threshold. If so, send an alarm message. When the surgical instrument to be guided collides with the obstacle, lock the surgical instrument to be guided, send an alarm message, and replan the movement path of the surgical instrument to be guided.

9. The readable storage medium according to claim 1, wherein: In the process of controlling the surgical instrument to be guided to perform corresponding movements, the following steps are also performed: Acquiring real-time position information of the surgical instrument to be guided; updating the motion path of the surgical instrument to be guided in real time according to the real-time posture information of the surgical instrument to be guided and its target posture information; and\or Based on the real-time posture information of the surgical instrument to be guided and its target posture information, the real-time posture deviation of the surgical instrument to be guided is calculated, and it is determined whether the real-time posture deviation is within a preset error range. If so, the movement of the surgical instrument to be guided is stopped.

10. The readable storage medium according to claim 1, wherein When the computer program is executed by a processor, the following steps are further implemented: Displaying the motion path of the surgical instrument to be guided; and / or After the surgical instrument to be guided moves into the current field of view of the endoscope, the target position and movement direction of the surgical instrument to be guided are displayed.

11. The readable storage medium according to claim 1, wherein: The step of controlling the surgical instrument to be guided to perform corresponding movement according to the movement path of the surgical instrument to be guided comprises: Manually controlling the surgical instrument to be guided to move accordingly according to the movement path of the surgical instrument to be guided; or According to the movement path of the surgical instrument to be guided, the surgical instrument to be guided is automatically controlled to perform corresponding movement.

12. The readable storage medium according to claim 11, wherein: Manually controlling the surgical instrument to be guided to perform corresponding movement according to the movement path of the surgical instrument to be guided includes: According to the movement path of the surgical instrument to be guided, a guiding force with the same direction as the movement direction of the surgical instrument to be guided is applied to the robotic arm where the surgical instrument to be guided is located, so as to guide the operator to manually control the surgical instrument to be guided to perform corresponding movement according to the movement path.

13. A surgical robot, characterized in that: It includes a controller and at least one robotic arm, wherein a surgical instrument and an endoscope are installed at the end of at least one of the robotic arms, the controller is coupled to the robotic arm, and the controller is configured to implement the steps described in the readable storage medium according to any one of claims 1 to 12.

Images