A method for processing a surgical instrument out of field of view, a surgical robot and an electronic device

Abstract

The application relates to a surgical instrument out-of-view processing method, a surgical robot and an electronic device. The method comprises the following steps: determining the position of the end of the surgical instrument in the image plane of the endoscope based on the position of the end of the surgical instrument in the spatial coordinate system of the endoscope; and determining whether the surgical instrument is out of the field of view of the endoscope based on the position of the end of the surgical instrument in the image plane of the endoscope and the resolution parameter of the display screen. The technical scheme of the application is more accurate and efficient in determining whether the surgical instrument is out of the field of view based on coordinate system position conversion.

Description

Technical Field

[0001] This application relates to the field of medical device technology, specifically to a method for handling surgical instruments that are out of sight, a surgical robot, and electronic equipment. Background Technology

[0002] During robot-assisted surgery, the surgeon operates surgical instruments from a control panel, viewing the endoscopic images. The control panel's image window displays the surgical field of view inside the patient's body; the surgeon cannot see information outside this field of view. When surgical instruments are moved outside the field of view, the surgeon cannot accurately determine their position, potentially causing damage to surrounding tissues. Therefore, accurately and efficiently assisting the surgeon in promptly detecting whether surgical instruments have moved out of the field of view can effectively reduce surgical risks. Summary of the Invention

[0003] To address the aforementioned technical problems, this application provides a method for handling surgical instruments that have moved out of the field of vision, a surgical robot, and an electronic device. The method determines whether a surgical instrument has moved out of the field of vision based on a coordinate system transformation, which is more accurate and efficient.

[0004] To address the aforementioned technical problems, this application provides a method for handling surgical instruments that have fallen out of the field of vision, applied to a surgical robot. The surgical robot includes surgical instruments, an endoscope, and a display screen. The display screen is used to display at least the images acquired by the endoscope. The method includes the following steps: Obtain the position of the distal end of the surgical instrument in the spatial coordinate system of the endoscope; Based on the position of the tip of the surgical instrument in the spatial coordinate system of the endoscope, the position of the tip of the surgical instrument in the image plane of the endoscope is determined. Based on the position of the tip of the surgical instrument in the image plane of the endoscope and the resolution parameters of the display screen, it is determined whether the surgical instrument has left the field of view of the endoscope.

[0005] In some embodiments, after determining whether the surgical instrument has moved out of the endoscope's field of view based on the position of the surgical instrument's tip in the endoscope's image plane and the resolution parameters of the display screen, the method further includes: When it is determined that the surgical instrument is out of the field of view of the endoscope, the relative orientation between the position of the end of the surgical instrument and the display center of the display screen is determined; Based on the relative orientation, the corresponding edges of the display screen and / or the display area of ​​the image are marked to at least indicate the orientation of the surgical instrument after it has left the field of vision.

[0006] In some embodiments, determining whether the surgical instrument has moved out of the endoscope's field of view based on the position of the surgical instrument's tip in the endoscope's image plane and the resolution parameters of the display screen includes: The position of the tip of the surgical instrument in the image plane of the endoscope is mapped to the pixel coordinate plane of the display screen to determine the position of the tip of the surgical instrument in the pixel coordinate plane of the display screen; Based on the position of the end of the surgical instrument in the pixel coordinate plane of the display screen and the resolution parameters of the display screen, it is determined whether the surgical instrument has left the field of view of the endoscope.

[0007] In some embodiments, determining the relative orientation between the position of the end of the surgical instrument and the display center of the display screen includes: A first line connecting the position of the surgical instrument tip in the image plane of the endoscope and the center point of the image plane is mapped to the pixel coordinate plane of the display screen to obtain a second line. The edge of the display screen that intersects with the second line is determined based on the slope of the second line to characterize the relative orientation between the position of the surgical instrument tip and the display center of the display screen. or, The position of the tip of the surgical instrument in the image plane of the endoscope is mapped to the pixel coordinate plane of the display screen to determine the position of the tip of the surgical instrument in the pixel coordinate plane of the display screen. Based on the slope of the third line connecting the position of the tip of the surgical instrument in the pixel coordinate plane of the display screen and the display center of the display screen, the edge of the display screen that intersects with the third line is determined to characterize the relative orientation between the position of the tip of the surgical instrument and the display center of the display screen. In some embodiments, the identification of the corresponding edge of the display screen and / or the corresponding edge of the display area of ​​the image based on the relative orientation includes: Determine the first intersection point between the edge of the display screen that intersects with the second connecting line and the second connecting line, mark the first intersection point, and / or mark the position of the corresponding edge of the display area of ​​the image at the position corresponding to the first intersection point; Determine the second intersection point between the edge of the display screen that intersects with the third connecting line and the third connecting line, mark the second intersection point, and / or mark the position of the corresponding edge of the display area of ​​the image at the position corresponding to the second intersection point; The corresponding edges of the display screen and / or the corresponding edges of the display area of ​​the image are marked as a whole.

[0008] In some embodiments, the identifier is deformed based on the distance the tip of the surgical instrument is out of the field of vision to characterize the change in the distance the tip of the surgical instrument is out of the field of vision; or, The identifier includes a first shape portion and a second shape portion. The first shape portion is deformed based on the distance of the end of the surgical instrument from the field of view to characterize the change in the distance of the end of the surgical instrument from the field of view. The second shape portion is deformed based on the pose of the end of the surgical instrument to characterize the change in the pose of the end of the surgical instrument.

[0009] In some embodiments, after identifying the corresponding edges of the display screen and / or the corresponding edges of the display area of ​​the image based on the relative orientation, the method further includes: The position and / or orientation of the endoscope are automatically adjusted based on the position of the distal end of the surgical instrument. Determine whether the tip of the surgical instrument has returned to the field of view of the endoscope; When the tip of the surgical instrument returns to the field of view of the endoscope, the endoscope is controlled to stop moving, and the marker disappears.

[0010] In some embodiments, automatically adjusting the position and / or orientation of the endoscope based on the position of the distal end of the surgical instrument includes: Based on the position of the end of the surgical instrument in the spatial coordinate system of the endoscope, determine the vector between the end of the surgical instrument and the origin of the spatial coordinate system of the endoscope; The position and / or orientation of the endoscope are adjusted based on the deviation between the coordinate axis parallel to the central axis of the endoscope in the spatial coordinate system of the endoscope and the vector.

[0011] This application also provides a surgical robot, which includes surgical instruments, an endoscope, a display screen and a controller. The display screen is used to display images acquired by the endoscope, and the controller executes a computer program to implement the method for handling surgical instruments out of the field of vision as described above.

[0012] This application also provides an electronic device, including a storage medium and a controller, wherein a computer program is stored on the storage medium, and when the computer program is executed by the controller, it implements the above-described method for removing surgical instruments from the field of vision.

[0013] This application also provides a storage medium storing a computer program, which, when executed by a processor, implements the above-described method for removing surgical instruments from the field of vision.

[0014] This application discloses a method for handling surgical instruments that have moved out of the field of view, a surgical robot, and an electronic device. The method includes: determining the position of the surgical instrument's tip in the endoscope's image plane based on the position of the tip of the surgical instrument in the endoscope's spatial coordinate system; and determining whether the surgical instrument has moved out of the endoscope's field of view based on the position of the surgical instrument's tip in the endoscope's image plane and the resolution parameters of the display screen. The technical solution of this application, which determines whether a surgical instrument has moved out of the field of view by performing position transformation based on the coordinate system, is more accurate and efficient. Attached Figure Description

[0015] Figure 1 This is a top view of a surgical robot arranged in an operating room according to an embodiment of this application.

[0016] Figure 2 This is a schematic diagram of the main control console of a surgical robot according to one embodiment of this application.

[0017] Figure 3 This is a schematic diagram of the operating device of a surgical robot according to an embodiment of this application.

[0018] Figure 4 This is a top view of a surgical robot arranged in an operating room, according to another embodiment of this application.

[0019] Figure 5 This is a schematic diagram of the structure of the operating device shown in another embodiment of this application.

[0020] Figure 6 This is a schematic diagram showing the positional relationship between surgical instruments and the endoscopic field of view.

[0021] Figure 7 This is a simplified diagram illustrating the kinematic relationship between surgical instruments and endoscopes.

[0022] Figure 8 This is a flowchart illustrating a method for handling surgical instruments that have fallen out of the field of vision, according to one embodiment.

[0023] Figure 9 This is a schematic diagram showing the positional relationship between the tip of a surgical instrument and the endoscopic field of view.

[0024] Figure 10 This is a schematic diagram of the connection structure of surgical instruments.

[0025] Figure 11 This is a flowchart illustrating a method for handling surgical instruments that have fallen out of the field of vision, according to another embodiment.

[0026] Figure 12 This is a schematic diagram showing a variation of the identifier according to another embodiment.

[0027] Figure 13This is a schematic diagram illustrating the principle of adjusting an endoscope according to another embodiment. Detailed Implementation

[0028] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. In this invention, "each" includes one or more items.

[0030] Figure 1 This is a top view schematic diagram of a surgical robot arranged in an operating room according to an embodiment of this application. Figure 1 As shown, a single-arm, single-port surgical robot includes a main console 20 and a slave operating device 10. The main console 20 is remotely connected to the slave operating device 10, allowing the surgeon S to remotely operate and control the slave operating device 10 from the main console 20. The main console 20 is configured to send control signals to the slave operating device 10 and display images acquired by the slave operating device 10 based on the surgeon S's operations. Through the main console 20, the surgeon S can observe three-dimensional images of the patient's body provided by the imaging system. By observing these three-dimensional images, the surgeon S can immerse themselves in the experience and control the slave operating device 10 to perform related operations (such as performing surgery or acquiring images of the patient's body).

[0031] The operating device 10 includes a control unit, a robotic arm 11, and a tool-holding device 12. The control unit can be located in the base of the operating device 10 or on the robotic arm 11. In one embodiment, the control unit is used to control the joint movement of the robotic arm 11 and the movement of the drive device in the tool-holding device 12. Multiple surgical tools can be mounted on the tool-holding device 12, and the drive device of the tool-holding device 12 is used to drive the surgical tools to perform various surgeries.

[0032] In one embodiment, the surgical robot further includes a gas inhalation device, a lumen assembly (not shown), and a cannula 13, the lumen assembly providing fluid communication between the cannula 13 and the gas inhalation device. The cannula 13 is connected to the distal end of the instrument holding device 12 and is inserted into the body cavity of the patient P lying on the operating table T. The end devices of multiple surgical instruments or cameras at the distal end of an endoscope extend through the cannula 13 into the body cavity of the patient P to perform surgery-related operations or acquire images of the patient P's internal environment.

[0033] In one embodiment, surgeon S can control the operating mode of the gas inhalation device via main console 20, such as injecting gas from a gas source into the body cavity of patient P to create an artificial pneumoperitoneum, or aspirating gas from the body cavity of patient P. Assistant A attaches surgical instruments 40 to or removes surgical instruments 40 from the instrument holding device 12 according to the surgical situation. Surgeon S, assistant A, and anesthesiologist B constitute a basic surgical team. Surgical instruments 40 can be surgical instruments such as electrocautery devices, forceps, staplers, and ultrasonic scalpels used to perform surgical operations, or imaging devices (such as endoscopes) or other surgical tools for acquiring images.

[0034] The main control console 20 is also remotely connected to the electronic device cart 30, which in turn is remotely connected to the slave operating device 10. The electronic device cart 30 may include an energy generating device, an image signal processing device, and the aforementioned gas blowing device. In this embodiment, the main control console 20, the slave operating device 10, and the electronic device cart 30 communicate remotely via wired Ethernet. However, remote communication is not limited to wired Ethernet; it can also be other wired methods, such as, but not limited to, serial port, CAN, RS485, RS232, USB, SPI, etc., or wireless communication methods, such as, but not limited to, 5G, WiFi, NB, Zigbee, Bluetooth, RFID, etc.

[0035] In one embodiment, such as Figure 2As shown, the main control console 20 includes a display device 21, an armrest 22, an input device 23, an observation device 24, and a control signal processing system 25. The display device 21 displays images acquired by the imaging system. The display device 21 can be an image source reflected into the eyepiece via multiple mirrors, or it can be a 3D display. The armrest 22 is used to support the arm and / or hand of the doctor (e.g., surgeon S) to allow for more comfortable operation of the input device 23. The observation device 24 is used to observe the images displayed on the display device. Depending on actual needs, the armrest or observation device 24 can be omitted, allowing for direct observation. The doctor S manipulates the surgical instruments of the secondary operating device 10 by operating the input device 23. The control signal processing system of the main control console 20 processes the input signal from the input device 23 and sends control commands to the secondary operating device. The secondary operating device 10 responds to the control commands of the main control console 20 and performs corresponding operations. In some embodiments, the control signal processing system 25 can also be located in the secondary operating device 10, for example, in the base of the secondary operating device 10. The control signal processing system 25 can be the same device as the control device described above.

[0036] Surgical robots typically also include an imaging system (not shown) that enables the surgeon S to view the surgical site from outside the patient's body. This imaging system typically includes a surgical tool 40 with video image acquisition capabilities (e.g., an image acquisition function) and one or more video display devices for displaying the acquired images. Generally, the surgical tool 40 with image acquisition capabilities includes optics for acquiring images of one or more imaging sensors (e.g., CCD or CMOS sensors) within the patient's body. These one or more imaging sensors can be positioned distal to the surgical tool 40 with image acquisition capabilities, and the signals generated by these sensors can be transmitted via cable or wirelessly for processing and display on the video display device.

[0037] In one embodiment, such as Figure 3 As shown, the operating device 10 is usually placed near the operating table T during the preoperative preparation process. After the doctor makes an incision 301 in the patient P's body, a part of the trocar 13 used to support the surgical instruments will be inserted into the patient P's body through the incision 301. After sealing, the medical staff will manually drag the robotic arm of the operating device 10 to assemble and dock with one end of the trocar 13 that has been inserted into the patient's body. This ensures that during the doctor's master-slave operation control of the robotic arm of the operating device 10, the robotic arm can rotate around the remote motion center 103 of the trocar 13 without causing any additional damage to the incision 301 of the patient P.

[0038] In one embodiment, the holding device 12 may be equipped with a plurality of surgical instruments 40, which enter the body through the incision 301 via the same cannula 13.

[0039] Figure 4 This is a top view schematic diagram of a surgical robot arranged in an operating room according to another embodiment of this application. Figure 4 The image shows a multi-hole surgical robot, which is similar to... Figures 1 to 3 The main difference between the single-port surgical robot shown is that the multi-port surgical robot's operating device 10 includes multiple robotic arms 11, and correspondingly, multiple cannulas (not shown). The multiple robotic arms 11 can have the same configuration or different configurations, and multiple surgical instruments are mounted on the multiple robotic arms 11. Specifically, as... Figure 5 As shown, the first instrument 310, the second instrument 320, the third instrument 330, and the imaging device 340 (e.g., an endoscope) of the plurality of surgical instruments 110 are detachably mounted on the plurality of robotic arms 11. In some other embodiments, the instruments and imaging devices of the plurality of surgical instruments are interchangeable with the mounted robotic arms 11.

[0040] like Figure 5 As shown, the controller 250 is configured to control the joint movements of the drive robotic arm 11 and the movements of surgical instruments in response to control signals from the main control console 20 or the slave operating device 10. The controller 250 may be located in the base of the slave operating device 10. In some embodiments, the controller 250 may also be located on each robotic arm 11. It is understood that the controller 250 may also be located in the main control console 20.

[0041] Figure 6 This is a schematic diagram showing the positional relationship between surgical instruments and the endoscopic field of view. For example... Figure 6 As shown, taking a single-port robot as an example, surgical instruments 110 and endoscopes 120 are mounted on the drive housing 101 of the robotic arm from the operating device 10 through a cannula 13. To accurately describe the position information of the remote motion center 103 of the robotic arm and the attitude information of the drive housing 101 relative to the chassis base coordinate system of the device 100, a remote motion center coordinate system is typically established at the position of the remote motion center 103. This coordinate system serves as the reference coordinate system for the motion control of the surgical instrument 110 and endoscope 120. Furthermore, to accurately describe the position and orientation motion information of the surgical instrument 110 and endoscope 120, a base coordinate system for each surgical instrument 110 and endoscope 120 is established. and the base coordinate system of endoscope 120 and the end coordinate system at the end of the surgical instrument 110 at point 111. And the coordinate system of the endoscope 120 The coordinate system of the endoscope 120 It can be built on the lens end face of the endoscope 120. The axis coincides with the central axis of the endoscope 120. The image information obtained by the endoscope 120 is displayed on the doctor's console observation window screen as a view of the field of view plane M. This field of view plane M is parallel to the lens plane at the end of the endoscope 120 and is aligned with the coordinate system of the end-view image of the endoscope 120. They are in a vertical relationship.

[0042] Figure 7 This is a simplified diagram illustrating the kinematic relationship between surgical instruments and the endoscope. (For example...) Figure 7 As shown, based on the mechanical transmission structure and joint motion information of the surgical instruments themselves, the kinematic relationship between the end-effector coordinate system and the base coordinate system of the surgical instruments can be calculated. Similarly, based on the mechanical transmission structure and joint motion information of the endoscope itself, the kinematic relationship between the endoscope's display coordinate system and the endoscope's base coordinate system can be calculated. Based on the kinematic relationship between the base coordinate system of the surgical instruments, the base coordinate system of the endoscope, and the remote motion center coordinate system of the robotic arm, the kinematic relationship between the end-effector coordinate system and the endoscope's display coordinate system can be calculated, ultimately obtaining the pose information of the end-effector tool in the endoscope's display coordinate system and the relative pose relationships between the various surgical instruments.

[0043] When surgical instruments are operated outside the field of vision, the surgeon cannot accurately obtain the position information of the surgical instruments, which can easily cause damage to the surrounding tissues outside the field of vision. Therefore, this application provides a technical solution that can accurately and efficiently obtain the relative positional relationship between the end of the surgical instrument and the current field of vision when it is outside the field of vision, and provide prompts. This helps the lead surgeon to promptly detect whether the surgical instrument has left the field of vision, and guides the surgeon to move the surgical instrument outside the field of vision into the field of vision, thereby reducing the risk of surgical operation.

[0044] Figure 8 This is a flowchart illustrating a method for handling surgical instruments that have fallen out of the field of vision, according to one embodiment. Figure 8 As shown, this application discloses a method for handling surgical instruments that have fallen out of the field of vision, applicable to single-port or multi-port surgical robots. The surgical robot includes surgical instruments and an endoscope. The method includes the following steps: S1, based on the kinematic relationship between the end-effector coordinate system and the endoscope image coordinate system, determine the first distance between the end-effector along the central axis of the endoscope and the endoscope image coordinate system plane; S5, when the first distance is less than or equal to 0, determine that the surgical instruments have left the field of view of the endoscope; S6 outputs a notification message indicating that surgical instruments have left the field of vision.

[0045] Among them, based on Figure 6 and Figure 7 The principle behind this method is based on the kinematic relationship between the end-effector coordinate system and the endoscope's display coordinate system. This allows for the determination of the end-effector's position within the endoscope's display coordinate system. Furthermore, it allows for the determination of the first distance between the end-effector's position along the endoscope's central axis and the endoscope's display coordinate system plane. The endoscope's display coordinate system plane refers to the plane perpendicular to the endoscope's central axis, where the origin of the display coordinate system is located. For example, when the z-axis of the display coordinate system coincides with the endoscope's central axis, the endoscope's display coordinate system plane is the plane containing the x-axis and y-axis.

[0046] Thus, the first distance can represent the distance between the tip of the surgical instrument and the end face of the endoscope lens. When the first distance is calculated, step S2 is executed to determine whether the first distance is greater than 0. When the first distance is less than or equal to 0, it means that the tip of the surgical instrument is located behind the end face of the endoscope lens. At this time, it is determined that the surgical instrument has been removed from the field of view of the endoscope.

[0047] In some embodiments, such as Figure 9 As shown, taking a circular field of view of the endoscope as an example, the coordinate system of the endoscope 120 is established on the lens end face M2 of the endoscope 120. The axis coincides with the central axis L1 of the endoscope 120, and the lens radius of the endoscope 120 is expressed as follows: The edge C of the field of view of endoscope 120 is determined by the field of view angle of endoscope 120. The decision is made. A virtual cross-section parallel to the lens end face M2 of the endoscope 120, where the distal end point b of the surgical instrument 110 is located, is taken as the first endoscopic field of view plane M1 where the distal end of the surgical instrument 110 is located. Based on the kinematic relationship between the distal end coordinate system of the surgical instrument 110 and the image coordinate system of the endoscope 120, the first distance between the distal end point b of the surgical instrument along the central axis L1 of the endoscope 120 and the image coordinate plane of the endoscope 120 (i.e., the lens end face M2) can be obtained. .

[0048] After confirming that the surgical instruments have left the endoscope's field of vision, step S6 is executed, outputting a notification message to indicate that the surgical instruments have left the field of vision. This notification message can be displayed as text or graphics on the screen, as an indicator added to the edge of the screen, or as feedback to the user via sound or other means. The notification message can be displayed in a fixed position on the screen or as dynamic information displayed in real-time as the instruments move. The method of adding an indicator to the edge of the screen can be... Figure 11 Steps S40-S50 in the illustrated embodiment are implemented (see description below for details).

[0049] When the first distance is greater than 0, it indicates that the end of the surgical instrument is in front of the endoscope lens. At this time, other distance information is further combined to determine whether the surgical instrument has left the endoscope's field of vision.

[0050] In some embodiments, the method of this application further includes: S3, when the first distance is greater than 0, obtain the first maximum distance between the edge of the endoscope field of view of the first endoscope field of view plane where the end of the surgical instrument is located and the central axis of the endoscope, and obtain the first actual distance from the end of the surgical instrument to the central axis of the endoscope within the first endoscope field of view plane. S5, when the first actual distance is greater than or equal to the first maximum distance, determine that the surgical instruments have left the field of view of the endoscope.

[0051] In some embodiments, the method further includes: S7, when the first distance is greater than 0 and the first actual distance is less than the first maximum distance, determine that the surgical instruments are within the field of view of the endoscope.

[0052] The first endoscopic field of view plane, where the distal end of the surgical instrument is located, is a virtual cross-section parallel to the end face of the endoscope lens. The first maximum distance between the edge of the endoscopic field of view plane and the central axis of the endoscope is determined based on the endoscope lens radius and field of view angle information. This first maximum distance determines the maximum image range that the endoscope can capture within the first endoscopic field of view plane. Therefore, after calculating the first actual distance and the first maximum distance, step S4 is executed to determine whether the first actual distance is greater than or equal to the first maximum distance. If so, it is determined that the surgical instrument has left the endoscope's field of view, and step S6 is executed, outputting a prompt message; otherwise, it is determined that the surgical instrument is within the endoscope's field of view, and no prompt message is output.

[0053] Specifically, such as Figure 9 As shown, based on the kinematic relationship between the coordinate system of the surgical instrument 110's distal end and the coordinate system of the endoscope 120's field of view, the first actual distance from the distal end of the surgical instrument 110 to the central axis L1 of the endoscope 120 within the first endoscopic field of view plane M1 can also be obtained as follows: Furthermore, based on the lens radius of the endoscope 120... Information and field of view of the endoscope The information also reveals the first maximum distance between the edge C of the endoscopic field of view in the first endoscopic field of view plane M1 and the central axis L1 of the endoscope 120. The first maximum distance is This determines the maximum image range that the endoscope 120 can capture within the first endoscopic field of view M1, by measuring the first actual distance. The first maximum distance By comparison, the relative positional relationship between the distal end of the surgical instrument 110 and the field of view of the endoscope 120 can be determined.

[0054] Specifically, if <E and If the value is >0, it is assumed that the tip of the surgical instrument 110 is within the field of view of the endoscope 120. In this case, the system does not output a prompt message indicating that the surgical instrument 120 has left the field of view. ≥ and If the value is >0, it is assumed that the end of the surgical instrument 110 is outside the field of view of the endoscope 120, and the system outputs a prompt message indicating that the surgical instrument 120 has left the field of view.

[0055] By employing the above method, distance transformation based on the coordinate system is used to determine whether surgical instruments have left the field of view, and a prompt message is output, making the process more accurate and efficient. Furthermore, it decouples the analysis process of whether the end of a surgical instrument has left the endoscope's field of view from the endoscope's image acquisition process, resulting in greater robustness.

[0056] Surgical instruments and endoscopes have basically the same main structure, including a drive unit, long shaft connecting rod, shoulder and elbow base, shoulder and elbow joint, wrist base and wrist joint. The difference lies in the end-effectors, which are end tools and camera lenses, respectively. Figure 10 This is a schematic diagram of the connection structure of surgical instruments. (For example...) Figure 10 As shown, taking surgical instruments as an example, the surgical instruments include a drive unit 1011, a long-axis connecting rod 116, a shoulder-elbow base 115, a shoulder-elbow joint 114, a wrist base 113, a wrist joint 112, and an end effector 111, which is the end tool. The drive unit 1011 mainly consists of active devices used to control the movement of the long-axis connecting rod 116, the shoulder-elbow joint 114, the wrist joint 112, and the end tool. The long-axis connecting rod 116 connects the shoulder-elbow base 115 and the drive unit 1011, and can achieve rotational freedom around its own axis L2 according to control commands. A shoulder-elbow base coordinate system is established at the position of the shoulder-elbow base 115. Establish a wrist base coordinate system at position 113 of the wrist base. (Equivalent to the distal coordinate system of shoulder-elbow joint 114), establish a coordinate system at the connection position of wrist joint 112 and distal end 111 (equivalent to the distal coordinate system of wrist joint 112), then based on Figure 7The coordinate system transformation relationship can be obtained to obtain the transformation relationship between the end coordinate system of the shoulder and elbow joint 114, the end coordinate system of the wrist joint 112 and the image coordinate system of the endoscope 120.

[0057] The postures of the shoulder-elbow joint 114 and wrist joint 112 reflect the posture of the distal end 111 of the surgical instrument. Normally, the surgical instrument is considered out of view when its distal end is determined to be out of sight. However, this only indicates the position of the distal end and does not reflect the instrument's posture. Therefore, it is necessary to further consider whether the wrist and shoulder-elbow joints are within the field of view to provide a more comprehensive picture of the instrument's current posture to the operator, enabling them to accurately control the direction of operation of the master and slave instruments.

[0058] In some embodiments, the surgical instrument further includes a wrist joint, and the end of the surgical instrument is mounted on the end of the wrist joint. The method of this application further includes: Based on the kinematic relationship between the distal end coordinate system of the wrist joint and the image coordinate system of the endoscope, a second distance is determined between the distal end of the wrist joint along the central axis of the endoscope and the image coordinate plane of the endoscope. When the second distance is less than or equal to 0, it is determined that the wrist joint is out of the endoscopic field of vision; When the second distance is greater than 0, the second maximum distance between the edge of the endoscope field of view in the second endoscope field of view plane where the end of the wrist joint is located and the central axis of the endoscope is obtained, and the second actual distance from the end of the wrist joint to the central axis of the endoscope in the second endoscope field of view plane is obtained. When the second actual distance is greater than or equal to the second maximum distance, it is determined that the wrist joint is out of the endoscopic field of view; Output a notification message indicating that the wrist joint has been out of sight.

[0059] The calculation of the second distance, the second maximum distance, and the second actual distance can be achieved using steps S1 and S3 described above. During the calculation process, the distal end of the wrist joint (i.e., Figure 10 The connection between the mid-end 111 and the wrist joint 112 is equivalent to the end of the surgical instrument in steps S1 and S3. The specific calculation process will not be repeated here.

[0060] In some embodiments, the surgical instrument further includes a shoulder-elbow joint, with one end of the wrist joint distal to the end of the surgical instrument mounted at the end of the shoulder-elbow joint. The method of this application further includes: Based on the kinematic relationship between the end-effector coordinate system and the endoscope's display coordinate system, the third distance between the end-effector of the shoulder and elbow joint along the central axis of the endoscope and the endoscope's display coordinate plane is determined. When the third distance is less than or equal to 0, it is determined that the shoulder and elbow joints are out of the endoscopic field of vision; When the third distance is greater than 0, obtain the third maximum distance between the edge of the endoscopic field of view of the third endoscopic field of view plane where the end of the shoulder and elbow joint is located and the central axis of the endoscope; and obtain the third actual distance from the end of the shoulder and elbow joint to the central axis of the endoscope in the third endoscopic field of view plane; when the third actual distance is greater than or equal to the third maximum distance, determine that the shoulder and elbow joint is out of the endoscopic field of view. Output a notification message indicating that the shoulder or elbow joint has left the field of vision.

[0061] The calculation of the third distance, the third maximum distance, and the third actual distance can be achieved using steps S1 and S3 described above. During the calculation process, the distal end of the shoulder and elbow joint (i.e., Figure 10 The wrist base 113 in the middle is equivalent to the end of the surgical instrument in steps S1 and S3. The specific calculation process will not be repeated here.

[0062] In some embodiments, the surgical instrument includes a distal end, a wrist joint, and a shoulder-elbow joint connected in sequence. The distal end is mounted on the distal end of the wrist joint, and the end of the wrist joint furthest from the distal end of the surgical instrument is mounted on the distal end of the shoulder-elbow joint. The method of this application further includes: Different prompts are output based on the combination of the surgical instrument tip, the wrist joint, and the shoulder and elbow joint where the view is out of the endoscope.

[0063] When the distal end of a surgical instrument, the wrist joint, and the shoulder / elbow joint are disengaged from the endoscope's field of vision in different combinations, the posture of the distal end of the instrument varies, making the operation of bringing the instrument back into the endoscope's field of vision varying in difficulty. Therefore, different prompts can be used to provide the operator with a more intuitive understanding. Different prompts include at least one of the following: different prompt text, different prompt volume, different flashing frequency of the prompt content, different font size of the prompt content, and different font color of the prompt content.

[0064] In some embodiments, different prompts are output based on the combination of the surgical instrument tip, the wrist joint, and the shoulder / elbow joint where the view is out of the endoscope, including at least one of the following: When the tip of the surgical instrument, the wrist joint, and the shoulder and elbow joints are all out of the field of vision, the first prompt message is output; When the tip of the surgical instrument leaves the field of vision, but the wrist and shoulder / elbow joints remain in the field of vision, a second prompt message is output. When the tip of the surgical instrument is not out of the field of vision, and the wrist joint and shoulder / elbow joint are both out of the field of vision, a third prompt message is output. The urgency level of the first, second, and third prompt messages decreases in that order.

[0065] In this way, the urgency of the surgical instrument tip, wrist joint, and shoulder / elbow joint being removed from the endoscope's field of vision in different combinations is distinguished. The first, second, and third prompts are output in a manner corresponding to different urgency levels, allowing the operator to receive feedback more intuitively and promptly.

[0066] The method for handling surgical instruments leaving the field of view in this embodiment includes: determining a first distance between the end of the surgical instrument along the central axis of the endoscope and the plane of the endoscope's image coordinate system based on the kinematic relationship between the coordinate system of the surgical instrument's distal end and the coordinate system of the endoscope's image coordinate system; determining that the surgical instrument has left the endoscope's field of view when the first distance is less than or equal to 0; and outputting a prompt message to indicate that the surgical instrument has left the field of view. The technical solution of this application, which determines whether the surgical instrument has left the field of view based on distance transformation of the coordinate system and outputs a prompt message, is more accurate and efficient.

[0067] Figure 11 This is a flowchart illustrating a method for handling surgical instruments that have fallen out of the field of vision, according to another embodiment. Figure 11 As shown, this application provides a method for handling surgical instruments that have fallen out of the field of vision, applicable to single-port or multi-port surgical robots. The surgical robot includes surgical instruments, an endoscope, and a display screen. The display screen is used to display at least the images acquired by the endoscope. The method includes the following steps: S10, Obtain the position of the end of the surgical instrument in the spatial coordinate system of the endoscope; S20, Based on the position of the end of the surgical instrument in the spatial coordinate system of the endoscope, determine the position of the end of the surgical instrument in the image plane of the endoscope; S30, based on the position of the surgical instrument tip in the image plane of the endoscope and the resolution parameters of the display screen, determines whether the surgical instrument has left the field of view of the endoscope.

[0068] Based on the mechanical transmission structure and joint motion information of the surgical instruments, the kinematic relationship between the instrument tip coordinate system and the instrument base coordinate system can be calculated. Similarly, based on the mechanical transmission structure and joint motion information of the endoscope, the kinematic relationship between the endoscope's spatial coordinate system and its base coordinate system can be calculated. Furthermore, based on the kinematic relationship between the instrument base coordinate system, the endoscope base coordinate system, and the robotic arm's remote motion center coordinate system, the kinematic relationship between the instrument tip coordinate system and the endoscope's spatial coordinate system can be calculated. Ultimately, the position and / or orientation of the surgical instrument tip in the endoscope's spatial coordinate system can be obtained. Therefore, during endoscope operation, through forward kinematics and coordinate transformation, the position of the surgical instrument tip can be converted to the endoscope's spatial coordinate system, thus obtaining the position of the surgical instrument tip in the endoscope's spatial coordinate system.

[0069] The image plane of an endoscope refers to the imaging plane. For example, when the endoscope is a digital camera, the image plane is the plane where the photosensitive element such as the CCD is located. After determining the position of the surgical instrument tip in the spatial coordinate system of the endoscope, based on the camera intrinsic parameter matrix of the endoscope, the position of the surgical instrument tip in the spatial coordinate system of the endoscope can be transformed into the position of the surgical instrument tip in the image plane of the endoscope.

[0070] The display screen's resolution parameter indicates its display range in the pixel coordinate plane. Based on the transformation relationship between the endoscope's image plane and the pixel coordinate system, it's possible to determine whether a surgical instrument has moved out of the endoscope's field of view, based on the position of the surgical instrument's tip in the endoscope's image plane and the display screen's resolution parameter. Specifically, if the position of the surgical instrument's tip is outside the display screen's display range, the surgical instrument is considered to be out of the endoscope's field of view; otherwise, it is not.

[0071] By using the above method, position transformation based on the transformation relationship between coordinate systems is used to determine whether the end of the surgical instrument has left the endoscope's field of view. This allows for more efficient and accurate relative position analysis during the movement of the surgical instrument. Furthermore, it decouples the analysis process of whether the end of the surgical instrument has left the endoscope's field of view from the endoscope's image acquisition process, resulting in greater robustness.

[0072] In some embodiments, after S30, the method further includes: S40, when it is determined that the surgical instrument has left the field of view of the endoscope, determine the relative orientation between the position of the end of the surgical instrument and the display center of the display screen; S50, based on relative orientation, marks are made on the corresponding edge of the display screen and / or the corresponding edge of the image display area to at least indicate the orientation of the surgical instruments after they have left the field of vision.

[0073] When it is determined that the surgical instrument is out of the endoscope's field of vision, the relative position between the end of the surgical instrument and the display center of the screen is determined. Based on the relative position, the operator can know in which direction the end of the surgical instrument is located with reference to the field of vision displayed on the screen. For example, it is located above, below, to the left, or to the right of the field of vision, or it is located to the right of the top of the field of vision, to the left of the top, above the left, below the left, etc.

[0074] After determining the relative orientation, markings are made on the corresponding edges of the display screen and / or the corresponding edges of the image display area based on the relative orientation to at least indicate the orientation of the surgical instrument after it leaves the field of view. For example, when the relative orientation is represented by the tip of the surgical instrument being above the field of view, the entire upper boundary of the display screen can be marked. Similarly, when the image acquired by the endoscope is not displayed in full screen, the entire upper boundary of the image display area can also be marked. As another example, when the relative orientation is represented by the tip of the surgical instrument being to the left of the upper edge of the field of view, only the left half of the upper boundary of the display screen can be marked. Furthermore, to more accurately represent the relative orientation, the marking may not be a complete segment; for example, it could be a marking displayed at a point on the boundary that provides a more precise indication of the orientation. In this way, by marking the corresponding boundaries based on relative orientation, operators can intuitively understand the situation of surgical instruments after they have left the endoscope's field of vision through the display screen, further improving the safety of the surgical procedure. Furthermore, it is also more efficient when operators need to manually maneuver surgical instruments back into the endoscope's field of vision.

[0075] In some embodiments, in addition to displaying the indicator, a voice prompt may also be output to alert the operator that the tip of the surgical instrument is currently out of the field of vision. Furthermore, the voice prompt may also indicate the procedure to bring the tip of the surgical instrument back into the endoscope's field of vision. The procedure may include manually adjusting the endoscope or triggering an automatic adjustment of the endoscope.

[0076] In some embodiments, S30, determining whether the surgical instrument has left the endoscopic field of view based on the position of the tip of the surgical instrument in the image plane of the endoscope and the resolution parameters of the display screen includes: The position of the tip of the surgical instrument in the image plane of the endoscope is mapped to the pixel coordinate plane of the display screen to determine the position of the tip of the surgical instrument in the pixel coordinate plane of the display screen. Based on the position of the surgical instrument tip in the pixel coordinate plane of the display screen and the resolution parameters of the display screen, it is determined whether the surgical instrument has left the field of view of the endoscope.

[0077] Based on the transformation relationship between the image plane of the endoscope and the pixel coordinate system, the position of the surgical instrument tip in the image plane of the endoscope can be mapped to the pixel coordinate plane of the display screen. This determines the position of the surgical instrument tip in the pixel coordinate plane of the display screen. Furthermore, by determining whether the position of the surgical instrument tip in the pixel coordinate plane of the display screen is within the display range corresponding to the resolution parameters of the display screen, it can be determined whether the surgical instrument has moved out of the endoscope's field of view. In other embodiments, the display range of the display screen can also be mapped to the image plane of the endoscope according to the display screen's resolution parameters. By determining whether the position of the surgical instrument tip in the image plane is within the range mapped from the display range of the display screen to the image plane of the endoscope, it can be determined whether the surgical instrument has moved out of the endoscope's field of view.

[0078] In some embodiments, S40, determining the relative orientation between the position of the end of the surgical instrument and the display center of the display screen includes: A first line connecting the position of the surgical instrument tip in the image plane of the endoscope and the center point of the image plane is mapped to the pixel coordinate plane of the display screen to obtain a second line. The edge of the display screen that intersects with the second line is determined based on the slope of the second line to characterize the relative orientation between the position of the surgical instrument tip and the display center of the display screen.

[0079] In this process, a first line is drawn connecting the tip of the surgical instrument to the center point of the image plane in the endoscope's image plane. This first line is then mapped onto the pixel coordinate plane of the display screen to obtain a second line. In the pixel coordinate plane, one endpoint of the second line is the center of the display area, thus intersecting with the boundary of the display area corresponding to the screen's resolution (the boundary of the display area corresponds to the boundary of the display screen). Using the center of the display area as coordinates in the pixel coordinate plane, the slope of the second line can be calculated. Based on the range of the slope of the second line, the intersection point of the second line with the display screen can be determined. This intersection point can then be used to characterize the relative orientation between the tip of the surgical instrument and the display center. For example, if the second line intersects the upper boundary, the tip of the surgical instrument is located above the field of view, and so on.

[0080] In some embodiments, S40, determining the relative orientation between the position of the end of the surgical instrument and the display center of the display screen includes: The position of the surgical instrument tip in the image plane of the endoscope is mapped to the pixel coordinate plane of the display screen to determine the position of the surgical instrument tip in the pixel coordinate plane of the display screen. Based on the slope of the third line connecting the position of the surgical instrument tip in the pixel coordinate plane of the display screen and the display center of the display screen, the edge of the display screen that intersects with the third line is determined to characterize the relative orientation between the position of the surgical instrument tip and the display center of the display screen. First, the position of the surgical instrument's tip in the endoscope's image plane is mapped to the display screen's pixel coordinate plane, determining the instrument's position within that plane. Then, a third line is drawn connecting the instrument's tip to the display screen's center in the pixel coordinate plane. Using the center of the display area as the coordinate system, the slope of this third line can be calculated. Based on the range of this slope, the intersection point of the third line with the display screen's center can be determined. This intersection point represents the relative orientation between the surgical instrument's tip and the display screen's center. For example, if the third line intersects the upper boundary, the surgical instrument's tip is located above the field of view, and so on.

[0081] It should be noted that the second and third lines are only used to distinguish the position of the surgical instrument tip in the pixel coordinate plane of the display screen and the display center of the display screen obtained by different methods. When the surgical instrument tips are in the same position, the second and third lines are the same line.

[0082] In some embodiments, S50, based on relative orientation, marking is performed on the corresponding edge of the display screen and / or the corresponding edge of the display area of ​​the image, including: Determine the first intersection point between the edge of the display screen that intersects with the second line and the second line, mark the first intersection point, and / or mark the position of the corresponding edge of the image display area at the position corresponding to the first intersection point; Determine the second intersection point between the edge of the display screen that intersects with the third line and the third line, mark the second intersection point, and / or mark the position of the corresponding edge of the image display area at the position corresponding to the second intersection point. Mark the corresponding edge of the display screen and / or the corresponding edge of the image display area in whole segments.

[0083] When using lines to determine intersections on the boundary, the markers are placed at the intersections of the corresponding edges on the display screen, and / or at the corresponding positions on the edges of the image display area corresponding to the intersections. This method of marking entire segments allows for more accurate indication of the relative orientation between the end of the surgical instrument and the display center. Alternatively, as described earlier, the corresponding edges of the display screen and / or the corresponding edges of the image display area can be marked entirely based on relative orientation. This marking can include marking the entire boundary segment or a portion of a boundary segment, such as marking the left half of the upper boundary.

[0084] In some embodiments, deformation is indicated based on the distance the distal end of the surgical instrument is out of the field of vision to characterize the change in the distance the distal end of the surgical instrument is out of the field of vision.

[0085] The marker can be a thickened line displayed at the corresponding edge, or an ellipse displayed at a point on the corresponding edge, but is not limited to these. As the distance of the surgical instrument's tip from the field of view changes, the line can become thicker or thinner, and the ellipse can become larger or smaller, thus more intuitively representing the change in distance of the surgical instrument's tip from the field of view. The distance includes the distance relative to the display center in the horizontal direction and / or the depth relative to the image plane in the vertical direction.

[0086] In some embodiments, the identifier includes a first shape portion and a second shape portion, wherein the first shape portion is deformed based on the distance of the surgical instrument tip from the field of vision to characterize the change in the distance of the surgical instrument tip from the field of vision; and the second shape portion is deformed based on the pose of the surgical instrument tip to characterize the change in the pose of the surgical instrument tip.

[0087] Please refer to the following: Figure 12 In (a), identifier 60 includes a first shape portion 21 and a second shape portion 62. The first shape portion 21 is, for example, an ellipse, which can be deformed based on the distance the tip of the surgical instrument is out of the field of vision to characterize the change in the distance the tip of the surgical instrument is out of the field of vision. For example, Figure 12 The surgical instrument 110 shown in (b) is relative to Figure 12 In (a) shown in the image, the surgical instrument 110 is closer to the boundary of the display area 211, therefore, Figure 12 The size of the first shape portion 61 shown in (b) is greater than Figure 12 The dimensions of the first shape portion 61 are shown in (a). The second shape portion 62 is deformed based on the pose of the end of the surgical instrument; for example, the second shape portion 62 is, for example, a triangle, the position of its vertices changing with the pose of the end of the surgical instrument. Figure 12 The surgical instrument 110 shown in (c) is relative to Figure 12If the surgical instrument 110 shown in (a) is deflected further away from the endoscope, then... Figure 12 The apex angles of the second shape portion 62 shown in (c) are opposite Figure 12 The position of the second shape portion 62 shown in (a) changes to characterize the change in the pose of the end of the surgical instrument.

[0088] In addition to indicating relative orientation through the above methods, the markings can also indicate changes in the distance the surgical instrument's tip has moved out of the field of vision, or simultaneously indicate changes in both the distance and position of the instrument's tip. This makes the markings more intuitive and further improves the safety of the surgical procedure. Furthermore, it makes manual manipulation of the surgical instrument to re-enter the endoscope's field of vision more efficient.

[0089] In some embodiments, S50, after identifying the corresponding edges of the display screen and / or the display area of ​​the image based on relative orientation, the method further includes: The position and / or orientation of the endoscope are automatically adjusted based on the position of the surgical instrument tip. Determine whether the tip of the surgical instrument has returned to the endoscope's field of view; When the tip of the surgical instrument returns to the endoscope's field of view, the endoscope is controlled to stop moving, and the marker disappears.

[0090] The procedure involves adjusting the position and / or orientation of the endoscope to bring the tip of the surgical instrument back into the endoscope's field of view. Once the tip of the surgical instrument is back in the field of view, the indicator disappears. The timing of this process can be determined by either stopping the endoscope when the tip enters the display area or by stopping it when the tip is centered within the display area; the specific location is not limited. The process of determining whether the tip of the surgical instrument has returned to the endoscope's field of view during the adjustment of the endoscope's position and / or orientation is the same as in steps S10-S30 and will not be repeated. Automatic instrument retrieval saves surgical time, reduces errors during instrument retrieval, and improves the safety of the surgical procedure.

[0091] In some embodiments, automatically adjusting the position and / or orientation of the endoscope based on the position of the distal end of the surgical instrument includes: Based on the position of the surgical instrument tip in the spatial coordinate system of the endoscope, determine the vector between the surgical instrument tip and the origin of the spatial coordinate system of the endoscope. The position and / or orientation of the endoscope are adjusted based on the deviation between the coordinate axes and vectors parallel to the central axis of the endoscope in the spatial coordinate system of the endoscope.

[0092] Among them, such as Figure 13 As shown, the position of the distal end of the surgical instrument 110 in the spatial coordinate system of the endoscope 120 is indicated by point b, the origin of the spatial coordinate system of the endoscope 120 is indicated by point O, and the vector between the distal end of the surgical instrument 110 and the origin of the spatial coordinate system of the endoscope 120 is... The coordinate axis of the endoscope 120 spatial coordinate system, parallel to the central axis of the endoscope, is, for example, the z-axis. The vector is... The orientation to be aligned with the z-axis of the spatial coordinate system of the endoscope 120 is determined by calculating the deviation angle between the z-axis and the vector. Based on inverse kinematics, the rotation angle of the drive motor of the endoscope 120 is calculated, thereby controlling the orientation of the endoscope 120 from state ① to state ②, thus bringing the tip of the surgical instrument 110 back into the field of view of the endoscope 120. It can be understood that during the movement of the endoscope 120 from state ① to state ②, the endoscope 120 can be stopped when the tip of the surgical instrument 110 enters the display range of the screen, or it can be stopped when the tip of the surgical instrument 110 is located in the center of the display range. No limitation is made here. When the endoscope 120 stops moving, its orientation may not necessarily be aligned with the target direction. Overlap, that is, before the endoscope 120 reaches state ②, but when the end of the surgical instrument 110 is already in a suitable position within the display range, the endoscope 120 can stop moving.

[0093] The method for handling surgical instruments out of the field of view in this embodiment includes: determining the position of the surgical instrument's tip in the endoscope's image plane based on the position of the surgical instrument's tip in the endoscope's spatial coordinate system; and determining whether the surgical instrument has left the endoscope's field of view based on the position of the surgical instrument's tip in the endoscope's image plane and the resolution parameters of the display screen. The technical solution of this application determines whether a surgical instrument has left the field of view based on coordinate system transformation, which is more accurate and efficient. Furthermore, it provides a more intuitive way to indicate the location of the out-of-field movement by marking the corresponding edges of the display screen and / or the corresponding edges of the image display area.

[0094] This application also provides a surgical robot, which includes surgical instruments, an endoscope, a display screen, and a controller. The display screen is used to display images acquired by the endoscope, and the controller executes a computer program to implement the method for handling surgical instruments out of the field of vision as described in the above embodiment.

[0095] This application also provides an electronic device, including a storage medium and a controller, wherein the storage medium stores a computer program, and when the computer program is executed by the controller, it implements the surgical instrument removal process described in the above embodiment.

[0096] This application also provides a storage medium storing a computer program, which, when executed by a processor, implements the surgical instrument removal process described in the above embodiments.

[0097] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for handling surgical instruments that have fallen out of the field of vision, characterized in that, Applied to a surgical robot, the surgical robot including surgical instruments, an endoscope, and a display screen, the display screen being used to display at least images acquired by the endoscope, the method includes the following steps: Obtain the position of the distal end of the surgical instrument in the spatial coordinate system of the endoscope; Based on the position of the tip of the surgical instrument in the spatial coordinate system of the endoscope, the position of the tip of the surgical instrument in the image plane of the endoscope is determined. Based on the position of the tip of the surgical instrument in the image plane of the endoscope and the resolution parameters of the display screen, it is determined whether the surgical instrument has left the field of view of the endoscope.

2. The method according to claim 1, characterized in that, Determining whether the surgical instrument has moved out of the endoscope's field of view based on the position of the surgical instrument's tip in the endoscope's image plane and the display screen's resolution parameters includes: The position of the tip of the surgical instrument in the image plane of the endoscope is mapped to the pixel coordinate plane of the display screen to determine the position of the tip of the surgical instrument in the pixel coordinate plane of the display screen; Based on the position of the end of the surgical instrument in the pixel coordinate plane of the display screen and the resolution parameters of the display screen, it is determined whether the surgical instrument has left the field of view of the endoscope.

3. The method according to claim 1, characterized in that, After determining whether the surgical instrument has moved out of the endoscope's field of view based on the position of the surgical instrument's tip in the endoscope's image plane and the display screen's resolution parameters, the method further includes: When it is determined that the surgical instrument is out of the field of view of the endoscope, the relative orientation between the position of the end of the surgical instrument and the display center of the display screen is determined; Based on the relative orientation, the corresponding edges of the display screen and / or the display area of ​​the image are marked to at least indicate the orientation of the surgical instrument after it has left the field of vision.

4. The method according to claim 3, characterized in that, Determining the relative orientation between the position of the end of the surgical instrument and the display center of the display screen includes: A first line connecting the position of the surgical instrument tip in the image plane of the endoscope and the center point of the image plane is mapped to the pixel coordinate plane of the display screen to obtain a second line. The edge of the display screen that intersects with the second line is determined based on the slope of the second line to characterize the relative orientation between the position of the surgical instrument tip and the display center of the display screen. or, The position of the tip of the surgical instrument in the image plane of the endoscope is mapped to the pixel coordinate plane of the display screen to determine the position of the tip of the surgical instrument in the pixel coordinate plane of the display screen. Based on the slope of the third line connecting the position of the tip of the surgical instrument in the pixel coordinate plane of the display screen and the display center of the display screen, the edge of the display screen that intersects with the third line is determined to characterize the relative orientation between the position of the tip of the surgical instrument and the display center of the display screen.

5. The method according to claim 4, characterized in that, The step of identifying the corresponding edge of the display screen and / or the corresponding edge of the display area of ​​the image based on the relative orientation includes: Determine the first intersection point between the edge of the display screen that intersects with the second connecting line and the second connecting line, mark the first intersection point, and / or mark the position of the corresponding edge of the display area of ​​the image at the position corresponding to the first intersection point; Determine the second intersection point between the edge of the display screen that intersects with the third connecting line and the third connecting line, mark the second intersection point, and / or mark the position of the corresponding edge of the display area of ​​the image at the position corresponding to the second intersection point; The corresponding edges of the display screen and / or the corresponding edges of the display area of ​​the image are marked as a whole.

6. The method according to any one of claims 3 to 5, characterized in that, The marking is deformed based on the distance the tip of the surgical instrument is removed from the field of vision, in order to characterize the change in the distance the tip of the surgical instrument is removed from the field of vision; or, The marking includes a first shape portion and a second shape portion, wherein the first shape portion is deformed based on the distance of the end of the surgical instrument from the field of vision, so as to characterize the change in the distance of the end of the surgical instrument from the field of vision; The second shape portion is deformed based on the pose of the end of the surgical instrument to characterize the change in the pose of the end of the surgical instrument.

7. The method according to any one of claims 3 to 6, characterized in that, After identifying the corresponding edges of the display screen and / or the corresponding edges of the display area of ​​the image based on the relative orientation, the method further includes: The position and / or orientation of the endoscope are automatically adjusted based on the position of the distal end of the surgical instrument. Determine whether the tip of the surgical instrument has returned to the field of view of the endoscope; When the tip of the surgical instrument returns to the field of view of the endoscope, the endoscope is controlled to stop moving, and the marker disappears.

8. The method according to claim 7, characterized in that, The automatic adjustment of the position and / or orientation of the endoscope based on the position of the distal end of the surgical instrument includes: Based on the position of the end of the surgical instrument in the spatial coordinate system of the endoscope, determine the vector between the end of the surgical instrument and the origin of the spatial coordinate system of the endoscope; The position and / or orientation of the endoscope are adjusted based on the deviation between the coordinate axis parallel to the central axis of the endoscope in the spatial coordinate system of the endoscope and the vector.

9. A surgical robot, characterized in that, The surgical robot includes surgical instruments, an endoscope, a display screen, and a controller. The display screen is used to display images acquired by the endoscope. When the controller executes a computer program, it implements the method for handling surgical instruments out of the field of vision as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, The device includes a storage medium and a controller, wherein the storage medium stores a computer program, which, when executed by the controller, implements the method for handling surgical instruments out of the field of vision as described in any one of claims 1 to 8.

11. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method for handling surgical instruments out of the field of vision as described in any one of claims 1 to 8.

Images