Endoscope control method and surgical robotic system

Abstract

The present disclosure relates to the field of medical devices, and discloses an endoscope control method. The method comprises: controlling endoscope movement, the endoscope comprising a main body, a first imaging unit, a second imaging unit, and an end instrument extending from a distal end of the main body; obtaining a first image from the first imaging unit; obtaining a second image from the second imaging unit, wherein the fields of view of the first image and the second image are different and comprise an image of the end instrument; generating a composite scene image to remove an actual image of the end instrument based on the first image and the second image; and generating a virtual image of the end instrument in the composite scene image.

Description

Technical Field

[0001] This disclosure relates to the field of medical devices, and more particularly to an endoscope control method and a surgical robot system. Background Technology

[0002] Modern laparoscopic minimally invasive surgery requires the use of endoscopes inserted into cavities to capture images. Sometimes, endoscopes also integrate distal instruments to facilitate both image acquisition and surgical procedures. However, distal instruments can limit the endoscope's field of view, thus affecting the accuracy of the surgeon's diagnostic work. Summary of the Invention

[0003] In some embodiments, this disclosure provides an endoscope control method. The method may include: controlling the movement of an endoscope, the endoscope including a body, a first imaging unit, a second imaging unit, and a distal end-effector extending from the body; acquiring a first image from the first imaging unit; acquiring a second image from the second imaging unit, wherein the first and second images have different fields of view and include an image of the distal end-effector; generating a synthetic scene image based on the first and second images to remove the actual image of the distal end-effector; and generating a virtual image of the distal end-effector in the synthetic scene image.

[0004] In some embodiments, this disclosure provides a robotic system comprising: a motion arm; an endoscope disposed at the end of the motion arm, the endoscope comprising: an actuator arm including at least one controllably bendable segment; a body disposed at the distal end of the actuator arm; a first imaging unit for capturing a first image; a second imaging unit for capturing a second image; and an end effector configured to extend from the distal end of the body; a control device configured to perform a method according to any one of the embodiments of this disclosure; and a display device for displaying images based on instructions output by the control device.

[0005] In some embodiments, this disclosure provides a computer device including: a memory for storing at least one instruction; and a processor coupled to the memory and configured to execute at least one instruction to perform a method according to any of some embodiments of this disclosure.

[0006] In some embodiments, this disclosure provides a computer-readable storage medium for storing at least one instruction, which, when executed by a computer, causes the computer to perform any of the methods described in some embodiments of this disclosure. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below. Obviously, the accompanying drawings described below only show some embodiments of this disclosure. For those skilled in the art, other embodiments can be obtained based on the content of the embodiments of this disclosure and these drawings without creative effort.

[0008] Figure 1 This diagram illustrates a structural block diagram of a robot system according to some embodiments of the present disclosure;

[0009] Figure 2 This diagram illustrates the structure of an endoscope according to some embodiments of the present disclosure;

[0010] Figure 3 A schematic diagram of an endoscope located within a body cavity according to some embodiments of the present disclosure is shown.

[0011] Figure 4 A schematic diagram showing the relative positional relationship between a first imaging unit, a second imaging unit, and an end effector according to some embodiments of the present disclosure;

[0012] Figure 5 This diagram illustrates the structure of an end-effector according to some embodiments of the present disclosure;

[0013] Figure 6 (a) Figure 6 (b) A schematic diagram of the structure of an end-effector according to other embodiments of the present disclosure is shown, wherein Figure 6 (a) is a front view of the end effector. Figure 6 (b) is a rear view of the end effector;

[0014] Figure 7 This diagram illustrates the structure of an endoscope body according to some embodiments of the present disclosure;

[0015] Figure 8 This diagram illustrates a structural schematic of an actuator arm assembly according to some embodiments of the present disclosure;

[0016] Figure 9 A schematic block diagram of a control device according to some embodiments of the present disclosure is shown;

[0017] Figure 10 A flowchart illustrating an endoscope control method according to some embodiments of the present disclosure is shown;

[0018] Figure 11 A flowchart illustrating a method for displaying scene images based on display mode instructions according to some embodiments of the present disclosure;

[0019] Figure 12A schematic diagram illustrating multi-scene display on a display device according to some embodiments of the present disclosure is shown;

[0020] Figure 13 A flowchart illustrating a method for generating a synthetic scene image based on a first image and a second image according to some embodiments of the present disclosure;

[0021] Figure 14 A flowchart illustrating a method for generating a three-dimensional synthetic scene image based on a first image and a second image according to some embodiments of the present disclosure;

[0022] Figure 15 A flowchart illustrating a method for generating depth maps based on the pose of an imaging unit according to some embodiments of the present disclosure;

[0023] Figure 16 A flowchart illustrating a method for generating a three-dimensional real-scene image based on a first image and / or a second image according to some embodiments of the present disclosure;

[0024] Figure 17 A flowchart illustrating a method for generating a virtual image of an end-effector in a synthetic scene image according to some embodiments of the present disclosure;

[0025] Figure 18 A schematic block diagram of a computer device according to some embodiments of the present disclosure is shown;

[0026] Figure 19 A schematic diagram of a robot system according to some embodiments of the present disclosure is shown. Detailed Implementation

[0027] To make the technical problems solved by this disclosure, the technical solutions adopted, and the technical effects achieved clearer, the technical solutions of the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely exemplary embodiments of this disclosure, and not all embodiments.

[0028] In the description of this disclosure, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this disclosure, it should be noted that unless otherwise expressly specified and limited, the terms "installed," "connected," "coupled," and "coupled" should be interpreted broadly. For example, they can refer to fixed connections or detachable connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.

[0029] In this disclosure, the end closer to the operator (e.g., a doctor) is defined as the proximal end, proximal or rear end, or posterior end, and the end closer to the surgical patient is defined as the distal end, distal or anterior end, or anterior end. Those skilled in the art will understand that embodiments of this disclosure can be used in medical devices or surgical robots, as well as in other non-medical devices.

[0030] In this disclosure, a reference coordinate system can be understood as a coordinate system capable of describing the pose of an object. Depending on the actual positioning requirements, the reference coordinate system can be selected with the origin of a virtual reference object or the origin of a physical reference object as its origin. In some embodiments, the reference coordinate system can be a world coordinate system, or a coordinate system in the space where a point on the main manipulator, motion arm, end effector, endoscope body, end effector, first imaging unit, second imaging unit, or cavity is located, or the operator's own perception coordinate system, etc.

[0031] In this disclosure, an object can be understood as an object or target that needs to be positioned, such as an actuator arm or the end effector of an actuator arm, or a point on a cavity. The pose of the actuator arm or a part thereof (e.g., the end effector) can refer to the pose of the coordinate system defined by the actuator arm, a part of the actuator arm, or a rigidly extended portion of the actuator arm (e.g., the body or end effector of an endoscope) relative to a reference coordinate system.

[0032] Figure 1 A structural block diagram of a robot system 100 according to some embodiments of the present disclosure is shown. In some embodiments, such as Figure 1As shown, the robot system 100 may include a master control carriage 110, a slave carriage 130, and a control device 120. The control device 120 can communicate with the master control carriage 110 and the slave carriage 130, for example, via cable or wireless connection, to achieve communication between them. The master control carriage 110 functions as the operating and interactive end of the robot system 100, and may include a master manipulator for remote operation by the operator, and an image display device. The slave carriage 130 functions as the working end of the robot system 100, including a motion arm for performing tasks, and an endoscope disposed at the end of the motion arm. The control device 120 enables master-slave mapping between the master manipulator in the master control carriage 110 and the endoscope in the slave carriage 130, allowing the master manipulator to control the movement of the endoscope. In some embodiments, the distal portion of the endoscope is configured to enter the operating area via a cavity through a cannula, sheath, or similar means, to capture images of the target area in the scene and generate two-dimensional or three-dimensional scene images, which are then displayed on a display device. The endoscope may include a distal end-effector, which may be a surgical tool such as a clamp, hemostatic device, or drug delivery device. In some embodiments, the robotic system 100 may be configured to process the captured scene images to generate various images, including actual scene images of the end-effector and / or composite scene images including virtual images of the end-effector, and selectively display them on a display device based on operational instructions. By controlling the robotic system 100, the end-effector can be operated on the surgical site (e.g., pathological tissue, etc.) with or without contact, while having a view of the operating area and surgical site. The cannula or sheath may be fixed to an opening (e.g., an incision or natural opening) on ​​a human or animal body; the cavity may be the trachea, esophagus, vagina, intestine, etc.; the operating area may be the area where the surgical procedure is performed; and the scene may be a cavity or the operating area. Those skilled in the art will understand that the master control carriage 110 and the driven carriage 130 may adopt other structures or forms, such as bases or supports. The master control carriage 110 and the driven carriage 130 may also be integrated into the same device.

[0033] Figure 2 A schematic diagram of the structure of an endoscope 200 according to some embodiments of the present disclosure is shown. Figure 3 This diagram illustrates the structure of an endoscope 200 according to some embodiments of the present disclosure located within a cavity A in a body (e.g., in a human or animal body). Figure 2 and Figure 3 As shown, the endoscope 200 may include an actuator arm 210 and an endoscope body 221. In some embodiments, the actuator arm 210 may be a controllably bendable continuum, and may include a segment that is bendable in at least one degree of freedom at its distal end (e.g., Figure 8As shown in the configuration 800, an endoscope body 221 is disposed at the distal end of the actuator arm 210. The actuator arm 210 may be disposed at the distal end of the motion arm of the driven trolley 130, and there is a master-slave motion mapping relationship between the pose of the master manipulator and the pose of the end of the actuator arm. In some embodiments, the actuator arm may change the direction of the endoscope body 221 based on the operation command issued by the operator to avoid important organs in the body or adapt to the curved and complex cavity A, so that the endoscope body 221 can advance to the operation area or retract from the cavity A. In some embodiments, the actuator arm may adjust the pose of the endoscope body 221 based on the operation command, so that the imaging unit on the endoscope body 221 (e.g., Figure 2 and Figure 3 The first imaging unit 230 and the second imaging unit 240 shown are... Figure 4 The first imaging unit 430 and the second imaging unit 440 shown photograph the operating area, and the end-effector (e.g., on the endoscope body 221) is positioned so that the end instrument (e.g., Figure 2 and Figure 3 The end effector 260 shown Figure 4 The end effector 460 shown Figure 5 The end device 500 shown Figure 6 (a) and Figure 6 (b) The end instrument 600 shown is aligned with the surgical site in the operating area.

[0034] The endoscope 200 may further include a first imaging unit 230, a second imaging unit 240, and a distal end instrument 260. In some embodiments, the endoscope 200 may further include at least one illumination unit 250. Figure 2 As shown, the endoscope body 221 is roughly cylindrical, and its cross-sectional shape can be circular or elliptical to meet different functional needs.

[0035] The first imaging unit 230 can be used to capture a first image, and the second imaging unit 240 can be used to capture a second image. In some embodiments, the first imaging unit 230 and the second imaging unit 240 can be, for example, a CCD camera, each including a set of image sensors and image lenses. The image lenses can be disposed at the distal end of the image sensors and aligned with at least a corresponding image sensor, thereby facilitating the image sensors to capture target areas in the scene through the image lenses. In some embodiments, the image lenses can include multiple convex lenses and concave lenses, which are distributed to form an optical imaging system. For example, the distal surface of the image lens can be a curved convex lens, such as a spherical lens, an ellipsoidal lens, a conical lens, a frustum lens, etc. The image lens can include at least one convex surface to increase the field of view that can be captured.

[0036] The illumination unit 250 provides illumination to facilitate imaging by the first imaging unit 230 and the second imaging unit 240. For example... Figure 2 As shown, in some embodiments, three illumination units 250 are arranged along the circumferential edge of the endoscope 200, located between two imaging units or between an imaging unit and the end instrument 260, but are not limited thereto. The number and arrangement of the illumination units 250 can be changed according to actual needs. For example, there may be two illumination units 250, located on the left and right sides of the endoscope 200. Alternatively, to further increase the illumination intensity, the number of illumination units 250 may be greater than three. In some embodiments, the cross-section of the illumination unit 250 may be crescent-shaped, thereby making full use of the space on the endoscope 200, which helps to achieve miniaturization of the endoscope and increase the illumination field of view, but is not limited thereto, the illumination unit 250 may also be other shapes. In some embodiments, the illumination unit 250 may include a light source and one or more optical fibers coupled to the light source, and an illumination channel for arranging the optical fibers may be formed inside the endoscope body 221. In some embodiments, the light source of the illumination unit 250 may be, for example, an LED light source.

[0037] In some embodiments, the distal instrument 260 may be configured to extend distally from the endoscope body 221 to perform surgical procedures, as described later. In some embodiments, the endoscope 200 may further include a ranging unit (not shown) for measuring the distance between the endoscope 200 and the surgical site. The ranging unit may be, for example, a ranging sensor such as a laser rangefinder. By providing the ranging unit, the distance between the endoscope 200 (e.g., the distal end face of the endoscope body 221) and the surgical site can be determined, thereby enabling the further determination of the distance between the distal instrument 260 and the surgical site.

[0038] Figure 4 A schematic diagram illustrating the relative positional relationship between a first imaging unit 430, a second imaging unit 440, and an end effector 460 according to some embodiments of the present disclosure is shown. Figure 4 As shown, in some embodiments, the first imaging unit 430 may be configured to be located on one side of the endoscope body 421 relative to the end instrument 460 and have a first field of view corresponding to the orientation of its own optical axis L1.

[0039] It should be understood that the first field of view of the first imaging unit 430 is formed in a roughly conical shape centered on the optical axis L1. Figure 4 The cross-section of the first field of view is schematically shown, and the plane containing this cross-section is perpendicular to the optical axis L1. Furthermore, it should be understood that the planes containing the first imaging unit 430 and the second imaging unit 440 (described below), the distal surface of the end effector 460, the first field of view, and the cross-section of the second field of view (described below) are not located on the same plane. This is for ease of description. Figure 4The plane containing the first imaging unit 430 and the second imaging unit 440 described below, the distal surface of the end effector 460, the first field of view, and the cross section of the second field of view described below are shown on the same plane.

[0040] The first field of view includes field of view 431 and field of view 432, where field of view 431 is the portion of the first field of view not obstructed by the end effector 460, and field of view 432 is the portion of the first field of view obstructed by the end effector 460. For example... Figure 4 As shown, the fields of view 431 and 432 cover the entire cross-section of the operating region B (or cavity A). The first imaging unit 430 can capture a first image of the operating region B (or cavity A) under the first field of view, the first image including an image of the end effector 460.

[0041] The second imaging unit 440 can be configured to be located on the opposite side of the endoscope body 421 relative to the end-effector 460, and has a second field of view corresponding to its own optical axis L2. Similar to the first imaging unit 430, the second field of view of the second imaging unit 440 is formed in a generally conical shape centered on the optical axis L2. Figure 4 A cross-section of the second field of view is schematically shown, the plane of which is perpendicular to the optical axis L2. The second field of view includes field of view 441 and field of view 442, where field of view 441 is the portion of the second field of view not obstructed by the end effector 460, and field of view 442 is the portion of the second field of view obstructed by the end effector 460. Figure 4 As shown, the fields of view 441 and 442 cover the entire cross-section of the operating region B (or cavity A). The second imaging unit 440 can capture a second image of the operating region B (or cavity A) under the second field of view. This second image has a different field of view from the first image and includes an image of the end effector 460.

[0042] In some embodiments, the optical axis L1 of the first imaging unit 430 and the optical axis L2 of the second imaging unit 440 may be parallel to the axis L0 of the endoscope body 421, respectively, and the axis L3 of the endoscope 460 may be parallel to the axis L0 of the endoscope body 421 and offset from the line connecting the first imaging unit 430 and the second imaging unit 440. For example, the first imaging unit 430, the second imaging unit 440, and the endoscope 460 may be configured such that the optical axis L1 of the first imaging unit 430, the optical axis L2 of the second imaging unit 440, and the axis L3 of the endoscope 460 are perpendicular to the distal surface of the endoscope body 421, the first imaging unit 430 and the second imaging unit 440 are symmetrically distributed with respect to the endoscope 460, and the axis L3 of the endoscope 460 is located below the line connecting the first imaging unit 430 and the second imaging unit 440. By configuring the first imaging unit 430 and the second imaging unit 440 with their optical axes parallel and symmetrically distributed on both sides of the end effector 460, the first image captured by the first imaging unit 430 and the second image captured by the second imaging unit 440 can be made symmetrical to each other, which helps to process the first image and the second image and can improve the image generation quality and image processing speed of the robot system.

[0043] Those skilled in the art will understand that although the first and second images are described as examples in this disclosure for ease of explanation, the embodiments of this disclosure can be applied to processing sequences of first and second images to form continuous video frame processing and display. Therefore, the capturing, processing, and display of sequences of first and second images fall within the scope of this disclosure and the protection scope of the claims of this disclosure.

[0044] In this disclosure, the end device (e.g., Figure 2 and Figure 3 The end effector 260 shown Figure 4 The end effector 460 shown Figure 5 The end device 500 shown Figure 6 (a) and Figure 6 (b) The end-effector 600 shown may include surgical tools such as hemostatic devices (e.g., electrocoagulation hemostatic devices), clamp devices, and drug delivery devices to meet different surgical needs. The following description uses an electrocoagulation hemostatic device as the end-effector.

[0045] In some embodiments, the distal end device can be configured to be fixedly connected proximally to the distal end of the endoscope body, thereby allowing the position of the distal end device to be changed by adjusting the position of the endoscope body, thus aligning the distal end device with the surgical site in the operating area. In some embodiments, the distal end device can be a bipolar electrocoagulation hemostasis device. For example, the distal end device may include at least one first electrode, at least one second electrode, and an insulating body, wherein the at least one first electrode and at least one second electrode are alternately disposed on the circumferentially outer side of the insulating body, and at least a portion of the first electrode and at least a portion of the second electrode are exposed. When a high-frequency current is applied, the at least partially exposed first electrode and the at least partially exposed second electrode form a circuit for electrocoagulation hemostasis.

[0046] Figure 5 A schematic diagram of the structure of an end effector 500 according to some embodiments of the present disclosure is shown. For example... Figure 5 As shown, in some embodiments, the end effector 500 is formed as a generally round-headed cylindrical shape. It should be understood that the end effector 500 includes, but is not limited to, the shape described above, and may also be hemispherical, frustum-shaped, etc. The end effector 500 includes at least one first electrode 511, at least one second electrode 512, and an insulating body 513. The insulating body 513 may be made of insulating materials such as plastic, ceramic, or mica. In some embodiments, at least one first electrode 511 and at least one second electrode 512 may be alternately arranged on the circumferential outer side of the insulating body 513. For example, the first electrode 511 and the second electrode 512 may be alternately arranged on the circumferential outer side of the insulating body 513 at predetermined intervals. In some embodiments, the first electrode 511 and the second electrode 512 may extend from the distal end to the proximal end along the generatrix of the cylinder in the circumferential direction of the insulating body 513, but are not limited thereto; the first electrode 511 and the second electrode 512 may also be formed as generally helical lines, alternately coiled around the circumferential outer side of the insulating body 513.

[0047] In some embodiments, the distal end device 500 may be mounted on the body of the endoscope, or it may be integrally formed with the body of the endoscope. For example, a mounting structure (e.g.,) may be formed at the rear end of the insulating body 513. Figure 7 (As shown), this is used to mount the end-effector 500 onto the body of the endoscope. For example, the insulating body 513 may have a recess as a mounting structure formed at its proximal end, and an internal thread may be formed on the circumferential inner wall of the recess, thereby threadedly connecting with the body of the endoscope. It should be understood that the mounting structure of the insulating body 513 is not limited to the above-described structure. For example, the mounting structure of the insulating body 513 may also be formed as a boss with external threads, or a structure that can engage with the body of the endoscope.

[0048] Figure 6 (a) Figure 6(b) A schematic diagram of the structure of an end-effector 600 according to other embodiments of the present disclosure is shown, wherein Figure 6 (a) is a front view of the end-effector 600. Figure 6 (b) is a rear view of the end-effector 600. For example... Figure 6 (a) and Figure 6 As shown in (b), the end effector 600 may include a first electrode 611 and a second electrode 612. In some embodiments, the first electrode 611 may be formed as a generally hollow, round-headed cylinder, and the second electrode 612 may be formed as a generally annular cross-section, and radially spaced around at least a portion of the first electrode 611. In some embodiments, the end effector 600 may further include an insulating body 613 disposed between the first electrode 611 and the second electrode 612. Similar to the insulating body 513, the insulating body 613 may be made of an insulating material such as plastic, ceramic, or mica.

[0049] In some embodiments, a mounting structure for mounting an endoscope is formed on the endoscope body. Figure 7 A schematic diagram of the structure of an endoscope body 721 according to some embodiments of the present disclosure is shown. Figure 7 As shown, the endoscope body 721 may include an inner annular component 7211 and an outer annular component 7212 for mounting end instruments (e.g., Figure 2 and Figure 3 The end effector 260 shown Figure 4 The end effector 460 shown Figure 5 The end device 500 shown Figure 6 (a) and Figure 6 (b) The end-effector 600 shown. For example, the first electrode 611 of the end-effector 600 can be mounted on the inner annular member 7211 of the endoscope body 721, and the second electrode 612 can be mounted on the outer annular member 7212. In some embodiments, threaded or engaging structures can be provided on the inner annular member 7211 or the outer annular member 7212, and the first electrode 611 or the second electrode 612 of the end-effector 600 can be mounted on the inner annular member 7211 and the outer annular member 7212 by means of threaded connection or engaging connection. It should be understood that the mounting structure on the body is not limited to the above-described structure, and any structure that enables the mounting of the end-effector is not outside the scope of this disclosure. For example, the inner annular member 7211 and the outer annular member 7212 can also be annular grooves recessed from the end face of the endoscope body 721.

[0050] Figure 8A schematic diagram of a segment 800 of an actuator arm 210 according to some embodiments of the present disclosure is shown. In some embodiments, the actuator arm 210 may include at least one segment 800. The segment 800 includes one or more structural bones 810, a fixing plate 830, and at least one spacer plate 820. The structural bone 810 passes through at least one spacer plate 820 and its end is fixedly connected to the fixing plate 830. The spacer plate 820 and the fixing plate 830 are spaced apart axially from the structural bone 810. The spacer plate 820 is provided with a through hole 8201 through which the structural bone 810 passes, and the fixing plate 830 is provided with a fixing hole 8301 for fixing the structural bone 810. The end of the structural bone 810 may be connected to the fixing plate 830, and the proximal end of the structural bone 810 may be connected to a drive mechanism (not shown). When the structural bone 810 is driven by the drive mechanism, the structural bone 810 may move along the through hole 8201. In some embodiments, at least one spacer plate 820 may form a continuous structure, such as a bellows. In some embodiments, the exterior of the component 800 may be covered with a covering layer or a cover.

[0051] In some embodiments, the number of structural bones 810 is one or more, and they are uniformly or non-uniformly distributed on the cross-section of the spacer disc, for example, located at the center or distributed circumferentially. In some embodiments, the number of spacer discs 820 is one or more. One or more through holes 8201 may be formed on the cross-section of the spacer disc 820, and the shape of the through holes 8201 matches or substantially matches the shape of the cross-section of the structural bones, so that the structural bones 810 can pass through the through holes 8201. In some embodiments, the number of through holes 8201 formed on the cross-section of the spacer disc 820 is the same as the number of structural bones 810, so that one structural bone 810 can pass through each through hole 8201. In some embodiments, the cross-sectional shape of the spacer disc 820 is circular, the through holes 8201 on the spacer disc are circular holes, and the cross-section of the structural bones 810 is circular. In other embodiments, the cross-sectional shape of the spacer disc 820 is rectangular, the through holes on the spacer disc are polygonal holes, and the cross-section of the structural bones is polygonal, etc.

[0052] In some embodiments, the actuator arm 210 may further include a plurality of segments 800 connected in series. For example, the arm body 210 may include two segments connected in series, with the spacer 820 of the distal segment serving as a fixing plate for the proximal segment. The inclusion of two or more segments 800 can increase the bending flexibility of the arm body 210.

[0053] In some embodiments, the body of the endoscope (e.g., Figure 2 and Figure 3 Endoscope body 221 shown Figure 4 The endoscope body 421 shown is Figure 7The endoscope body 721 shown can be located at the distal end of the actuator arm 210. For example, the body can be fixedly mounted on the mounting plate 830 of the component 800, and the body can move with the distal end of the actuator arm.

[0054] In this disclosure, the robot system may also include a control device for controlling the robot system based on an endoscopic control method. Figure 9 A structural block diagram of a control device 900 according to some embodiments of the present disclosure is shown. Figure 9 As shown, the control device 900 may include a motion control module 910, a pose determination module 920, an image processing module 930, a virtual image generation module 940, and a scene output module 950. The motion control module 910 can receive input motion control commands to drive the actuator arm 210. In some embodiments, the motion control module 910 can be connected to... Figure 1 The control device 120 shown is communicatively connected and receives motion control commands input by the operator through the main operator, and sends drive signals to the drive device to drive the actuator 210. In some embodiments, the pose determination module 920 can determine the pose of the end of the actuator 210 based on the target pose of the end of the actuator 210 corresponding to the pose of the main operator, and then determine the pose of the endoscope body 221 and the first imaging unit 230, the second imaging unit 240 and the end instrument 260 disposed on the endoscope body 221. In some embodiments, the pose determination module 920 can also receive signals from a pose sensor, such as an electromagnetic pose sensor or a fiber optic sensor, to determine the pose of the actuator 210 and the endoscope body 221. The image processing module 930 can be configured to receive signals from the first imaging unit (e.g., Figure 2 The first imaging unit 230 shown or Figure 4 The first imaging unit 430 shown receives the first image and receives it from the second imaging unit (e.g., Figure 2 The second imaging unit 240 shown Figure 4 The second imaging unit 440 shown receives the second image and generates a synthetic scene image and an actual scene image based on the first and second images. The virtual image generation module 940 can be used to generate an end effector (e.g., [missing information]) in the synthetic scene image. Figure 2 and Figure 3 The end effector 260 shown Figure 4 The end effector 460 shown Figure 5 The end device 500 shown Figure 6 (a) and Figure 6(b) The virtual image of the end effector 600 shown. The scene output module 950 can switch between outputting a composite scene image with virtual images or an actual scene image to the display device according to instructions, or simultaneously output both a composite scene image with virtual images and an actual scene image to the display device. It should be understood that the control device of this disclosure includes, but is not limited to, the above-described structure, and any control device capable of realizing the robot system is not outside the scope of this disclosure.

[0055] In some embodiments, the robot system may also include a display device (e.g., Figure 12 The display device 1200 shown is used to display images based on instructions output by the control device 900.

[0056] Some embodiments of this disclosure provide an endoscope control method. Figure 10 A flowchart illustrating an endoscope control method 1000 according to some embodiments of the present disclosure is provided. In some embodiments, some or all steps of method 1000 may be performed by a robotic system (e.g., Figure 1 The robot system 100 shown Figure 19 The control device (e.g., for the robot system 1900 shown) Figure 9 The control device 900 shown or Figure 19 The method 1000 is executed by the control device 1970 shown. The control device may include a computing device. The method 1000 may be implemented by software, firmware, and / or hardware. In some embodiments, the method 1000 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., a dedicated processor). Figure 19 The control device (1970) shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0057] refer to Figure 10 In step 1001, the endoscope is controlled to move. The endoscope may include a main body, a first imaging unit, a second imaging unit, and a distal end instrument extending from the end of the main body. In some embodiments, in response to a motion control command input by the operator via a master manipulator, a drive signal may be sent to a drive device to drive a motion arm or an actuator arm, thereby controlling the movement of the endoscope. In some embodiments, based on the master-slave motion mapping relationship between the pose of the master manipulator and the pose of the distal end of the actuator arm, a target pose of the distal end of the actuator arm may be determined according to the pose of the master manipulator, and a drive signal for controlling the movement of the endoscope may be determined based on the current pose and the target pose of the distal end of the actuator arm. In some embodiments, step 1001 may be performed by a control device 900 (e.g., a motion control module 910).

[0058] Continue to refer to Figure 10In step 1003, a first image is obtained from the first imaging unit. In some embodiments, the first imaging unit is configured to be located on one side of the endoscope body relative to the end-effector. As the endoscope moves within the scene, the first imaging unit continuously captures the first image in a first field of view, and the control device 900 (e.g., image processing module 930) can receive the first image from the first imaging unit. In some embodiments, the end-effector is located within the first field of view of the first imaging unit, and the first image includes an image of the end-effector captured from one side of the body.

[0059] Continue to refer to Figure 10 In step 1005, a second image is obtained from the second imaging unit, wherein the fields of view of the first image and the second image are different and include an image of the endoscope. In some embodiments, the second imaging unit is configured to be located on the opposite side of the endoscope body relative to the endoscope. As the endoscope moves within the scene, the second imaging unit continuously captures the second image in a second field of view different from the first field of view of the first imaging unit, and the control device 900 (e.g., image processing module 930) can receive the second image from the second imaging unit. In some embodiments, the endoscope is located within the second field of view of the second imaging unit, and the second image includes an image of the endoscope captured from the opposite side of the body.

[0060] Continue to refer to Figure 10 In step 1007, a synthetic scene image is generated based on the first image and the second image to remove the actual image of the end effector. In this disclosure, due to the occlusion by the end effector, neither the first imaging unit nor the second imaging unit can capture the entire scene. In some embodiments, the control device 900 (e.g., image processing module 930) can use computer vision processing to fill in the portion of the other image occluded by the end effector using either the first image or the second image, thereby generating a two-dimensional or three-dimensional synthetic scene image with the end effector removed. For example, an exemplary method for generating a synthetic scene image based on the first image and the second image may include, as shown below... Figure 14 The method 1400 is shown. In some embodiments, the computer vision processing may include a feature point detection algorithm that can extract feature points from a first image and a second image for matching, thereby achieving two-dimensional stitching of the first image and the second image. In some embodiments, the computer vision processing may include an image sequence optical flow reconstruction algorithm that can determine the depth of a pixel in scene space based on the optical flow of the pixel in the image, thereby reconstructing the scene in three dimensions. By generating a synthetic scene image, a more complete scene image, at least partially unobstructed by the end effector, can be displayed on a display device, helping the operator to observe the cavity and operating area without obstruction.

[0061] Continue to refer to Figure 10In step 1009, a virtual image of the end effector is generated in the synthesized scene image. For example, an exemplary method for generating a virtual image of the end effector in the synthesized scene image may include, for instance... Figure 17 The method 1700 is shown. In some embodiments, the control device 900 (e.g., virtual image generation module 940) can generate a virtual image of the end-effector at the position corresponding to the end-effector in the synthesized scene image using a real-time rendering method. By generating a virtual image of the end-effector in the synthesized scene image, the actual position and size of the end-effector can be indicated to the operator without obstructing the operator's view of the scene, thereby avoiding collisions with the walls of the cavity or operating area due to the inability to see the end-effector during operation.

[0062] In some embodiments, method 1000 may further include switching scene modes based on display mode instructions. Figure 11 A flowchart illustrating a method 1100 for displaying scene images based on display mode instructions according to some embodiments of the present disclosure is shown. In some embodiments, some or all of the steps in method 1100 may be performed by a robot system (e.g., Figure 1 The robot system 100 shown Figure 19 The control device (e.g., for the robot system 1900 shown) Figure 9 The control device 900 shown or Figure 19 The method 1100 is executed by the control device 1970 shown. The control device may include a computing device. The method 1100 may be implemented by software, firmware, and / or hardware. In some embodiments, the method 1100 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., a dedicated processor). Figure 19 The control device (1970) shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0063] refer to Figure 11 In step 1101, a real-scene image is generated based on the first image and / or the second image to display the actual image of the end effector. In some embodiments, the first image captured by the first imaging unit or the second image captured by the second image unit can be used as a two-dimensional real-scene image, which includes the actual image of the end effector. In some embodiments, the control device 900 (e.g., image processing module 930) can generate a three-dimensional real-scene image based on the first image or the second image using a computer vision algorithm. For example, an exemplary method for generating a three-dimensional real-scene image based on the first image or the second image may include, for example, Figure 16Method 1600. In some embodiments, the computer vision algorithm may include an image sequence optical flow reconstruction algorithm, which can determine the depth of a pixel in the scene space based on the optical flow of a pixel in a first image or a second image, thereby reconstructing the actual scene in three dimensions. In some embodiments, two actual scene images with different fields of view may be generated simultaneously based on the first image and the second image for side-by-side display on a display device.

[0064] Continue to refer to Figure 11 In step 1103, in response to a display mode instruction, a composite scene image and / or an actual scene image with a virtual image of the end-effector is displayed. The display mode instruction may include, for example, at least one of a motion control instruction, an end-effector operation instruction, and a display mode selection instruction.

[0065] Motion mode commands are used to control the movement of the endoscope within the cavity. In some embodiments, the motion control commands for controlling the movement of the endoscope may include a first endoscope motion command, which may include a control command for endoscope feed or steering. In some embodiments, the first endoscope motion command may be determined based on the master-slave motion mapping relationship between the pose of the master manipulator and the pose of the end-effector, and the first endoscope motion command may be, for example, a target pose of the end-effector of the endoscope or a drive signal associated with the target pose. In some embodiments, method 1100 may further include, in response to the first endoscope motion command, displaying a synthetic scene image of a virtual image of the end-effector, and, based on the first endoscope motion command, controlling the endoscope to move toward the operating area. For example, a first endoscope movement command can be determined based on the operator's operation of the main manipulator. The scene output module 950 of the control device 900 can respond to the first endoscope movement command by outputting a composite scene image with a virtual image of the end-effector to the display device for display. The motion control module 910 of the control device 900 can control the endoscope's feed or rotation based on the first endoscope movement command. By displaying a composite scene image with the end-effector removed, the operator can avoid operating the endoscope's feed or rotation with an incomplete field of vision, thus avoiding unnecessary surgical risks.

[0066] End-device operation commands can be used to control the operation of an end-device. In some embodiments, method 1100 may further include displaying an actual scene image in response to an end-device operation command, or displaying a composite scene image with a virtual image of the end-device in response to termination of the end-device operation command. In some embodiments, the end-device operation command may include an activation command for the end-device, which indicates the initiation of end-device operation. When the end-device is an electrocoagulation hemostasis device (e.g., Figure 5 The end device 500 shown Figure 6 (a) and Figure 6 In the case of the end-effector 600 shown in (b), the activation command may be, for example, turning on the power. In some embodiments, the scene output module 950 of the control device 900 may, in response to the end-effector operation command, display an actual scene image or switch the display screen from a composite scene image of a virtual image with the end-effector to an actual scene image, thereby allowing the operator to perform surgical procedures while observing the end-effector, which helps to improve operational accuracy. The termination of the end-effector operation command means that the operator has completed or temporarily stopped the end-effector operation. When the end-effector is an electrocoagulation hemostasis device (e.g., Figure 5 The end device 500 shown Figure 6 (a) and Figure 6 In the case of the end-effector 600 shown in (b), the termination of the end-effector operation command can be, for example, by disconnecting the power supply. In some embodiments, the scene output module 950 of the control device 900 can, in response to the termination of the end-effector operation command, display a composite scene image with a virtual image of the end-effector or switch the display screen from the actual scene image to a composite scene image with a virtual image of the end-effector, thereby allowing the operator to confirm the effectiveness of the surgical procedure with a full field of view. In some embodiments, the end-effector operation command may have higher priority than the first endoscope movement command in the display mode control. In this way, when the end-effector is triggered to start working, the operator can see the actual situation inside the body more intuitively.

[0067] Those skilled in the art will understand that, in this disclosure, display mode control priority refers to the higher priority of a display mode instruction when multiple display mode instructions exist simultaneously. In some embodiments, the operator can control the endoscope's advance or directional movement via a first endoscope movement instruction while issuing an end-device operation instruction, thereby achieving a first compound operation of controlling the endoscope's movement while activating the end-device for work. For example, when the end-device is an electrocoagulation hemostasis device, the operator can control the endoscope's advance while activating the electrocoagulation hemostasis device, thereby achieving slight contact and compression of the tissue by the electrocoagulation hemostasis device. By prioritizing the display of actual scene images based on end-device operation instructions, it can be ensured that the operator performs surgical procedures based on the actual situation inside the body. Similarly, for other compound operations involving multiple operations, the display mode can also be controlled based on the display mode control priority of various operations or operation instructions.

[0068] In some embodiments, the motion control command for controlling the movement of the endoscope may further include a second endoscope motion command, which may include controlling the endoscope to move away from the operating area. For example, the second endoscope motion command may be a retraction command. In some embodiments, the endoscope motion command may be determined based on the master-slave motion mapping relationship between the pose of the master manipulator and the pose of the end effector arm, and the endoscope motion command may be, for example, a target pose of the end effector arm of the endoscope or a drive signal associated with the target pose.

[0069] In some embodiments, method 1100 may further include controlling the endoscope to move away from the operating area based on a second endoscope movement command, and displaying a composite scene image of a virtual image with an endoscope in response to the endoscope moving away from the operating area beyond a threshold. The endoscope moving away from the operating area beyond the threshold may include determining whether the distance the endoscope has moved back exceeds the threshold, or determining whether the cumulative value of the position change of the master manipulator used to control the endoscope movement exceeds the threshold, the position change corresponding to the second endoscope movement command (e.g., a retraction command). For example, in multiple control loops, a retraction command proportional to the position change of the master manipulator may be determined based on the operator's operation on the master manipulator. The motion control module 910 of the control device 900 can control the endoscope to move away from the operating area based on a retraction command. The control device 900 can obtain (e.g., store in memory) the retraction position change of the main operator in each control cycle and accumulate these position changes. When the accumulated position change exceeds a predetermined threshold, the scene output module 950 of the control device 900 can display a composite scene image with a virtual image of the endoscope, or switch the display from the actual scene image to a composite scene image with a virtual image of the endoscope, facilitating operator observation and preventing damage to internal organs or cavities during retraction. This display operation for the second endoscope motion command can have a higher priority in display mode control than other motion control commands and endoscope operation commands. This allows for timely and automatic adjustment of the display mode when the operator intends to retract the endoscope, facilitating observation of the internal condition.

[0070] In some embodiments, the operator can simultaneously issue an end-device operation command and control the endoscope retraction via a second endoscope movement command to achieve a second compound operation of controlling endoscope movement while the end-device is activated for work. For example, when the end-device is a clamping device, the operator can control the endoscope to retract while activating the clamping device to hold tissue, thereby achieving slight traction and dissection of the tissue by the clamping device. By displaying a composite scene image based on the end-device operation command when the endoscope is less than a threshold away from the operation area, and displaying a virtual image with the end-device attached, preferentially based on the second endoscope movement command when the endoscope is more than a threshold away from the operation area, the operator can be automatically provided with an actual or complete field of view inside the body for observation. In some embodiments, method 1100 may further include terminating the end-device operation command in response to the endoscope being more than a threshold away from the operation area. This prevents the end-device from damaging internal organs or cavities during the retraction of the endoscope.

[0071] In some embodiments, the motion control commands for controlling the movement of the endoscope may further include an automatic exit command, which can be used to control the endoscope to automatically exit from the body. In some embodiments, method 1100 may further include controlling the endoscope to exit the body based on the automatic exit command, and displaying a composite scene image with a virtual image of the endoscope with an endoscope in response to the automatic exit command. The automatic exit command allows the operator to quickly exit the endoscope. Similarly, the automatic exit command may have higher priority than other motion control commands (e.g., first endoscope motion command, second endoscope motion command, etc.) and endoscope operation commands in display mode control. In addition, the automatic exit command may automatically terminate other motion control commands and endoscope operation commands that are being executed, and may disable other motion control commands, endoscope operation commands, or display mode selection commands. Alternatively, the automatic exit command may also allow the triggering of a display mode selection command during execution, so that the operator can switch display modes for better observation of the body.

[0072] Display mode selection instructions can be used to manually trigger display mode switching, for example, by manual input from an operator. In some embodiments, display mode selection instructions may include at least one of a composite scene display instruction, a real scene display instruction, and a multi-scene display instruction. Specifically, the composite scene display instruction displays a composite scene image of a virtual image with an end-effector, the real scene display instruction displays a real scene image, and the multi-scene display instruction simultaneously displays both the composite scene image of the virtual image with an end-effector and the real scene image. For example, a multi-scene display instruction may display at least a portion of the real scene image in a first window of the display device and at least a portion of the composite scene image of the virtual image with an end-effector in a second window of the display device.

[0073] In some embodiments, the display mode selection command may have a higher priority than other display mode commands in terms of display mode control, such as the first endoscope movement command, the second endoscope movement command, the end-device operation command, and the automatic exit command, etc. Because the display mode selection command requires operator intervention and expresses the operator's direct display needs, it has a higher priority during operation.

[0074] Figure 12 This diagram illustrates multi-scene display on a display device 1200 according to some embodiments of the present disclosure. For example... Figure 12 As shown, in some embodiments, the display device 1200 may include a first window 1210 and a second window 1220, with the first window 1210 surrounding the second window 1220 from the outside, forming a so-called picture-in-picture display. In some embodiments, at least a portion of the actual scene image may be displayed in the first window 1210, and at least a portion of a composite scene image of a virtual image with an endoscope may be displayed in the second window 1220. For example, the scene output module 950 of the control device 900 may, in response to a multi-scene display command, simultaneously output the actual scene image and the composite scene image of a virtual image with an endoscope to the display device 1200. The display device 1200 may display a portion of the actual scene image in the first window 1210, and display, for example, the surgical site in the composite scene image of a virtual image with an endoscope in the second window 1220. This display method allows the operator to simultaneously see the environment around the endoscope and the surgical site in front of the endoscope, improving operational accuracy while suppressing discomfort caused to the operator by repeatedly switching images. It should be understood that the ways to present two scene images simultaneously on a display device include, but are not limited to, the methods described above. For example, the display device 1200 may also display the first window 1210 and the second window 1220 side by side in a split-screen manner.

[0075] This disclosure provides some embodiments of a method for generating a composite scene image based on a first image and a second image. Figure 13 A flowchart illustrating a method 1300 for generating a synthetic scene image based on a first image and a second image according to some embodiments of the present disclosure is shown. In some embodiments, some or all of the steps in method 1300 may be performed by a robotic system (e.g., Figure 1 The robot system 100 shown Figure 19 The control device (e.g., of the robot system 1900 shown) Figure 9 The control device 900 shown is... Figure 19The method 1300 is executed by the control device 1970 shown. The control device may include a computing device. The method 1300 may be implemented by software, firmware, and / or hardware. In some embodiments, the method 1300 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., a dedicated processor). Figure 19 The control device (1970) shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0076] refer to Figure 13 In step 1301, a supplementary image is determined based on the first image or the second image. The supplementary image includes the portion of the second image or the first image that is obscured by the end-effector. In some embodiments, the supplementary image may be determined based on the first image, and this supplementary image includes the portion of the second image that is obscured by the end-effector. For example, such as... Figure 4 As shown, the first imaging unit 430 can capture a first image within a first field of view (e.g., the sum of fields of view 431 and 432). The first image includes a first environmental image (e.g., an image of the cavity wall) located within field of view 431 and an image of the end-effector 460 located within field of view 432. The first environmental image may include an image located within field of view 431' (a portion of field of view 431), which is a supplementary image used to synthesize with a second image captured by the second imaging unit 440, corresponding to the portion of the second image obscured by the end-effector 460. Similarly, in some embodiments, a supplementary image may be determined based on the second image, which includes the portion of the first image obscured by the end-effector. For example, the second imaging unit 440 can capture a second image within a second field of view (e.g., the sum of fields of view 441 and 442). The second image includes a second environmental image (e.g., an image of the cavity wall) located within field of view 441 and an image of the end-effector 460 located within field of view 442. The second environmental image may include an image located within the field of view 441' (a portion of the field of view 441), which is a supplementary image used to synthesize with the first image captured by the first imaging unit 430, corresponding to the portion of the first image that is obscured by the end effector 460.

[0077] In some embodiments, the position of the supplementary image in the first image or the position of the supplementary image in the second image can be determined based on the spatial positional relationship between the first imaging unit 430, the second imaging unit 440 and the end effector 460, thereby separating the supplementary image from the first image or the second image. The supplementary image can be used to stitch together with the second environmental image in the second image or the first environmental image in the first image to generate a stitched image.

[0078] Continue to refer to Figure 13In step 1303, a first environmental image or a second environmental image is determined based on the first image or the second image. The first environmental image and the second environmental image do not include the image of the end-device. In some embodiments, the image of the end-device 460 can be removed from the first image or the second image based on the difference between the environmental image and the image of the end-device 460 to obtain the first environmental image in the first image or the second environmental image in the second image. For example, the image of the end-device 460 can be removed from the first image or the second image based on color features, boundary features, texture features, or spatial relationship features to generate the first environmental image or the second environmental image.

[0079] Continue to refer to Figure 13 In step 1305, the first environmental image or the second environmental image and the supplementary image are stitched together to generate a stitched image. The following explanation uses the stitching of the first environmental image and the supplementary image as an example to illustrate the generation of the stitched image. It should be understood that the stitched image can also be generated by stitching the second environmental image and the supplementary image.

[0080] In some embodiments, feature points can be extracted from the first environment image and the supplementary image using a feature point detection algorithm. The feature point detection algorithm can be any one of the following: Harris (corner detection), SIFT (Scale Invariant Feature Transform), SURF (Speeded-Up Robust Features), and ORB (Oriented Fast and Rotated Brief). For example, feature points can be extracted from the edges of the first environment image and the supplementary image using a feature point detection algorithm, and a feature point database can be established based on the feature point data structure. The feature point data structure can include the feature point's position coordinates, scale, orientation, and feature vector, etc.

[0081] In some embodiments, a feature matching algorithm can be used to perform feature matching on feature points of the edges of the first environment image and the supplementary image, thereby determining the correlation between the edges of the first environment image and the edges of the supplementary image. The feature matching algorithm can be any one of the following: brute-force matching algorithm, cross-matching algorithm, KNN (k-nearest neighbor classification) matching algorithm, and RANSAC (Random Sample Consensus) matching algorithm.

[0082] In some embodiments, a registration image can be generated based on a first environmental image and / or a supplementary image. For example, the transformation relationship between the first image coordinate system and the second image coordinate system can be determined based on the spatial positional relationship between the first imaging unit 430 and the second imaging unit 440, and the supplementary image in the second image coordinate system can be transformed into an image in the first image coordinate system based on this transformation relationship to generate a registration image for image fusion with the first environmental image. In some embodiments, both the first environmental image and the supplementary image can be transformed into images in the reference coordinate system based on the transformation relationship between the first image coordinate system, the second image coordinate system, and a reference coordinate system (e.g., the coordinate system of the end of the endoscope).

[0083] In some embodiments, a stitched image can be generated by stitching together a first environment image and a registration image. For example, the edges of the first environment image and the registration image can be aligned and stitched together based on successfully matched feature points in the first environment image and the supplementary image to generate a stitched image. In some embodiments, the generated stitched image can be a two-dimensional stitched image. In some embodiments, the two-dimensional stitched image can serve as a two-dimensional composite scene image.

[0084] In some embodiments, method 1300 may further include processing the first environment image, the second environment image, the supplementary image, or the stitched image to generate a three-dimensional synthetic scene image. Figure 14 A flowchart illustrating a method 1400 for generating a three-dimensional synthetic scene image based on a first image and a second image according to some embodiments of the present disclosure is provided. In some embodiments, some or all of the steps in method 1400 may be performed by a robotic system (e.g., Figure 1 The robot system 100 shown Figure 19 The control device (e.g., for the robot system 1900 shown) Figure 9 The control device 900 shown or Figure 19 The method 1400 is executed by the control device 1970 shown. The control device may include a computing device. The method 1400 may be implemented by software, firmware, and / or hardware. In some embodiments, the method 1400 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., a dedicated processor). Figure 19 The control device (1970) shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0085] refer to Figure 14 In step 1401, for at least one of the first environmental image, the second environmental image, the supplementary image, or the stitched image, the optical flow field of the image is determined based on the image and the previous frame image (two consecutive frames). The optical flow field includes the optical flow of multiple pixels in the image. The following explanation uses the first environmental image in the first image as an example.

[0086] In some embodiments, as the endoscope (e.g., endoscope 420) moves within the cavity, a first imaging unit (e.g., first imaging unit 430) captures images of the cavity environment with a continuously changing field of view (corresponding to the direction of the optical axis), resulting in a plurality of first images arranged in a frame sequence. A first environment image can be determined by removing images of the end-effector (e.g., end-effector 460) from the first images. The method for determining the first environment image can be implemented similarly to step 1303 in method 1300.

[0087] Pixels in the first environmental image correspond to object points in the environment. In the sequence of the first environmental images, pixels move between adjacent frames (e.g., the previous frame and the current frame of the image), generating optical flow. This optical flow is a two-dimensional vector describing the positional changes of the pixels, corresponding to the three-dimensional motion vector of the object points in the environment, and is the projection of the three-dimensional motion vector of the object points onto the image plane. In some embodiments, the optical flow of pixels in the first environmental image can be calculated using the previous and current frames of the first environmental image. In some embodiments, by calculating the optical flow of multiple pixels in the first environmental image, the optical flow field of the first environmental image can be obtained. The optical flow field is the instantaneous velocity field generated by the movement of pixels in the first environmental image on the image plane, including the instantaneous motion vector information of the pixels, such as the direction and speed of the pixel's movement.

[0088] Continue to refer to Figure 14 In step 1403, a depth map of the image is generated based on the optical flow field of the image and the pose of the imaging unit corresponding to the image. This depth map includes the depth of object points corresponding to multiple pixels. In some embodiments, the depth value of the object point in the cavity environment corresponding to the pixel can be determined based on the optical flow of the pixel in the optical flow field of the first environment image and the pose change of the first imaging unit, thereby generating a depth map of the first environment image based on the depth of the object point. For example, an exemplary method for generating a depth map of an image based on the optical flow field and the pose of the imaging unit may include, as shown below... Figure 15 Method 1500.

[0089] Figure 15 A flowchart illustrating a method 1500 for generating a depth map based on the pose of an optical flow field and an imaging unit according to some embodiments of the present disclosure is provided. In some embodiments, some or all of the steps in method 1500 may be performed by a robotic system (e.g., Figure 1 The robot system 100 shown Figure 19 The control device (e.g., for the robot system 1900 shown) Figure 9 The control device 900 shown or Figure 19The method 1500 is executed by the control device 1970 shown. The control device may include a computing device. The method 1500 may be implemented by software, firmware, and / or hardware. In some embodiments, the method 1500 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., a dedicated processor). Figure 19 The control device (1970) shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0090] refer to Figure 15 In step 1501, the focus of the optical flow field is determined based on the optical flow field of the image. For example, when generating the optical flow field based on a first environmental image, during endoscope advance or retraction, the optical flows of multiple pixels in the first environmental image are not parallel to each other, and the extensions of the optical flow vectors converge at the focus of the optical flow field, which is a fixed point in the optical flow field. In some embodiments, the optical flow vectors in the optical flow field may have a corresponding relationship with the focus, and each optical flow vector may converge to a different focus. In some embodiments, such as during endoscope advance, the focus of the optical flow field may include an expansion focus (FOE), which is the convergence point of the optical flow vectors extending in the opposite direction. In some embodiments, such as during endoscope retraction, the focus of the optical flow field may include a contraction focus (FOC), which is the convergence point of the optical flow vectors extending in the forward direction.

[0091] Continue to refer to Figure 15 In step 1503, based on the focal point of the optical flow field, the distances between multiple pixels and the focal point are determined. For example, in the first environmental image, the distances between multiple pixels and the focal point can be determined in the first image coordinate system.

[0092] Continue to refer to Figure 15 In step 1505, based on the optical flow field of the image, the velocities of multiple pixels in the optical flow field are determined. The optical flow velocity of a pixel can be the ratio between the distance the pixel moves in the optical flow field (the length of the optical flow) and the time interval between two consecutive frames. In some embodiments, in the first environmental image, the distances moved by multiple pixels in the optical flow field can be determined in the first image coordinate system. In some embodiments, the time interval between two consecutive frames (the time of each frame) can be, for example, 1 / 60 of a second, but is not limited thereto and can be appropriately adjusted according to imaging requirements.

[0093] Continue to refer to Figure 15In step 1507, the velocity of the imaging unit is determined based on the pose of the imaging unit corresponding to the image. In some embodiments, the first image coordinate system and the coordinate system of the endoscope's main body have a predetermined transformation relationship, and the pose of the first imaging unit can be calculated based on the pose of the end body. In some embodiments, the distance the first imaging unit moves can be determined from the pose of the first imaging unit, thereby determining the velocity of the first imaging unit based on the distance the first imaging unit moves and the time interval between two consecutive frames.

[0094] Continue to refer to Figure 15 In step 1509, a depth map of the image is determined based on the distances between multiple pixels and the focal point, the velocities of the multiple pixels in the optical flow field, and the velocity of the imaging unit. In some embodiments, in the optical flow field generated from the first environment image, the depth value (depth information) of an object point can be determined based on the distance between a pixel and the focal point, the velocity of that pixel in the optical flow field, and the velocity of the first imaging unit. The depth value of the object point can be the distance between the object point and the image plane of the first imaging unit. By calculating the depth value of the object point for each pixel in the first environment image, a depth map of the first environment image can be obtained.

[0095] In some embodiments, the distances of multiple pixels moving within the optical flow field can be determined based on the optical flow field of the image, and the distances of the imaging unit moving can be determined based on the pose of the imaging unit corresponding to the image. Thus, the depth map of the image is determined based on the distances between the multiple pixels and the focal point, the distances of the multiple pixels moving within the optical flow field, and the distances of the imaging unit moving.

[0096] Continue to refer to Figure 14 In step 1405, based on the depth map of the image and the pixel coordinates of multiple pixels, the spatial coordinates of the object points corresponding to the multiple pixels are determined. In some embodiments, the spatial coordinates of the object points can be determined based on the pixel coordinates of the pixels in the first environment image in the first image coordinate system and the depth value of the object points corresponding to the pixels, thereby realizing the transformation of the two-dimensional pixels in the first environment image to three-dimensional coordinates.

[0097] Continue to refer to Figure 14 In step 1407, color information of multiple pixels is obtained based on the image. In some embodiments, a color feature extraction algorithm can be used to extract the color information of pixels in the first environmental image. The color feature extraction algorithm can be any of the following methods: color histogram, color set, color moment, color aggregation vector, etc.

[0098] Continue to refer to Figure 14In step 1409, point cloud fusion is performed on the image based on the color information of multiple pixels and the spatial coordinates of object points to generate a three-dimensional point cloud. In some embodiments, the spatial coordinates of object points can be transformed based on the intrinsic parameter matrix of the first imaging unit, and point cloud fusion is performed on the first environmental image based on the color information of the pixels to generate a three-dimensional point cloud, which includes the three-dimensional spatial coordinates and color information of the object points. In some embodiments, the intrinsic parameters of the first imaging unit can be known or obtained through calibration.

[0099] In some embodiments, method 1400 can be used to process a first environment image or a second environment image and a supplementary image to generate a three-dimensional point cloud. For example, in some embodiments, feature extraction and stereo matching can be performed on the three-dimensional point clouds of the first environment image and the supplementary image to stitch the first environment image and the supplementary image together, thereby generating a three-dimensional stitched image. In some embodiments, method 1400 can also be used to process a registration image generated based on the supplementary image to generate a three-dimensional point cloud, and feature extraction and stereo matching can be performed on the three-dimensional point clouds of the first environment image and the registration image to stitch together and generate a three-dimensional stitched image. It should be understood that a three-dimensional stitched image can also be generated by stitching together a second environment image in a second image and its corresponding supplementary image.

[0100] In some embodiments, a two-dimensional or three-dimensional stitched image can be used as a composite scene image. In some embodiments, a two-dimensional composite scene image can also be processed to generate a three-dimensional composite scene image. For example, method 1400 can be used to process the stitched image generated in method 1300 to achieve the conversion of the composite scene image from two-dimensional to three-dimensional.

[0101] In some embodiments, method 1100 may further include generating a three-dimensional real-scene image based on at least one of a first image or a second image. Figure 16 A flowchart illustrating a method 1600 for generating a three-dimensional real-world scene image based on a first image and / or a second image according to some embodiments of the present disclosure is provided. In some embodiments, some or all steps of method 1600 may be performed by a robotic system (e.g., Figure 1 The robot system 100 shown Figure 19 The control device (e.g., for the robot system 1900 shown) Figure 9 The control device 900 shown or Figure 19 The method 1600 is executed by the control device 1970 shown. The control device may include a computing device. The method 1600 may be implemented by software, firmware, and / or hardware. In some embodiments, the method 1600 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., a dedicated processor). Figure 19The control device (1970) shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0102] refer to Figure 16 In step 1601, for either the first or second image, the optical flow field of the image is determined based on the image and its previous frame. The optical flow field includes the optical flow of multiple pixels in the image. In some embodiments, step 1601 can be implemented similarly to step 1401 in method 1400.

[0103] Continue to refer to Figure 16 In step 1603, a depth map of the image is generated based on the optical flow field of the image and the pose of the imaging unit corresponding to the image. The depth map includes the depth of object points corresponding to multiple pixels. In some embodiments, step 1603 can be implemented similarly to step 1403 in method 1400.

[0104] Continue to refer to Figure 16 In step 1605, based on the depth map of the image and the pixel coordinates of multiple pixels, the object point spatial coordinates corresponding to the multiple pixels are determined. In some embodiments, step 1605 can be implemented similarly to step 1405 in method 1400.

[0105] Continue to refer to Figure 16 In step 1607, color information of multiple pixels is obtained based on the image. In some embodiments, step 1607 can be implemented similarly to step 1407 in method 1400.

[0106] Continue to refer to Figure 16 In step 1609, point cloud fusion is performed on the image based on the color information of multiple pixels and the spatial coordinates of object points to generate a three-dimensional point cloud. In some embodiments, step 1609 can be implemented similarly to step 1409 in method 1400.

[0107] This disclosure provides some embodiments of a method for generating a virtual image of an end-effector in a synthetic scene image. Figure 17 A flowchart illustrating a method 1700 for generating a virtual image of an end effector in a synthetic scene image according to some embodiments of the present disclosure is shown. In some embodiments, some or all of the steps in method 1700 may be performed by a robotic system (e.g., Figure 1 The robot system 100 shown Figure 19 The control device (e.g., for the robot system 1900 shown) Figure 9 The control device 900 shown or Figure 19The method 1700 is executed by the control device 1970 shown. The control device may include a computing device. The method 1700 may be implemented by software, firmware, and / or hardware. In some embodiments, the method 1700 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., a dedicated processor). Figure 19 The control device (1970) shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0108] refer to Figure 17 In step 1701, the position and size of the end effector in the synthesized scene image are determined. In some embodiments, the position and size of the end effector in the synthesized scene image can be determined based on the inherent parameters of the end effector, which may include the positional parameters of the end effector on the subject (e.g., the relative positional relationship with the first imaging unit and the second imaging unit), orientation parameters, and size parameters. For example, the inherent parameters of the end effector may be known or obtained through calibration. In some embodiments, the position and size of the end effector in the synthesized scene image may also be determined based on the edges of the first environment image, the second environment image, or the supplementary image.

[0109] Continue to refer to Figure 17 In step 1703, a virtual image of the end effector is generated in the composite scene image. In some embodiments, the virtual image of the end effector can be generated in the composite scene image through real-time rendering. For example, the virtual image of the end effector can be generated for each frame of the composite scene image. In some embodiments, the virtual image of the end effector may include outlines and / or transparent entities to indicate the end effector. This allows the position and size of the end effector to be shown without obstructing the operator's view.

[0110] In some embodiments, method 1000 may further include generating a first virtual ruler for indicating distance along the axial direction of a virtual image of the end-effector in the synthesized scene image. This first virtual ruler can be generated along the contour of the virtual image of the end-effector to indicate the length of the end-effector, helping to improve the operator's operational accuracy. In some embodiments, the method for generating the first virtual ruler can be implemented similarly to step 1703 in method 1700.

[0111] In some embodiments, method 1000 may further include determining the distance between the distal end of the instrument and the surgical site, and updating a first virtual ruler to a second virtual ruler based on the distance between the distal end of the instrument and the surgical site. For example, the distance between the distal end of the instrument and the surgical site can be measured by a ranging unit on the instrument, and a second virtual ruler can be generated based on that distance. The second virtual ruler may include information indicated by the first virtual ruler and distance information between the distal end of the instrument and the surgical site. By updating the first virtual ruler to the second virtual ruler, the length of the distal instrument and the distance between the distal instrument and the surgical site can be shown simultaneously, which helps to further improve the operator's operational accuracy. In some embodiments, the method for generating the second virtual ruler can be implemented similarly to step 1703 in method 1700.

[0112] In some embodiments of this disclosure, a computer device is also provided, including a memory and a processor. The memory may be used to store at least one instruction, and the processor is coupled to the memory for executing the at least one instruction to perform some or all of the steps in the method of this disclosure, such as... Figure 10 , Figure 11 , Figures 13-17 Some or all of the steps in the method disclosed herein.

[0113] Figure 18 A schematic block diagram of a computer device 1800 according to some embodiments of the present disclosure is shown. See also Figure 18 The computer device 1800 may include a central processing unit (CPU) 1801, a system memory 1804 including random access memory (RAM) 1802 and read-only memory (ROM) 1803, and a system bus 1805 connecting the various components. The computer device 1800 may also include an input / output system and a mass storage device 1807 for storing an operating system 1813, application programs 1814, and other program modules 1815. The input / output devices include an input / output control unit 1810 mainly composed of a display 1808 and input devices 1809.

[0114] Mass storage device 1807 is connected to central processing unit 1801 via a mass storage control device (not shown) connected to system bus 1805. Mass storage device 1807 or computer-readable media provides non-volatile storage for computer devices. Mass storage device 1807 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0115] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The aforementioned system memories and mass storage devices can be collectively referred to as memory.

[0116] Computer device 1800 can be connected to network 1812 via network interface unit 1811 connected to system bus 1805.

[0117] The system memory 1804 or mass storage device 1807 is also used to store one or more instructions. The central processing unit 1801 implements all or part of the steps of the methods in some embodiments of this disclosure by executing the one or more instructions.

[0118] In some embodiments of this disclosure, a computer-readable storage medium is also provided, storing at least one instruction that is executed by a processor to cause a computer to perform some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 10 , Figure 11 , Figures 13-17 Some or all of the steps in the disclosed method. Examples of computer-readable storage media include memory for computer programs (instructions), such as read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage devices.

[0119] Figure 19 A schematic diagram of a robot system 1900 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [reference needed]. Figure 19The robot system 1900 may include a motion arm 1940, an endoscope 1930, a control device 1970, and a display device 1990. The endoscope 1930 includes a drive unit 1920 and an actuator arm 1950. An endoscope body 1960 is disposed at the distal end of the actuator arm 1950, and the endoscope 1930 includes an end-effector 1980 for performing surgical procedures. The display device 1990 is used to display images output by the endoscope 1930. The control device 1970 is configured to connect to the motion arm 1940 and the drive unit 1920 to control the movement of the actuator arm 1950, and is communicatively connected to the endoscope 1930 to process the images output by the endoscope 1930. The control device 1970 is used to perform some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 10 , Figure 11 , Figures 13-17 Some or all of the steps in the method disclosed herein.

[0120] Note that the above are merely exemplary embodiments and technical principles of this disclosure. Those skilled in the art will understand that this disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this disclosure. Therefore, although this disclosure has been described in detail through the above embodiments, this disclosure is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this disclosure, the scope of which is determined by the scope of the appended claims.

Claims

1. A robot system, characterized in that, include: Movement arm; An endoscope, disposed at the end of the moving arm, the endoscope comprising: An actuator arm, including at least one segment capable of controlled bending; The main body is located at the distal end of the actuator arm; A first imaging unit for capturing a first image; A second imaging unit for capturing a second image, wherein the fields of view of the first image and the second image are different and include an image of the end effector; and The end effector is configured to extend from the distal end of the body; A display device for displaying images; and A control device is communicatively connected to the endoscope and the display device. The control device is configured to control the movement of the endoscope, obtain a first image from the first imaging unit, obtain a second image from the second imaging unit, generate a synthetic scene image based on the first image and the second image to remove the actual image of the end instrument, and generate a virtual image of the end instrument in the synthetic scene image. The control device is also configured to: Based on the first image or the second image, a supplementary image is determined, the supplementary image including the portion of the second image or the first image that is obscured by the end-effector; Based on the first image or the second image, a first environmental image or a second environmental image is determined, wherein the first environmental image and the second environmental image do not include an image of the end effector; and A stitched image is generated by stitching together the first environmental image or the second environmental image and the supplementary image. For at least one of the first image, the second image, the first environment image, the second environment image, the supplementary image, or the stitched image, Based on the image and the previous frame of the image, the optical flow field of the image is determined, and the optical flow field includes the optical flow of multiple pixels in the image; Based on the optical flow field of the image and the pose of the imaging unit corresponding to the image, a depth map of the image is generated, and the depth map includes the depth of the object points corresponding to the plurality of pixels; Based on the depth map of the image and the pixel coordinates of the plurality of pixels, determine the object point spatial coordinates of the object point corresponding to the plurality of pixels; Based on the image, the color information of the plurality of pixels is obtained, and the color information of the pixels in the first environment image is extracted using a color feature extraction algorithm; and Based on the color information of the multiple pixels and the spatial coordinates of the object points, the image is fused into a point cloud to generate a 3D point cloud. Based on the optical flow field of the image, determine the focal point of the optical flow field; Based on the focal point of the optical flow field, the distance between the plurality of pixels and the focal point is determined; Based on the optical flow field of the image, the velocity of the plurality of pixels in the optical flow field is determined; Based on the pose of the imaging unit corresponding to the image, the velocity of the imaging unit is determined; and The depth map of the image is determined based on the distance between the plurality of pixels and the focal point, the velocity of the plurality of pixels in the optical flow field, and the velocity of the imaging unit.

2. The robot system according to claim 1, characterized in that, The control device is also configured to: Based on the first image and / or the second image, generate an actual scene image to display the actual image of the end effector; as well as In response to a display mode command, the composite scene image with the virtual image and / or the actual scene image are displayed.

3. The robot system according to claim 2, characterized in that, The display mode instructions include motion control instructions for controlling the movement of the endoscope; The motion control command includes a first endoscope motion command, which includes a feed command for controlling the endoscope's advance or a steering command for controlling the endoscope's rotation. The control device is further configured to: In response to the first endoscope motion command, the synthetic scene image with the virtual image is displayed; and Based on the first endoscope movement command, the endoscope is controlled to move towards the operating area.

4. The robot system according to claim 3, characterized in that, The motion control command further includes a second endoscope motion command, which includes a retraction command for controlling the retraction of the endoscope, and the control device is further configured to: Based on the second endoscope movement command, control the endoscope to move away from the operating area; and In response to the endoscope moving away from the operating area beyond a threshold, the synthetic scene image with the virtual image is displayed.

5. The robot system according to claim 4, characterized in that, The display mode command also includes: End-effector operation commands, used to control the operation of the end-effector; and / or The display mode selection command is used to select the display mode.

6. The robot system according to claim 5, characterized in that, The control device is also configured to: In response to the end-device operation command, display the actual scene image; or In response to the termination of the end-device operation command, the synthetic scene image with the virtual image is displayed.

7. The robot system according to claim 4, characterized in that, The control device is also configured to: Determine whether the distance the endoscope has retracted exceeds a threshold; or Determine whether the cumulative value of the position change of the master operator used for master-slave control of the endoscope, corresponding to the second endoscope movement command, exceeds a threshold.

8. The robot system according to claim 4, characterized in that, The motion control commands also include automatic exit commands. The control device is also configured to: Based on the aforementioned automatic exit command, the endoscope is controlled to exit the body; and In response to the automatic exit command, a composite scene image with a virtual image of the end effector is displayed.

9. The robot system according to claim 5, characterized in that, The display mode control priority of the display mode instruction includes at least one of the following: The display mode control priority of the end-device operation command is higher than the display mode control priority of the first endoscope movement command. The display mode control priority of the second endoscope movement command is higher than the display mode control priority of the end-device operation command; or The display mode selection command has a higher display mode control priority than the second endoscope movement command.

10. The robot system according to claim 5, characterized in that, The display mode selection instruction includes at least one of the following: A composite scene display instruction is used to display the composite scene image containing the virtual image; The actual scene display command is used to display the actual scene image; or A multi-scene display instruction is used to display at least a portion of the actual scene image in a first window and at least a portion of the composite scene image with the virtual image in a second window.

11. The robot system according to claim 1, characterized in that, The control device is also configured to: Determine the position and size of the end effector in the synthesized scene image; and A virtual image of the end effector is generated in the synthesized scene image.

12. The robot system according to claim 11, characterized in that, The virtual image of the end effector includes outlines and / or transparent entities to show the end effector; and / or In the synthesized scene image, a first virtual ruler for indicating distance is generated along the axial direction of the virtual image of the end effector.

13. The robot system according to claim 12, characterized in that, The control device is also configured to: Determine the distance between the endoscope and the surgical site; and Based on the distance between the endoscope and the surgical site, the first virtual ruler is updated to the second virtual ruler.

14. The robot system according to claim 1, characterized in that, The optical axes of the first imaging unit and the second imaging unit are respectively parallel to the axis of the main body; and The axis of the end effector is parallel to the axis of the body and deviates from the line connecting the first imaging unit and the second imaging unit.

15. The robot system according to claim 1, characterized in that, The end effector includes at least one first electrode, at least one second electrode, and an insulating body, wherein the at least one first electrode and the at least one second electrode are alternately disposed on the circumferentially outer side of the insulating body, and at least a portion of the first electrode and at least a portion of the second electrode are exposed; and / or The endoscope body includes an inner annular component and an outer annular component. The distal end instrument includes a first electrode and a second electrode. The first electrode is mounted on the inner annular component, and the second electrode is mounted on the outer annular component and radially spaced around at least a portion of the first electrode; and / or The actuator arm includes at least one structural bone, a fixing plate, and at least one spacer plate, wherein the at least one structural bone passes through the at least one spacer plate and its end is fixedly connected to the fixing plate.

16. A computer device, comprising: Memory, used to store at least one instruction; as well as A processor, coupled to the memory, is configured to execute the at least one instruction to perform the following steps: The endoscope of the surgical robot system controls the movement of the endoscope, which includes a main body, a first imaging unit, a second imaging unit, and an end instrument extending from the distal end of the main body. Obtain a first image from the first imaging unit; A second image is obtained from the second imaging unit, wherein the fields of view of the first image and the second image are different and include an image of the endoscope's end instrument; A synthetic scene image is generated based on the first image and the second image to remove the actual image of the end effector; A virtual image of the end effector is generated in the synthesized scene image; Based on the first image or the second image, a supplementary image is determined, the supplementary image including the portion of the second image or the first image that is obscured by the end-effector; Based on the first image or the second image, a first environmental image or a second environmental image is determined, wherein the first environmental image and the second environmental image do not include an image of the end effector; and A stitched image is generated by stitching together the first environmental image or the second environmental image and the supplementary image. For at least one of the first image, the second image, the first environment image, the second environment image, the supplementary image, or the stitched image, Based on the image and the previous frame of the image, the optical flow field of the image is determined, and the optical flow field includes the optical flow of multiple pixels in the image; Based on the optical flow field of the image and the pose of the imaging unit corresponding to the image, a depth map of the image is generated, and the depth map includes the depth of the object points corresponding to the plurality of pixels; Based on the depth map of the image and the pixel coordinates of the plurality of pixels, determine the object point spatial coordinates of the object point corresponding to the plurality of pixels; Based on the image, the color information of the plurality of pixels is obtained, and the color information of the pixels in the first environment image is extracted using a color feature extraction algorithm; and Based on the color information of the multiple pixels and the spatial coordinates of the object points, the image is fused into a point cloud to generate a 3D point cloud. Based on the optical flow field of the image, determine the focal point of the optical flow field; Based on the focal point of the optical flow field, the distance between the plurality of pixels and the focal point is determined; Based on the optical flow field of the image, the velocity of the plurality of pixels in the optical flow field is determined; Based on the pose of the imaging unit corresponding to the image, the velocity of the imaging unit is determined; and The depth map of the image is determined based on the distance between the plurality of pixels and the focal point, the velocity of the plurality of pixels in the optical flow field, and the velocity of the imaging unit.

17. The computer device according to claim 16, characterized in that, The steps also include: Based on the first image and / or the second image, generate an actual scene image to display the actual image of the end effector; and In response to a display mode instruction, the synthetic scene image with the virtual image and / or the actual scene image are displayed, the display mode instruction including motion control instructions for controlling the movement of the endoscope.

18. The computer device according to claim 17, characterized in that, The motion control command includes a first endoscope motion command, which includes a feed command for controlling the endoscope's advance or a steering command for controlling the endoscope's rotation. Displaying the synthetic scene image with the virtual image and / or the actual scene image in response to the display mode command includes: In response to the first endoscope motion command, the synthetic scene image with the virtual image is displayed; and Based on the first endoscope movement command, the endoscope is controlled to move towards the operating area.

19. The computer device according to claim 18, characterized in that, The motion control command further includes a second endoscope motion command, which includes a retraction command for controlling the endoscope to retract. The method of displaying the synthetic scene image with the virtual image and / or the actual scene image in response to the display mode command also includes: Based on the second endoscope movement command, control the endoscope to move away from the operating area; and In response to the endoscope moving away from the operating area beyond a threshold, the synthetic scene image with the virtual image is displayed.

20. A computer-readable storage medium for storing at least one instruction, which, when executed by a computer, causes the computer to perform the following steps: The endoscope of the surgical robot system controls the movement of the endoscope, which includes a main body, a first imaging unit, a second imaging unit, and an end instrument extending from the distal end of the main body. Obtain a first image from the first imaging unit; A second image is obtained from the second imaging unit, wherein the fields of view of the first image and the second image are different and include an image of the endoscope's end instrument; A synthetic scene image is generated based on the first image and the second image to remove the actual image of the end effector; A virtual image of the end effector is generated in the synthesized scene image; Based on the first image or the second image, a supplementary image is determined, the supplementary image including the portion of the second image or the first image that is obscured by the end-effector; Based on the first image or the second image, a first environmental image or a second environmental image is determined, wherein the first environmental image and the second environmental image do not include an image of the end effector; and A stitched image is generated by stitching together the first environmental image or the second environmental image and the supplementary image. For at least one of the first image, the second image, the first environment image, the second environment image, the supplementary image, or the stitched image, Based on the image and the previous frame of the image, the optical flow field of the image is determined, and the optical flow field includes the optical flow of multiple pixels in the image; Based on the optical flow field of the image and the pose of the imaging unit corresponding to the image, a depth map of the image is generated, and the depth map includes the depth of the object points corresponding to the plurality of pixels; Based on the depth map of the image and the pixel coordinates of the plurality of pixels, determine the object point spatial coordinates of the object point corresponding to the plurality of pixels; Based on the image, the color information of the plurality of pixels is obtained, and the color information of the pixels in the first environment image is extracted using a color feature extraction algorithm; and Based on the color information of the multiple pixels and the spatial coordinates of the object points, the image is fused into a point cloud to generate a 3D point cloud. Based on the optical flow field of the image, determine the focal point of the optical flow field; Based on the focal point of the optical flow field, the distance between the plurality of pixels and the focal point is determined; Based on the optical flow field of the image, the velocity of the plurality of pixels in the optical flow field is determined; Based on the pose of the imaging unit corresponding to the image, the velocity of the imaging unit is determined; and The depth map of the image is determined based on the distance between the plurality of pixels and the focal point, the velocity of the plurality of pixels in the optical flow field, and the velocity of the imaging unit.

21. The computer-readable storage medium according to claim 20, characterized in that, The steps also include: Based on the first image and / or the second image, generate an actual scene image to display the actual image of the end effector; and In response to a display mode instruction, the synthetic scene image with the virtual image and / or the actual scene image are displayed, the display mode instruction including motion control instructions for controlling the movement of the endoscope.

22. The computer-readable storage medium according to claim 21, characterized in that, The motion control command includes a first endoscope motion command, which includes a feed command for controlling the endoscope's advance or a steering command for controlling the endoscope's rotation. Displaying the synthetic scene image with the virtual image and / or the actual scene image in response to the display mode command includes: In response to the first endoscope motion command, the synthetic scene image with the virtual image is displayed; and Based on the first endoscope movement command, the endoscope is controlled to move towards the operating area.

23. The computer-readable storage medium according to claim 22, characterized in that, The motion control command further includes a second endoscope motion command, which includes a retraction command for controlling the endoscope to retract. The method of displaying the synthetic scene image with the virtual image and / or the actual scene image in response to the display mode command also includes: Based on the second endoscope movement command, control the endoscope to move away from the operating area; and In response to the endoscope moving away from the operating area beyond a threshold, the synthetic scene image with the virtual image is displayed.

Images