Field of view display method and surgical robotic system

Abstract

The application provides a visual field display method and a surgical robot system, acquires a first endoscope image captured by an endoscope of the surgical robot system at a first position; displays a first image with a first visual field area on a first display and a second display based on the first endoscope image respectively; when detecting a device switching, controls the endoscope of the surgical robot system to move to a second position, and captures a second endoscope image with a visual field area larger than the first endoscope image; switches the second display to display a second image with a second visual field area based on the second endoscope image, and the second visual field area is larger than the first visual field area. The scheme moves the endoscope to capture an endoscope image with a larger visual field area, and switches the display of the image with a larger visual field area based on the endoscope image captured after the movement to assist the user in observing the position of the device end in real time, avoids the device from touching the human body by mistake, reduces the risk of injury by mistake, and improves the user experience.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, specifically to a visual field display method and a surgical robot system. Background Technology

[0002] During the operation of the surgical robot, if the surgeon needs to change surgical instruments, an assistant must perform the replacement at one end of the imaging tower. After the surgical instrument is removed and replaced, the assistant reinserts the instrument based on their clinical experience. The insertion position is entirely based on guesswork, which may lead to accidental injury to organs due to limited field of vision during reinsertion, resulting in a high risk of injury and a poor user experience. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a field of view display method and a surgical robot system to reduce the risk of accidental injury and improve the user experience when switching instruments.

[0004] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0005] According to a first aspect, one embodiment provides a method for displaying endoscopic images in a surgical robot system, the surgical robot system including at least a first display mounted on a surgeon's workstation and a second display mounted on an external imaging tower; the method includes:

[0006] Acquire the first endoscopic image captured by the endoscope of the surgical robot system at the first position;

[0007] Based on the first endoscopic image, a first image with a first field of view is displayed on the first display and the second display, respectively;

[0008] When instrument switching is detected, the endoscope of the surgical robot system is controlled to move to a second position and a second endoscopic image is captured; the field of view of the second endoscopic image captured by the endoscope of the surgical robot system at the second position is larger than the field of view of the first endoscopic image captured at the first position.

[0009] Based on the second endoscopic image, the second display is switched to show a second image with a second field of view, which is larger than the first field of view.

[0010] In one embodiment, after detecting instrument switching, controlling the endoscope of the surgical robot system to move to a second position and capturing a second endoscopic image, the method further includes: maintaining the first display on the first screen to show a third image with the first field of view based on the second endoscopic image.

[0011] In one embodiment, maintaining the first display of a third image having the first field of view region based on the second endoscopic image includes:

[0012] The second endoscopic image is magnified and cropped to obtain the third image having the first field of view area, and then displayed on the first display.

[0013] In one embodiment, when instrument switching is detected, the endoscope of the surgical robot system is controlled to move to a second position and a second endoscopic image is captured, including:

[0014] When a user-triggered operation characterizing device switching is detected, device switching is determined;

[0015] The endoscope of the surgical robot system is controlled to move to the second position, and a second endoscopic image is captured.

[0016] In one embodiment, it further includes:

[0017] When the instrument switching is detected to be complete, the endoscope of the surgical robot system is controlled to move back to the first position and the first endoscopic image captured at the first position is acquired; based on the first endoscopic image, the first image with the first field of view is restored and displayed on the first display and the second display respectively.

[0018] In one embodiment, upon detecting the completion of instrument switching, the endoscope of the surgical robot system is controlled to move back to the first position, and the first endoscopic image captured at the first position is acquired, including:

[0019] When a user-triggered operation indicating the completion of device switching is detected, it is determined that the device switching is complete.

[0020] The endoscope of the surgical robot system is controlled to move back to the first position, and the first endoscopic image captured at the first position is obtained.

[0021] In one embodiment, the content of the second image in the second field of view includes the content of the first image in the first field of view.

[0022] In one embodiment, the first field of view is rectangular and the second field of view is circular.

[0023] According to a second aspect, one embodiment provides a method for displaying an endoscopic image field of view, the method comprising:

[0024] Acquire the first endoscopic image captured by the endoscope of the surgical robot system at the first position;

[0025] A first image with a first field of view is displayed based on the first endoscopic image;

[0026] When instrument switching is detected, the endoscope of the surgical robot system is controlled to move to a second position and a second endoscopic image is captured; the field of view of the second endoscopic image captured by the endoscope of the surgical robot system at the second position is larger than the field of view of the first endoscopic image captured at the first position.

[0027] Based on the second endoscopic image, a second image with a second field of view is switched and displayed, the second field of view being larger than the first field of view.

[0028] In one embodiment, when instrument switching is detected, the endoscope of the surgical robot system is controlled to move to a second position and a second endoscopic image is captured, including:

[0029] When a user-triggered operation characterizing device switching is detected, device switching is determined;

[0030] The endoscope of the surgical robot system is controlled to move to the second position, and a second endoscopic image is captured.

[0031] In one embodiment, it further includes:

[0032] When the instrument switching is detected to be complete, the endoscope of the surgical robot system is controlled to move back to the first position, and the first endoscopic image captured at the first position is acquired; the first image with the first field of view is restored and displayed based on the first endoscopic image.

[0033] In one embodiment, upon detecting the completion of instrument switching, the endoscope of the surgical robot system is controlled to move back to the first position, and the first endoscopic image captured at the first position is acquired, including:

[0034] When a user-triggered operation indicating the completion of device switching is detected, it is determined that the device switching is complete.

[0035] The endoscope of the surgical robot system is controlled to move back to the first position, and the first endoscopic image captured at the first position is obtained.

[0036] In one embodiment, the content of the second image in the second field of view includes the content of the first image in the first field of view.

[0037] In one embodiment, the first field of view is rectangular and the second field of view is circular.

[0038] In one embodiment, the surgical robot system includes a first display mounted on the doctor's workstation and a second display mounted on an external imaging tower;

[0039] A first image with a first field of view is displayed based on the first endoscopic image;

[0040] When instrument switching is detected, the endoscope of the surgical robot system is moved to the second position and a second endoscopic image is captured.

[0041] Based on the second endoscopic image, a second image with a second field of view is switched and displayed; including:

[0042] Based on the first endoscopic image, the first image having the first field of view is displayed on the first display and the second display, respectively;

[0043] When instrument switching is detected, the endoscope of the surgical robot system is controlled to move to the second position and a second endoscopic image is captured. Based on the second endoscopic image, only the second display switches to display the second image with the second field of view area.

[0044] In one embodiment, after detecting instrument switching, controlling the endoscope of the surgical robot system to move to a second position and capturing a second endoscopic image, the method further includes: maintaining the first display on the first screen to show a third image with the first field of view based on the second endoscopic image.

[0045] In one embodiment, maintaining the first display of a third image having the first field of view region based on the second endoscopic image includes:

[0046] The second endoscopic image is magnified and cropped to obtain the third image having the first field of view area, and then displayed on the first display.

[0047] According to a third aspect, one embodiment provides a method for displaying the field of view of an endoscopic image, the method comprising:

[0048] Acquire the first endoscopic image captured by the endoscope at the first position;

[0049] A first image with a first field of view is displayed based on the first endoscopic image;

[0050] When a first specific event is detected, the endoscope is controlled to move to a second position and a second endoscopic image is captured; the field of view of the second endoscopic image captured by the endoscope at the second position is larger than the field of view of the first endoscopic image captured at the first position.

[0051] Based on the second endoscopic image, a second image with a second field of view is switched and displayed, the second field of view being larger than the first field of view.

[0052] In one embodiment, upon detecting a first specific event, controlling the endoscope to move to a second position includes:

[0053] When instrument switching is detected, or when a user-triggered operation to activate the field of view switching function is detected, the endoscope is controlled to move to the second position.

[0054] In one embodiment, after switching to display a second image having a second field of view, the method further includes:

[0055] Upon detecting a second specific event, the endoscope is controlled to move back to the first position and the first image with the first field of view is restored.

[0056] In one embodiment, upon detecting a second specific event, controlling the endoscope to move back to the first position and restoring the display of the first image having the first field of view includes:

[0057] When instrument switching is detected to be complete, or when a user-triggered operation to disable the field of view switching function is detected, the endoscope is controlled to move back to the first position and the first image with the first field of view area is restored.

[0058] According to a fourth aspect, one embodiment provides a method for displaying endoscopic images in a surgical robot system, the surgical robot system including at least a first display mounted on a surgeon's workstation and a second display mounted on an external imaging tower; the method includes:

[0059] Acquire the first endoscopic image captured by the endoscope of the surgical robot system at the first position;

[0060] Based on the first endoscopic image, a first image with a first field of view is displayed on the first display and the second display, respectively;

[0061] When a device change is detected, a device change assistance operation is performed;

[0062] When a user-triggered operation to activate the field of view switching function is detected, the endoscope of the surgical robot system is controlled to move to the second position and capture a second endoscopic image; the field of view of the second endoscopic image captured by the endoscope of the surgical robot system at the second position is larger than the field of view of the first endoscopic image captured at the first position.

[0063] Based on the second endoscopic image, the second display is switched to show a second image with a second field of view, which is larger than the first field of view.

[0064] According to a fifth aspect, one embodiment provides a surgical robot system, the surgical robot system including a processor and a memory; the memory includes computer-readable instructions stored thereon, the computer-readable instructions implementing the endoscopic image field of view display method described in any of the preceding claims when executed by the processor.

[0065] As can be seen from the above technical solutions, the embodiments of the present invention have the following advantages:

[0066] When the surgical robot system detects the switching of instruments, it moves the endoscope to capture an endoscopic image with a larger field of view. Based on the endoscopic image captured after the movement, it switches to display an image with a larger field of view to help the user observe the position of the instrument end in real time, avoid accidental contact between the instrument and the human body, reduce the risk of accidental injury, and improve the user experience. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0068] Figure 1 This is a schematic diagram of the structure of a surgical robot system provided in an embodiment of the present invention;

[0069] Figure 2 This is a first flowchart of a method for displaying the endoscopic image field of view of a surgical robot system according to an embodiment of the present invention;

[0070] Figure 3 An example diagram of a first image having a first field of view, provided for an embodiment of the present invention;

[0071] Figure 4 An example diagram showing a second image having a second field of view, provided for an embodiment of the present invention;

[0072] Figure 5 An example diagram showing a second image with a second field of view, captured at a second location, provided in an embodiment of the present invention;

[0073] Figure 6 Another example of a display of a second image with a second field of view obtained at a second location, provided in an embodiment of the present invention;

[0074] Figure 7 This is a workflow example diagram of the endoscopic image field of view display method provided in an embodiment of the present invention;

[0075] Figure 8 This is an example diagram of the field of view switching of the second display provided in an embodiment of the present invention;

[0076] Figure 9 This is a second flowchart of an endoscopic image field of view display method provided in an embodiment of the present invention;

[0077] Figure 10 This is a third flowchart of an endoscopic image field of view display method provided in an embodiment of the present invention;

[0078] Figure 11 This is a fourth flowchart of a method for displaying endoscopic images in a surgical robot system according to an embodiment of the present invention;

[0079] Figure 12 A fifth flowchart of a method for displaying endoscopic images in a surgical robot system according to an embodiment of the present invention;

[0080] Figure 13 An example diagram showing the light-sensing range, first field of view, and second field of view of an image sensor provided in an embodiment of the present invention;

[0081] Figure 14 Another example display diagram of a second image having a second field of view provided in an embodiment of the present invention;

[0082] Figure 15 This is yet another example of a display of a second image having a second field of view, provided in an embodiment of the present invention.

[0083] Figure 16 Another workflow example diagram of the endoscopic image field of view display method provided in the embodiments of the present invention;

[0084] Figure 17 This is another example diagram of field-of-view switching for the second display provided in an embodiment of the present invention;

[0085] Figure 18 A sixth flowchart of an endoscopic image field of view display method provided in an embodiment of the present invention;

[0086] Figure 19 A seventh flowchart of an endoscopic image field of view display method provided in an embodiment of the present invention;

[0087] Figure 20 The eighth flowchart is provided for an embodiment of the present invention for a method of displaying endoscopic images in a surgical robot system. Detailed Implementation

[0088] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0089] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0090] During the operation of surgical robots (such as laparoscopic surgical robots), if the surgeon needs to change surgical instruments, an assistant must perform the instrument replacement at one end of the imaging tower. After the surgical instruments are removed and replaced, they need to be reinserted into the patient. Because the traditional 3D imaging range is relatively small, the assistant relies on their clinical experience to reinsert the instruments, often guessing the placement. This can lead to accidental injury to organs due to limited visibility during the reinsertion process.

[0091] The inventors discovered through research that existing surgical robots have a position memory function, allowing surgical instruments to reach the surrounding area before removal when inserted for the second time. While this provides sufficient space for the surgical instruments, the relative positions of organs within the human body may change at any time. Despite providing space for the second insertion, organs may still be accidentally damaged. Another existing surgical robot has a position memory function and a target position marking function, which automatically stops when it encounters an obstacle in the second insertion path. However, this cannot completely avoid the risk of accidental injury through vision.

[0092] Based on the above research, this solution proposes an endoscopic image field of view display method and a surgical robot system. When the surgical robot system detects the switching of instruments, it moves the endoscope to capture an endoscopic image with a larger field of view. Based on the endoscopic image captured after the movement, it switches to display an image with a larger field of view to assist the user in observing the position of the instrument end in real time, avoiding accidental contact between the instrument and the human body, reducing the risk of accidental injury and improving the user experience.

[0093] Applying some embodiments of the present invention, one practical application scenario is as follows: a first endoscopic image captured by the endoscope of a surgical robot at a first position is displayed on a first display and a second display, respectively, with a first field of view area; when switching instruments, the endoscope is controlled to move to a second position and capture a second endoscopic image with a field of view area larger than the first endoscopic image; based on the second endoscopic image, the second display is switched to display a second image with a second field of view area larger than the first field of view area, thereby assisting the user in observing the position of the instrument end in real time and avoiding accidental contact between the instrument and the human body.

[0094] The present invention will now be described in detail through various embodiments.

[0095] Figure 1 This is a schematic diagram of the structure of a surgical robot system according to an embodiment of the present invention. The surgical robot system may include a doctor's operating table 101, an external imaging tower 102 (also called an imaging tower), a surgical robotic arm 103, an imaging system 104 (usually a 3D imaging system or 3D imaging system), a first display 105 disposed on the doctor's operating table 101, and a second display 106 disposed on the external imaging tower 102; the imaging system 104 includes at least an endoscope, a processor, and a memory.

[0096] The surgeon in the surgical team controls the surgical robotic arm 103 to perform the surgery through the doctor's operating table 101; the imaging system 104 collects image information during the surgery and sends the collected image information to the first display 105 and the second display 106 for display.

[0097] The surgeon observes the surgical procedure through the first monitor 105 on the surgeon's console 101, while other members of the surgical team (such as assistants) observe the procedure through the second monitor 106 on the external imaging tower 102.

[0098] The memory in the imaging system 104 includes computer-readable instructions stored thereon, which, when executed by the processor in the imaging system 104, implement the endoscopic image field of view display method provided in any of the following method embodiments.

[0099] It should be noted that, Figure 1 The structure of the surgical robot system shown is for illustrative purposes only and may include more advanced features. Figure 1 Showing more or fewer parts, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented in hardware and / or software.

[0100] See Figure 2This diagram illustrates a first flowchart of an endoscopic image field of view display method for a surgical robot system according to an embodiment of the present invention. The surgical robot system includes at least a first display mounted on the surgeon's workstation and a second display mounted on an external imaging tower. Figure 2 Includes the following steps:

[0101] Step S201: Acquire the first endoscopic image taken by the endoscope of the surgical robot system at the first position.

[0102] In the specific implementation of step S201, during the operation, a first endoscopic image is acquired by the endoscope (e.g., stereoscopic endoscope) of the surgical robot system (e.g., laparoscopic surgical robot system) at a first position. The endoscope is equipped with an image sensor, and the first endoscopic image is acquired by the image sensor.

[0103] Step S202: Based on the first endoscopic image, display the first image with the first field of view on the first display and the second display respectively.

[0104] In the specific implementation step S202, a first image with a first field of view is selected from the first endoscopic image captured, and then the first image with the first field of view is displayed on a first monitor set on the doctor's operating table and a second monitor set on an external imaging tower.

[0105] It should be noted that conventional displays are usually rectangular in shape. In some specific embodiments, when the first display and the second display show a first image with a first field of view, in order to further improve the user experience, the first field of view can be rectangular (or other shapes), so that the first image with the first field of view can be displayed on both the first display and the second display.

[0106] Step S203: When instrument switching is detected, the endoscope of the surgical robot system is controlled to move to the second position and a second endoscopic image is captured; the field of view of the second endoscopic image captured by the endoscope of the surgical robot system at the second position is larger than the field of view of the first endoscopic image captured at the first position.

[0107] It should be noted that the procedure for switching instruments in a surgical robot system is as follows: after determining the need for instrument switching, the user (such as an assistant) unlocks the instrument to be disassembled on the surgical robotic arm, disassembles it, installs the new instrument on the surgical robotic arm, and then locks the newly installed instrument.

[0108] In the specific implementation of step S203, after the first image with a first field of view is displayed on the first and second displays, when instrument switching is detected, the endoscope of the surgical robot system is controlled to move from the first position to the second position, and the endoscope is controlled to capture a second endoscopic image at the second position.

[0109] In some specific embodiments, when a user-triggered operation to characterize instrument switching is detected, instrument switching is determined, the endoscope of the surgical robot system is controlled to move to a second position, and a second endoscopic image is captured.

[0110] For example: when the user triggers the unlocking of the device, the device switching is determined, and the endoscope is controlled to move to the second position and a second endoscopic image is captured.

[0111] Specifically, when instrument switching is detected, the endoscope controlling the surgical robot system moves axially from the first position to the second position outside the body. That is, the endoscope moves from the first position to the second position along the central axis of the endoscope outside the body and takes a second endoscopic image.

[0112] It should be noted that the imaging range of an endoscope is different at different positions, which means that the field of view of the endoscopic images taken at different positions is different. As the endoscope moves towards the outside of the body to the second position, the imaging range of the endoscope is increased. Therefore, the field of view of the second endoscopic image taken at the second position is larger than the field of view of the first endoscopic image taken at the first position.

[0113] The distance between the first position and the second position can be preset. That is, the endoscope moves a preset distance along the central axis of the endoscope in the external direction from the first position to reach the second position. When the endoscope moves from the first position to the second position, in order to further ensure that the display content of the first display remains unchanged in subsequent steps, the endoscope is controlled to rotate a preset angle during the process of moving from the first position to the second position.

[0114] Step S204: Based on the second endoscopic image, switch the second display to show a second image with a second field of view, the second field of view being larger than the first field of view.

[0115] In the specific implementation of step S204, after the endoscope is controlled to capture a second endoscopic image at the second position, a second image with a second field of view is selected from the second endoscopic image, and the second display is controlled to switch to display the second image with the second field of view to expand the field of view.

[0116] It should be noted that the field of view of the second endoscopic image captured at the second position is larger than the field of view of the first endoscopic image captured at the first position; therefore, the second field of view is larger than the first field of view.

[0117] In some specific embodiments, the content of the second image in the second field of view includes the content of the first image in the first field of view. Switching the second display from displaying the first image with the first field of view to displaying the second image with the second field of view can expand the field of view displayed on the second display, thereby helping the user to observe the position of the instrument end to avoid accidental injury.

[0118] In other embodiments, the first field of view can be rectangular, and the second field of view can be circular or rectangular; however, whether the second field of view is circular or rectangular, it is larger than the first field of view, so that the field of view can be expanded after the second display is switched to display a second image with the second field of view.

[0119] It should be noted that the field of view of a circular second field of view is larger than that of a rectangular second field of view. In practical applications, the shape of the second field of view can be set according to the actual situation.

[0120] For example, if a second image with a second field of view needs to fill the entire second display, then the second field of view can be rectangular; if a larger field of view is needed, then the second field of view can be circular.

[0121] For example: assuming both the first and second visual fields are rectangular, a stereoscopic endoscope captures a first endoscopic image at a first position. From this first endoscopic image, a region is selected as follows... Figure 3 A first image with a first field of view is shown, and the first image with the first field of view is displayed on a first display and a second display, wherein, Figure 3 Point A in the diagram is the first position;

[0122] When instrument switching is detected, the stereoscopic endoscope is controlled to move axially from the first position to the second position outside the body, and a second endoscopic image is captured. From the second endoscopic image, an image is selected as follows: Figure 4 The second image shown has a second field of view. The second display is switched to show the second image with the second field of view, wherein, Figure 4 Point A is the first position, and point B is the second position.

[0123] from Figure 3 and Figure 4As shown in the diagram, switching the second display to show a second image with a second field of view expands the field of view, allowing the user to observe the position of the instrument's end in a second image with a larger field of view.

[0124] It should be noted that when the second field of view is circular, the second display switches to show the second image with the second field of view in the following two ways:

[0125] In the first scenario, the size of the second field of view is less than or equal to the size of the second display. When the second display shows a second image with the second field of view, the second display can display the complete second image with the second field of view (i.e., display a complete circular second image).

[0126] For example Figure 5 As can be seen from the example image of the second image with a second field of view taken at the second position, when the size of the second field of view is less than or equal to the size of the second display, the second display can display the complete second image with the second field of view.

[0127] In the second scenario, the size of the second field of view is larger than the size of the second display. When the second display shows a second image with a second field of view, the second display can display most of the content of the second image with the second field of view (i.e., display most of the second image). The field of view corresponding to the "most of the content of the second image" is still larger than the first field of view.

[0128] For example Figure 6 Another example of a display of a second image with a second field of view, taken at a second location, shows that the size of the second field of view is larger than the size of the second display, and most of the content of the second image with the second field of view is displayed on the second display.

[0129] Whether displaying the complete second image or most of its content, the second display can expand the field of view.

[0130] In some specific embodiments, after the endoscope is moved to the second position and a second endoscopic image is captured, in order to ensure that the content displayed on the first monitor set on the doctor's operating table remains unchanged (the purpose is to ensure that the surgeon's field of vision does not change), a third image with a first field of view is maintained on the first monitor based on the second endoscopic image. The content of the third image with the first field of view includes the content of the first image with the first field of view.

[0131] The method for maintaining the display of a third image with a first field of view on the first display based on the second endoscope image is as follows: the second endoscope image is magnified and cropped to obtain a third image with a first field of view, and the third image with a first field of view is displayed on the first display.

[0132] Specifically, the magnification ratio (also known as magnification factor) is calculated based on the distance from the first position to the second position (equivalent to the movement distance of the endoscope) and the working distance of the endoscope at the first position.

[0133] For example, the magnification ratio m can be calculated by m = 1 + sin(60°) * Y / X, where X is the working distance of the endoscope at the first position and Y is the distance from the first position to the second position.

[0134] The second endoscope image is magnified according to the magnification ratio calculated above. A third image with a first field of view is cropped from the magnified second endoscope image and displayed on the first display.

[0135] For example, after magnifying the second endoscope image by m times (m is the magnification ratio), the top center area of ​​the second endoscope image magnified by m times is cropped to obtain a third image with a first field of view, and the third image with the first field of view is displayed on the first display.

[0136] In some specific embodiments, after the second display is switched to display a second image with a second field of view, when the instrument switching is detected to be completed, the endoscope of the surgical robot system is controlled to move back to the first position and the first endoscopic image captured by it at the first position is acquired; based on the first endoscopic image, the first image with the first field of view is restored to be displayed on the first display and the second display respectively.

[0137] Specifically, when a user-triggered operation to indicate the completion of instrument switching is detected, the instrument switching is determined to be completed; the endoscope of the surgical robot system is controlled to move back to the first position, and the first endoscopic image captured by it at the first position is acquired; based on the first endoscopic image, the first image with the first field of view is restored and displayed on the first display and the second display respectively.

[0138] For example: when the user triggers the locking operation of the instrument, the instrument switching is completed. At this time, the endoscope is controlled to move back to the first position from the second position along the axial direction of the endoscope in the body. That is, the endoscope is controlled to move back to the first position from the second position along the central axis of the endoscope in the body. The first endoscopic image captured by the endoscope in the first position is acquired. Then, based on the first endoscopic image, the first image with the first field of view area is restored and displayed on the first display and the second display respectively.

[0139] In the above Figure 2 Based on the steps described, one application scenario of this solution is as follows: An endoscope captures a first endoscopic image at a first position. Based on this first endoscopic image, a first image with a first field of view is displayed on a first monitor on the doctor's workstation and a second monitor on an external imaging tower. When switching instruments, the endoscope is controlled to move from the first position outward along the central axis of the endoscope to a second position and capture a second endoscopic image. Based on the second endoscopic image, the second monitor is switched to display a second image with a second field of view to expand the observation field of the external imaging tower, assisting the assistant in observing the instrument's end position to avoid accidental injury. The second endoscopic image is magnified and cropped to obtain a third image with a first field of view. The third image with a first field of view is maintained on the first monitor, ensuring the doctor's observation field of view remains unchanged. After instrument switching is completed, the endoscope is controlled to move back to the first position from the second position inward along the central axis of the endoscope, acquiring the first endoscopic image captured at the first position. Based on this first endoscopic image, the first image with a first field of view is restored to be displayed on both the first and second monitors.

[0140] To better explain the above application scenarios, the following will illustrate... Figure 7 The workflow example diagram shown is illustrated below for the endoscopic image field of view display method:

[0141] Acquire the first endoscopic image taken at the first position; determine whether to switch instruments.

[0142] If no instruments are switched, based on the first endoscopic image, the first display on the doctor's console displays the first image with the first field of view, and the second display on the external imaging tower displays the first image with the first field of view.

[0143] If the instrument is switched, the endoscope moves from the first position to the second position and takes a second endoscopic image. Based on the second endoscopic image, the second display of the external imaging tower switches to display a second image with a second field of view. The second endoscopic image is magnified and cropped so that the first display of the doctor's operating table maintains the display of a third image with a first field of view, and it is determined whether the instrument switching is complete.

[0144] If the instrument switching is completed, the endoscope moves from the second position back to the first position and acquires the first endoscopic image taken in the first position. Based on the first endoscopic image, the first display on the doctor's operating table and the second display on the external imaging tower resume displaying the first image with the first field of view area.

[0145] If the instrument switching is not completed, the first monitor on the doctor's operating table continues to display the third image with the first field of view, and the second monitor on the external imaging tower continues to display the second image with the second field of view.

[0146] From the above Figure 7 Based on the workflow of the endoscopic image field of view display method shown, the field of view switching of the second display of the external imaging tower is as follows: Figure 8 As shown, Figure 8 Point A in the diagram is the first position. Figure 8 Point B in the diagram is the second position. Figure 8 Both the first and second visual fields of view are rectangular. When no instrument is switched, the second display of the external imaging tower displays the first image with the first visual field based on the first endoscopic image captured by the endoscope at the first position. When switching instruments, the endoscope is controlled to move from the first position to the second position to capture a second endoscopic image with a larger field of view. Based on the second endoscopic image, the second display of the external imaging tower switches to display the second image with the second visual field. When the instrument switching is completed, the endoscope is controlled to move from the second position back to the first position and capture the first endoscopic image. Based on the first endoscopic image, the second display of the external imaging tower resumes displaying the first image with the first visual field.

[0147] from Figure 8 As shown in the diagram of the second display of the external imaging tower, when switching instruments, the endoscope is moved to allow the second display of the external imaging tower to switch to display a second image with a wider field of view. This allows instruments outside the first field of view to be identified and observed, avoiding accidental injury during instrument switching. Throughout the process, the first display on the doctor's console maintains a constant field of view, without affecting or interfering with the display content on the doctor's console.

[0148] See Figure 9 This diagram illustrates a second flowchart of an endoscopic image field of view display method provided by an embodiment of the present invention. Figure 9 Includes the following steps:

[0149] Step S901: Acquire the first endoscopic image taken by the endoscope of the surgical robot system at the first position.

[0150] In the specific implementation of step S901, during the operation, a first endoscopic image is acquired by the endoscope (e.g., stereoscopic endoscope) of the surgical robot system (e.g., laparoscopic surgical robot system) at a first position.

[0151] Step S902: Display a first image with a first field of view based on the first endoscopic image.

[0152] In the specific implementation step S902, a first image with a first field of view is selected from the first endoscopic image captured, and the first image with the first field of view is displayed.

[0153] Step S903: When instrument switching is detected, the endoscope of the surgical robot system is controlled to move to the second position and a second endoscopic image is captured; the field of view of the second endoscopic image captured by the endoscope of the surgical robot system at the second position is larger than the field of view of the first endoscopic image captured at the first position.

[0154] In the specific implementation of step S903, when instrument switching is detected, the endoscope of the surgical robot system is controlled to move from the first position to the second position, and the second endoscopic image captured by the endoscope at the second position is acquired.

[0155] In some specific embodiments, when a user-triggered operation to characterize instrument switching is detected, instrument switching is determined; the endoscope of the surgical robot system is controlled to move to a second position, and a second endoscopic image is captured.

[0156] Step S904: Based on the second endoscope image, switch to display a second image with a second field of view, the second field of view being larger than the first field of view.

[0157] In the specific implementation of step S904, after the endoscope is controlled to capture a second endoscopic image at the second position, a second image with a second field of view is selected from the second endoscopic image, and the second image with the second field of view is switched to be displayed to expand the field of view.

[0158] In some specific embodiments, the content of the second image in the second field of view includes the content of the first image in the first field of view;

[0159] In some other specific embodiments, the first field of view is rectangular and the second field of view is circular.

[0160] After switching to display a second image with a second field of view, when the instrument switching is detected to be complete, the endoscope of the surgical robot system is controlled to move back to the first position, and the first endoscopic image captured by it at the first position is acquired; specifically, when the user-triggered operation to indicate the completion of the instrument switching is detected, the instrument switching is determined to be complete; the endoscope of the surgical robot system is controlled to move back to the first position, and the first endoscopic image captured by it at the first position is acquired.

[0161] After controlling the endoscope to move from the second position back to the first position and acquiring the first endoscopic image captured by the endoscope at the first position, the first image with the first field of view is restored and displayed based on the first endoscopic image.

[0162] In some specific embodiments, the surgical robot system includes a first display mounted on the doctor's operating table and a second display mounted on an external imaging tower; therefore, the specific implementation of the above steps S902 and S904 is as follows: based on the first endoscopic image, a first image with a first field of view is displayed on the first display and the second display respectively; when instrument switching is detected, the endoscope of the surgical robot system is controlled to move to a second position and a second endoscopic image is captured; based on the second endoscopic image, only the second display switches to display a second image with a second field of view.

[0163] In other words, based on the first endoscopic image, a first image with a first field of view is displayed on the first and second displays respectively; when instrument switching is detected, the endoscope is controlled to move to the second position and a second endoscopic image is captured. Based on the second endoscopic image, only the second display set on the external imaging tower is controlled to switch to display the second image with a second field of view, while maintaining the observation field of view of the first display set on the doctor's operating table. This allows the assistant to observe the position of the instrument end through the second image with a larger field of view to avoid accidental injury, and also keeps the surgeon's observation field of view unchanged (i.e., does not affect or interfere with the display content on the doctor's operating table).

[0164] After the endoscope is moved to the second position and a second endoscopic image is captured, in order to maintain the field of view of the first display unchanged, in some specific embodiments, a third image with a first field of view is maintained on the first display based on the second endoscopic image. Specifically, the second endoscopic image is magnified and cropped to obtain the third image with the first field of view, and then displayed on the first display.

[0165] Furthermore, after controlling the endoscope to move to the second position and taking a second endoscopic image, the magnification ratio is calculated based on the distance from the first position to the second position (equivalent to the movement distance of the endoscope) and the working distance of the endoscope at the first position; the second endoscopic image is magnified according to the magnification ratio, a third image with a first field of view is cropped from the magnified second endoscopic image, and the third image with the first field of view is displayed on the first display, thereby keeping the observation field of view of the first display unchanged.

[0166] See Figure 10 This diagram illustrates a third flowchart of an endoscopic image field of view display method provided by an embodiment of the present invention. Figure 10 Includes the following steps:

[0167] Step S1001: Acquire the first endoscopic image taken by the endoscope at the first position.

[0168] In the specific implementation of step S1001, during the operation, a first endoscopic image is acquired by an endoscope (e.g., a stereoscopic endoscope) at a first position.

[0169] Step S1002: Display a first image with a first field of view based on the first endoscopic image.

[0170] In the specific implementation step S1002, a first image with a first field of view is selected from the first endoscopic image captured, and the first image with the first field of view is displayed.

[0171] Step S1003: When the first specific event is detected, the endoscope is controlled to move to the second position and a second endoscopic image is captured; the field of view of the second endoscopic image captured by the endoscope at the second position is larger than the field of view of the first endoscopic image captured at the first position.

[0172] In the specific implementation step S1003, after displaying the first image with the first field of view, when the first specific event is detected, the endoscope is controlled to move from the first position to the second position, and the second endoscopic image captured by the endoscope at the second position is acquired.

[0173] In some specific embodiments, the first specific event is "instrument switching" or "operation to activate the field of view switching function"; after displaying the first image with the first field of view area, when instrument switching is detected, or when an operation triggered by the user to activate the field of view switching function is detected, the endoscope is controlled to move to the second position.

[0174] For example, a user can activate the field of view switching function on the doctor's operating table or external imaging tower. When the activation of the field of view switching function is detected, the endoscope is controlled to move from the first position to the second position, and the second endoscopic image taken by the endoscope in the second position is acquired.

[0175] Step S1004: Based on the second endoscope image, switch to display a second image with a second field of view, the second field of view being larger than the first field of view.

[0176] In the specific implementation step S1004, after the endoscope is controlled to capture a second endoscopic image at the second position, a second image with a second field of view is selected from the second endoscopic image, and the second image with the second field of view is switched to expand the field of view. The content of the second image with the second field of view includes the content of the first image with the first field of view.

[0177] In some specific embodiments, after switching to display a second image with a second field of view, when a second specific event is detected, the endoscope is controlled to move back to the first position and the display of the first image with the first field of view is restored.

[0178] The second specific event is either "completing instrument switching" or "operation to turn off the field of view switching function"; after switching to display a second image with a second field of view area, when instrument switching is detected as completed, or when user-triggered operation to turn off the field of view switching function is detected, the endoscope is controlled to move back to the first position and the first image with the first field of view area is restored.

[0179] Specifically, when the instrument switching is detected to be completed, or when the user triggers an operation to turn off the field of view switching function, the endoscope is controlled to move from the second position back to the first position, and a first endoscopic image captured by the endoscope in the first position is acquired. Based on the first endoscopic image, a first image with a first field of view area is restored and displayed.

[0180] For example, after switching to display a second image with a second field of view, the user can turn off the field of view switching function on the doctor's console or external imaging tower. When the field of view switching function is detected to be turned off, the endoscope is controlled to move from the second position back to the first position and the first image with the first field of view is restored.

[0181] The activation of the functions corresponding to the first and second specific events can be achieved by triggering physical or virtual buttons, or by automatically detecting certain related actions.

[0182] As can be seen from the above example, users can actively activate the field-of-view switching function to switch to displaying a second image with a second field of view, or users can actively deactivate the field-of-view switching function after switching to displaying a second image with a second field of view to restore the display of the first image with a first field of view.

[0183] See Figure 11 This diagram illustrates a fourth flowchart of an endoscopic image field of view display method for a surgical robot system according to an embodiment of the present invention. The surgical robot system includes at least a first display mounted on the surgeon's workstation and a second display mounted on an external imaging tower. Figure 11 Includes the following steps:

[0184] Step S1101: Acquire the first endoscopic image taken by the endoscope of the surgical robot system at the first position.

[0185] In the specific implementation of step S1101, during the operation, a first endoscopic image is acquired by the endoscope (e.g., stereoscopic endoscope) of the surgical robot system (e.g., laparoscopic surgical robot system) at a first position.

[0186] Step S1102: Based on the first endoscopic image, display the first image with the first field of view on the first display and the second display respectively.

[0187] In the specific implementation step S1103, a first image with a first field of view is selected from the first endoscopic image captured, and then the first image with the first field of view is displayed on a first monitor set on the doctor's operating table and a second monitor set on an external imaging tower.

[0188] Step S1103: When instrument switching is detected, perform instrument switching auxiliary operation.

[0189] In the specific implementation of step S1103, when instrument switching is detected, an instrument switching auxiliary operation is executed to assist in the instrument switching.

[0190] Specifically, the instrument switching auxiliary operation is performed as follows: when an instrument is detected on the unlocked surgical robotic arm, an instrument presence detection is triggered to check whether the instrument is in place; when an instrument is detected to be out of place (indicating that the instrument has been removed), a first prompt message is displayed on the first and second displays to indicate that the instrument has been removed; for example, when an instrument is detected to be out of place, the first prompt message "Instrument removed" is displayed in the bottom status bar of the first and second displays.

[0191] After the first prompt message is displayed, when it is detected that the instrument on the surgical robotic arm is locked and that the instrument is in place, a second prompt message is displayed on the first and second displays to indicate that the instrument is installed in place; for example: after the first prompt message is displayed, when it is detected that the instrument on the surgical robotic arm is locked and that the instrument is in place, the second prompt message "Instrument is in place" is displayed in the bottom status bar of the first and second displays.

[0192] Step S1104: When the user triggers an operation to start the field of view switching function, the endoscope of the surgical robot system is controlled to move to the second position and a second endoscopic image is captured. The field of view of the second endoscopic image captured by the endoscope of the surgical robot system at the second position is larger than the field of view of the first endoscopic image captured at the first position.

[0193] In the specific implementation of step S1104, after the first image with a first field of view is displayed on the first and second displays, if the operation triggered by the user to start the field of view switching function is detected, the endoscope of the surgical robot system is controlled to move from the first position to the second position, and the endoscope is controlled to take a second endoscopic image at the second position.

[0194] Step S1105: Based on the second endoscopic image, switch the second display to show a second image with a second field of view, the second field of view being larger than the first field of view.

[0195] In the specific implementation step S1105, based on the second endoscopic image captured by the endoscope at the second position, the second display is switched to display a second image with a second field of view area.

[0196] Accordingly, after switching the second display to show the second image with the second field of view, if the user triggers an operation to turn off the field of view switching function, the endoscope is controlled to move back to the first position and acquire the first endoscopic image captured at the first position. Based on the first endoscopic image, the second display is restored to show the first image with the first field of view.

[0197] Combining the surgical robot system's instrument switching process with some embodiments of the present invention, one practical application scenario is as follows: based on the first endoscopic image captured by the endoscope at the first position, the first image with a first field of view is displayed on the first display at the doctor's operating table and the second display at the external imaging tower.

[0198] After determining the need for instrument switching, the assistant unlocks the instrument to be disassembled on the surgical robotic arm and then disassembles it, triggering an instrument presence detection to check if the instrument is in place. When an instrument is detected to be out of place (indicating that the instrument has been disassembled) and an operation to activate the field of view switching function is detected, a first prompt message is displayed on the first and second displays to indicate that the instrument has been disassembled. The endoscope is controlled to move from the first position to the second position to capture a second endoscopic image with a larger field of view. Based on the second endoscopic image, the second display of the external imaging tower switches to display a second image with a second field of view. The second endoscopic image is magnified and cropped so that the first display on the doctor's operating table maintains the display of a third image with a first field of view.

[0199] After the assistant completes the instrument installation and locks the instrument again, when the instrument is detected in place and the operation of turning off the field of view switching function is detected, a second prompt message is displayed on the first and second displays to indicate that the instrument is installed in place. The assistant controls the endoscope to move from the second position back to the first position and acquires the first endoscopic image taken at the first position. Based on the first endoscopic image, the first display on the doctor's operating table and the second display on the external imaging tower resume displaying the first image with the first field of view area. Throughout the entire process, the first display on the doctor's operating table maintains the same field of view and does not affect or interfere with the display content on the doctor's operating table.

[0200] The above is a partial description of the endoscopic image field of view display method. Through the content of the above embodiments, it is possible to expand the field of view by moving the endoscope during the operation.

[0201] It is worth noting that the field of view can be expanded without moving the endoscope. The following is a detailed explanation of how to expand the field of view without moving the endoscope.

[0202] See Figure 12 This diagram illustrates a fifth flowchart of an endoscopic image field of view display method for a surgical robot system according to an embodiment of the present invention. The surgical robot system includes at least a first display mounted on the surgeon's workstation and a second display mounted on an external imaging tower. Figure 12 Includes the following steps:

[0203] Step S1201: Acquire endoscopic images captured by the endoscope of the surgical robot system.

[0204] In the specific implementation of step S1201, during the operation, endoscopic images are acquired by the endoscope (e.g., stereoscopic endoscope) of the surgical robot system (e.g., laparoscopic surgical robot system). The endoscope is equipped with an image sensor, and the endoscopic images are acquired by the image sensor.

[0205] Step S1202: Based on the endoscopic image, display the first image with the first field of view on the first display and the second display respectively.

[0206] In the specific implementation step S1202, a second image with a second field of view is selected from the captured endoscope image. This second field of view is smaller than the light-sensitive range of the endoscope's image sensor.

[0207] Select a first image with a first field of view from a second image having a second field of view, wherein the second field of view is larger than the first field of view.

[0208] After obtaining a first image with a first field of view, the first image with the first field of view is displayed on a first display and a second display, respectively.

[0209] In some specific embodiments, since the second field of view is larger than the first field of view, and the first image having the first field of view is selected from the second image having the second field of view, the content of the second image having the second field of view includes the content of the first image having the first field of view.

[0210] It should be noted that conventional displays are usually rectangular in shape. In some specific embodiments, when the first display and the second display show a first image with a first field of view, in order to further improve the user experience, the first field of view can be rectangular (or other shapes), so that the first image with the first field of view can be displayed on both the first display and the second display.

[0211] The endoscope's image sensor has a field of view that ensures optimal image quality, and the second field of view can be either the "field of view that ensures optimal image quality" or a portion thereof.

[0212] Conventionally, the "field of view that ensures the best image quality" is usually a circular field of view, so the second field of view can be circular (or other shapes).

[0213] For example Figure 13 As shown in the example diagram of the light-sensing range, first field of view, and second field of view of the image sensor, the second field of view is: the 100% field of view area within the light-sensing range of the image sensor that ensures optimal image quality (which can be simply referred to as the 1.0F circular field of view area, where 1.0 represents 100% and F represents field); the first field of view area is a rectangular area within the second field of view area; under normal 3D observation, the first image with this first field of view area is displayed on the first display and the second display.

[0214] It should be noted that the above Figure 13 The specific ranges of the light-sensing range, the first field of view, and the second field of view of the image sensor shown are for illustrative purposes only; the first field of view can also be the inscribed circle within the rectangular light-sensing range of the image sensor, and the second field of view can also be the inscribed rectangle within the circular first field of view.

[0215] Step S1203: When instrument switching is detected, the second display is switched to display a second image with a second field of view, which is larger than the first field of view.

[0216] It should be noted that the procedure for switching instruments in a surgical robot system is as follows: after determining the need for instrument switching, the user (such as an assistant) unlocks the instrument to be disassembled on the surgical robotic arm, disassembles it, installs the new instrument on the surgical robotic arm, and then locks the newly installed instrument.

[0217] In the specific implementation step S1203, when the instrument switching is detected, the second display of the external imaging tower is switched to display a second image with a second field of view. That is, the second display switches from displaying a first image with a first field of view to displaying a second image with a second field of view, so that the second display realizes the expansion of the observation field of view.

[0218] In some specific embodiments, when a user-triggered operation to characterize device switching is detected, device switching is determined, and the second display is switched to display a second image with a second field of view.

[0219] For example, when a user-triggered unlocking operation is detected, a device switching is determined. At this time, the second display switches to show a second image with a second field of view to expand the field of view. Alternatively, the robotic arm can detect whether the device is in place to determine whether a device switching operation is being performed.

[0220] It should be noted that when the second field of view is circular, the second display switches to show the second image with the second field of view in the following two ways:

[0221] In the first scenario, the size of the second field of view is less than or equal to the size of the second display. When the second display shows a second image with the second field of view, the second display can display the complete second image with the second field of view (i.e., display the complete second image).

[0222] For example Figure 14 As can be seen from another example diagram of the second image with a second field of view, when the size of the second field of view is less than or equal to the size of the second display, the second display can display the complete second image with the second field of view.

[0223] In the second scenario, the size of the second field of view is larger than the size of the second display. When the second display shows a second image with a second field of view, the second display can display most of the content of the second image with the second field of view (i.e., display most of the second image). The field of view corresponding to the "most of the content of the second image" is still larger than the first field of view.

[0224] For example Figure 15As can be seen from another example of the provided second image with a second field of view, the size of the second field of view is larger than the size of the second display, and most of the content of the second image with the second field of view is displayed on the second display.

[0225] Whether displaying the complete second image or most of its content, the second display can expand the field of view.

[0226] In other specific embodiments, when a device switching is detected, the second display is switched to display a second image with a second field of view, while the first display remains displaying a first image with a first field of view.

[0227] In other words, when switching instruments, the second display switches to display a second image with a second field of view (the field of view of the second display is expanded), while the first display still displays the first image with a first field of view (the field of view of the first display remains unchanged). This allows the assistant to observe the position of the instrument end through the second image with a larger field of view to avoid accidental injury, while also keeping the surgeon's field of view unchanged (i.e., focusing on the first image with the first field of view).

[0228] In some specific embodiments, after the second display is switched to display a second image with a second field of view, when the device switching is detected to be complete, the second display is switched back to display a first image with a first field of view.

[0229] Specifically, when a user-triggered operation to indicate the completion of device switching is detected, the device switching is determined to be complete, and the second display is restored to show the first image with the first field of view area.

[0230] For example, when a user-triggered locking operation on the device is detected, the device switching is confirmed to be complete, and the second display is restored to show the first image with the first field of view area.

[0231] It is worth noting that in practical applications, when switching instruments, the first display can also switch from displaying a first image with a first field of view to a second image with a second field of view. After the instrument switching is completed, the first display will resume displaying the first image with the first field of view. However, the most preferred implementation is to maintain the first display on the first display showing the first image with the first field of view throughout the entire surgical procedure, so that the surgeon's field of view does not change.

[0232] In the above Figure 12Based on the content shown in each step, one application scenario of this solution is as follows: the first monitor on the doctor's operating table and the second monitor on the external imaging tower display a first image with a rectangular field of view (first field of view); when switching instruments, the second monitor switches to display a second image with a 1.0F circular field of view (second field of view) to expand the observation field of view of the external imaging tower, assisting the assistant in observing the position of the instrument end to avoid accidental injury, while the first monitor continues to display the first image with a rectangular field of view (first field of view); after the instrument switching is completed, the second monitor resumes displaying the first image with a rectangular field of view (first field of view).

[0233] To better explain the above application scenarios, the following will illustrate... Figure 16 Another workflow example diagram illustrating the endoscopic image field of view display method is shown: acquiring endoscopic images; determining whether to switch instruments.

[0234] If no instruments are switched, the first monitor on the doctor's console displays a first image with a rectangular field of view (first field of view), and the second monitor on the external imaging tower displays the first image with a rectangular field of view.

[0235] If the instrument is switched, the first display on the doctor's console continues to display the first image with a rectangular field of view (i.e., the field of view of the first display remains unchanged), while the second display on the external imaging tower switches to display the second image with a 1.0F circular field of view (second field of view) to expand the field of view of the external imaging tower.

[0236] After the second display on the external imaging tower switches to display the second image of the 1.0F circular field of view area, it is determined whether the instrument switching is complete;

[0237] Once the instrument switching is complete, the first monitor on the doctor's console continues to display the first image with a rectangular field of view, while the second monitor on the external imaging tower resumes displaying the first image with a rectangular field of view (the first field of view).

[0238] If the instrument switching is not completed, the first monitor on the doctor's operating table continues to display the first image with a rectangular field of view, while the second monitor on the external imaging tower continues to display the second image with a 1.0F circular field of view (second field of view).

[0239] In the above Figure 16 Based on the workflow of the endoscopic image field of view display method shown, another field of view switching situation of the second display of the external imaging tower is as follows: Figure 17 As shown; Figure 17 In the absence of equipment switching, the second display on the external imaging tower shows a first image with a rectangular field of view (first field of view). Figure 17(The dashed rectangular area); when switching instruments, the second display on the external imaging tower switches to display a second image with a 1.0F circular field of view (second field of view); when the instrument switching is complete, the second display resumes displaying the first image with a rectangular field of view (first field of view). Figure 17 (The dashed rectangle portion).

[0240] from Figure 17 As shown in the diagram of the second display of the external imaging tower, when switching instruments, the second display of the external imaging tower switches to display a second image with a wider field of view. This allows for the identification and observation of instruments outside the first field of view, thus avoiding accidental injury during instrument switching.

[0241] Combining the instrument switching process of the surgical robot system, and applying some embodiments of the present invention, one practical application scenario is as follows: After determining the instrument switching requirement, the assistant unlocks the instrument to be disassembled on the surgical robotic arm and then disassembles the instrument. The second display of the external imaging tower switches to display a second image with a second field of view, expanding the field of view to assist the assistant in completing the instrument switching; during the process of the assistant disassembling the instrument and reinstalling the new instrument, the second display continues to display the second image with the second field of view; after the assistant completes the instrument installation and relocks the instrument, the second display resumes displaying the first image with the first field of view; throughout the entire process, the first display on the doctor's operating table always maintains the display of the first image with the first field of view, without affecting or interfering with the display content on the doctor's operating table.

[0242] See Figure 18 This illustrates a sixth flowchart of an endoscopic image field of view display method provided in an embodiment of the present invention. Figure 18 Includes the following steps:

[0243] Step S1801: Acquire endoscopic images captured by the endoscope of the surgical robot system.

[0244] In the specific implementation of step S1801, during the operation, an endoscopic image is acquired by the endoscope (e.g., stereoscopic endoscope) of the surgical robot system. The endoscope is equipped with an image sensor, and the endoscopic image is acquired by the image sensor.

[0245] Step S1802: Display a first image with a first field of view based on the endoscopic image.

[0246] In the specific implementation step S1802, a second image with a second field of view is selected from the endoscope image. This second field of view is smaller than the light-sensitive range of the endoscope's image sensor.

[0247] Select a first image with a first field of view from a second image having a second field of view, wherein the second field of view is larger than the first field of view.

[0248] After obtaining the first image with the first field of view, the first image with the first field of view is displayed.

[0249] In some specific embodiments, since the second field of view is larger than the first field of view, and the first image having the first field of view is selected from the second image having the second field of view, the content of the second image having the second field of view includes the content of the first image having the first field of view.

[0250] In other specific embodiments, the first field of view can be rectangular (or other shapes), and the second field of view can be circular (or other shapes).

[0251] Step S1803: When instrument switching is detected, a second image with a second field of view is switched to be displayed, and the second field of view is larger than the first field of view.

[0252] In the specific implementation step S1803, when instrument switching is detected, the display of a first image with a first field of view is switched to the display of a second image with a second field of view, thereby expanding the field of view.

[0253] In some specific embodiments, when a user-triggered operation to characterize device switching is detected, device switching is determined, and a second image with a second field of view is displayed.

[0254] In other specific embodiments, after switching to display a second image with a second field of view, when the device switching is detected to be complete, the display of a first image with a first field of view is restored.

[0255] Specifically, when a user-triggered operation is detected to indicate the completion of device switching, the device switching is determined to be complete, and the first image with the first field of view is restored.

[0256] In some specific embodiments, the surgical robot system includes a first display set on the doctor's operating table and a second display set on an external imaging tower; therefore, the specific implementation of the above steps S1802 and S1803 is as follows.

[0257] Based on the endoscopic image, the first image with a first field of view is displayed on the first display and the second display respectively; when instrument switching is detected, only the second display switches to display the second image with a second field of view.

[0258] In other words, a first image with a first field of view is displayed on the first display and the second display respectively; when instrument switching is detected, only the second display set on the external imaging tower switches to display the second image with a second field of view, while the first display set on the doctor's operating table maintains the display of the first image with the first field of view. This allows the assistant to observe the position of the instrument end through the second image with a larger field of view to avoid accidental injury, and also keeps the surgeon's field of view unchanged (i.e., does not affect or interfere with the display content on the doctor's operating table).

[0259] See Figure 19 This illustrates a seventh flowchart of an endoscopic image field of view display method provided by an embodiment of the present invention. Figure 19 Includes the following steps:

[0260] Step S1901: Acquire endoscopic images.

[0261] In the specific implementation of step S1901, during the operation, an endoscopic image is acquired by an endoscope (e.g., a stereo endoscope). The endoscope is equipped with an image sensor, and the endoscopic image is acquired by the image sensor.

[0262] Step S1902: Display a first image with a first field of view based on the endoscopic image.

[0263] In the specific implementation step S1902, a second image with a second field of view is selected from the endoscope image. This second field of view is smaller than the light-sensitive range of the image sensor of the endoscope.

[0264] Select a first image with a first field of view from a second image having a second field of view, wherein the second field of view is larger than the first field of view.

[0265] After obtaining the first image with the first field of view, the first image with the first field of view is displayed.

[0266] Step S1903: When a first specific event is detected, switch to display a second image with a second field of view, the second field of view being larger than the first field of view.

[0267] In the specific implementation step S1903, when a first specific event is detected, the display of a first image with a first field of view is switched to the display of a second image with a second field of view, thereby expanding the field of view.

[0268] In some specific embodiments, the first specific event is "instrument switching" or "operation to activate the field of view switching function"; after displaying the first image with the first field of view area, when instrument switching is detected, or when an operation triggered by the user to activate the field of view switching function is detected, the second image with the second field of view area is switched to be displayed.

[0269] For example, users can activate the field-of-view switching function on the doctor's console or external imaging tower to switch to displaying a second image with a second field of view, that is, users can actively switch to displaying a second image with a second field of view.

[0270] In some specific embodiments, after switching to display a second image with a second field of view, when a second specific event is detected, the display of a first image with a first field of view is restored.

[0271] The second specific event is either "completing device switching" or "operation to turn off the field of view switching function"; after switching to display a second image with a second field of view area, when device switching is detected as complete, or when user-triggered operation to turn off the field of view switching function is detected, the first image with a first field of view area is restored to display.

[0272] For example, after switching to display a second image with a second field of view, the user can turn off the field of view switching function on the doctor's console or external imaging tower, thereby restoring the display of the first image with the first field of view. In other words, the user can actively restore the display of the first image with the first field of view.

[0273] The activation of the functions corresponding to the first and second specific events can be achieved by triggering physical or virtual buttons, or by automatically detecting certain related actions.

[0274] See Figure 20 This diagram illustrates an eighth flowchart of an endoscopic image field of view display method for a surgical robot system according to an embodiment of the present invention. The surgical robot system includes at least a first display mounted on the surgeon's workstation and a second display mounted on an external imaging tower. Figure 20 Includes the following steps:

[0275] Step S2001: Acquire endoscopic images captured by the endoscope of the surgical robot system.

[0276] In the specific implementation of step S2001, during the operation, endoscopic images are acquired by the endoscope (e.g., stereoscopic endoscope) of the surgical robot system (e.g., laparoscopic surgical robot system). The endoscope is equipped with an image sensor, and the endoscopic images are acquired by the image sensor.

[0277] Step S2002: Based on the endoscopic image, display the first image with the first field of view on the first display and the second display respectively.

[0278] In the specific implementation step S2002, a second image with a second field of view is selected from the endoscope image. This second field of view is smaller than the light-sensitive range of the endoscope's image sensor.

[0279] Select a first image with a first field of view from a second image having a second field of view, wherein the second field of view is larger than the first field of view.

[0280] After obtaining a first image with a first field of view, the first image with the first field of view is displayed on a first display and a second display, respectively.

[0281] Step S2003: When instrument switching is detected, perform instrument switching auxiliary operation.

[0282] In the specific implementation of step S2003, when instrument switching is detected, an instrument switching auxiliary operation is executed to assist in the instrument switching.

[0283] Specifically, the instrument switching auxiliary operation is performed as follows: when an instrument is detected on the unlocked surgical robotic arm, an instrument presence detection is triggered to check whether the instrument is in place; when an instrument is detected to be out of place (indicating that the instrument has been removed), a first prompt message is displayed on the first and second displays to indicate that the instrument has been removed; for example, when an instrument is detected to be out of place, the first prompt message "Instrument removed" is displayed in the bottom status bar of the first and second displays.

[0284] After the first prompt message is displayed, when it is detected that the instrument on the surgical robotic arm is locked and that the instrument is in place, a second prompt message is displayed on the first and second displays to indicate that the instrument is installed in place; for example: after the first prompt message is displayed, when it is detected that the instrument on the surgical robotic arm is locked and that the instrument is in place, the second prompt message "Instrument is in place" is displayed in the bottom status bar of the first and second displays.

[0285] Step S2004: When a user-triggered operation to activate the field-of-view switching function is detected, the second display is switched to display a second image with a second field-of-view area, which is larger than the first field-of-view area.

[0286] In the specific implementation of step S2004, after the first display and the second display show the first image with the first field of view area, if the operation triggered by the user to start the field of view switching function is detected, the second display will switch to show the second image with the second field of view area, that is, the user can actively switch to show the second image with the second field of view area to expand the field of view.

[0287] Accordingly, after switching the second display to show the second image with the second field of view, when an operation triggered by the user to turn off the field of view switching function is detected, the second display will restore the display to show the first image with the first field of view.

[0288] Combining the surgical robot system's instrument switching process, and applying some embodiments of the present invention, one practical application scenario is as follows: After determining the instrument switching requirement, the assistant unlocks the instrument to be disassembled on the surgical robotic arm and then disassembles the instrument, triggering an instrument presence detection to check whether the instrument is in place; when the instrument is detected to be out of place (indicating that the instrument has been disassembled) and the operation of activating the field of view switching function is detected, a first prompt message is displayed on the first and second displays to indicate that the instrument has been disassembled, and the second display of the external imaging tower is switched to display a second image with a second field of view area to expand the field of view range to assist the assistant in completing the instrument switching.

[0289] During the process of the assistant disassembling and reinstalling the new instrument, the second display continuously shows a second image with a second field of view. After the assistant completes the instrument installation and locks the instrument again, when the instrument is detected in place and the operation of turning off the field of view switching function is detected, a second prompt message is displayed on the first and second displays to indicate that the instrument is installed in place, and the second display is restored to displaying the first image with a first field of view. Throughout the entire process, the first display on the doctor's operating table always maintains the display of the first image with a first field of view, without affecting or interfering with the display content on the doctor's operating table.

[0290] The above is a partial description of the endoscopic image field of view display method. Through the content of the above embodiments, it is possible to expand the field of view during surgery.

[0291] In summary, this solution proposes an endoscopic image field of view display method and a surgical robot system. When the surgical robot system detects the switching of instruments, it moves the endoscope to capture an endoscopic image with a larger field of view. Based on the endoscopic image captured after the movement, it switches to display an image with a larger field of view to assist the user in observing the position of the instrument end in real time, avoiding accidental contact between the instrument and the human body, reducing the risk of accidental injury, and improving the user experience.

[0292] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0293] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0294] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for displaying endoscopic images in a surgical robot system, wherein the surgical robot system includes at least a first display mounted on a surgeon's worktable and a second display mounted on an external imaging tower; characterized in that, The method includes: Acquire the first endoscopic image captured by the endoscope of the surgical robot system at the first position; Based on the first endoscopic image, a first image with a first field of view is displayed on the first display and the second display, respectively; When instrument switching is detected, the endoscope of the surgical robot system is controlled to move to a second position and a second endoscopic image is captured; the field of view of the second endoscopic image captured by the endoscope of the surgical robot system at the second position is larger than the field of view of the first endoscopic image captured at the first position. Based on the second endoscopic image, the second display is switched to show a second image with a second field of view, which is larger than the first field of view.

2. The method according to claim 1, characterized in that, When instrument switching is detected, after controlling the endoscope of the surgical robot system to move to the second position and capturing a second endoscopic image, the system further includes: maintaining the first display on the first screen to show a third image with the first field of view based on the second endoscopic image.

3. The method according to claim 2, characterized in that, Maintaining the first display of a third image having the first field of view based on the second endoscopic image; including: The second endoscopic image is magnified and cropped to obtain the third image having the first field of view area, and then displayed on the first display.

4. The method according to any one of claims 1-3, characterized in that, When instrument switching is detected, the endoscope of the surgical robot system is moved to a second position, and a second endoscopic image is captured, including: When a user-triggered operation characterizing device switching is detected, device switching is determined; The endoscope of the surgical robot system is controlled to move to the second position, and a second endoscopic image is captured.

5. The method according to any one of claims 1-4, characterized in that, Also includes: When the instrument switching is detected to be complete, the endoscope of the surgical robot system is controlled to move back to the first position and the first endoscopic image captured at the first position is acquired; based on the first endoscopic image, the first image with the first field of view is restored and displayed on the first display and the second display respectively.

6. The method according to claim 5, characterized in that, Upon detection of completed instrument switching, the endoscope of the surgical robot system is controlled to move back to the first position, and the first endoscopic image captured at the first position is acquired, including: When a user-triggered operation indicating the completion of device switching is detected, it is determined that the device switching is complete. The endoscope of the surgical robot system is controlled to move back to the first position, and the first endoscopic image captured at the first position is obtained.

7. The method according to any one of claims 1-6, characterized in that, The content of the second image in the second field of view includes the content of the first image in the first field of view.

8. The method according to any one of claims 1-7, characterized in that, The first field of view is rectangular, and the second field of view is circular.

9. A method for displaying the field of view of an endoscopic image, characterized in that, The method includes: Acquire the first endoscopic image captured by the endoscope of the surgical robot system at the first position; A first image with a first field of view is displayed based on the first endoscopic image; When instrument switching is detected, the endoscope of the surgical robot system is controlled to move to a second position and a second endoscopic image is captured; the field of view of the second endoscopic image captured by the endoscope of the surgical robot system at the second position is larger than the field of view of the first endoscopic image captured at the first position. Based on the second endoscopic image, a second image with a second field of view is switched and displayed, the second field of view being larger than the first field of view.

10. The method according to claim 9, characterized in that, When instrument switching is detected, the endoscope of the surgical robot system is moved to a second position, and a second endoscopic image is captured, including: When a user-triggered operation characterizing device switching is detected, device switching is determined; The endoscope of the surgical robot system is controlled to move to the second position, and a second endoscopic image is captured.

11. The method according to claim 9 or 10, characterized in that, Also includes: When the instrument switching is detected to be complete, the endoscope of the surgical robot system is controlled to move back to the first position, and the first endoscopic image captured at the first position is acquired; the first image with the first field of view is restored and displayed based on the first endoscopic image.

12. The method according to claim 11, characterized in that, Upon detection of completed instrument switching, the endoscope of the surgical robot system is controlled to move back to the first position, and the first endoscopic image captured at the first position is acquired, including: When a user-triggered operation indicating the completion of device switching is detected, it is determined that the device switching is complete. The endoscope of the surgical robot system is controlled to move back to the first position, and the first endoscopic image captured at the first position is obtained.

13. The method according to any one of claims 9-12, characterized in that, The content of the second image in the second field of view includes the content of the first image in the first field of view.

14. The method according to any one of claims 9-13, characterized in that, The first field of view is rectangular, and the second field of view is circular.

15. The method according to any one of claims 9-14, characterized in that, The surgical robot system includes a first display mounted on the doctor's operating table and a second display mounted on an external imaging tower; A first image with a first field of view is displayed based on the first endoscopic image; When instrument switching is detected, the endoscope of the surgical robot system is moved to the second position and a second endoscopic image is captured. Based on the second endoscopic image, a second image with a second field of view is switched and displayed; include: Based on the first endoscopic image, the first image having the first field of view is displayed on the first display and the second display, respectively; When instrument switching is detected, the endoscope of the surgical robot system is controlled to move to the second position and a second endoscopic image is captured. Based on the second endoscopic image, only the second display switches to display the second image with the second field of view area.

16. The method according to claim 15, characterized in that, When instrument switching is detected, after controlling the endoscope of the surgical robot system to move to the second position and capturing a second endoscopic image, the system further includes: maintaining the first display on the first screen to show a third image with the first field of view based on the second endoscopic image.

17. The method according to claim 16, characterized in that, Maintaining the first display of a third image having the first field of view based on the second endoscopic image; including: The second endoscopic image is magnified and cropped to obtain the third image having the first field of view area, and then displayed on the first display.

18. A method for displaying the field of view of an endoscopic image, characterized in that, The method includes: Acquire the first endoscopic image captured by the endoscope at the first position; A first image with a first field of view is displayed based on the first endoscopic image; When a first specific event is detected, the endoscope is controlled to move to a second position and a second endoscopic image is captured; the field of view of the second endoscopic image captured by the endoscope at the second position is larger than the field of view of the first endoscopic image captured at the first position. Based on the second endoscopic image, a second image with a second field of view is switched and displayed, the second field of view being larger than the first field of view.

19. The method according to claim 18, characterized in that, Upon detecting a first specific event, controlling the endoscope to move to a second position includes: When instrument switching is detected, or when a user-triggered operation to activate the field of view switching function is detected, the endoscope is controlled to move to the second position.

20. The method according to claim 18 or 19, characterized in that, After switching to display a second image with a second field of view, the following is also included: Upon detecting a second specific event, the endoscope is controlled to move back to the first position and the first image with the first field of view is restored.

21. The method according to claim 17, characterized in that, Upon detecting a second specific event, controlling the endoscope to move back to the first position and restoring the display of the first image with the first field of view includes: When instrument switching is detected to be complete, or when a user-triggered operation to disable the field of view switching function is detected, the endoscope is controlled to move back to the first position and the first image with the first field of view area is restored.

22. A method for displaying endoscopic images in a surgical robot system, wherein the surgical robot system includes at least a first display mounted on a surgeon's worktable and a second display mounted on an external imaging tower; characterized in that, The method includes: Acquire the first endoscopic image captured by the endoscope of the surgical robot system at the first position; Based on the first endoscopic image, a first image with a first field of view is displayed on the first display and the second display, respectively; When a device change is detected, a device change assistance operation is performed; When a user-triggered operation to activate the field of view switching function is detected, the endoscope of the surgical robot system is controlled to move to the second position and capture a second endoscopic image; the field of view of the second endoscopic image captured by the endoscope of the surgical robot system at the second position is larger than the field of view of the first endoscopic image captured at the first position. Based on the second endoscopic image, the second display is switched to show a second image with a second field of view, which is larger than the first field of view.

23. A surgical robot system, characterized in that, The surgical robot system includes a processor and a memory; the memory includes computer-readable instructions stored thereon, which, when executed by the processor, implement the endoscopic image field of view display method according to any one of claims 1-22.

Images