Multi-image superposition display system and method, observation bin and doctor console

By using a multi-image overlay display system in the surgical robot system, the surgical environment and menu interface are synchronized by reflector and mirror technology, the transmission delay problem is solved and the smoothness and safety of surgical operations are improved.

CN118938498APending Publication Date: 2024-11-12SHANDONG WEIGAO SURGICAL ROBOT CO LTD

Patent Information

Application Number
CN202411280498.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-13
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

When the prior art adds menu options on the surgical video screen, the transmission delay of the surgical video screen increases, affecting the smoothness and safety of surgical operations.

Method used

Using a multi-image overlay display system, the surgical environment image and menu interface image are presented through the first and second display devices respectively, and light is transmitted using reflectors and mirrors, so that the two form superimposed virtual images in front of the eyepiece, avoiding the software synthesis method transmission delay.

Benefits of technology

It effectively reduces the transmission delay of surgical video images, improves the smoothness and safety of surgical operations, and ensures the synchronous display of surgical environment images and menu interface images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118938498A_ABST
    Figure CN118938498A_ABST
Patent Text Reader

Abstract

The invention provides a multi-image superposition display system and method, an observation bin and a doctor console, and relates to the technical field of image processing, the multi-image superposition display system comprises a first display device configured to present a first image; a first reflector arranged at an angle to the first display device such that the first reflector delivers the first image light from the first display device toward the second reflector; a second display device configured to present a second image such that the first reflector delivers second image light from the second display device toward the second reflector; and a second reflector configured to simultaneously receive the first image light and the second image light from the first reflector. According to the invention, the operation environment image and the menu interface image can be transmitted at the same time in the surgical operation scene, there is no successive problem, the transmission delay of the operation video picture can be effectively reduced, and the smoothness and safety of the operation are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multi-image overlay display system, method, observation chamber and doctor console. Background Art

[0002] A surgical robot system usually includes a master-hand console for the doctor to operate and a slave-hand console located on the patient's side. When using the surgical robot system to perform surgery, the surgeon sits at the remote master-hand console and observes the condition of the lesion area in real time through the display device of the master-hand console. The surgeon uses a remote control operation mode to manipulate the main arm mechanism at the master-hand console to control the surgical instruments on the slave-hand console to complete various surgical operations.

[0003] In minimally invasive surgery, the surgical video images captured by the 3D endoscopic imaging system are transmitted to the display device of the main operating table in real time for display, which can provide the surgeon with an immersive and real three-dimensional spatial experience. Generally, while providing the surgeon with the surgical video images, menu options will be superimposed on the endoscopic video images to facilitate the surgeon to control and change the surgical instruments and surgical methods.

[0004] In the prior art, software synthesis is usually used to add menu options to the surgical video screen. Compared with only displaying the surgical video screen, this method will increase the transmission delay of the entire surgical video screen, thereby reducing the smoothness of the surgical operation and affecting the overall safety of the operation. Summary of the invention

[0005] The purpose of the present application is to provide a multi-image overlay display system, method, observation chamber and doctor console, which can enable the surgical environment image and menu interface image to be transmitted simultaneously in a surgical operation scene without any order problem, and can effectively reduce the transmission delay of the surgical video screen and improve the fluency and safety of the surgical operation.

[0006] To achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0007] A multi-image overlay display system, comprising:

[0008] A first display device configured to present a first image;

[0009] a first reflector disposed at an angle to the first display device such that the first reflector transmits first image light from the first display device toward a second reflector;

[0010] a second display device configured to present a second image, the second display device being disposed at a primary virtual image position of the first display device with respect to the first reflector, such that the first reflector transmits second image light from the second display device toward the second reflector;

[0011] The second reflector is configured to simultaneously receive the first image light and the second image light from the first reflector, and to form a superimposed virtual image of the two in front of the eyepiece so that the observer's eyes can perceive them.

[0012] A multi-image overlay display system, comprising:

[0013] A first display device configured to present a surgical environment image of a surgical site;

[0014] a first reflector disposed at an angle to the first display device so that the first reflector transmits surgical environment image light from the first display device toward a second reflector;

[0015] a second display device configured to present a menu interface image including menu information, the second display device being disposed at a position of a primary virtual image of the first display device with respect to the first reflector, so that the first reflector transmits menu interface image light from the second display device toward the second reflector;

[0016] The second reflector is configured to simultaneously receive the surgical environment image light and the menu interface image light from the first reflector, and to form a superimposed virtual image of the two in front of the eyepiece so that the observer's eyes can perceive them.

[0017] A multi-image overlay display system comprises: two groups of optical components symmetrically arranged with respect to a vertical plane and having the same structure, each of the optical components comprising:

[0018] A first display device is configured to present a surgical environment image of a surgical site, wherein the surgical environment images corresponding to the two groups of the first display devices have a lateral parallax;

[0019] a first reflector disposed at an angle to the first display device so that the first reflector transmits surgical environment image light from the first display device toward a second reflector;

[0020] a second display device configured to present a menu interface image including menu information, the second display device being disposed at a position of a primary virtual image of the first display device with respect to the first reflector, so that the first reflector transmits menu interface image light from the second display device toward the second reflector;

[0021] The second reflector is configured to simultaneously receive the surgical environment image light and the menu interface image light from the first reflector, and to form a superimposed virtual image of the two in front of the eyepiece so that the observer's eyes can perceive them.

[0022] Furthermore, the first display device and the second display device are both configured as flat-panel display screens, and the display ranges of the two are set to be consistent.

[0023] Further, the first reflector is configured as a semi-transparent and semi-reflective mirror, for reflecting the first image light from the first display device and transmitting the second image light from the second display device;

[0024] And / or, the second reflector is configured as a mirror.

[0025] Furthermore, the first reflector is configured as a semi-transparent and semi-reflective mirror, and a ratio of transmittance to reflectivity of the first reflector ranges from 10:90 to 90:10.

[0026] Furthermore, the first display devices corresponding to the two groups of optical components are arranged on the same plane.

[0027] A multi-image overlay display method, the method comprising:

[0028] Acquire a surgical environment image, and display the surgical environment image;

[0029] Acquire a menu interface image, and display the menu interface image, wherein the surgical environment image and the menu interface image are displayed separately and independently;

[0030] Based on different light transmission paths and equal optical path lengths, the surgical environment image and the menu interface image are synchronously transmitted to form a superimposed virtual image of the two in front of the eyepiece that can be perceived by the observer's eyes.

[0031] Furthermore, based on different light transmission paths and equal optical path lengths, the step of synchronously transmitting the surgical environment image and the menu interface image to form a superimposed virtual image of the two in front of the eyepiece that can be perceived by the observer's eyes includes:

[0032] Setting a secondary light action plane, wherein the menu interface image is located at a primary virtual image position of the surgical environment image with respect to the primary light action plane;

[0033] One of the surgical environment image light and the menu interface image light is reflected at the primary light action plane, and the other is transmitted at the same time, and the reflected light and the transmitted light reach the secondary light action plane at the same time, and form the superimposed virtual image of the two in front of the eyepiece.

[0034] A multi-image overlay display method, the method comprising:

[0035] Acquire the left eye surgical environment image and display the left eye surgical environment image, and acquire the right eye surgical environment image and display the right eye surgical environment image;

[0036] Acquire the left-eye menu interface image and display the left-eye menu interface image, and acquire the right-eye menu interface image and display the right-eye menu interface image at the same time; wherein the left-eye and right-eye surgical environment images and the left-eye and right-eye menu interface images are displayed separately;

[0037] The left-eye surgical environment image and the right-eye surgical environment image are respectively reflected by the first reflector and then reach the second reflector;

[0038] The left-eye menu interface image and the right-eye menu interface image are respectively transmitted through the first reflector and then reach the second reflector;

[0039] The second reflector corresponding to the left eye simultaneously receives the left eye surgical environment image and the left eye menu interface image, and forms a left eye superimposed virtual image of the two in front of the eyepiece;

[0040] The second reflector corresponding to the right eye receives the right eye surgical environment image and the right eye menu interface image simultaneously, and forms a right eye superimposed virtual image of the two in front of the eyepiece.

[0041] An observation chamber, comprising: a housing and the aforementioned multi-image overlay display system;

[0042] The housing has a receiving space for receiving the multi-image overlay display system, and an eyepiece hole for installing an eyepiece is provided on the housing;

[0043] The first display device is close to the top surface of the housing and arranged parallel to the top surface.

[0044] A doctor's console, comprising: an image acquisition system and the observation chamber according to claim 11;

[0045] The image acquisition system is communicatively connected to the first display device and the second display device, and is configured to acquire a surgical environment image of the surgical scene and transmit it to the first display device, while acquiring status information of the robotic arm and / or the endoscopic imaging system and transmitting it to the second display device.

[0046] A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the aforementioned method.

[0047] The image overlay display system, method, observation chamber, and doctor console provided in the embodiments of the present application have at least the following beneficial effects:

[0048] A second display device is arranged at the position of the primary virtual image of the first display device with respect to the first reflector. Through the first reflector, on the one hand, the first image from the first display device can be transmitted, and on the other hand, the second image from the second display device can be transmitted. The present application adopts a light transmission method, which can make the first image and the second image reach the second reflector at the same time, and be reflected by the second reflector at the same time, forming a superimposed virtual image of the two in front of the eyepiece that can be perceived by the observer's eyes. In this process, since the first image and the second image are transmitted at the same time, there is no problem of displaying them in sequence, which can effectively reduce the transmission delay of the surgical video screen and improve the fluency and safety of the surgical operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A schematic diagram of the structure of a multi-image overlay display system provided in an embodiment of the present application;

[0051] Figure 2 A front view of a multi-image overlay display system provided in an embodiment of the present application;

[0052] Figure 3 An optical path diagram of a multi-image overlay display system provided in an embodiment of the present application;

[0053] Figure 4 One of the flow charts of the multi-image overlay display method provided in the embodiment of the present application;

[0054] Figure 5 The second flowchart of the multi-image overlay display method provided in the embodiment of the present application;

[0055] Figure 6 The third flowchart of the multi-image overlay display method provided in the embodiment of the present application;

[0056] Figure 7 A simplified flowchart of the multi-image overlay display method provided in an embodiment of the present application;

[0057] Figure 8 A schematic diagram of a menu interface image in an embodiment of the present application.

[0058] icon:

[0059] 10-Observe the image plane position;

[0060] 100 - first display screen; 200 - second display screen; 300 - semi-transparent and semi-reflective mirror; 400 - reflecting mirror. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0062] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0064] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the invention is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0065] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components are required to be absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0066] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0067] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0068] In general technology, the surgical robot mainly includes a master-hand operation table, a slave-hand operation table and an image processing trolley. The operator can operate the master-hand operation table at a remote end, and control the slave-hand operation table through the master-slave control relationship between the master-hand operation table and the slave-hand operation table. The image processing trolley can provide the operator with auxiliary surgical environment images, etc. The slave-hand operation table includes at least one image arm and an instrument arm, wherein the image arm is equipped with an endoscopic imaging system. During the operation, the endoscopic imaging system and the surgical instruments enter the patient's position through the wound on the patient's body respectively. The endoscopic imaging system is connected to the display device in communication. The endoscopic imaging system can obtain images of the surgical site (such as surgical instruments, human tissues, blood vessels, etc.), and the captured images are processed by the processor for display by the display device (configured to display images visible to the operator), for example, on the master-hand operation table or on another suitable display device located locally and / or remotely to display to the operator. The endoscopic imaging system can be a stereoscopic endoscope. In the case of using a stereoscopic endoscope, the captured images can be processed by the processor to present a coordinated stereoscopic image of the surgical site to the surgeon. During the operation, the instrument arm moves according to the movement of the main arm mechanism on the doctor's main hand operation table to achieve the corresponding surgical operation.

[0069] It can be understood that surgical instruments are coupled to the instrument arms, and the multiple instrument arms can be multiple identical surgical instruments and / or multiple different surgical instruments, such as forceps, clip appliers, clamps, retractors, cauterization instruments, suction instruments, suturing devices, anastomosis devices, cutting devices, etc.; similarly, in the case of multiple endoscopic imaging systems, the multiple endoscopic imaging systems can be multiple identical and / or multiple different endoscopic imaging systems.

[0070] In this embodiment, the main hand operating table includes an observation chamber, and a left eyepiece hole and a right eyepiece hole are provided on the front of the observation chamber. A left eyepiece viewer is provided at the left eyepiece hole, and a right eyepiece viewer is provided at the right eyepiece hole. Both the left eye viewer and the right eye viewer include optical lenses. The doctor can view the three-dimensional view of the surgical scene that can achieve depth perception presented to the doctor by the observation chamber through the left and right eye viewers.

[0071] The present application provides a multi-image overlay display system, comprising: a first display device, configured to present a first image; a first reflector, arranged at an angle to the first display device, so that the first reflector transmits the first image light from the first display device toward the second reflector; a second display device, configured to present a second image, the second display device being arranged at a primary virtual image position of the first display device relative to the first reflector, so that the first reflector transmits the second image light from the second display device toward the second reflector; and the second reflector, configured to simultaneously receive the first image light and the second image light from the first reflector, and to form a superimposed virtual image of the two in front of the eyepiece so that the observer's eyes can perceive them.

[0072] It should be noted that the front of the eyepiece refers to the direction from the eyepiece away from the operator when the operator sits facing the main hand console, and the direction from the eyepiece towards the operator is the front, and the direction from the eyepiece towards the operator is the back.

[0073] The above scheme can simultaneously superimpose the first image and the second image to form a superimposed virtual image of the two for the observer to watch, wherein the first image and the second image can be different types of images, which can be appropriately set according to the usage scenario. For example, in a surgical operation scenario, the first image can be a surgical environment image showing the surgical site, and the second image can be a menu interface image showing the instrument arm, the state of the surgical instrument (opening and closing state), and the surgical instrument information (name, etc.), such as the rotation angle of the endoscope imaging system, the cutting and coagulation state of the instrument, the master-slave hand switching state, etc.

[0074] Furthermore, in order to display a three-dimensional scene, the multi-image overlay display system needs to be equipped with two sets of optical components. Please continue to refer to Figure 1 The two groups of optical components are symmetrically arranged relative to the vertical plane and have the same structure, and the first images corresponding to the two groups of first display devices have horizontal parallax.

[0075] Exemplarily, the first display device and the second display device may both be flat screens, such as liquid crystal screens, plasma screens, touch screens, etc., or may be other display devices capable of displaying images, such as projectors, etc.; Figure 3The first reflector is configured as a semi-transparent and semi-reflective mirror 300, that is, it can both reflect and transmit light. Specifically, it can reflect the first image light from the first display device and transmit the second image light from the second display device; the second reflector is configured as a reflector 400 for reflecting light.

[0076] In this embodiment, the ratio of transmittance to reflectivity of the first reflector ranges from 10:90 to 90:10. Exemplarily, the ratio between the two can be 20:80, 30:70, 40:60, 50:50, 60:40, 70:30 or 80:20.

[0077] Specifically, refer to Figure 1 and Figure 2 The multi-image overlay display system of this embodiment includes two first display screens 100, two second display screens 200, two semi-transparent and semi-reflective mirrors 300 and two reflecting mirrors 400, wherein the semi-transparent and semi-reflective mirrors 300 are arranged at an angle with the corresponding first display screens 100, the reflecting mirrors 400 are arranged at an angle with the corresponding semi-transparent and semi-reflective mirrors 300, and the second display screens 200 are arranged at the primary virtual image position of the corresponding first display screen 100 with respect to the corresponding semi-transparent and semi-reflective mirrors 300.

[0078] In a surgical operation scene, after the surgical environment image is displayed on the corresponding first display screen 100, it is reflected by the corresponding semi-transparent and semi-reflective mirror 300 and the reflective mirror 400 in sequence to reach the observation image plane position 10 to form a virtual image of the surgical environment. At the same time, after the menu interface image is displayed on the corresponding second display screen 200, it is transmitted by the corresponding semi-transparent and semi-reflective mirror 300 and then reflected by the reflective mirror 400 to reach the observation image plane position 10 to form a virtual image of the menu interface, and forms a superimposed image effect with the virtual image of the surgical environment.

[0079] In other embodiments, the first display screen 100 may be used to display the menu interface image, and the second display screen 200 may be used to display the surgical environment image. The specific principle is the same as above and will not be repeated here.

[0080] In this embodiment, two first display screens 100 are arranged at intervals on the left and right to receive two images with lateral parallax, namely, a left-eye surgical environment image and a right-eye surgical environment image; two semi-transparent and semi-reflective mirrors 300 are arranged in a one-to-one correspondence with the two first display screens 100.

[0081] In a specific application, after the left-eye surgical environment image is displayed on the first display screen 100 on the left, it is reflected by the semi-transparent and semi-reflective mirror 300 on the left and the reflective mirror 400 on the left in sequence to reach the observation image plane position 10 to form a left-eye surgical environment virtual image; after the left-eye menu interface image is displayed on the second display screen 200 on the left, it is transmitted by the semi-transparent and semi-reflective mirror 300 on the left, and then reflected by the reflective mirror 400 on the left to reach the observation image plane position 10 to form a left-eye menu interface virtual image, and is superimposed with the left-eye surgical environment virtual image to form a left-eye superimposed virtual image. Similarly, after the right eye surgical environment image is displayed on the right first display screen 100, it is reflected by the right half-transparent mirror 300 and the right reflector 400 in sequence to reach the observation image plane position 10 to form the right eye surgical environment virtual image; after the right eye menu interface image is displayed on the right second display screen 200, it is transmitted by the right half-transparent mirror 300 and then reflected by the right reflector 400 to reach the observation image plane position 10 to form the right eye menu interface virtual image, and is superimposed with the right eye surgical environment virtual image to form the right eye superimposed virtual image. The aforementioned left eye superimposed virtual image and right eye superimposed virtual image are synthesized at the observation image plane position 10.

[0082] When the operator's binoculars are aligned with the left and right viewers of the observation chamber, a synthetic stereoscopic image visible to the operator can be viewed. The stereoscopic image includes a menu interface image in the surgical environment image to provide the operator with an intuitive, stereoscopic image display.

[0083] Reference Figure 4 This embodiment also provides a multi-image overlay display method, which specifically includes the following steps:

[0084] Step S11, acquiring a surgical environment image and displaying the surgical environment image.

[0085] In this embodiment, the surgical environment image of the surgical site can be collected by the endoscopic imaging system, and after the image is processed, it is transmitted to the corresponding first display screen 100 through a signal line and displayed on the first display screen 100.

[0086] Step S12, obtaining a menu interface image, and displaying the menu interface image, wherein the surgical environment image and the menu interface image are displayed separately and independently.

[0087] In this embodiment, a signal acquisition sensor may be provided at corresponding positions of the endoscope imaging system, the instrument arm, etc., and the signal data collected by the signal acquisition sensor may be transmitted to the corresponding second display screen 200 in the form of a menu interface image and displayed on the second display screen 200.

[0088] Step S13, based on different light transmission paths and equal optical path lengths, the surgical environment image and the menu interface image are synchronously transmitted to form a superimposed virtual image of the two in front of the eyepiece that can be perceived by the observer's eyes.

[0089] The multi-image overlay display method of this embodiment does not need to adopt software synthesis. The surgical environment image and the menu interface image are displayed separately and independently, and based on different light transmission paths and equal optical path lengths, the surgical environment image and the menu interface image are synchronously transmitted. During the synchronous transmission process, the transmission delay of the surgical environment image is not increased, and a superimposed virtual image of the two is formed in front of the eyepiece so that the observer's eyes can perceive it.

[0090] Further, see Figure 5 The step of synchronously transmitting the surgical environment image and the menu interface image based on different light transmission paths and equal optical path lengths to form a superimposed virtual image of the two in front of the eyepiece that can be perceived by the observer's eyes includes:

[0091] Step S131, setting a secondary light action plane, wherein the menu interface image is located at a primary virtual image position of the surgical environment image with respect to a primary light action plane.

[0092] The aforementioned light action plane can be regarded as a transmission surface or a reflection surface to realize the transmission of light. The menu interface image is set at the position of the primary virtual image of the surgical environment image with respect to the primary light action plane, so that the optical path length from the surgical environment image to the primary light action plane is equal to the optical path length from the menu interface image to the primary light action plane, and the two are different light transmission paths.

[0093] Step S132, one of the surgical environment image light and the menu interface image light is reflected at the primary light action plane, and the other is transmitted at the same time, and the reflected light and the transmitted light reach the secondary light action plane at the same time, and form a superimposed virtual image of the two in front of the eyepiece.

[0094] Specifically, when the surgical environment image light passes through the primary light action plane, the primary light action plane can reflect it and reflect it to the secondary light action plane. At the same time, when the menu interface image light passes through the primary light action plane, the primary light action plane can transmit it and transmit it to the secondary light action plane.

[0095] As an alternative, when the surgical environment image light passes through the primary light action plane, the primary light action plane can transmit it and transmit it to the secondary light action plane. At the same time, when the menu interface image light passes through the primary light action plane, the primary light action plane can reflect it and reflect it to the secondary light action plane. Based on the above method, more specific implementation steps include:

[0096] Step 21, acquiring a left-eye surgical environment image and displaying the left-eye surgical environment image, and acquiring a right-eye surgical environment image and displaying the right-eye surgical environment image at the same time.

[0097] When acquiring the left and right eye surgical environment images, they can be acquired separately through two CMOS sensors, or simultaneously through one CMOS sensor. The two CMOS sensors or one CMOS sensor are communicatively connected with the corresponding processor, and the images are processed and then transmitted. Among them, the images displayed by the left eye surgical environment image and the right eye surgical environment image have lateral parallax.

[0098] Step 22, obtain the left-eye menu interface image and display the left-eye menu interface image, and simultaneously obtain the right-eye menu interface image and display the right-eye menu interface image; wherein the left-eye and right-eye surgical environment images and the left-eye and right-eye menu interface images are displayed independently and separately.

[0099] When acquiring the left and right menu interface images, image information can be collected by the same sensor and simultaneously transmitted to different second display screens 200 for display, wherein the images displayed by the left menu interface image and the right menu interface image are the same menu interface.

[0100] Step 23: the left-eye surgical environment image and the right-eye surgical environment image are respectively reflected by the first reflector and then reach the second reflector.

[0101] There are two first reflectors, which are identically configured to receive the left-eye surgical environment image or the right-eye surgical environment image respectively.

[0102] Step 24, the left-eye menu interface image and the right-eye menu interface image are respectively transmitted through the first reflector and then reach the second reflector.

[0103] There are two first reflectors, which are identically configured to receive the left-eye menu interface image or the right-eye menu interface image respectively.

[0104] Step 25, the second reflector corresponding to the left eye receives the left eye surgical environment image and the left eye menu interface image simultaneously, and forms a left eye superimposed virtual image of the two in front of the eyepiece.

[0105] The display range size of the left-eye surgical environment image and the left-eye menu interface image is the same, so as to ensure that the menu interface is at the intended designated position on the screen finally seen by the human eye.

[0106] Step 26, the second reflector corresponding to the right eye receives the right eye surgical environment image and the right eye menu interface image simultaneously, and forms a right eye superimposed virtual image of the two in front of the eyepiece.

[0107] Similarly, the display range size of the right-eye surgical environment image and the right-eye menu interface image is the same to ensure that the menu interface is at the intended designated position on the screen finally seen by the human eye.

[0108] It should be noted that in order to ensure that the display range sizes of the surgical environment image and the menu interface image are the same, the display range size of the first display screen 100 needs to be set to be consistent with the display range size of the second display screen 200.

[0109] Combination Figure 1 , Figure 3 and Figure 7 , the left-eye surgical environment image is reflected by the left semi-transparent mirror 300 and the left-side reflective mirror 400 in turn to reach the observation image plane position 10 to form the left-eye surgical environment virtual image; the left-eye menu interface image is transmitted by the left semi-transparent mirror 300, and then reflected by the left-side reflective mirror 400 to reach the observation image plane position 10 to form the left-eye menu interface virtual image, and is superimposed with the left-eye surgical environment virtual image to form the left-eye superimposed virtual image. Similarly, the right-eye surgical environment image is reflected by the right semi-transparent mirror 300 and the right-side reflective mirror 400 in turn to reach the observation image plane position 10 to form the right-eye surgical environment virtual image; the right-eye menu interface image is transmitted by the right semi-transparent mirror 300, and then reflected by the right-side reflective mirror 400 to reach the observation image plane position 10 to form the right-eye menu interface virtual image, and is superimposed with the right-eye surgical environment virtual image to form the right-eye superimposed virtual image. The aforementioned left-eye superimposed virtual image and right-eye superimposed virtual image are combined at the observation image plane position 10 to form a stereoscopic image visible to the operator.

[0110] Specifically, refer to Figure 8 , which shows a synthesized menu image after the left-eye menu interface image and the right-eye menu interface image are synthesized; the synthesized menu image includes multiple information fields, which can realize the status information of the instrument arm and the image arm, including: instrument name, instrument installation and connection status, left and right position status, power-on status, pause status, endoscope status, moving status and fault status, etc.; wherein, the blank area in the figure is used to display the dynamic picture collected by the endoscope imaging system, that is, the surgical environment image, and the distal part of the surgical instrument corresponding to multiple instrument arms.

[0111] It should be noted that surgical instruments are not limited to forceps, clip applicators, clamps, retractors, cauterization instruments, suction instruments, suturing devices, anastomosis devices, cutting devices, etc. The information field corresponds to the surgical instrument and includes the name of the surgical instrument, which can be any name listed above for the surgical instrument.

[0112] In this embodiment, the information field is displayed around the periphery of the surgical scene image; the information field may be superimposed above or below the surgical scene image or may be outside the field of view and border the peripheral edge of the field of view. Additionally or alternatively, the information field may be displayed at a position very close to the surgical instrument corresponding thereto. In other embodiments, the information field may be displayed along the same peripheral edge of the surgical scene image, for example, along the bottom edge, the left edge, the right edge, or the top edge.

[0113] An embodiment of the present application also provides an observation chamber, comprising: a shell and the multi-image overlay display system in the aforementioned embodiment; the shell has a storage space for accommodating the multi-image overlay display system, and an eyepiece hole for installing an eyepiece is opened on the shell; the first display device is arranged close to the top surface of the shell and parallel to the top surface. Such an arrangement is conducive to the layout of the multi-image overlay display system inside the shell, and also facilitates the installation and fixation of the multi-image overlay display system.

[0114] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above method. The specific implementation can be found in the aforementioned method embodiment, which will not be repeated here.

[0115] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0116] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present application.

[0117] The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-image overlay display system, characterized in that: include: A first display device configured to present a first image; a first reflector disposed at an angle to the first display device such that the first reflector transmits first image light from the first display device toward a second reflector; a second display device configured to present a second image, the second display device being disposed at a primary virtual image position of the first display device with respect to the first reflector, such that the first reflector transmits second image light from the second display device toward the second reflector; The second reflector is configured to simultaneously receive the first image light and the second image light from the first reflector, and to form a superimposed virtual image of the two in front of the eyepiece so that the observer's eyes can perceive them.

2. A multi-image overlay display system, characterized in that: include: A first display device configured to present a surgical environment image of a surgical site; a first reflector disposed at an angle to the first display device so that the first reflector transmits surgical environment image light from the first display device toward a second reflector; a second display device configured to present a menu interface image including menu information, the second display device being disposed at a position of a primary virtual image of the first display device with respect to the first reflector, so that the first reflector transmits menu interface image light from the second display device toward the second reflector; The second reflector is configured to simultaneously receive the surgical environment image light and the menu interface image light from the first reflector, and to form a superimposed virtual image of the two in front of the eyepiece so that the observer's eyes can perceive them.

3. A multi-image overlay display system, characterized in that: include: Two groups of optical components arranged symmetrically with respect to a vertical plane and having the same structure, each of the optical components comprising: A first display device is configured to present a surgical environment image of a surgical site, wherein the surgical environment images corresponding to the two groups of the first display devices have a lateral parallax; a first reflector disposed at an angle to the first display device so that the first reflector transmits surgical environment image light from the first display device toward a second reflector; a second display device configured to present a menu interface image including menu information, the second display device being disposed at a position of a primary virtual image of the first display device with respect to the first reflector, so that the first reflector transmits menu interface image light from the second display device toward the second reflector; The second reflector is configured to simultaneously receive the surgical environment image light and the menu interface image light from the first reflector, and to form a superimposed virtual image of the two in front of the eyepiece so that the observer's eyes can perceive them.

4. The multi-image overlay display system according to any one of claims 1 to 3, characterized in that: The first display device and the second display device are both configured as flat-panel display screens, and the display ranges of the two are set to be consistent.

5. The multi-image overlay display system according to any one of claims 1 to 3, characterized in that: The first reflector is configured as a semi-transparent and semi-reflective mirror for reflecting the first image light from the first display device and transmitting the second image light from the second display device; And / or, the second reflector is configured as a mirror.

6. The multi-image overlay display system according to claims 1-3, characterized in that: The first reflector is configured as a semi-transparent and semi-reflective mirror, and a ratio of transmittance to reflectivity of the first reflector ranges from 10:90 to 90:

10.

7. The multi-image overlay display system according to claim 3, characterized in that: The first display devices corresponding to the two groups of optical components are arranged on the same plane.

8. A multi-image overlay display method, characterized in that: The method comprises: Acquire a surgical environment image, and display the surgical environment image; Acquire a menu interface image, and display the menu interface image, wherein the surgical environment image and the menu interface image are displayed separately and independently; Based on different light transmission paths and equal optical path lengths, the surgical environment image and the menu interface image are synchronously transmitted to form a superimposed virtual image of the two in front of the eyepiece that can be perceived by the observer's eyes.

9. The multi-image overlay display method according to claim 8, characterized in that: Based on different light transmission paths and equal optical path lengths, the step of synchronously transmitting the surgical environment image and the menu interface image to form a superimposed virtual image of the two in front of the eyepiece that can be perceived by the eyes of the observer includes: Setting a secondary light action plane, wherein the menu interface image is located at a primary virtual image position of the surgical environment image with respect to the primary light action plane; One of the surgical environment image light and the menu interface image light is reflected at the primary light action plane, and the other is transmitted at the same time, and the reflected light and the transmitted light reach the secondary light action plane at the same time, and form the superimposed virtual image of the two in front of the eyepiece.

10. A multi-image overlay display method, characterized in that: The method comprises: Acquire the left eye surgical environment image and display the left eye surgical environment image, and acquire the right eye surgical environment image and display the right eye surgical environment image; Acquire the left-eye menu interface image and display the left-eye menu interface image, and acquire the right-eye menu interface image and display the right-eye menu interface image at the same time; wherein the left-eye and right-eye surgical environment images and the left-eye and right-eye menu interface images are displayed separately; The left-eye surgical environment image and the right-eye surgical environment image are respectively reflected by the first reflector and then reach the second reflector; The left-eye menu interface image and the right-eye menu interface image are respectively transmitted through the first reflector and then reach the second reflector; The second reflector corresponding to the left eye simultaneously receives the left eye surgical environment image and the left eye menu interface image, and forms a left eye superimposed virtual image of the two in front of the eyepiece; The second reflector corresponding to the right eye receives the right eye surgical environment image and the right eye menu interface image simultaneously, and forms a right eye superimposed virtual image of the two in front of the eyepiece.

11. An observation chamber, characterized in that: include: A housing and a multi-image overlay display system according to any one of claims 1 to 7; The housing has a receiving space for receiving the multi-image overlay display system, and an eyepiece hole for installing an eyepiece is provided on the housing; The first display device is close to the top surface of the housing and arranged parallel to the top surface.

12. A doctor's console, characterized in that: include: An image acquisition system and an observation chamber as claimed in claim 11; The image acquisition system is communicatively connected to the first display device and the second display device, and is configured to acquire a surgical environment image of the surgical scene and transmit it to the first display device, while acquiring status information of the robotic arm and / or the endoscopic imaging system and transmitting it to the second display device.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 8 to 10.

Citation Information

Patent Citations

  • Displaying device and surgical robot

    CN110101455A

  • Virtual reality display equipment, picture presentation method and device and storage medium

    CN113419350A

  • System and method for navigating on-screen menus in teleoperation medical systems

    CN115426965A

  • Surgical visualizations systems and displays

    US20160220324A1

Cited By

  • Multi-image overlay display system and method, observation chamber, and surgeon console

    WO2026056735A1