Multi-image overlay display system and method, observation chamber, and surgeon console

Abstract

A multi-image overlay display system and method, an observation chamber, and a surgeon console, relating to the technical field of image processing. The multi-image overlay display system comprises: first display devices (100), configured to present a first image; first reflectors (300), respectively arranged at an angle relative to the corresponding first display devices (100), such that the first reflectors (300) transmit, respectively towards second reflectors (400), first image light from the first display devices (100); second display devices (200), configured to present a second image, such that the first reflectors (300) transmit, respectively towards the second reflectors (400), second image light from the second display devices (200); and the second reflectors (400), configured to simultaneously receive both the first image light and the second image light from the first reflectors (300). In a surgical scenario, the multi-image overlay display system enables simultaneous transmission of both a surgical environment image and a menu interface image, eliminating any problem related to sequential processing. This effectively reduces the transmission delay of surgical video streams, thereby improving the smoothness and safety of surgical operations.

Description

Multi-image overlay display system, method, viewing pod and surgeon console TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a multi-image overlay display system, method, viewing pod and surgeon console. BACKGROUND

[0002] Surgical robot systems generally include a master console for a surgeon to operate and a slave console located on the patient side. When a surgical robot system is used for surgery, a surgical operator sits beside the master console at a remote end, observes the situation of a lesion area in real time through a display device of the master console, adopts a remote control operation mode, and controls a master arm mechanism at the master console to control surgical instruments on the slave console to complete various surgical operations.

[0003] In minimally invasive surgery, a surgical video picture taken by a 3D endoscope imaging system is transmitted to a display device of the master console in real time for display, which can provide the surgical operator with a real and stereoscopic spatial experience. Generally, while providing the surgical operator with the surgical video picture, menu options are superimposed on the endoscope video picture to facilitate the surgical operator to control and change the surgical instruments and the surgical method.

[0004] In the prior art, a software synthesis method is generally used to add menu options to the surgical video picture. Compared with only displaying the surgical video picture, this method increases the transmission delay of the entire surgical video picture, thereby reducing the smoothness of the surgical operation and affecting the safety of the overall surgery. SUMMARY

[0005] The present application aims to provide a multi-image overlay display system, method, viewing pod and surgeon console, which can simultaneously propagate surgical environment images and menu interface images in a surgical scene without any sequence problems, effectively reducing the transmission delay of the surgical video picture and improving the smoothness and safety of the surgical operation.

[0006] To achieve the above-mentioned purpose, the embodiments of the present application adopt the following technical solutions:

[0007] A multi-image overlay display system comprises:

[0008] a first display device configured to present a first image;

[0009] a first reflector arranged at an angle to the first display device such that the first reflector passes first image light from the first display device towards a second reflector;

[0010] a second display device configured to present a second image, the second display device disposed at a primary virtual image position of the first display device with respect to the first reflector such that the first reflector passes second image light from the second display device toward the second reflector;

[0011] a second reflector configured to simultaneously receive the first image light and the second image light from the first reflector and to form a superimposed virtual image of both perceivable by an observer's eye in front of an eyepiece.

[0012] A multi-image superimposition display system, comprising:

[0013] a first display device configured to present a surgical environment image of a surgical site;

[0014] a first reflector angularly arranged with the first display device such that the first reflector passes surgical environment image light from the first display device toward a second reflector;

[0015] a second display device configured to present a menu interface image containing menu information, the second display device disposed at a primary virtual image position of the first display device with respect to the first reflector such that the first reflector passes menu interface image light from the second display device toward the second reflector;

[0016] a second reflector configured to simultaneously receive the surgical environment image light and the menu interface image light from the first reflector and to form a superimposed virtual image of both perceivable by an observer's eye in front of an eyepiece.

[0017] A multi-image superimposition display system, comprising: two sets of optical assemblies symmetrically arranged with respect to a vertical plane and structurally identical, each of the optical assemblies comprising:

[0018] a first display device configured to present a surgical environment image of a surgical site, wherein the surgical environment images corresponding to the two sets of the first display devices have lateral parallax;

[0019] a first reflector angularly arranged with the first display device such that the first reflector passes surgical environment image light from the first display device toward a second reflector;

[0020] a second display device configured to present a menu interface image containing menu information, the second display device disposed at a primary virtual image position of the first display device with respect to the first reflector such that the first reflector passes menu interface image light from the second display device toward the second reflector;

[0021] A second reflector configured to simultaneously receive the surgical environment image light and the menu interface image light from the first reflector and to form a superimposed virtual image of both that is perceivable by an observer's eye in front of the eyepiece.

[0022] Further, the first display device and the second display device are both configured as flat display screens, and display ranges of both are set to be consistent.

[0023] Further, the first reflector is configured as a semi-transparent semi-reflective mirror for reflecting the first image light from the first display device and transmitting the second image light from the second display device.

[0024] And / or, the second reflector is configured as a reflective mirror.

[0025] Further, the first reflector is configured as a semi-transparent semi-reflective mirror, and a ratio of transmittance to reflectance of the first reflector ranges from 10:90 to 90:10.

[0026] Further, the first display devices corresponding to the two groups of optical assemblies are coplanarly arranged.

[0027] A multi-image superimposition display method, the method comprising:

[0028] acquiring a surgical environment image and displaying the surgical environment image;

[0029] acquiring a menu interface image and displaying the menu interface image, wherein the surgical environment image and the menu interface image are both independently displayed.

[0030] synchronously transmitting the surgical environment image and the menu interface image based on different light transmission paths and equal optical path lengths to form a superimposed virtual image of both that is perceivable by an observer's eye in front of the eyepiece.

[0031] Further, the step of synchronously transmitting the surgical environment image and the menu interface image based on different light transmission paths and equal optical path lengths to form a superimposed virtual image of both that is perceivable by an observer's eye in front of the eyepiece comprises:

[0032] setting two light action planes, wherein the menu interface image is located at a first virtual image position of the surgical environment image with respect to a first light action plane;

[0033] reflecting one of the surgical environment image light and the menu interface image light at the first light action plane while transmitting the other, and the reflected light and the transmitted light simultaneously reach a second light action plane to form the superimposed virtual image of both in front of the eyepiece.

[0034] A multi-image superimposed display method, the method comprising:

[0035] acquiring a left-eye surgical environment image and displaying the left-eye surgical environment image, while acquiring a right-eye surgical environment image and displaying the right-eye surgical environment image;

[0036] acquiring a left-eye menu interface image and displaying the left-eye menu interface image, while acquiring a right-eye menu interface image and displaying the right-eye menu interface image; wherein the left-eye and right-eye surgical environment images and the left-eye and right-eye menu interface images are independently displayed separately;

[0037] the left-eye surgical environment image and the right-eye surgical environment image each reach a second reflector after being reflected by a first reflector respectively;

[0038] the left-eye menu interface image and the right-eye menu interface image each reach the second reflector after being transmitted by the first reflector respectively;

[0039] the second reflector corresponding to the left eye simultaneously receives the left-eye surgical environment image and the left-eye menu interface image, and forms a left-eye superimposed virtual image of the two in front of the ocular lens;

[0040] the second reflector corresponding to the right eye simultaneously receives the right-eye surgical environment image and the right-eye menu interface image, and forms a right-eye superimposed virtual image of the two in front of the ocular lens.

[0041] An observation bin comprising: a shell and the aforementioned multi-image superimposed display system;

[0042] the shell has a containing space for containing the multi-image superimposed display system, and an ocular lens hole for installing an ocular lens is formed on the shell;

[0043] the first display device is arranged close to and parallel to the top surface of the shell.

[0044] A doctor's console comprising: an image acquisition system and the observation bin of claim 11;

[0045] the image acquisition system is communicatively connected with the first display device and the second display device, and the image acquisition system is configured to acquire a surgical environment image of a surgical scene and transmit the surgical environment image to the first display device, while acquiring state information of a mechanical arm and / or an endoscope imaging system and transmitting the state information to the second display device.

[0046] A computer readable storage medium, characterized in that the computer readable storage medium stores computer executable instructions, and the computer executable instructions, when invoked and executed by a processor, cause the processor to implement the aforementioned method.

[0047] The image superimposed display system, method, observation cabin and doctor console provided by the embodiment of the application have at least the following beneficial effects.

[0048] The second display device is arranged at the first virtual image position of the first display device relative to the first reflector. Through the first reflector, the first image from the first display device can be transmitted on one hand, and the second image from the second display device can be transmitted on the other hand. The first image and the second image can reach the second reflector at the same time by using the light transmission mode, and the second reflector can reflect the two images at the same time, so that the superimposed virtual image of the two images can be formed in front of the ocular lens and can be perceived by the observer's eyes. In this process, since the first image and the second image are transmitted at the same time, there is no problem of displaying in sequence, and the transmission delay of the surgical video image can be effectively reduced, and the fluency and safety of the surgical operation can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the specific embodiments of the application or the technical solutions in the prior art, the drawings needed to be used in the specific embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0050] Fig. 1 is a structural schematic diagram of a multi-image superimposed display system provided by the embodiment of the application;

[0051] Fig. 2 is an elevation view of the multi-image superimposed display system provided by the embodiment of the application;

[0052] Fig. 3 is an optical path diagram of the multi-image superimposed display system provided by the embodiment of the application;

[0053] Fig. 4 is a flowchart of a multi-image superimposed display method provided by the embodiment of the application;

[0054] Fig. 5 is a flowchart of a multi-image superimposed display method provided by the embodiment of the application;

[0055] Fig. 6 is a flowchart of a multi-image superimposed display method provided by the embodiment of the application;

[0056] Fig. 7 is a simplified flowchart of a multi-image superimposed display method provided by the embodiment of the application;

[0057] Fig. 8 is a schematic diagram of a menu interface image in the embodiment of the application.

[0058] Icon:

[0059] 10-observation image plane position;

[0060] 100 - first display screen; 200 - second display screen; 300 - half mirror; 400 - mirror. DETAILED DESCRIPTION

[0061] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0062] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0063] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0064] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship when the product of the present application is usually placed, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0065] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components must be absolutely horizontal or overhanging, but can be slightly inclined. For example, "horizontal" only means that its direction is relatively more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0066] In the description of the present application, it is also necessary to explain that, unless otherwise explicitly specified and limited, the terms "arrange", "install", "connect", "connect" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0067] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following examples and features in the examples can be combined with each other without conflict.

[0068] In general technology, the surgical robot mainly includes a master operating table, a slave operating table and an image processing trolley. An operator can remotely operate the master operating table to control the slave operating table through the master-slave control relationship between the master operating table and the slave operating table. The image processing trolley can provide the operator with auxiliary surgical environment images, etc. The slave operating table includes at least one image arm and an instrument arm, wherein the image arm is loaded with an endoscope imaging system. During surgery, the endoscope imaging system and the surgical instrument respectively enter the patient's disease position through the incision on the patient's body. The endoscope imaging system is in communication connection with a display device. The endoscope imaging system can acquire images of the surgical site (such as surgical instruments, human tissues, blood vessels, etc.), and process the captured images through a processor for display by the display device (configured to display images visible to the operator), for example, on the master operating table or on another suitable display device located locally and / or remotely to the operator. The endoscope imaging system can be a stereo endoscope. In the case of using a stereo endoscope, the captured images can be processed by the processor to present the surgical site to the surgeon in a coordinated stereo image. The instrument arm follows the movement of the master arm mechanism operated by the doctor on the master operating table during the surgical process to achieve the corresponding surgical operation.

[0069] It can be understood that the instrument arm is coupled with a surgical instrument, and the plurality of instrument arms can be a plurality of same surgical instruments and / or a plurality of different surgical instruments, such as forceps, clip appliers, clamps, retractors, cauterizing instruments, suction instruments, suturing devices, anastomosis devices, cutting devices, etc. Similarly, in the case of multiple endoscope imaging systems, the multiple endoscope imaging systems can be multiple same and / or multiple different endoscope imaging systems.

[0070] In the embodiment, the master manipulator console comprises an observation bin, the front of the observation bin is provided with a left eye lens hole and a right eye lens hole, a left eye viewer is arranged at the left eye lens hole, and a right eye viewer is arranged at the right eye lens hole, the left eye viewer and the right eye viewer each comprise an optical lens, and a doctor can watch a three-dimensional view of a surgical scene capable of realizing depth perception presented to the doctor by the observation bin through the left and right eye viewers.

[0071] The application provides a multi-image superimposed display system, comprising: a first display device configured to present a first image; a first reflector arranged at an angle with the first display device, such that the first reflector transmits first image light from the first display device towards a second reflector; a second display device configured to present a second image, the second display device being disposed at a virtual image position of the first display device with respect to the first reflector, such that the first reflector transmits second image light from the second display device towards the second reflector; and a second reflector configured to simultaneously receive the first image light and the second image light from the first reflector and to form a superimposed virtual image of both that is perceivable by an observer's eye in front of an eyepiece.

[0072] It should be noted that the front of the eyepiece refers to a direction away from the operator when the operator sits facing the master manipulator console, and a direction close to the operator is the back of the eyepiece.

[0073] The foregoing scheme can superimpose the first image and the second image together to form a superimposed virtual image of both for an observer to watch, wherein the first image and the second image can be different types of images, and specific settings can be made according to the use scenario. For example, in a surgical scene, the first image can be a surgical environment image showing a surgical site, and the second image can be a menu interface image showing the state (open / close state) of a surgical instrument, such as the rotation angle of an endoscopic imaging system, the cutting and coagulation state of an instrument, the master-slave hand switching state, and the like.

[0074] Further, in order to display a three-dimensional scene, the multi-image superimposed display system needs to be provided with two sets of optical assemblies, please continue to refer to FIG. 1, the two sets of optical assemblies are symmetrically arranged with respect to a vertical plane and have the same structure, and the first images corresponding to the two sets of first display devices have horizontal parallax.

[0075] Exemplarily, the first display device and the second display device can each be a flat display screen, such as a liquid crystal display screen, a plasma display screen, a touch screen, or the like, or can be other display devices capable of displaying images, such as a projector or the like; referring to FIG. 3, the first reflector is configured as a half-transmission half-reflection mirror 300, that is, capable of reflecting light and transmitting light, specifically, capable of reflecting the first image light from the first display device and transmitting the second image light from the second display device; the second reflector is configured as a reflection mirror 400 for reflecting light.

[0076] In this embodiment, the ratio of the transmittance to the reflectance of the first reflector ranges from 10:90 to 90:10, and exemplarily, the ratio can be 20:80, 30:70, 40:60, 50:50, 60:40, 70:30, or 80:20.

[0077] Specifically, referring to FIGS. 1 and 2, the multi-image superimposed display system of this embodiment includes two first display screens 100, two second display screens 200, two half-transmission half-reflection mirrors 300, and two reflection mirrors 400, wherein the half-transmission half-reflection mirror 300 is arranged at an angle with respect to the corresponding first display screen 100, the reflection mirror 400 is arranged at an angle with respect to the corresponding half-transmission half-reflection mirror 300, and the second display screen 200 is arranged at a virtual image position of the corresponding first display screen 100 with respect to the corresponding half-transmission half-reflection mirror 300.

[0078] In a surgical operation scenario, after the surgical environment image is displayed by the corresponding first display screen 100, the surgical environment image is sequentially reflected by the corresponding half-transmission half-reflection mirror 300 and the reflection mirror 400 to reach the observation image plane position 10 to form a surgical environment virtual image, and at the same time, after the menu interface image is displayed by the corresponding second display screen 200, the menu interface image is transmitted by the corresponding half-transmission half-reflection mirror 300 and then reflected by the reflection mirror 400 to reach the observation image plane position 10 to form a menu interface virtual image, and the menu interface virtual image and the surgical environment virtual image form a superimposed image effect.

[0079] In other embodiments, the first display screen 100 can also display the menu interface image, and the second display screen 200 can display the surgical environment image, and the specific principle is the same as above, which will not be described herein.

[0080] In this embodiment, the two first display screens 100 are arranged at left and right intervals, for receiving two images with horizontal parallax, that is, a left-eye surgical environment image and a right-eye surgical environment image; the two half-transmission half-reflection mirrors 300 are arranged in one-to-one correspondence with the two first display screens 100.

[0081] In a specific application, the left eye surgical environment image displayed by the left first display screen 100 is reflected by the left semi-transmissive half mirror 300 and the left mirror 400 in turn to reach the observation image plane position 10 to form a left eye surgical environment virtual image; the left eye menu interface image displayed by the left second display screen 200 is transmitted by the left semi-transmissive half mirror 300 and reflected by the left mirror 400 to reach the observation image plane position 10 to form a left eye menu interface virtual image, which is superimposed with the left eye surgical environment virtual image to form a left eye superimposed virtual image. Similarly, the right eye surgical environment image displayed by the right first display screen 100 is reflected by the right semi-transmissive half mirror 300 and the right mirror 400 in turn to reach the observation image plane position 10 to form a right eye surgical environment virtual image; the right eye menu interface image displayed by the right second display screen 200 is transmitted by the right semi-transmissive half mirror 300 and reflected by the right mirror 400 to reach the observation image plane position 10 to form a right eye menu interface virtual image, which is superimposed with the right eye surgical environment virtual image to form a right eye superimposed virtual image. The left eye superimposed virtual image and the right eye superimposed virtual image are combined at the observation image plane position 10.

[0082] When the eyes of the operator are aligned with the left and right eye viewers of the observation bin, a combined stereoscopic image that can be visible to the operator can be viewed, which includes the menu interface image in the surgical environment image to provide the operator with an intuitive and stereoscopic image display.

[0083] Referring to FIG. 4, the embodiment further provides a multi-image superimposed display method, specifically including the following steps:

[0084] Step S11, acquiring a surgical environment image and displaying the surgical environment image.

[0085] In the embodiment, the surgical environment image of the surgical site can be acquired by an endoscope imaging system, and the image is processed and transmitted to the corresponding first display screen 100 through a signal line, and displayed on the first display screen 100.

[0086] Step S12, acquiring a menu interface image and displaying the menu interface image, wherein the surgical environment image and the menu interface image are independently displayed.

[0087] In the embodiment, signal acquisition sensors can be arranged at corresponding positions of the endoscope imaging system, instrument arm, etc., and the signal data acquired by the signal acquisition sensors are transmitted to the corresponding second display screen 200 in the form of a menu interface image, and displayed on the second display screen 200.

[0088] Step S13, based on different light transmission paths and equal optical path lengths, the surgical environment image and the menu interface image are synchronously transmitted to form an overlaid virtual image of both that can be perceived by the observer's eyes in front of the ocular lens.

[0089] By using the multi-image superimposed display method of the embodiment, without using the software synthesis manner, the surgical environment image and the menu interface image are independently displayed, and based on different light transmission paths and equal light path lengths, the surgical environment image and the menu interface image are synchronously transmitted, in the synchronous transmission process, the transmission delay of the surgical environment image is not increased, and the two form superimposed virtual images in front of the ocular lens which can be perceived by the observer's eyes.

[0090] Further, with reference to FIG. 5, based on different light transmission paths and equal light path lengths, the surgical environment image and the menu interface image are synchronously transmitted to form superimposed virtual images of the two in front of the ocular lens which can be perceived by the observer's eyes, and the step includes:

[0091] In step S131, two light action planes are set, wherein the menu interface image is located at a primary virtual image position of the surgical environment image relative to the primary light action plane.

[0092] The aforementioned light action plane can be regarded as a transmission surface or a reflection surface to realize the transmission of light. The menu interface image is arranged at the primary virtual image position of the surgical environment image relative to the primary light action plane, so that the light path length of the surgical environment image to the primary light action plane is equal to the light path length of the menu interface image to the primary light action plane, and the two are different light transmission paths.

[0093] In step S132, one of the surgical environment image light and the menu interface image light is reflected at the primary light action plane, and the other is transmitted, and the reflected light and the transmitted light reach the secondary light action plane at the same time, and form superimposed virtual images of the two in front of the ocular lens.

[0094] Specifically, when the surgical environment image light passes through the primary light action plane, the primary light action plane can reflect it to the secondary light action plane, and at the same time, when the menu interface image light passes through the primary light action plane, the primary light action plane can transmit it to the secondary light action plane.

[0095] As an alternative, when the surgical environment image light passes through the primary light action plane, the primary light action plane can transmit it to the secondary light action plane, and at the same time, when the menu interface image light passes through the primary light action plane, the primary light action plane can reflect it to the secondary light action plane. Based on the foregoing method, the more specific implementation steps include:

[0096] In step 21, the left-eye surgical environment image is obtained and displayed, and at the same time, the right-eye surgical environment image is obtained and displayed.

[0097] In the acquisition of left and right eye surgery environment images, two CMOS sensors can be used to acquire the images respectively, or one CMOS sensor can be used to acquire the images simultaneously, and the two CMOS sensors or the one CMOS sensor are in communication connection with the corresponding processors, and the images are transmitted after being processed, wherein the left eye surgery environment image and the right eye surgery environment image have lateral parallax.

[0098] Step 22, acquire a left eye menu interface image and display the left eye menu interface image, and acquire a right eye menu interface image and display the right eye menu interface image; wherein the left and right eye surgery environment images and the left and right eye menu interface images are independently displayed.

[0099] In the acquisition of left and right menu interface images, the same sensor can be used to collect image information, and the image information is transmitted to different second display screens 200 for display simultaneously, wherein the left eye menu interface image and the right eye menu interface image display the same menu interface.

[0100] Step 23, the left eye surgery environment image and the right eye surgery environment image respectively reach the second reflector after being reflected by the first reflector.

[0101] Among them, the first reflector is provided with two, and is the same setting, respectively used for receiving left eye surgery environment image or right eye surgery environment image.

[0102] Step 24, the left eye menu interface image and the right eye menu interface image respectively reach the second reflector after being transmitted by the first reflector.

[0103] Among them, the first reflector is provided with two, and is the same setting, respectively used for receiving left eye menu interface image or right eye menu interface image.

[0104] Step 25, the second reflector corresponding to the left eye simultaneously receives the left eye surgery environment image and the left eye menu interface image, and forms a left eye superimposed virtual image of the two in front of the ocular lens.

[0105] Among them, the display range size of the left eye surgery environment image and the left eye menu interface image is the same, so as to ensure that the menu interface is in the intended designated position on the final human eye viewing screen.

[0106] Step 26, the second reflector corresponding to the right eye simultaneously receives the right eye surgery environment image and the right eye menu interface image, and forms a right eye superimposed virtual image of the two in front of the ocular lens.

[0107] Similarly, the display range size of the right eye surgery environment image and the right eye menu interface image is the same, so as to ensure that the menu interface is in the intended designated position on the final human eye viewing screen.

[0108] It should be noted that, in order to ensure that the display range sizes of the surgical environment image and the menu interface image are the same, the display range size of the first display screen 100 needs to be set to be consistent with the display range size of the second display screen 200.

[0109] In combination with FIGS. 1, 3, and 7, the left-eye surgical environment image is reflected by the left half-mirror 300 and the left mirror 400 in sequence to reach the observation image plane position 10 to form a left-eye surgical environment virtual image; the left-eye menu interface image is transmitted by the left half-mirror 300 and then reflected by the left mirror 400 to reach the observation image plane position 10 to form a left-eye menu interface virtual image, which is superimposed with the left-eye surgical environment virtual image to form a left-eye superimposed virtual image. Similarly, the right-eye surgical environment image is reflected by the right half-mirror 300 and the right mirror 400 in sequence to reach the observation image plane position 10 to form a right-eye surgical environment virtual image; the right-eye menu interface image is transmitted by the right half-mirror 300 and then reflected by the right mirror 400 to reach the observation image plane position 10 to form a right-eye menu interface virtual image, which is superimposed with the right-eye surgical environment virtual image to form a right-eye superimposed virtual image. The left-eye superimposed virtual image and the right-eye superimposed virtual image are combined at the observation image plane position 10 to form a stereoscopic image visible to the operator.

[0110] Specifically, referring to FIG. 8, a synthesized menu image after the left-eye menu interface image and the right-eye menu interface image are synthesized is shown; the synthesized menu image includes a plurality of information fields, which can realize the state information of the instrument arms and the image arm, including: instrument name, instrument installation connection state, left and right position state, power-on state, pause state, endoscope state, movement state, and fault state, etc.; wherein the blank area in the figure is used to display the dynamic picture collected by the endoscope imaging system, i.e., the surgical environment image, and the distal end part of the surgical instrument corresponding to the plurality of instrument arms.

[0111] It should be noted that the surgical instrument is not limited to forceps, clip appliers, clamps, retractors, cauterizing instruments, suction instruments, suturing devices, anastomosis devices, cutting devices, etc., and the information field corresponds to the surgical instrument and includes the name of the surgical instrument, which can be any of the names listed above about the surgical instrument.

[0112] In this embodiment, the information fields are displayed around the periphery of the surgical scene image; the information fields can be superimposed above, below, or outside the field of view, bordering the peripheral edge of the field of view. Additionally or alternatively, the information fields can be displayed in close proximity to the surgical instruments to which they respectively correspond. In other embodiments, the information fields can be displayed along the same peripheral edge of the surgical scene image, e.g., along the bottom edge, the left edge, the right edge, or the top edge.

[0113] The embodiment of the present application further provides an observation bin, comprising: a shell and the multi-image superimposed display system in the foregoing embodiment; the shell has a containing space for containing the multi-image superimposed display system, and an ocular hole for mounting an ocular lens is formed on the shell; the first display device is arranged close to and parallel to the top surface of the shell, so that the layout of the multi-image superimposed display system in the shell is facilitated, and the installation and fixation of the multi-image superimposed display system are facilitated.

[0114] The embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions, when the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to realize the method, and the specific implementation can be referred to the foregoing method embodiment, and will not be described here.

[0115] Unless specifically stated, the relative steps, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0116] If the functions are realized in the form of software function units and sold or used as independent products, the functions can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on such understanding, the technical solutions of the present application essentially or the parts of the prior art that make contributions or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application.

[0117] The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0118] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-image superimposed display system, characterized by, Comprising: a first display device configured to present a first image; a first reflector arranged at an angle to the first display device such that the first reflector passes first image light from the first display device toward a second reflector; a second display device configured to present a second image, the second display device disposed at a primary virtual image position of the first display device with respect to the first reflector such that the first reflector passes second image light from the second display device toward the second reflector; a second reflector configured to simultaneously receive the first image light and the second image light from the first reflector and to form a superimposed virtual image of both that is perceivable by an observer's eye in front of an eyepiece.

2. A multi-image superimposed display system, characterized by comprising: Comprising: a first display device configured to present a surgical environment image of a surgical site; a first reflector arranged at an angle to the first display device such that the first reflector passes surgical environment image light from the first display device toward a second reflector; a second display device configured to present a menu interface image containing menu information, the second display device disposed at a primary virtual image position of the first display device with respect to the first reflector such that the first reflector passes menu interface image light from the second display device toward the second reflector; a second reflector configured to simultaneously receive the surgical environment image light and the menu interface image light from the first reflector and to form a superimposed virtual image of both that is perceivable by an observer's eye in front of an eyepiece.

3. A multi-image superimposed display system, characterized by comprising: Comprising: two sets of optical components arranged symmetrically with respect to a vertical plane and identical in structure, each of the optical components comprising: a first display device configured to present a surgical environment image of a surgical site, wherein the surgical environment images corresponding to the two sets of the first display devices have lateral parallax; a first reflector arranged at an angle to the first display device such that the first reflector passes surgical environment image light from the first display device toward a second reflector; a second display device configured to present a menu interface image containing menu information, the second display device disposed at a primary virtual image position of the first display device with respect to the first reflector such that the first reflector passes menu interface image light from the second display device toward the second reflector; a second reflector configured to simultaneously receive the surgical environment image light and the menu interface image light from the first reflector and to form a superimposed virtual image of both that is perceivable by an observer's eye in front of an eyepiece.

4. The multi-image overlay display system of any of claims 1-3, wherein, The first display device and the second display device are both configured as flat display screens, and the display ranges of both are set to be consistent.

5. The multi-image overlay display system of any of claims 1-3, wherein, The first reflector is configured as a half-transmission half-reflection mirror for reflecting the first image light from the first display device and transmitting the second image light from the second display device; and / or, the second reflector is configured as a reflection mirror.

6. The multi-image overlay display system of claims 1-3, wherein, The first reflector is configured as a half-transmission half-reflection mirror, and the ratio of the transmission rate to the reflection rate of the first reflector ranges from 10:90 to 90:

10.

7. The multi-image overlay display system of claim 3, wherein, The first display device corresponding to the optical assembly of the two groups is coplanarly arranged.

8. A multi-image superimposed display method characterized by comprising: The method comprises: acquiring a surgical environment image and displaying the surgical environment image; acquiring a menu interface image and displaying the menu interface image, wherein the surgical environment image and the menu interface image are independently displayed separately; synchronously transmitting the surgical environment image and the menu interface image based on different light transmission paths and equal optical path lengths to form an overlaid virtual image of both in front of the ocular lens, which can be perceived by the observer's eyes.

9. The multi-image superimposition display method according to claim 8, wherein The step of synchronously transmitting the surgical environment image and the menu interface image based on different light transmission paths and equal optical path lengths to form an overlaid virtual image of both in front of the ocular lens, which can be perceived by the observer's eyes, comprises: setting two light action planes, wherein the menu interface image is located at a first virtual image position of the surgical environment image with respect to a first light action plane; reflecting one of the surgical environment image light and the menu interface image light at the first light action plane while transmitting the other, and the reflected light and the transmitted light reach a second light action plane at the same time and form the overlaid virtual image of both in front of the ocular lens.

10. A multi-image superimposed display method characterized by comprising: The method comprises: acquiring a left-eye surgical environment image and displaying the left-eye surgical environment image, and simultaneously acquiring a right-eye surgical environment image and displaying the right-eye surgical environment image; acquiring a left-eye menu interface image and displaying the left-eye menu interface image, and simultaneously acquiring a right-eye menu interface image and displaying the right-eye menu interface image; wherein the left-eye and right-eye surgical environment images and the left-eye and right-eye menu interface images are independently displayed separately; the left-eye surgical environment image and the right-eye surgical environment image each respectively reach a second reflector after being reflected by a first reflector; the left-eye menu interface image and the right-eye menu interface image each respectively reach the second reflector after being transmitted by the first reflector; the second reflector corresponding to the left eye simultaneously receives the left-eye surgical environment image and the left-eye menu interface image and forms a left-eye overlaid virtual image of both in front of the ocular lens; the second reflector corresponding to the right eye simultaneously receives the right-eye surgical environment image and the right-eye menu interface image and forms a right-eye overlaid virtual image of both in front of the ocular lens.

11. An observation bin, characterized by, It comprises: a housing and a multi-image superimposed display system according to any one of claims 1-7; the housing has a containing space for containing the multi-image superimposed display system, and an ocular lens hole is formed on the containing space for mounting the ocular lens; the first display device is arranged close to and parallel to the top surface of the housing.

12. A physician's console, characterized by It comprises: an image acquisition system and an observation cabin according to claim 11; the image acquisition system is communicatively connected with the first display device and the second display device, and is configured to acquire a surgical environment image of a surgical scene and transmit it to the first display device, and simultaneously acquire state information of a mechanical arm and / or an endoscope imaging system and transmit it to the second display device.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions that, when invoked and executed by the processor, cause the processor to implement the method of any one of claims 8-10.

Images