2D / 3D display fusion surgery remote guidance system

Abstract

This invention provides a 2D / 3D display fusion-based remote surgical guidance system, comprising: a surgical acquisition end for acquiring three-dimensional light field information and two-dimensional image information of the surgical area; a data processing end for receiving two-dimensional image information and external guidance information, and generating two-dimensional enhanced information; a surgeon's workstation including a first display unit for reconstructing a three-dimensional light field scene based on the three-dimensional light field information; a second display unit for projecting the two-dimensional enhanced information; and an optical fusion unit for optically superimposing the two-dimensional enhanced information onto the corresponding spatial position of the three-dimensional light field scene for fusion display. By superimposing the two-dimensional enhanced information, including the enhanced information from external guidance, onto the corresponding spatial position of the three-dimensional light field scene through the optical fusion unit, three-dimensional anatomical depth and high-resolution two-dimensional guidance details can be perceived without switching between two-dimensional and three-dimensional display modes, compensating for the single-viewpoint resolution loss of simple three-dimensional light field display and improving surgical efficiency.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and in particular to a 2D / 3D display fusion surgical remote guidance system. Background Technology

[0002] Endoscopic surgery, as a minimally invasive technique, is applied in various fields, including transurethral resection of the prostate (TURP) in urology, transurethral resection of bladder tumors (TURP), percutaneous nephrolithotomy (PCNL), ureteroscopic nephrolithotomy, and hysteroscopic surgery in gynecology. Due to the high learning curve of laparoscopic or thoracic surgeries, doctors require long-term guidance from experienced physicians after short-term endoscopic training. Furthermore, the surgical process often needs to be broadcast remotely for sharing.

[0003] Currently, remote illustration technology guides doctors to annotate and interact with two-dimensional images. Surgeons need to switch from three-dimensional to two-dimensional display mode to view the guidance content in front of a three-dimensional monitor, and the repeated switching reduces surgical efficiency. Summary of the Invention

[0004] This invention provides a 2D / 3D display fusion surgical remote guidance system to solve the problem that in the prior art, remote illustration technology guides doctors to annotate and interact with two-dimensional images. Surgeons need to switch from three-dimensional display to two-dimensional display mode repeatedly to view the guidance content in front of a three-dimensional monitor, which reduces surgical efficiency.

[0005] This invention provides a 2D / 3D display fusion-based remote surgical guidance system, comprising: The surgical acquisition end is used to acquire three-dimensional light field information and two-dimensional image information of the surgical area; The data processing end is used to receive the two-dimensional image information and guidance information from the outside, and generate two-dimensional enhancement information; The surgeon's workstation includes a two-dimensional-three-dimensional fused light field display, which includes a first display unit, a second display unit, and an optical fusion unit; The first display unit is used to reconstruct a three-dimensional light field scene based on the three-dimensional light field information; The second display unit is used to project the two-dimensional enhanced information; An optical fusion unit is used to superimpose the two-dimensional enhancement information onto the corresponding spatial position of the three-dimensional light field scene in an optical form for fusion display.

[0006] According to the present invention, a 2D / 3D display fusion surgical remote guidance system is provided, wherein the first display unit comprises: A rear projection projector is used to project the three-dimensional light field information; A three-dimensional light control device is disposed downstream of the optical path of the rear projection projector to spatially modulate the projected light of the rear projection projector and control the reconstruction of a three-dimensional light field scene with multiple discrete viewpoints in the viewing area.

[0007] According to the present invention, a 2D / 3D display fusion surgical remote guidance system is provided, wherein the three-dimensional light control device is a cylindrical lens grating or a microlens array.

[0008] According to the present invention, a 2D / 3D display fusion surgical remote guidance system is provided, wherein the optical fusion unit comprises: A semi-reflective and semi-transparent film is disposed on the viewing side of the three-dimensional light control device; The second display unit includes a front projection projector, the light path of which passes through the three-dimensional light control device after being reflected by the semi-reflective and semi-transparent film. The optical path of the rear projection projector passes sequentially through the semi-reflective and semi-transparent film and the three-dimensional light control device, so that the two-dimensional enhancement information and the light of the three-dimensional light field scene are spatially fused at the three-dimensional light control device.

[0009] According to the 2D / 3D display fusion surgical remote guidance system provided by the present invention, the single-viewpoint resolution reconstructed by the first display unit satisfies the following relationship with the resolution and the number of reconstructed viewpoints of the rear projection projector: P1 = P2 / N; Where P1 represents the single-viewpoint resolution reconstructed by the first display unit, P2 represents the resolution of the rear projection projector, and N represents the number of reconstructed viewpoints. The two-dimensional enhancement information provided by the second display unit is used to improve the effective single-viewpoint resolution perceived by the viewer.

[0010] A 2D / 3D display fusion surgical remote guidance system according to the present invention further includes: The doctor's workstation is connected to the data processing terminal for receiving and displaying the two-dimensional image information, and generating guidance information for illustrating and annotating the two-dimensional image information in response to interactive operations.

[0011] According to the present invention, a 2D / 3D display fusion surgical remote guidance system is provided, wherein the data processing terminal is specifically used for: The three-dimensional light field information is encoded to generate a three-dimensional light field signal suitable for reconstruction by the first display unit; The two-dimensional image information and the guidance information are combined to perform image integration to generate two-dimensional enhancement information.

[0012] According to the 2D / 3D display fusion surgical remote guidance system provided by the present invention, the data processing terminal is further used for: Based on human eye tracking information, a two-dimensional view corresponding to the current viewing point is extracted from the three-dimensional light field information, and the two-dimensional view is sent to the guiding doctor's terminal as the two-dimensional image information.

[0013] According to the present invention, a 2D / 3D display fusion surgical remote guidance system is provided, wherein the surgical acquisition end includes: a light field camera equipped with a microlens array for acquiring three-dimensional light field information from different spatial positions and angles of the surgical area; the surgical acquisition end downsamples and encodes the three-dimensional light field information and transmits it to the data processing end.

[0014] According to the 2D / 3D display fusion surgical remote guidance system provided by the present invention, the surgeon's workstation also includes an operating handle for controlling the surgical robot in the surgical acquisition terminal to perform surgical operations.

[0015] The present invention provides a 2D / 3D display fusion surgical remote guidance system. Through an optical fusion unit, it superimposes two-dimensional enhanced information, which includes external guidance information, onto the corresponding spatial position of the three-dimensional light field scene. This allows for simultaneous perception of three-dimensional anatomical depth and high-resolution two-dimensional guidance details without switching between two-dimensional and three-dimensional display modes. Visually, this compensates for the single-viewpoint resolution loss of simple three-dimensional light field display, thereby improving surgical efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the 2D / 3D display fusion surgical remote guidance system provided by the present invention; Figure 2 This is a schematic diagram of the structure of the two-dimensional-three-dimensional fused light field display provided by the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] Figure 1 This is a schematic diagram of the 2D / 3D display fusion surgical remote guidance system provided by the present invention. Figure 2 This is a schematic diagram of the structure of the two-dimensional-three-dimensional fused light field display provided by the present invention.

[0020] like Figure 1 As shown in this embodiment, a 2D / 3D display fusion surgical remote guidance system is provided. The core of this system is to achieve the fusion display of two-dimensional enhanced information and three-dimensional light field scene, allowing surgeons to take into account both depth perception and high-resolution guidance details without switching display modes. It can be applied to remote training, intraoperative interactive live broadcast, etc., and is especially suitable for procedures such as transurethral resection of the prostate in urology, transurethral resection of bladder tumors, percutaneous nephrolithotomy, ureteroscopic nephrolithotomy, and hysteroscopic surgery in gynecology. The 2D / 3D display fusion surgical remote guidance system includes: a surgical acquisition terminal 1, used to acquire three-dimensional light field information and two-dimensional image information of the surgical area; a data processing terminal 2, used to receive two-dimensional image information and guidance information from the outside, and generate two-dimensional enhanced information; and a surgeon's workstation 3, including a two-dimensional-three-dimensional fused light field display, which includes a first display unit, a second display unit, and an optical fusion unit; the first display unit is used to reconstruct a three-dimensional light field scene based on the three-dimensional light field information; the second display unit is used to project the two-dimensional enhanced information; and the optical fusion unit is used to superimpose the two-dimensional enhanced information in an optical form onto the corresponding spatial position of the three-dimensional light field scene for fusion display.

[0021] Specifically, such as Figure 1 As shown, a communication connection is established between the surgical acquisition terminal 1 and the data processing terminal 2, primarily for acquiring three-dimensional light field information and two-dimensional image information of the surgical area. The surgical acquisition terminal 1 includes a light field camera equipped with a microlens array and a surgical robot. The light field camera, due to its small size, is particularly suitable for close-range acquisition within the patient's body. Through the microlens array at its front end, it can capture three-dimensional light field information of the surgical area at different spatial positions and angles in real time. This information includes key three-dimensional data such as the depth and contour of the lesion. Simultaneously, the light field camera extracts key viewpoint two-dimensional information of the surgical area, serving as the core content of the two-dimensional image information. After acquisition, the three-dimensional light field information and two-dimensional image information are transmitted to the data processing terminal 2.

[0022] Data processing terminal 2 is the information processing hub of the system. It establishes communication with surgical acquisition terminal 1, surgeon's workstation 3, and external guidance information sources. It is responsible for receiving information from surgical acquisition terminal 1 and combining it with external guidance information to generate displayable 2D enhanced information. Data processing terminal 2 first receives the 3D light field information and 2D image information transmitted from surgical acquisition terminal 1, performing preliminary noise reduction and calibration processing on this raw information to ensure accuracy. Subsequently, data processing terminal 2 receives external guidance information, such as annotations and diagrams from the surgeon, integrating and optimizing this guidance information with the previously received 2D image information to ultimately generate 2D enhanced information containing guidance details. By fusing 2D image information and external guidance information into unified 2D enhanced information, information fragmentation is avoided, while ensuring accurate loading of guidance information.

[0023] The surgeon's workstation 3 is the core component for direct operation and observation by the surgeon. It communicates with the data processing unit 2 and includes a 2D-3D fused light field display and a control handle. The 2D-3D fused light field display is crucial for achieving seamless switching between 2D and 3D modes; its internal structure is shown in the attached diagram. Figure 2 As shown, it consists of a first display unit, a second display unit, and an optical fusion unit.

[0024] The first display unit is primarily used to reconstruct a 3D light field scene based on 3D light field information, enabling surgeons to gain depth perception. Combined with... Figure 2 After receiving the three-dimensional light field related signals transmitted from data processing terminal 2, it converts these signals into a visualized three-dimensional light field scene through its internal display mechanism. Figure 2 The reconstructed 3D images inside the laparoscopy room can realistically reproduce the spatial structure of the surgical area, including the precise depth and location of the lesion, the adjacent relationship of surrounding blood vessels, and the direction of tissue. Moreover, the reconstructed three-dimensional images do not have convergence-accommodation conflicts, so surgeons will not feel dizzy when viewing them for a long time.

[0025] The function of the second display unit is to project the two-dimensional augmented information generated by the data processing terminal 2, combined with... Figure 2 It corresponds to the forward projection module, which accurately receives the two-dimensional augmented information transmitted from the data processing terminal 2, that is... Figure 2 The system uses 2D remote illustrations and resolution-enhanced images, then projects this information in high resolution to ensure that guidance details such as lesion annotations and operational prompts are clearly visible. The key effect of this part is to provide high-resolution guidance content, compensating for the limitations of simple 3D displays in terms of detail resolution.

[0026] The optical fusion unit is the core component for achieving seamless fusion of two-dimensional and three-dimensional information. Figure 2The core component is a semi-reflective, semi-transparent film, positioned on the viewing side of the 3D light control device. Its main function is to precisely superimpose the 2D enhanced information projected by the second display unit onto the corresponding spatial location of the 3D light field scene reconstructed by the first display unit in an optical form. Simply put, it allows 2D guidance information to be superimposed onto the corresponding location in the 3D scene. For example, at the location of the lesion in the 3D scene, the resection area marked by the surgeon is directly superimposed, providing the surgeon with a fused image that combines 3D depth with clear guidance details. This fusion method eliminates the need for surgeons to repeatedly switch between 3D and 2D guidance displays. They can accurately determine the lesion depth and surrounding structures through the 3D light field scene and obtain precise guidance through the superimposed 2D enhanced information, effectively improving surgical efficiency while avoiding the risk of operational errors caused by switching modes.

[0027] The collaborative workflow, which involves collecting information at the surgical acquisition end 1, processing it at the data processing end 2 to generate two-dimensional enhanced information, and then integrating and displaying it at the surgeon's workstation 3 for the surgeon to operate, achieves the synchronous presentation of three-dimensional depth perception and high-resolution two-dimensional guidance. The surgeon does not need to switch display modes, and the visual effect compensates for the single-viewpoint resolution loss of simple three-dimensional light field display. This not only improves the accuracy of surgical operations but also reduces surgical time and lowers surgical risks.

[0028] Furthermore, based on the above embodiments, the first display unit in this embodiment includes: a rear projection projector for projecting three-dimensional light field information; and a three-dimensional light control device disposed downstream of the optical path of the rear projection projector for spatially modulating the projected light of the rear projection projector and controlling the reconstruction of a three-dimensional light field scene with multiple discrete viewpoints in the viewing area.

[0029] Specifically, the rear projection projector receives the 3D light field signal generated by the data processing terminal 2, converts these signals into light rays, and projects them towards the semi-reflective membrane and the 3D light control device. These light rays carry the original information of the 3D light field scene, which is the basis for reconstructing the 3D image inside the laparoscopy within the 3D scene. After receiving the light rays projected by the rear projection projector, the 3D light control device performs spatial modulation on the light rays. According to the preset viewpoint distribution rules, it distributes the light rays to different spatial directions, controlling the light rays to reconstruct a 3D light field scene with multiple discrete viewpoints within the viewing area, that is, the surgeon's observation range. This allows the surgeon to see a clear 3D scene from different positions within the viewing area, and the multiple discrete viewpoints improve the accuracy of the stereoscopic effect, effectively avoiding visual fatigue.

[0030] Among them, the 3D light control device is either a lenticular lens grating or a microlens array. The lenticular lens grating, as a 3D light control device, uses a cylindrical lens structure to perform directional separation and convergence operations on the light from the rear projection projector, precisely guiding light from different viewpoints to the corresponding positions in the viewing area, achieving 3D light field reconstruction from multiple discrete viewpoints. The microlens array, as a 3D light control device, uses a distributed array of tiny lenses to perform fine-tuning operations on the light. Each microlens corresponds to one or more viewpoints, enabling more precise control of the light propagation direction and improving the resolution and stereoscopic effect of the 3D scene.

[0031] Furthermore, based on the above embodiments, the optical fusion unit in this embodiment includes: a semi-reflective and semi-transparent film, disposed on the viewing side of the three-dimensional light control device; the second display unit includes a front projection projector, the light path of the front projection projector passes through the three-dimensional light control device after being reflected by the semi-reflective and semi-transparent film; wherein, the light path of the rear projection projector passes through the semi-reflective and semi-transparent film and the three-dimensional light control device in sequence, so that the two-dimensional enhancement information and the light of the three-dimensional light field scene are spatially fused at the three-dimensional light control device.

[0032] Specifically, the core component of the optical fusion unit is a semi-reflective film. This film employs a neutral beam splitting design that balances transmission and reflection intensity, ensuring that the brightness of the three-dimensional light field transmitted from the rear projection matches the brightness of the two-dimensional enhanced information light reflected from the front projection. The semi-reflective film is positioned on the viewing side of the three-dimensional light control device, with its tilt angle precisely calibrated. Its function is to allow the light projected by the front projection projector, after reflection, to converge with the light transmitted from the rear projection on the same optical plane of the three-dimensional light control device, thereby achieving the fusion of two-dimensional and three-dimensional light.

[0033] The rear projection projector receives the encoded three-dimensional light field signal from the data processing terminal 2 and projects the light carrying multi-viewpoint three-dimensional information along the path of the rear projection, the semi-reflective and semi-transparent membrane, and the three-dimensional light control device. The three-dimensional light control device spatially modulates the light according to the preset viewpoint distribution rules, guiding the three-dimensional light from different viewpoints to the corresponding angle of the viewing area, forming a multi-viewpoint three-dimensional light field scene covering the surgeon's observation range, which is the reconstructed 3D image inside the laparoscopy.

[0034] The front projection projector receives the two-dimensional enhanced information generated by the data processing unit 2, which includes instructions for doctors to annotate and resolution enhancement details. It then projects the light carrying the two-dimensional enhanced information along the path of the front projection, the semi-reflective membrane, and the three-dimensional light control device. When the light hits the semi-reflective membrane, it undergoes specular reflection. The propagation direction of the reflected light is consistent with that of the rear projection light, and it is incident perpendicularly onto the three-dimensional light control device.

[0035] Data processing terminal 2 performs viewpoint calibration on the projected two-dimensional enhanced information based on real-time human eye tracking information. It maps the pixel coordinates of the two-dimensional image to the three-dimensional viewpoint coordinates of the surgeon, ensuring that the reflected two-dimensional light rays can be accurately superimposed on the corresponding spatial position of the three-dimensional light field scene, such as the edge of the lesion, the branch point of the blood vessel, and other key areas, to avoid spatial misalignment.

[0036] The three-dimensional light control device simultaneously receives two types of light rays and completes spatial fusion. The fusion process based on the lenticular lens grating is as follows: After receiving the rear-projected three-dimensional light, the lenticular lens grating decomposes the incident multi-viewpoint mixed light into independent discrete viewpoint light. Lenticular lenses at different positions project light into corresponding directions within the surgeon's field of vision. The light from the left lenticular lens is deflected towards the surgeon's left eye, the light from the right lenticular lens is deflected towards the surgeon's right eye, and the lenticular lenses in the middle area sequentially cover the transitional viewpoint between the left and right eyes, forming a continuous three-dimensional depth perception.

[0037] Two-dimensional enhancement rays incident on the same cylindrical lens as the three-dimensional light field rays are modulated by the cylindrical lens at the exact same projection angle as the corresponding three-dimensional viewpoint rays. Two-dimensional pixel rays incident on the left cylindrical lens are projected towards the surgeon's left eye along with the three-dimensional viewpoint rays, while two-dimensional pixel rays incident on the right cylindrical lens are projected towards the right eye along with the three-dimensional viewpoint rays, ensuring that the two-dimensional information is synchronized with the viewpoint of the three-dimensional scene.

[0038] Three-dimensional viewpoint rays and two-dimensional pixel rays, deflected by the same cylindrical lens, exit along the same optical path, forming a mixed ray carrying depth and guidance information. All cylindrical lenses perform this process synchronously, ultimately forming a fused light field in the viewing area where each viewpoint is superimposed with corresponding two-dimensional guidance details. The mixed ray received by the surgeon's left eye contains the three-dimensional depth of the left viewpoint and the corresponding two-dimensional annotation, while the mixed ray received by the right eye contains the three-dimensional depth of the right viewpoint and the corresponding two-dimensional annotation. Through binocular parallax fusion, a stereoscopic perception is formed that precisely covers the two-dimensional guidance information on the three-dimensional scene.

[0039] The fusion process based on microlens arrays is as follows: After receiving the rear-projected three-dimensional light rays, each microlens focuses the incident multi-directional three-dimensional light rays to a preset discrete viewpoint position. Microlenses in different regions focus the light rays to the corresponding positions in the viewing area, and through the synergistic effect of all microlenses, a multi-viewpoint three-dimensional light field is constructed.

[0040] Two-dimensional enhancement rays incident on the same microlens, corresponding to a specific pixel position in a three-dimensional viewpoint, are focused by the microlens to a spatial position exactly the same as the three-dimensional viewpoint. Three-dimensional rays focused to a particular viewpoint, after being focused by the microlens, intersect at the same spatial point with the corresponding two-dimensional annotation rays, achieving fusion.

[0041] The three-dimensional focused rays and two-dimensional focused rays output by each microlens do not separate or appear as ghosting after emission because their focal points completely overlap. Instead, they form a composite ray with superimposed information. The composite rays from all the microlenses together constitute a fused light field. When the surgeon observes, the three-dimensional scene seen from each perspective can be accurately superimposed with the corresponding two-dimensional guidance information. The guidance information fits closely with the three-dimensional scene without any floating or offset phenomena.

[0042] In this system, when data processing terminal 2 sends signals to the rear projection projector and the front projection projector, it uses the same clock triggering mechanism to ensure that the 3D light field rays and the 2D enhanced rays are simultaneously incident on the 3D light control device. This avoids asynchrony issues during dynamic fusion due to timing differences, and is particularly suitable for scenarios where the surgical area is dynamically changing during surgery. The composite light rays, after being fused by the 3D light control device, are transmitted in a straight line along the path from the 3D light control device to the surgeon's eye. Because the light rays have achieved spatial alignment and pixel matching, the human eye can simultaneously perceive 3D depth and 2D details without additional adjustment of eyeball convergence. The 3D light field rays provide the spatial distance between the lesion and surrounding tissues, as well as the depth level of blood vessel orientation, while the 2D enhanced rays provide high-resolution guidance and annotation and a clear presentation of the lesion's fine structure, compensating for the insufficient single-viewpoint resolution of the 3D light field.

[0043] The projector projects the corresponding two-dimensional viewpoint image calculated by human eye tracking. Guidance information transmitted via remote live streaming terminal The light is projected onto a semi-reflective membrane through a 3D light-controlling device, reflected by the membrane, and then passes through the light-controlling device again before reaching the viewer's eye. The information loaded by the front projection projector and the 3D light field image... Overlay display as a blended image Integrating as shown in Equation 1: (1) After the two are integrated, the surgeon sees a unified image combining a three-dimensional scene and two-dimensional guidance. The markings are directly aligned with the lesion surface, and the operation prompts follow the direction of the blood vessels, eliminating the need to switch display modes. Furthermore, because the light fusion process is visually seamless, surgeons will not experience eye strain even after prolonged viewing, making it particularly suitable for the extended procedures required in endoscopic surgery.

[0044] Furthermore, based on the above embodiments, the single-viewpoint resolution reconstructed by the first display unit in this embodiment satisfies the relationship (2) with the resolution of the rear projection projector and the number of reconstructed viewpoints: P1 = P2 / N (2) Where P1 represents the single-viewpoint resolution reconstructed by the first display unit, P2 represents the resolution of the rear projection projector, and N represents the number of reconstructed viewpoints. The two-dimensional enhancement information provided by the second display unit is used to improve the effective single-viewpoint resolution perceived by the viewer.

[0045] Specifically, as can be seen from formula (2), the single-viewpoint resolution decreases as the number of reconstructed viewpoints increases. If more viewpoints are needed to enhance the sense of depth, for example, by adding more viewpoints... Figure 2 The more realistic the reconstructed 3D image within the laparoscopy is, the lower the single-viewpoint resolution will be. The 2D enhanced information projected by the second display unit (forward projection) is fused with the 3D light field scene reconstructed by the first display unit—the reconstructed 3D image within the laparoscopy—to improve the effective single-viewpoint resolution perceived by the surgeon. In other words, the 3D light field scene provides depth information, but its single-viewpoint resolution is limited, while the 2D enhanced information is high-resolution, containing clear details and guidance. After fusion, the surgeon can perceive 3D depth while simultaneously seeing details through the superimposed 2D enhanced information, such as minute lesions and fine branches of blood vessels. This compensates for the single-viewpoint resolution loss of the 3D light field display without reducing the number of viewpoints or the sense of depth. This allows the surgeon to obtain accurate depth perception while also seeing high-resolution guidance details and surgical area details, facilitating precise judgment and operation during surgery.

[0046] Furthermore, based on the above embodiments, such as Figure 1 As shown, this embodiment also includes: a doctor guidance terminal 4, which is communicatively connected to the data processing terminal 2, for receiving and displaying two-dimensional image information, and generating guidance information for graphical annotation of the two-dimensional image information in response to interactive operations.

[0047] Specifically, the doctor's workstation 4 establishes a communication connection with the data processing terminal 2, receiving two-dimensional image information transmitted from the data processing terminal 2 in real time. This two-dimensional image information consists of key viewpoint images of the surgical area captured by the surgical acquisition terminal 1, allowing the doctor to clearly observe the real-time surgical situation. The doctor performs interactive operations using the interactive device on the doctor's workstation 4, such as a touchscreen or electronic pen, marking lesion locations, delineating surgical areas, and writing operational precautions on the two-dimensional image. These interactive operations generate corresponding guidance information. Subsequently, the doctor's workstation 4 transmits the generated guidance information back to the data processing terminal 2 in real time, for integration and generation of enhanced two-dimensional information.

[0048] When the system is in live interactive mode, the live interactive terminal will replace or work in parallel with the guiding doctor's workstation 4. The live interactive terminal receives two-dimensional image information and some three-dimensional light field-related live data transmitted by the data processing terminal 2, and simultaneously pushes the real-time surgical footage to multiple viewing terminals, such as other doctors' training equipment and academic exchange terminals, through the live streaming platform. Viewers can send simple interactive information through the message and annotation functions of the live interactive terminal. After being filtered and integrated by the data processing terminal 2, this information will also be incorporated into the two-dimensional enhanced information, enabling multi-user sharing and interactive communication.

[0049] By adding a fourth working terminal for supervising physicians, interaction between surgeons and supervising physicians is realized, and live streaming sharing and interaction among multiple users are also supported. Supervising physicians can provide precise guidance without being present at the surgical site, and viewers can learn about the surgical process in real time. This is especially suitable for scenarios such as endoscopic surgery that require long-term guidance, training, and academic sharing.

[0050] Furthermore, based on the above embodiments, the data processing terminal 2 in this embodiment is specifically used for: encoding the three-dimensional light field information to generate a three-dimensional light field signal suitable for reconstruction by the first display unit; and performing image integration operations on the two-dimensional image information and the guidance information to generate two-dimensional enhancement information.

[0051] Specifically, after receiving the three-dimensional light field information transmitted from the surgical acquisition terminal 1, the data processing terminal 2 performs encoding processing. Based on the display parameters of the first display unit, such as display resolution and viewpoint distribution requirements, it performs format conversion, data compression, and signal modulation on the three-dimensional light field information to generate a three-dimensional light field signal suitable for the first display unit to reconstruct the three-dimensional light field scene. Through encoding processing, it ensures that the first display unit can accurately identify the three-dimensional light field information, avoiding distortion of the three-dimensional scene due to signal mismatch.

[0052] After receiving the 2D image information from the surgical acquisition terminal 1 and the guidance information from the doctor's workstation 4 / live interactive terminal, data processing terminal 2 performs image integration calculation. Using the 2D image information as a base layer, the guidance information, such as annotations, comments, and live interactive messages, is superimposed onto the base layer with pixel-level precision. Simultaneously, image enhancement processing is performed to improve contrast and sharpen edges, making the guidance information clearer, ultimately generating enhanced 2D information. Through image integration calculation, the 2D image information and guidance information are perfectly integrated, avoiding any deviation or blurring of the guidance information, ensuring that the surgeon can clearly observe the details of the surgical area and the guidance content in the fused image.

[0053] Furthermore, based on the above embodiments, the data processing terminal 2 in this embodiment is also used to: extract a two-dimensional view corresponding to the current viewing point from the three-dimensional light field information based on human eye tracking information, and send the two-dimensional view as two-dimensional image information to the doctor's working terminal 4.

[0054] Specifically, data processing terminal 2 receives eye-tracking information from the eye-tracking module in real time, including the surgeon's eye position and viewing angle. Based on this eye-tracking information, it accurately extracts a two-dimensional view corresponding to the surgeon's current viewpoint from the three-dimensional light field information transmitted from surgical acquisition terminal 1. Subsequently, the extracted two-dimensional view is sent as the official two-dimensional image information to the supervising physician's workstation 4 / live broadcast interaction terminal. This ensures that the two-dimensional image seen by the supervising physician and live broadcast viewers is consistent with the surgeon's current viewpoint, avoiding misalignment of guidance information due to inconsistent viewpoints. This makes remote guidance and live broadcast interaction more accurate and reduces surgical risks caused by information discrepancies.

[0055] Furthermore, based on the above embodiments, such as Figure 1 As shown, the surgical acquisition terminal 1 in this embodiment includes: a light field camera equipped with a microlens array, used to acquire three-dimensional light field information from different spatial positions and angles of the surgical area; the surgical acquisition terminal 1 downsamples and encodes the three-dimensional light field information and transmits it to the data processing terminal 2. The surgeon's workstation 3 also includes an operating handle, used to control the surgical robot in the surgical acquisition terminal 1 to perform surgical operations.

[0056] Specifically, the surgical acquisition terminal 1 includes a light field camera equipped with a microlens array and a surgical robot. The miniaturized light field camera can be inserted into the patient's body and simultaneously captures three-dimensional light field information of different spatial positions and angles in the surgical area, as well as two-dimensional information of key viewpoints, through the front-end microlens array. The three-dimensional light field information covers data such as lesion depth, contour, and tissue adjacency. After acquisition, the surgical acquisition terminal 1 performs downsampling encoding processing on the three-dimensional light field information, optimizing the data format and compressing the data volume while retaining key depth information. Subsequently, the encoded three-dimensional light field information and key viewpoint two-dimensional information are transmitted in real time to the data processing terminal 2, and some two-dimensional information is simultaneously transmitted to the doctor's work terminal 4, providing a precise data source for three-dimensional scene reconstruction, two-dimensional enhancement information generation, remote guidance, and live streaming.

[0057] like Figure 1 As shown, the surgeon's workstation 3 is equipped with a control handle, which establishes a communication connection with the surgical robot at the surgical acquisition end 1. While viewing the fused image displayed on the 2D-3D fused light field display, the surgeon performs various surgical operations via the control handle. The control handle captures the pose information of the movements in real time and converts it into standardized control signals, which are then transmitted to the surgical robot. After receiving the control signals, the surgical robot performs the corresponding surgical operation within the patient's body.

[0058] The surgical acquisition terminal 1 collects information, the data processing terminal 2 processes signals, the surgeon's workstation 3 displays and operates the data, and the surgical robot executes actions, forming a closed-loop link. The light field camera provides real-time feedback on the post-operative scene information, enabling dynamic linkage. This solves the problems of inaccurate information perception, asynchronous guidance, and operational deviations in traditional endoscopic surgery, improving surgical safety and efficiency, and providing technical support for surgical training and remote live streaming.

[0059] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0060] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 2D / 3D display fusion-based remote surgical guidance system, characterized in that, include: The surgical acquisition end is used to acquire three-dimensional light field information and two-dimensional image information of the surgical area; The data processing end is used to receive the two-dimensional image information and guidance information from the outside, and generate two-dimensional enhancement information; The surgeon's workstation includes a two-dimensional-three-dimensional fused light field display, which includes a first display unit, a second display unit, and an optical fusion unit; The first display unit is used to reconstruct a three-dimensional light field scene based on the three-dimensional light field information; The second display unit is used to project the two-dimensional enhanced information; An optical fusion unit is used to superimpose the two-dimensional enhancement information onto the corresponding spatial position of the three-dimensional light field scene in an optical form for fusion display.

2. The 2D / 3D display fusion surgical remote guidance system according to claim 1, characterized in that, The first display unit includes: A rear projection projector is used to project the three-dimensional light field information; A three-dimensional light control device is disposed downstream of the optical path of the rear projection projector to spatially modulate the projected light of the rear projection projector and control the reconstruction of a three-dimensional light field scene with multiple discrete viewpoints in the viewing area.

3. The 2D / 3D display fusion-based remote surgical guidance system according to claim 2, characterized in that, The three-dimensional light control device is a cylindrical lens grating or a microlens array.

4. The 2D / 3D display fusion surgical remote guidance system according to claim 3, characterized in that, The optical fusion unit includes: A semi-reflective and semi-transparent film is disposed on the viewing side of the three-dimensional light control device; The second display unit includes a front projection projector, the light path of which passes through the three-dimensional light control device after being reflected by the semi-reflective and semi-transparent film. The optical path of the rear projection projector passes sequentially through the semi-reflective and semi-transparent film and the three-dimensional light control device, so that the two-dimensional enhancement information and the light of the three-dimensional light field scene are spatially fused at the three-dimensional light control device.

5. The 2D / 3D display fusion surgical remote guidance system according to claim 4, characterized in that, The single-viewpoint resolution reconstructed by the first display unit satisfies the following relationship with the resolution and number of reconstructed viewpoints of the rear projection projector: P1 = P2 / N; Where P1 represents the single-viewpoint resolution reconstructed by the first display unit, P2 represents the resolution of the rear projection projector, and N represents the number of reconstructed viewpoints. The two-dimensional enhancement information provided by the second display unit is used to improve the effective single-viewpoint resolution perceived by the viewer.

6. The 2D / 3D display fusion surgical remote guidance system according to claim 1, characterized in that, Also includes: The doctor's workstation is connected to the data processing terminal for receiving and displaying the two-dimensional image information, and generating guidance information for illustrating and annotating the two-dimensional image information in response to interactive operations.

7. The 2D / 3D display fusion surgical remote guidance system according to claim 6, characterized in that, The data processing terminal is specifically used for: The three-dimensional light field information is encoded to generate a three-dimensional light field signal suitable for reconstruction by the first display unit; The two-dimensional image information and the guidance information are combined to perform image integration to generate two-dimensional enhancement information.

8. The 2D / 3D display fusion surgical remote guidance system according to claim 7, characterized in that, The data processing terminal is also used for: Based on human eye tracking information, a two-dimensional view corresponding to the current viewing point is extracted from the three-dimensional light field information, and the two-dimensional view is sent to the guiding doctor's terminal as the two-dimensional image information.

9. The 2D / 3D display fusion surgical remote guidance system according to any one of claims 1-8, characterized in that, The surgical acquisition terminal includes: a light field camera equipped with a microlens array, used to acquire three-dimensional light field information from different spatial positions and angles of the surgical area; the surgical acquisition terminal downsamples and encodes the three-dimensional light field information and transmits it to the data processing terminal.

10. The 2D / 3D display fusion surgical remote guidance system according to any one of claims 1-8, characterized in that: The surgeon's workstation also includes an operating handle for controlling the surgical robot in the surgical acquisition unit to perform surgical operations.

Images