Holographic three-dimensional medical data visualization method and system capable of achieving mid-air gesture interaction

Abstract

The present application relates to the technical field of human-computer interaction, and specifically to a holographic three-dimensional medical data visualization method and system capable of achieving mid-air gesture interaction. The method comprises the following steps: firstly, acquiring user environment information and user motion information, wherein the user environment information is image information of a real scene around a user, and the user wears a mixed reality display device; then, on the basis of the user environment information and the user motion information, generating a virtual image by using a three-dimensional interpolation volume rendering method; next, by using an optical model, projecting the virtual image into the visual field of the user, and merging the virtual image with the real scene to obtain a mixed reality rendered three-dimensional model; and finally, on the basis of the mixed reality rendered three-dimensional model, capturing a gesture of the user by using a gesture recognition method, and performing mid-air gesture interaction. The present application achieves mid-air gesture interaction between a user and a virtual scene by means of efficient three-dimensional object volume rendering and a real-time accurate interaction method design.

Description

Holographic three-dimensional medical data visualization method and system capable of air gesture interaction Technical Field

[0001] The embodiments of the present application relate to the field of human-computer interaction technology, and in particular to a holographic three-dimensional medical data visualization method and system capable of gesture interaction through air. Background Art

[0002] With the continuous development and innovation of digital medicine, extended reality (XR) has gradually become a key technology to assist surgeons in clinical diagnosis and treatment. Thanks to the interdisciplinary integration of medicine and computer science, multimodal medical data obtained from medical imaging, such as computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound (US), and computer-rendered and reconstructed models, have enabled three-dimensional visualization of medical data, providing broad opportunities for the development of XR-assisted medicine.

[0003] XR-based medical interactive visualization systems equipped with HoloLen 2 head-mounted displays have demonstrated their potential for presenting high-quality 3D patient data. Building on 3D visualization, surgical simulations provide doctors with an immersive surgical training environment, allowing them to manipulate 3D objects within the environment. This enhances their visual experience, helps them make intuitive diagnoses, and effectively avoids the risks associated with real-world surgery. For holographic XR medical visualization systems, efficient preoperative volume rendering and real-time, accurate intraoperative interaction are key to assisting doctors in making surgical diagnoses.

[0004] Unlike traditional two-dimensional displays of medical data, XR can more realistically and accurately represent three-dimensional anatomical structures during intraoperative diagnosis and preoperative surgical planning. With the development of GPU technology, volume rendering has made significant progress in recent years. Volume rendering utilizes direct volume rendering techniques to enhance visual and spatial perception from three-dimensional datasets. Existing methods for volume rendering using GPUs have never fully achieved the same quality and performance as those used in medical augmented reality. Proposed methods are limited to rendering polygonal surface data. While relevant anatomy and structures can be depicted, these methods are too time-consuming to meet the efficient volume rendering requirements of XR-assisted diagnostic systems. Technical issues

[0005] The embodiments of the present application provide a holographic three-dimensional medical data visualization method and system that can interact with gestures in the air. Through efficient three-dimensional object rendering and real-time and accurate interaction method design, users can move, rotate, scale and cut virtual tissues from any angle, manipulate virtual surgical instruments in real time and observe the patient's lesion condition, thereby realizing gesture interaction between users and virtual scenes in the air. Technical Solutions

[0006] To solve the above technical problems, in the first aspect, an embodiment of the present application provides a holographic three-dimensional medical data visualization method that can interact with gestures through the air, including the following steps: first, obtaining user environment information and user motion information; the user environment information is image information of the real scene around the user; the user wears a mixed reality display device; then, based on the user environment information and user motion information, a stereo interpolation volume rendering method is used to generate a virtual image; next, an optical model is used to project the virtual image into the user's field of view, and the virtual image and the real scene are fused to obtain a three-dimensional model rendered by mixed reality; finally, based on the three-dimensional model rendered by mixed reality, the user's gestures are captured by a gesture recognition method and gesture interaction is performed through the air to realize interaction between the user and the virtual scene.

[0007] In some exemplary embodiments, the mixed reality display device is a head-mounted mixed reality display device HoloLens 2; and the optical model is an optical model in the head-mounted mixed reality display device HoloLens 2.

[0008] In some exemplary embodiments, the mixed reality display device has a built-in depth camera, an inertial measurement unit, and multiple sensors to obtain user environment information and user motion information.

[0009] In some exemplary embodiments, a recognition camera and a tracking camera are also provided in the mixed reality display device; the recognition camera is used to detect and recognize specific images, objects or scenes in the real scene; the tracking camera is used to track the position and direction of the camera, and place the virtual image in the real world according to the output of the recognition camera to achieve virtual-reality registration.

[0010] In some exemplary embodiments, a virtual image and a real scene are fused to obtain a three-dimensional model of mixed reality rendering, including: generating the virtual image into a three-dimensional model of the virtual image through 3D modeling and rendering technology, adjusting the properties of the virtual image in real time, and fusing the three-dimensional model of the virtual image with the real scene to obtain a three-dimensional model of mixed reality rendering; wherein the properties of the virtual image include the position, size and transparency of the virtual image.

[0011] In some exemplary embodiments, the three-dimensional model rendered by the mixed reality includes three-dimensional models at multiple different angles; wherein the three-dimensional models at different angles correspond to the user's observation angle.

[0012] In some exemplary embodiments, when performing air gesture interaction, the user uses gestures to move, rotate, scale, and shear the virtual tissue in the three-dimensional model rendered by the mixed reality from any angle, manipulate virtual surgical instruments in real time, and observe the patient's lesion condition.

[0013] In some exemplary embodiments, after obtaining the three-dimensional model of mixed reality rendering, the method further includes: evaluating the three-dimensional model of mixed reality rendering.

[0014] In some exemplary embodiments, evaluating the three-dimensional model rendered by the mixed reality includes: using medical images of three-dimensional CT original data, medical images of three-dimensional CT reconstructed data, and medical images of three-dimensional ultrasound original data as comparison data, and evaluating the efficiency and real-time performance of the three-dimensional model rendered by the mixed reality.

[0015] On the second aspect, the embodiments of the present application also provide a holographic three-dimensional medical data visualization system that can interact with gestures through the air, including a mixed reality display device and a gesture recognition system connected to the mixed reality display device; the mixed reality display device is worn on the user's head; the mixed reality display device has a built-in depth camera, an inertial measurement unit and multiple sensors for obtaining user environmental information and user motion information; the user environmental information is image information of the real scene around the user; the mixed reality display device also uses a stereo interpolation volume rendering method to generate a virtual image based on the user environmental information and user motion information; and uses an optical model to project the virtual image into the user's field of view, and fuse the virtual image and the real scene to obtain a three-dimensional model rendered by mixed reality; the gesture recognition system is used to capture the user's gestures and perform gesture interaction through the air based on the three-dimensional model rendered by mixed reality through a gesture recognition method, so as to realize interaction between the user and the virtual scene. Beneficial effects

[0016] The technical solution provided by the embodiments of the present application has at least the following advantages:

[0017] An embodiment of the present application provides a holographic three-dimensional medical data visualization method and system that can interact with gestures through the air. The method includes the following steps: first, obtaining user environment information and user motion information; the user environment information is image information of the real scene around the user; the user wears a mixed reality display device; then, based on the user environment information and user motion information, a stereo interpolation volume rendering method is used to generate a virtual image; next, an optical model is used to project the virtual image into the user's field of view, and the virtual image and the real scene are fused to obtain a three-dimensional model rendered by mixed reality; finally, based on the three-dimensional model rendered by mixed reality, the user's gestures are captured by a gesture recognition method and gesture interaction is performed through the air to realize interaction between the user and the virtual scene.

[0018] In order to enable doctors to better perceive the spatial structure of the patient's target area, this application proposes a holographic three-dimensional medical data intuitive visualization system under air gesture interaction suitable for surgical disciplines, covering preoperative planning and holographic three-dimensional visualization of tissue structure, using augmented reality technology to help surgeons predict tissue structure information and cut tissue organs under the guidance of virtual information. Combined with alignment tracking technology and air gesture interaction, it assists surgeons in performing surgical operations naturally, thereby transforming traditional surgery into predictable and accurately guided surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] One or more embodiments are exemplarily described by the pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Unless otherwise stated, the pictures in the drawings do not constitute proportional limitations.

[0020] FIG1 is a flow chart of a holographic three-dimensional medical data visualization method capable of air gesture interaction provided by an embodiment of the present application;

[0021] FIG2 is a schematic diagram of image recognition and virtual-real registration based on Vuforia provided in one embodiment of the present application;

[0022] FIG3 is a schematic diagram of the holographic gesture interaction principle provided by an embodiment of the present application. Modes for Carrying Out the Invention

[0023] As can be seen from the background, existing volume rendering methods utilize direct volume rendering techniques to enhance visual and spatial perception from 3D datasets. When using GPUs for volume rendering, existing methods have never fully achieved the same quality and performance as those used in medical augmented reality. Consequently, existing methods suffer from the technical problem of being limited to rendering polygonal surface data and failing to meet the high-efficiency volume rendering requirements of XR-assisted diagnostic systems.

[0024] Unlike traditional two-dimensional display technologies for medical data, XR can more realistically and accurately present three-dimensional anatomical structures during intraoperative diagnosis and preoperative surgical planning. With the development of GPU technology, volume rendering technology has made tremendous progress in recent years. Direct volume rendering technology is used to enhance visual and spatial perception from three-dimensional datasets. A related technology proposed is ClearView, which is used for non-segmented, real-time, focal and background visualization of three-dimensional data. ClearView draws the viewer's attention to the focal area, where the important features of the three-dimensional data are embedded. The transparency of the surrounding environment layer is adjusted based on the curvature characteristics of the surrounding layer or the distance from the focal layer.

[0025] Most early interaction technologies were designed from a device-centric perspective, such as using two-dimensional touchscreen input, as well as styluses, keyboards, and device sensors to interact with virtual objects. Another related technology, after a comprehensive analysis of immersive visualization, proposed: immersive interactive interfaces can utilize people's stereoscopic vision to display data, shifting the data representation space from a two-dimensional plane to the three-dimensional space surrounding the user; provide mobility, so that the user's physical work environment is no longer limited to a fixed office desk; and provide natural interaction methods such as gestures to make interaction more intuitive, allowing users to interact with data through multiple channels in parallel.

[0026] Because existing methods have never fully achieved the same quality and performance as those used in medical augmented reality, proposed methods are limited to rendering polygonal surface data. Although relevant anatomy and structures can be depicted, these methods are too time-consuming to meet the efficient volume rendering requirements of XR-assisted diagnosis systems. Through analysis and research of relevant scholars and case studies, medical-assisted diagnosis systems should be efficient and have as fast rendering speed as possible.

[0027] To solve the above technical problems, the embodiment of the present application provides a holographic three-dimensional medical data visualization method and system that can interact with gestures in the air. The method includes the following steps: first, obtaining user environment information and user motion information; the user environment information is image information of the real scene around the user; the user wears a mixed reality display device; then, based on the user environment information and user motion information, a stereo interpolation volume rendering method is used to generate a virtual image; next, an optical model is used to project the virtual image into the user's field of view, and the virtual image and the real scene are fused to obtain a three-dimensional model rendered by mixed reality; finally, based on the three-dimensional model rendered by mixed reality, the user's gesture is captured by a gesture recognition method and gesture interaction is performed in the air to realize the interaction between the user and the virtual scene. The embodiment of the present application provides a holographic three-dimensional medical data visualization method and system that can interact with gestures in the air. Through efficient three-dimensional object rendering and real-time and accurate interaction method design, the user can move, rotate, scale and cut virtual tissue from any angle, manipulate virtual surgical instruments in real time and observe the patient's lesion condition, thereby realizing gesture interaction between the user and the virtual scene in the air.

[0028] The following detailed description of the various embodiments of the present application is provided in conjunction with the accompanying drawings. However, those skilled in the art will appreciate that many technical details are provided in the various embodiments of the present application to facilitate a better understanding of the present application. However, even without these technical details and the various variations and modifications based on the following embodiments, the technical solutions claimed in the present application can still be implemented.

[0029] 1 , an embodiment of the present application provides a holographic three-dimensional medical data visualization method capable of air gesture interaction, including:

[0030] Step S1: Obtain user environment information and user motion information; the user environment information is image information of the real scene around the user; the user wears a mixed reality display device.

[0031] Step S2: Based on the user environment information and the user motion information, a stereo interpolation volume rendering method is used to generate a virtual image.

[0032] Step S3: Use an optical model to project the virtual image into the user's field of view, and fuse the virtual image with the real scene to obtain a three-dimensional model of mixed reality rendering.

[0033] Step S4: Based on the three-dimensional model rendered by mixed reality, the user's gestures are captured through gesture recognition methods and air gesture interaction is performed to enable interaction between the user and the virtual scene.

[0034] In some embodiments, the mixed reality display device is a head-mounted mixed reality display device HoloLens 2; and the optical model is an optical model in the head-mounted mixed reality display device HoloLens 2.

[0035] This application proposes an interactive three-dimensional medical data holographic visualization system adapted for use with the HoloLens2 head-mounted display device. Through efficient three-dimensional object volume rendering and real-time and accurate interactive method design, the system enables surgeons to move, rotate, scale, and shear virtual tissues from any angle, manipulate virtual surgical instruments in real time, and observe the patient's lesions, making intuitive and accurate diagnosis of areas of interest. Through experiments based on three-dimensional CT raw data, three-dimensional CT reconstruction data, and three-dimensional ultrasound raw data on HoloLens2, compared with advanced volume rendering algorithm models, the rendering effect and speed of the method of this application have certain advantages, achieving rendering speeds of 42.00±5.80 FPS, 60.00±1.80 FPS, and 42.05±5.55 FPS on HoloLens2, respectively, verifying the feasibility of the system of this application.

[0036] In some embodiments, the mixed reality display device has a built-in depth camera, an inertial measurement unit, and multiple sensors to obtain user environment information and user motion information.

[0037] Specifically, this application uses Microsoft's new generation mixed reality head-mounted display device HoloLens2 to achieve mixed reality display of three-dimensional models. Based on the technology of HoloLens2 to merge virtual images with real scenes, its principle is mainly to use multiple sensors such as depth cameras, inertial measurement units and optical sensors to obtain image information of real scenes and user motion information, and then transmit this information to a computer for processing and analysis, and finally generate a virtual image and project it into the user's field of view through the optical device of HoloLens2 to achieve a mixed reality display effect. In the mixed reality display of HoloLens2, the fusion of virtual images and real scenes is the key to achieving mixed reality. To achieve this goal, HoloLens2 adopts a variety of technologies, including optical technology, sensor technology and computer graphics technology. Among them, optical technology is the basis of mixed reality display. It uses transparent waveguides to project virtual images into the user's field of view, and realizes the fusion of virtual images and real scenes through optical principles such as reflection and refraction.

[0038] Sensor technology perceives the user's surroundings and their movements, transmitting this information to a computer for processing and analysis. HoloLens 2 utilizes a variety of sensors, including a depth camera, inertial measurement unit, and optical sensors. These sensors can acquire real-time information about the user's position, posture, and gestures, enabling precise tracking and positioning. This allows the position and size of the virtual image to be adjusted to seamlessly blend with the real scene. Computer graphics are used to generate and display the virtual image in the user's field of view. In HoloLens 2, 3D modeling and rendering technologies are used to generate a realistic three-dimensional model of the virtual image, and its position, size, transparency, and other attributes can be adjusted in real time to achieve integration with the real scene.

[0039] In some embodiments, the mixed reality display device is further provided with a recognition camera and a tracking camera; the recognition camera is used to detect and recognize specific images, objects or scenes in the real scene; the tracking camera is used to track the position and direction of the camera, and place the virtual image in the real world according to the output of the recognition camera to achieve virtual-reality registration.

[0040] The holographic 3D medical data visualization method designed in this application, which enables gesture interaction through air, utilizes the Vuforia engine, as shown in Figure 2. Vuforia uses image recognition and tracking capabilities through the mixed reality device's built-in cameras (recognition camera and tracking camera) to locate specific patterns. Vuforia's basic architecture consists of a recognition engine and a tracking engine. The recognition engine detects and recognizes specific images, objects, or scenes in the real world. The tracking engine is responsible for tracking the camera's position and orientation and accurately placing virtual content in the real world based on the recognition engine's output. This method enables precise virtual-real registration.

[0041] In some embodiments, the virtual image and the real scene are fused to obtain a three-dimensional model of mixed reality rendering, including: generating the virtual image into a three-dimensional model of the virtual image through 3D modeling and rendering technology, and adjusting the properties of the virtual image in real time, and fusing the three-dimensional model of the virtual image with the real scene to obtain a three-dimensional model of mixed reality rendering; wherein the properties of the virtual image include the position, size and transparency of the virtual image.

[0042] In some embodiments, the three-dimensional model rendered by the mixed reality includes three-dimensional models at multiple different angles; wherein the three-dimensional models at different angles correspond to the user's observation angle.

[0043] Specifically, in the holographic 3D medical data visualization method with gesture interaction provided in the embodiments of the present application, the aforementioned illumination model is used for mixed reality rendering using the HoloLens 2. After the model is rendered, the model is displayed at different angles based on the viewing angle and position of the HoloLens 2 wearer.

[0044] In some embodiments, when performing air gesture interaction, the user uses gestures to move, rotate, scale, and shear the virtual tissue in the three-dimensional model rendered by the mixed reality from any angle, manipulate virtual surgical instruments in real time, and observe the patient's lesion condition.

[0045] Specifically, this application provides a holographic 3D medical data visualization method that enables gesture-based interaction. Human-computer interaction utilizes gesture-based portable holographic visualization. This gesture-based portable holographic visualization technology is a human-computer interaction method based on holographic projection and gesture recognition. It primarily projects a virtual 3D scene onto a real scene, then uses gesture recognition technology to capture user gestures and enable interaction. This allows for a natural and intuitive interaction between the user and the virtual scene, enabling more refined and flexible operations.

[0046] Interaction requires capturing the user's hand movements to identify gestures and convert them into corresponding commands to control the virtual scene. By training and recognizing a large amount of gesture data, the accuracy, real-time performance, and stability of gesture recognition can be improved.

[0047] Figure 2 illustrates the principles of gesture interaction. A gesture recognition system uses a camera or depth sensor to capture the user's hand movements and converts them into digital signals. These signals are fed into a computer and processed using algorithms and models to identify different gestures. These gestures can represent various commands, such as swiping, zooming, and rotating. The gesture recognition system undergoes extensive training to identify different gestures.

[0048] Computer vision technology helps gesture recognition systems identify specific patterns and shapes in images, allowing them to accurately identify different gestures. For example, it can detect information such as finger position, gesture direction, speed, and size, helping the system determine the user's gesture. The interactive method in this application supports movement and rotation of the entire model, as well as scaling key local areas.

[0049] Additionally, users can perform cuts with hand gestures, allowing surgeons to view various cross-sections of the target area, allowing for comprehensive observation and analysis of the patient's lesions.

[0050] In some embodiments, after obtaining the three-dimensional model of mixed reality rendering, the method further includes: evaluating the three-dimensional model of mixed reality rendering.

[0051] In some embodiments, the three-dimensional model rendered by mixed reality is evaluated, including: using medical images of three-dimensional CT original data, medical images of three-dimensional CT reconstructed data, and medical images of three-dimensional ultrasound original data as comparison data, and evaluating the efficiency and real-time performance of the three-dimensional model rendered by mixed reality.

[0052] This application uses two types of medical data, CT images and ultrasound images, to test and evaluate the efficiency and real-time performance of the 3D model rendered by mixed reality provided by this application. For each type of medical data, the experiment was conducted according to the following steps:

[0053] (1) Obtain 3D CT raw data with a size of (270, 512, 512), 3D CT reconstructed data with a slice resolution of 512×512 and a slice thickness of 0.7-0.8 mm, or 3D ultrasound raw data with a resolution of 183×115×126 FPS.

[0054] (2) Adjust the transfer function to emphasize and classify the features of interest in the data to truly restore the material of areas such as bones, blood vessels, or soft tissues.

[0055] (3) Surgeons wearing HoloLens2 interact with medical 3D data.

[0056] The efficiency and real-time performance of the medical interactive visualization method are tested and evaluated below.

[0057] To verify the efficiency of medical interactive visualization methods, this application designed an experiment to compare three typical medical images: 3D CT raw data, 3D CT reconstructed data, and 3D ultrasound raw data. The rendering results were obtained using six volume rendering algorithm models. These included a base model (no illumination), an illumination model, a back-to-front direct volume rendering model, a back-to-front direct volume rendering + cubic convolution interpolation model, an early ray termination model + cubic convolution interpolation model, and an early ray termination model. The algorithm of this application has the highest contrast resolution, allowing for clearer visualization of tiny blood vessels and structures in the heart, and enabling appropriate texture analysis within the same tissue with homogeneous features. The algorithm model of this application is more suitable for myocardial assessment. A sphere centered on the focal point can focus on the focal area, displaying various tissue structures in a 3D composite manner, highlighting their boundaries and contours.

[0058] In order to verify the real-time performance of the medical interactive visualization method, this application conducted 20 interactive visualization experiments, recorded and calculated the average frames per second, and evaluated the real-time performance of the medical interactive visualization system of this application. Under the premise that the maximum displayable frame rate of HoloLens2 is 60FPS, for 3D CT raw data, the basic model achieved the highest rendering speed of 59.95±2.25FPS, which is close to the highest resolution that HoloLens 2 can display, followed by the early ray termination model (method of this application) at 40.00±5.80FPS. However, from the perspective of visual authenticity restoration, the basic model cannot provide an effective spatial reference, and this application is significantly better than the basic model in this respect. For 3D ultrasound raw data and 3D CT reconstructed data, the model of this application achieved the highest rendering speed, which was 60.00±1.80FPS and 42.05±5.55FPS respectively, and in ultrasound images, the rendering speed reached the highest display frame rate of HoloLens2. Therefore, compared with some other five representative methods, the method of the present application can achieve the best rendering speed and has better real-time performance. The medical interactive visualization system provided by the present application can quickly respond to changes in the posture of three-dimensional data.

[0059] In summary, to enable doctors to better perceive the spatial structure of the patient's target area, a holographic three-dimensional medical data visualization system suitable for surgical medicine that is interactive through air gestures has been developed. This system covers preoperative planning and holographic three-dimensional visualization of tissue structures. Augmented reality technology is used to help surgeons perceive tissue structure information and cut tissues and organs under the guidance of virtual information. Combined with registration tracking technology and air gesture interaction, it assists surgeons in performing surgical operations naturally and intuitively, thereby transforming traditional surgery into predictable and accurately guided surgery. Based on the characteristics of surgical medicine, a suitable medical visualization prototype system has been developed, and related interaction methods have been formulated. The feasibility of the system has been verified using 3D CT raw data, 3D CT reconstruction data, and 3D ultrasound raw data. On this basis, in-depth research and exploration have been carried out to determine its practical application value in clinical practice, and continuous improvement and optimization have been carried out, laying a solid foundation for its future.

[0060] The present invention utilizes mid-air gesture interaction and three-dimensional medical data visualization technology to assist doctors in achieving intuitive diagnosis. The method includes two strategies: stereo interpolation volume rendering based on the illumination model and holographic mid-air gesture interaction. We conducted multiple experiments for each strategy to verify their effectiveness. We used a variety of illumination models and rendering methods to perform volume rendering on three types of medical images. Compared with some representative methods, our method can achieve the fastest rendering speed while ensuring the visualization effect. At the same time, the interaction algorithm can meet the real-time and accuracy requirements of assisting doctors in diagnosis. Therefore, our method can provide users with more intuitive three-dimensional medical data visualization effects in the holographic XR environment.

[0061] An embodiment of the present application also provides a holographic three-dimensional medical data visualization system that can interact with gestures through the air, including a mixed reality display device and a gesture recognition system connected to the mixed reality display device; the mixed reality display device is worn on the user's head; the mixed reality display device has a built-in depth camera, an inertial measurement unit and multiple sensors for obtaining user environment information and user motion information; the user environment information is image information of the real scene around the user; the mixed reality display device also uses a stereo interpolation volume rendering method to generate a virtual image based on the user environment information and user motion information; and uses an optical model to project the virtual image into the user's field of view, and fuses the virtual image and the real scene to obtain a three-dimensional model rendered by mixed reality; the gesture recognition system is used to capture the user's gestures through a gesture recognition method based on the three-dimensional model rendered by mixed reality and perform gesture interaction through the air, so as to realize interaction between the user and the virtual scene.

[0062] Compared to existing technologies, the holographic three-dimensional medical data visualization system with mid-air gesture interaction provided by this application has the following advantages: Although many medical visualization systems currently exist, most of them are based on two-dimensional screen displays, and their holographic display of medical images is still significantly limited, making it impossible to achieve multi-perspective interactive surgical operations. Compared with the mid-air gesture interaction used in our system, surgeons use a mouse to operate on the screen, which greatly limits the surgeon's field of view. This interaction method is far from the natural method used in real-life surgical scenes, limiting the surgeon's workspace for medical image analysis and increasing the risk of potential diagnostic errors due to visual impairments caused by two-dimensional images. In contrast, in our system, surgeons wearing HoloLens 2 can move freely while viewing the virtual-physical fusion scene and performing surgical tasks. Our system can improve the comfort of doctors during diagnosis and thus optimize patient treatment outcomes. Surgeons can intuitively present medical knowledge to patients, helping them better understand and accept treatment, thereby reducing their psychological stress and anxiety.

[0063] Based on the above technical solution, the embodiment of the present application provides a holographic three-dimensional medical data visualization method and system that can interact with gestures through the air. The method includes the following steps: first, obtaining user environment information and user motion information; the user environment information is image information of the real scene around the user; the user wears a mixed reality display device; then, based on the user environment information and user motion information, a stereo interpolation volume rendering method is used to generate a virtual image; next, an optical model is used to project the virtual image into the user's field of view, and the virtual image and the real scene are fused to obtain a three-dimensional model rendered by mixed reality; finally, based on the three-dimensional model rendered by mixed reality, the user's gestures are captured by a gesture recognition method and gesture interaction is performed through the air to realize interaction between the user and the virtual scene.

[0064] In order to enable doctors to better perceive the spatial structure of the patient's target area, this application proposes a holographic three-dimensional medical data intuitive visualization system under air gesture interaction suitable for surgical disciplines, covering preoperative planning and holographic three-dimensional visualization of tissue structure, using augmented reality technology to help surgeons predict tissue structure information and cut tissue organs under the guidance of virtual information. Combined with alignment tracking technology and air gesture interaction, it assists surgeons in performing surgical operations naturally, thereby transforming traditional surgery into predictable and accurately guided surgery.

[0065] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned method embodiments can be implemented by instructing the relevant hardware through a program. The program is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps in the above-mentioned method embodiments of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0066] Those skilled in the art will appreciate that the above-described embodiments are specific examples for implementing the present application, and that in actual applications, various changes in form and detail may be made thereto without departing from the spirit and scope of the present application. Any person skilled in the art may make changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be subject to the scope defined in the claims.

Claims

1. A holographic three-dimensional medical data visualization method capable of air gesture interaction, characterized in that: The following steps are involved: Acquire user environment information and user motion information; the user environment information is image information of a real scene around the user; the user wears a mixed reality display device; Based on the user environment information and the user motion information, a virtual image is generated by using a stereo interpolation volume rendering method; The virtual image is projected into the user's field of vision using an optical model, and the virtual image is fused with the real scene to obtain a three-dimensional model rendered by mixed reality; Based on the three-dimensional model rendered by the mixed reality, the user's gestures are captured through a gesture recognition method and gesture interaction is performed in the air, thereby realizing interaction between the user and the virtual scene.

2. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 1 is characterized in that: The mixed reality display device is a head-mounted mixed reality display device HoloLens2; The optical model is an optical model in the head-mounted mixed reality display device HoloLens2.

3. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 1 is characterized in that: The mixed reality display device has a built-in depth camera, an inertial measurement unit and multiple sensors to obtain the user environment information and the user motion information.

4. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 3 is characterized in that: The mixed reality display device is also provided with a recognition camera and a tracking camera; the recognition camera is used to detect and recognize specific images, objects or scenes in the real scene; the tracking camera is used to track the position and direction of the camera, and place the virtual image in the real world according to the output of the recognition camera to achieve virtual-real registration.

5. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 1, characterized in that: The virtual image and the real scene are integrated to obtain a three-dimensional model of mixed reality rendering, including: The virtual image is generated into a three-dimensional model of the virtual image through 3D modeling and rendering technology, and the properties of the virtual image are adjusted in real time, and the three-dimensional model of the virtual image is integrated with the real scene to obtain a three-dimensional model of mixed reality rendering; wherein the properties of the virtual image include the position, size and transparency of the virtual image.

6. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 1, characterized in that: The three-dimensional model rendered by the mixed reality includes three-dimensional models at multiple different angles; wherein, The three-dimensional models at different angles correspond to the user's viewing angle.

7. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 1, characterized in that: When performing air gesture interaction, the user uses gestures to move, rotate, scale and cut the virtual tissue in the three-dimensional model rendered by the mixed reality from any angle, manipulate virtual surgical instruments in real time and observe the patient's lesion condition.

8. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 1, characterized in that: After obtaining the 3D model of mixed reality rendering, it also includes: The three-dimensional model of the mixed reality rendering is evaluated.

9. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 8, characterized in that: The three-dimensional model rendered by the mixed reality is evaluated, including: Using medical images of 3D CT original data, medical images of 3D CT reconstructed data, and medical images of 3D ultrasound original data as comparison data, the efficiency and real-time performance of the 3D model rendered by the mixed reality are evaluated respectively.

10. A holographic three-dimensional medical data visualization system capable of air gesture interaction, characterized in that: A mixed reality display device and a gesture recognition system connected to the mixed reality display device; The mixed reality display device is worn on the user's head; the mixed reality display device has a built-in depth camera, an inertial measurement unit and multiple sensors for obtaining user environment information and user motion information; the user environment information is image information of the real scene around the user; The mixed reality display device also generates a virtual image using a stereo interpolation volume rendering method according to the user environment information and the user motion information; and projects the virtual image into the user's field of view using an optical model, and fuses the virtual image with the real scene to obtain a three-dimensional model rendered by mixed reality; The gesture recognition system is used to capture the user's gestures through a gesture recognition method according to the three-dimensional model rendered by the mixed reality and perform gesture interaction in the air, thereby realizing interaction between the user and the virtual scene.

Images