A mobile terminal-based augmented reality intracranial lesion visualization positioning method

Abstract

The application provides a mobile terminal-based augmented reality intracranial lesion visualization positioning method, comprising the following steps: S1, acquiring a patient's brain multi-modal image sequence; S2, obtaining a corresponding three-dimensional model according to the image sequence; S3, exporting the three-dimensional model and transmitting it to a cloud server; S4, loading the three-dimensional model to a three-dimensional engine of a mobile terminal by logging in; S5, in the three-dimensional engine of the mobile terminal, adjusting the three-dimensional model to a user-setted surgery-planned observation view angle, display scale and transparency state, then taking a screenshot; switching to an augmented reality mode, enabling a mobile terminal camera to collect a real scene video stream, superimposing the adjusted three-dimensional model on the real scene video stream, realizing fusion display of the virtual three-dimensional model and the real environment, and performing alignment and overlap; based on the visual guidance after the overlap, marking a lesion contour on a patient's body surface; if the three-dimensional model screenshot does not meet the actual requirements of surgery, returning to re-adjust the three-dimensional model and performing subsequent steps. The method of the application significantly reduces the hardware performance requirements of the mobile terminal, effectively avoids problems such as screen freezing and delay caused by insufficient device computing power, and at the same time solves the previous problem of repeatedly converting the work flow between the computer and the mobile terminal during positioning operation, ensuring the smoothness of real-time display.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical image processing technology, specifically an augmented reality method for visualizing and locating intracranial lesions based on a mobile terminal. Background Technology

[0002] Precise preoperative localization of intracranial lesions is a fundamental prerequisite for the success of minimally invasive neurosurgical procedures. With the continuous development of neurosurgery, surgeries are becoming increasingly refined, individualized, and minimally invasive. While removing intracranial lesions, greater emphasis is placed on preserving neurological function to improve the patient's quality of life. Achieving refined and minimally invasive surgery requires not only meticulous surgical technique but also precise lesion localization. Given the complex anatomy and function of the brain and the limited space available for neurosurgical operations, accurate preoperative localization of intracranial lesions is essential to maximize lesion removal while minimizing damage to normal tissues. Traditional methods of intracranial lesion localization require neurosurgeons to rely on anatomical knowledge and clinical surgical experience, combined with preoperative two-dimensional imaging data such as cranial CT and MRI scans, and indirectly determine the approximate location of the lesion by referencing the characteristics of the skull surface. However, this method is highly subjective, and the accuracy is closely related to the surgeon's clinical experience. Different surgeons use different methods, and sometimes the localization process is time-consuming and the accuracy is unsatisfactory.

[0003] In recent years, virtual reality (VR) and augmented reality (AR) technologies have been used for three-dimensional visualization of medical images. Among them, augmented reality (AR) technology can provide neurosurgeons with intuitive and three-dimensional guidance for lesion localization by overlaying and fusing virtual medical image information with the patient's real anatomical structure in real time.

[0004] However, current augmented reality (AR) technologies applied to intracranial lesion localization have certain limitations. Some technologies require complex and expensive equipment, and the operation process consumes a lot of manpower and time, which limits their application in county-level hospitals and emergency scenarios. Currently, the main software for intracranial lesion localization based on augmented reality principles and relying on mobile phone applications is the "Sina" software. This software needs to rely on computer 3D model display software (such as "Slicer" software) to display intracranial lesions and anatomical structures such as the scalp simultaneously. After adjusting to a suitable angle, a screenshot is taken, and then the image is transmitted to the mobile phone. The "Sina" software is used to switch to the screenshot and integrate it with the target object in the actual scene before performing surface localization. The problems and shortcomings of the above method are as follows: ① It requires the use of a computer to present a 3D model, and the screenshot is then transmitted to the mobile phone. The whole process is complicated and requires that a computer with 3D model display software installed must be available near the application scene. ② If, in a real-world scenario, the previously captured image is found to be unsuitable or the location is inconvenient, requiring a new screenshot, the process must be repeated on the computer. After the screenshot is completed, it must be transferred back to the mobile phone, resulting in extremely cumbersome switching between devices. This makes it impossible for doctors to directly manipulate images on their mobile devices in real-time during clinical scenarios, especially in preoperative discussions, doctor-patient communication, medical teaching, telemedicine communication, or when temporary adjustments are needed during surgery. They must rely on computers or other media, leading to a disjointed workflow and impacting location efficiency and clinical applicability. Summary of the Invention

[0005] The purpose of this invention is to provide an augmented reality visualization and localization method for intracranial lesions based on mobile terminals, in order to solve the problems of existing augmented reality methods for intracranial lesion localization in clinical applications, such as the large number of devices required and the inconvenience of transportation, the cumbersome process of modifying selected images and the lack of a coherent workflow, the low efficiency of lesion localization, the inconvenience of academic exchanges and doctor-patient communication, and the limited clinical applicability.

[0006] This invention is implemented as follows: A method for visualizing and locating intracranial lesions based on augmented reality using a mobile terminal, comprising the following steps: S1. Obtain multimodal image sequences of the patient's brain; S2. The multimodal image sequence is registered using one of the sequences as the center coordinate sequence, and the scalp, skull and lesion tissue are segmented and reconstructed based on the registration result to obtain the corresponding three-dimensional model. S3. Export the three-dimensional model in a standardized format and transmit it to the cloud server through the medical imaging interaction platform; S4. By logging into the mobile terminal of the medical imaging interaction platform, obtain and display a list of three-dimensional models associated with patient information; in response to the user's selection command for the target model and the three-dimensional visualization trigger operation, load the three-dimensional model into the three-dimensional engine of the mobile terminal. S5. In the 3D engine of the mobile terminal, after adjusting the 3D model to the viewing angle, display ratio and transparency state set by the user; in response to the user's mode switching command, switch to augmented reality mode, enable the mobile terminal camera to capture the video stream of the real scene, and overlay the adjusted 3D model on the video stream of the real scene to achieve the fusion display of the virtual 3D model and the real environment. S6. Visually align and overlap the superimposed 3D model on the mobile terminal display screen with the real-time captured image of the patient's surgical site on the mobile terminal display screen. S7. After alignment and overlap, determine whether the matching degree between the 3D model and the real-time image of the corresponding surgical site of the patient collected on the mobile terminal display screen meets the alignment accuracy required by clinical practice; if it does, mark the lesion outline on the patient's body surface based on the visual guidance after overlap.

[0007] Furthermore, the present invention can be implemented according to the following technical solution: In step S3, the medical image interaction platform is a dedicated application deployed on a web platform.

[0008] In step S4, the 3D engine is Three.js, a WebGL-based 3D engine that runs in the embedded browser kernel of the mobile terminal or the WebView component of the native application.

[0009] In step S5, in the mobile terminal's local 3D engine, in response to the user's save trigger operation, the 3D model is adjusted to the user-set viewing angle, display ratio, and transparency state. The 3D model image frame of the current rendering viewport is captured and saved to the local storage medium. In response to the user's mode switching command, the system switches to augmented reality mode, enables the mobile terminal's camera to capture the video stream of the real scene, and selects the 3D model image frame from the local storage medium as a virtual layer to be superimposed on the real scene video stream based on the user's selection command, thereby realizing the fusion display of the virtual 3D model image frame and the real environment.

[0010] In step S5, the 3D model in the 3D engine is set to the user-defined viewing angle, display ratio, and transparency state by adjusting the camera parameters and model transformation matrix. Then, the 3D model image frame of the current rendering viewport is captured by the readPixels method of the 3D engine renderer or the off-screen rendering target. The 3D model image frame is encoded into PNG or JPEG format and automatically saved to the application's private directory or system album on the local storage medium.

[0011] In step S5, the camera component is initialized to acquire a video stream of the real scene as a camera layer, and three-dimensional model image frames are read from the local storage medium based on user selection instructions as three-dimensional model image frame layers. A rendering container is created using a front-end layout framework to overlay the camera layer and the 3D model image frame layer. The camera layer serves as the bottom background, and the 3D model image frame layer serves as the upper overlay layer. The 3D model image frame layer is styled using CSS style sheets. The style adjustments include setting the transform property to control the rotation of the viewing angle, setting the scale property to control the display ratio, and setting the opacity property to control the transparency, thereby achieving a fusion display of the virtual 3D model image frame and the real environment.

[0012] In step S6, the spatial position and angle of the mobile terminal relative to the patient's surgical site are manually adjusted so that the three-dimensional model image frame superimposed on the mobile terminal display screen visually overlaps with the real-time image of the patient's surgical site captured on the mobile terminal display screen.

[0013] In step S6, by adjusting the display parameters of the virtual layer of the three-dimensional model image frame, including scaling, rotation angle and transparency, the three-dimensional model image frame superimposed on the mobile terminal display screen is visually overlapped with the real-time image of the patient's surgical site captured on the mobile terminal display screen.

[0014] In step S7, it is determined whether the matching degree between the three-dimensional model image frame and the real-time image of the patient's surgical site captured on the mobile terminal display screen meets the alignment accuracy required for clinical surgery. If it does, the lesion outline is marked on the patient's body surface based on the visual guidance after overlap. If the matching is still difficult or the accuracy is unsatisfactory after repeated matching, it can return to step S6 for readjustment. When the error originates from the superimposed three-dimensional model image frame or the surgical plan needs to be changed, it returns to step S5 to readjust the three-dimensional model and execute the subsequent steps.

[0015] An augmented reality visualization and localization device for intracranial lesions based on a mobile terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements an augmented reality visualization and localization method for intracranial lesions based on a mobile terminal as described in any of the preceding claims.

[0016] In the method of this invention, new screenshots of the three-dimensional model can be updated on a mobile terminal (smartphone, tablet, etc.) at any time according to the needs of the surgical plan or the real-world scenario. Users can adjust the surgical approach or positioning angle based on the patient's latest image data and can directly capture updated three-dimensional model images on the mobile terminal in real time to quickly replace the original selected images. During craniotomy, the mobile terminal can be operated to display models of the skull and related structures and select appropriate images for locating key holes, leveraging the role of neuronavigation to guide the surgery. The entire process does not rely on other bulky professional workstations or additional electronic devices, greatly improving the flexibility and scenario adaptability of clinical target localization operations.

[0017] Specifically, mobile terminals can be used to match 3D images of intracranial lesions, scalp, important blood vessels, and functional areas with the patient's skull in a 1:1 ratio, thus enabling visual localization of intracranial lesions on the patient's head. Before surgery, by adjusting the size and angle of the 3D model on the mobile terminal, the scalp image is stably superimposed on the patient's actual skull. Then, a marker is used to mark the surface location of the projected intracranial lesion and trace its outline on the scalp, which then guides the design of the surgical incision.

[0018] Compared to traditional preoperative discussions and the frequent modification of parameters and switching of imaging data on computer imaging workstations to display 3D images, mobile terminals offer a visualization of intracranial lesions without time or space limitations. Doctors can use mobile devices to present 3D models for preoperative discussions anytime, anywhere. This convenient, real-time modifiable, and interactive augmented reality visualization method simplifies procedures, reduces reliance on complex and expensive equipment, and improves the efficiency of preoperative intracranial lesion localization. It can also be used in various clinical scenarios such as preoperative discussions, doctor-patient communication, medical teaching, telemedicine consultations, and intraoperative plan adjustments, improving the quality and effectiveness of these clinical tasks. This localization method, with its more convenient operation, more stable performance, and more reasonable cost structure, is easier to promote and apply in small and medium-sized hospitals and emergency settings, possessing high promotional value and feasibility.

[0019] In terms of doctor-patient communication, doctors can use mobile devices to more accurately and clearly explain surgical plans, related risks, and precautions to patients and their families in various scenarios, based on visualized intracranial lesion models. In medical education, young doctors can actively participate in preoperative discussions using visualized intracranial lesion models displayed on mobile devices, thereby accumulating clinical experience and deepening their professional knowledge. In telemedicine communication, by sharing visualized intracranial lesion models displayed on mobile devices in real time, experts in different regions can successfully overcome geographical barriers and work together to develop the most suitable surgical plan for patients.

[0020] The method of this invention eliminates the need for complex 3D rendering pipelines, requiring only 2D image overlay. This significantly reduces the hardware performance requirements of mobile terminals, effectively avoiding issues such as screen stuttering and latency caused by insufficient device computing power, and ensuring smooth real-time display. This allows the mode to run stably on a wider range of mobile terminals, including some mid-to-low-end smartphones or tablets, greatly improving the technology's versatility and ease of clinical application. Simultaneously, the lightweight computation process reduces device power consumption, extends the battery life of mobile terminals, and can meet the needs of more complex clinical application scenarios.

[0021] The method of this invention remains usable even in complex environments such as operating rooms with no network or poor network signal. Its core lies in the fact that the mobile device itself loads complete key data information of the 3D model, eliminating the need for real-time data transmission and calculation via a cloud server. Medical staff only need to successfully load the 3D model once. Even if the network environment in the operating room is unstable or completely offline, doctors can view the 3D model anytime, anywhere, take screenshots according to the surgical plan, and then proceed to the positioning stage. This avoids delays in the surgical process caused by network latency or interruption, ensuring the continuity of the surgery and emergency response capabilities. It is suitable for operating room scenarios with weak network infrastructure or complex electromagnetic environments. Simultaneously, the local loading mode effectively reduces the risk of privacy leaks during data transmission, complying with medical data security regulations.

[0022] The operating room is well-lit, and the pre-screening method of this invention avoids the impact of lighting changes on the stability of virtual object display. It prevents issues such as blurred edges, color distortion, or display fragmentation caused by light interference, maintaining a clear and stable visual presentation at all times, effectively reducing the risk of visual errors due to lighting interference. The method also offers good viewing angle stability, as the screenshot locks in the optimal viewing angle used in pre-operative planning, avoiding model viewpoint drift caused by hand tremors in real-time AR. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0024] Figure 2 This is a schematic diagram of the three-dimensional model imported into the medical imaging interaction platform in this invention.

[0025] Figure 3 This is a schematic diagram of the 3D model loaded into the 3D engine of the mobile terminal in this invention.

[0026] Figure 4 This is a schematic diagram illustrating how, in this invention, a 3D model image frame is superimposed as a virtual layer onto a real-world video stream to mark the outline of a lesion on the patient's body surface. Detailed Implementation

[0027] This invention provides an augmented reality method for visualizing and locating intracranial lesions based on a mobile terminal, such as... Figure 1 As shown, it includes the following steps: S1. Acquire multimodal brain imaging sequences from the patient. These sequences are the CT and MRI images from the patient's preoperative examination. The brain imaging data is retrieved from the image library in DICOM format (the standard format for medical imaging data) via a portable hard drive to ensure the integrity, accuracy, and compatibility of the brain imaging data for subsequent 3D reconstruction and augmented reality localization processing steps.

[0028] S2. Perform multimodal registration of the multimodal image sequence with one of them as the center coordinate sequence, and segment and reconstruct the scalp, skull and lesion tissue based on the registration results to obtain the corresponding three-dimensional model.

[0029] Perform image registration and 3D reconstruction steps: Spatial registration is performed between the CT image sequence and the MRI image sequence using either sequence as the center coordinate sequence, and tissue segmentation is performed on the registered image sequence to establish a 3D model of the scalp, skull and lesion tissue; Specifically, the above image sequences are imported into the 3DSlicer software. First, the "General Registration (Elastix)" module is used to register CT and MRI data (i.e., using any one of the image sequences as the central coordinate sequence, and registering all other image sequences to this central coordinate sequence). Then, the "Segment Editor" module is used to create three-dimensional models of lesions such as scalp, skull, tumors, or hematomas.

[0030] The General Registration (Elastix) module is the core registration extension module of the 3DSlicer software. It is based on the open-source library Elastix and is a public standard functional module of 3DSlicer software, used for registration of multimodal medical images (CT / MRI / PET, etc.). The Segment Editor module is the core segmentation module of the 3DSlicer software, providing segmentation tools for building 3D models.

[0031] S3. Export the 3D model in a standardized format and transmit it to the cloud server through the medical imaging interaction platform.

[0032] like Figure 2 As shown, this medical imaging interaction platform is a dedicated web-based application. The transmission process includes user authentication, patient information association and binding, and encrypted transmission and cloud storage. User authentication is performed during the registration phase based on relevant information for qualification verification. Login verification is completed through account and password methods to ensure that only authorized personnel can access the platform.

[0033] After registration, users log in to the medical imaging interaction platform to link and bind patient information. This involves entering relevant patient information, including patient name, age, gender, mobile phone number, hospital number, and model number, thus creating patient identification information. This patient identification information is then mapped and associated with the metadata of the 3D model file, generating a unique index code to ensure a unique correspondence between "one patient, one model."

[0034] The 3D model is exported in STL format and transmitted to a cloud server via a pre-defined medical imaging interaction platform, where patient information and the 3D model are shared. Each user's uploaded 3D model is stored in their own model database, which cannot be seen or opened by others.

[0035] S4. By logging into the mobile terminal of the medical imaging interaction platform, obtain and display a list of 3D models associated with patient information; in response to the user's selection command for the target model and the 3D visualization trigger operation, load the 3D model into the 3D engine of the mobile terminal.

[0036] Specifically, users access a pre-defined medical imaging interaction platform via their mobile devices. After logging in with their username and password, a list of 3D models associated with patient information is displayed. Within this list, selecting the target patient model loads it into the mobile device's local 3D engine. Once loaded, the previously uploaded 3D model is displayed on the phone screen. Figure 3 As shown.

[0037] This 3D engine, Three.js, is based on WebGL and runs within the embedded browser kernel of mobile devices or the WebView component of native applications. It supports interactive operations such as rotation, scaling, and translation of 3D models. Users can also zoom in on the model using two-finger touch, allowing them to view model details from different angles. The engine also supports a transparent display mode, allowing users to adjust the transparency parameters of different tissue structures to clearly present the relationship between internal anatomical structures and external morphology. Users can view the 3D model from any angle using a single finger touch, and edit the transparency using the slider at the bottom of the 3D engine app interface to control the display level of each model. Users can also zoom in and out on the model.

[0038] S5. In the 3D engine of the mobile terminal, in response to the user's save trigger operation, the 3D model is adjusted to the user-set viewing angle, display ratio and transparency state, the 3D model image frame of the current rendering viewport is captured and saved to the local storage medium; in response to the user's mode switching command, the mode is switched to augmented reality mode, the mobile terminal camera is enabled to capture the video stream of the real scene, and the 3D model image frame is selected from the local storage medium as a virtual layer and superimposed on the real scene video stream based on the user's selection command, so as to realize the fusion display of the virtual 3D model image frame and the real environment.

[0039] This user-defined viewing angle is set based on the key anatomical structure exposure angles required for minimally invasive clinical surgery, such as the intraoperative head positioning and optimal tumor exposure angle in the planning of intracranial tumor surgery. At the same time, the display ratio of the 3D model can be adjusted by using two-finger zoom to ensure that the model details are clearly presented without excessively occupying the display area. The transparency of the model can also be adjusted to an appropriate state by using sliders or preset parameters to clearly distinguish the spatial relationships of different tissue layers, such as making the bone structure completely transparent to show the course of its internal blood vessels.

[0040] The save trigger operation includes clicking the preset UI save button or performing a preset gesture operation. After adjusting the camera parameters and model transformation matrix, the 3D model in the Three.js scene graph is set to the user-defined viewing angle, display ratio, and transparency state. Then, the 3D model image frame of the current rendering viewport is captured by the Three.js renderer's readPixels method or the off-screen rendering target (WebGLRenderTarget). The 3D model image frame is encoded into PNG or JPEG format and automatically saved to the application's private directory or system album on the local storage medium.

[0041] The overlay and fusion of real-world video streams and 3D model image frames is mainly based on real-time video streams captured by mobile cameras. The original static screenshot background is replaced with dynamic real-world images. Through frame synchronization and layer fusion technology, the screenshot content and the real-world background of the camera are overlaid in real time, ensuring smooth images, clear layers, and stable rendering.

[0042] At the technical level, the mode switching instructions include clicking the preset AR mode button, initializing the camera component to capture the real scene video stream as the camera layer, and reading the three-dimensional model image frame from the local storage medium as the three-dimensional model image frame layer based on the user selection instructions. A rendering container is created using a front-end layout framework to overlay the camera layer and the 3D model image frame layer. The camera layer serves as the bottom background, and the 3D model image frame layer serves as the upper overlay layer. The 3D model image frame layer is styled using CSS style sheets. The style adjustments include setting the transform property to control the rotation of the viewing angle, setting the scale property to control the display ratio, and setting the opacity property to control the transparency, thereby achieving a fusion display of the virtual 3D model image frame and the real environment.

[0043] On the operational level, pressing the AR mode function key on the mobile 3D engine interface manually switches to the augmented reality mode interface. Users can then see the real-world scene captured by the phone's camera. Pressing the "Open" function key at the bottom of the rendering engine interface allows users to select the previously captured 3D model image from their photo album. Clicking "OK" returns the phone to the augmented reality mode interface, officially entering augmented reality mode. At this point, users can simultaneously observe the relationship between the tumor and other landmark features such as scalp or facial contours presented in the 3D model image frame, as well as the real-time captured images of the real-world environment. The virtual layer superimposed on the real-world video stream does not change with the camera's viewing angle. In other words, users can see the previously captured image on their phone screen, while the background is the real-world image of the patient's examination site captured by the camera in real time.

[0044] Based on real-time preview footage from a mobile camera, this system enables customized image overlay rendering, spatial positioning, and dynamic adjustment of visual attributes. It supports geometric transformations such as translation, scaling, and rotation of the overlaid image via touch interaction, and allows adjustment of image transparency. This enables precise alignment and semi-transparent overlay display of the reference image and camera view, meeting the needs of scenarios such as visual comparison and assisted positioning.

[0045] S6. By manually adjusting the spatial position and angle of the mobile terminal relative to the patient's surgical site, or by adjusting the display parameters of the three-dimensional model image frame on the mobile phone screen, including scaling ratio, rotation angle and transparency, the three-dimensional model screenshot on the mobile terminal display screen is visually superimposed on the actual image of the patient's surgical site on the screen.

[0046] Specifically, by placing the phone in a suitable position, the head image in the screenshot taken in the previous step is completely integrated with the patient's actual head in the background real scene, that is, the virtual skin or bony landmarks in the screenshot coincide with the corresponding skin or bony landmarks in reality.

[0047] Alternatively, by adjusting the display parameters of the 3D model image frame, including scaling, rotation angle, and transparency, the 3D model screenshot on the mobile terminal display screen can be visually merged with the actual image of the patient's surgical site.

[0048] During this process, doctors can simultaneously observe the fused image on the phone screen and perform localization operations by comparing it with the actual anatomical locations on the patient's head. For example, when locating a convexity meningioma, doctors can fine-tune the phone's angle or the scaling of the virtual layer to precisely align the 3D scalp model in the screenshot with the patient's scalp in the real-world scene. They can also adjust the transparency of the virtual layer to ensure that the real skin texture and the virtual model structure are clearly visible and do not obscure each other. By repeatedly adjusting the spatial position of the mobile terminal and the display parameters of the virtual layer, until key anatomical landmarks in the 3D model screenshot (such as the tip of the nose, eye socket, auricle, occipital protuberance, etc.) completely overlap with their corresponding landmarks on the patient's real skull, a precise match between the virtual 3D model and the real surgical anatomical structure is achieved.

[0049] S7. After alignment and overlap, determine whether the matching degree between the 3D model image frame and the real-time image of the corresponding surgical site (skull) of the patient captured on the mobile terminal display meets the clinically required alignment accuracy (all surface features are completely aligned); if so, based on the visual guidance after overlap, mark the lesion outline on the patient's body surface, such as... Figure 4 As shown; if matching difficulties or unsatisfactory accuracy still occur after repeated matching, you can return to step S6 to readjust; if the error originates from the superimposed three-dimensional model image frames or if the surgical plan needs to be changed, return to step S5 to readjust the three-dimensional model and execute the subsequent steps.

[0050] Determining whether the alignment accuracy between the 3D model image frame and the real-time image of the patient's surgical site captured on the mobile terminal display meets the clinical requirements is accomplished through human observation. Doctors, relying on their professional anatomical knowledge and clinical experience, meticulously and precisely align the surface features of the virtual 3D model superimposed on the display with the corresponding surface features of the real patient's surgical site. When the edge contours and other landmark structures of the scalp virtual model perfectly match the real patient's scalp and corresponding surface landmark structures, the doctor considers the matching accuracy to meet clinical requirements. Based on the visual guidance after overlap, the doctor marks the lesion outline on the patient's scalp surface. When marking the lesion outline on the patient's surface, the doctor uses key individualized feature points around the lesion area in the superimposed 3D model on the display as a visual reference, and uses a dedicated marker pen or other marking tools to accurately trace the corresponding surgical site on the patient's scalp. During the marking process, the doctor continuously monitors the overlap between the real-time image on the display and the virtual model, ensuring that the marking action highly matches the lesion range indicated by the 3D model. This provides intuitive and accurate surface localization information for subsequent clinical operations, such as surgical incision planning and puncture.

[0051] Conversely, if significant deviations are found, or the virtual scalp model is not perfectly aligned with the actual head, the matching accuracy is deemed insufficient. Return to step S6 for readjustment.

[0052] If the error is determined to stem from inappropriate selection of 3D image frames, such as key anatomical structures in the image frame not meeting the actual requirements of the patient's surgery, then it is necessary to return to step S5, that is, to recapture 3D image frames of the patient's surgical site using the mobile terminal. During the recapture process, the patient's 3D model needs to be adjusted to more closely match the patient's scalp surgical site position required during surgery before taking the screenshot, thereby ensuring the accuracy and reliability of the entire positioning and marking process.

Claims

1. A method for visualizing and locating intracranial lesions based on augmented reality using a mobile terminal, characterized in that, Includes the following steps: S1. Obtain multimodal image sequences of the patient's brain; S2. The multimodal image sequence is registered using one of the sequences as the center coordinate sequence, and the scalp, skull and lesion tissue are segmented and reconstructed based on the registration result to obtain the corresponding three-dimensional model. S3. Export the three-dimensional model in a standardized format and transmit it to the cloud server through the medical imaging interaction platform; S4. By logging into the mobile terminal of the medical imaging interaction platform, obtain and display a list of three-dimensional models associated with patient information; in response to the user's selection command for the target model and the three-dimensional visualization trigger operation, load the three-dimensional model into the three-dimensional engine of the mobile terminal. S5. In the 3D engine of the mobile terminal, after adjusting the 3D model to the viewing angle, display ratio and transparency state set by the user; in response to the user's mode switching command, switch to augmented reality mode, enable the mobile terminal camera to capture the video stream of the real scene, and overlay the adjusted 3D model on the video stream of the real scene to achieve the fusion display of the virtual 3D model and the real environment. S6. Visually align and overlap the superimposed 3D model on the mobile terminal display screen with the real-time captured image of the patient's surgical site on the mobile terminal display screen. S7. After alignment and overlap, determine whether the matching degree between the 3D model and the real-time image of the corresponding surgical site of the patient collected on the mobile terminal display screen meets the alignment accuracy required by clinical practice; if it does, mark the lesion outline on the patient's body surface based on the visual guidance after overlap.

2. The augmented reality method for locating and visualizing intracranial lesions based on a mobile terminal according to claim 2, characterized in that, In step S3, the medical image interaction platform is a web-based application.

3. The augmented reality method for locating and visualizing intracranial lesions based on a mobile terminal according to claim 2, characterized in that, in In step S4, the 3D engine is Three.js, a WebGL-based 3D engine that runs in the embedded browser kernel of the mobile terminal or the WebView component of the native application.

4. The augmented reality method for locating and visualizing intracranial lesions based on a mobile terminal according to claim 1, characterized in that, in In step S5, in the local 3D engine of the mobile terminal, in response to the user's save trigger operation, the 3D model is adjusted to the user-set viewing angle, display ratio, and transparency state, and the 3D model image frame of the current rendering viewport is captured and saved to the local storage medium; in response to the user's mode switching command, the system switches to augmented reality mode, enables the mobile terminal camera to capture the video stream of the real scene, and selects the 3D model image frame from the local storage medium as a virtual layer to be superimposed on the real scene video stream based on the user's selection command, thereby realizing the fusion display of the virtual 3D model image frame and the real environment.

5. The augmented reality method for visualizing and locating intracranial lesions based on a mobile terminal according to claim 4, characterized in that, In step S5, the 3D model in the 3D engine is set to the user-defined viewing angle, display ratio, and transparency state by adjusting the camera parameters and model transformation matrix. Then, the 3D model image frame of the current rendering viewport is captured by the readPixels method of the 3D engine renderer or the off-screen rendering target. The 3D model image frame is encoded into PNG or JPEG format and automatically saved to the application's private directory or system album on the local storage medium.

6. The augmented reality method for visualizing and locating intracranial lesions based on a mobile terminal according to claim 4, characterized in that, in In step S5, the camera component is initialized to acquire a video stream of the real scene as a camera layer, and the three-dimensional model image frame is read from the local storage medium as a three-dimensional model image frame layer based on the user's selection instruction. A rendering container is created using a front-end layout framework to overlay the camera layer and the 3D model image frame layer. The camera layer serves as the bottom background, and the 3D model image frame layer serves as the upper overlay layer. The 3D model image frame layer is styled using CSS style sheets. The style adjustments include setting the transform property to control the rotation of the viewing angle, setting the scale property to control the display ratio, and setting the opacity property to control the transparency, thereby achieving a fusion display of the virtual 3D model image frame and the real environment.

7. The augmented reality method for locating and visualizing intracranial lesions based on a mobile terminal according to claim 4, characterized in that, in In step S6, the spatial position and angle of the mobile terminal relative to the patient's surgical site are manually adjusted so that the three-dimensional model image frame superimposed on the mobile terminal display screen visually overlaps with the real-time image of the patient's surgical site captured on the mobile terminal display screen.

8. The augmented reality method for locating and visualizing intracranial lesions based on a mobile terminal according to claim 4, characterized in that, in In step S6, by adjusting the display parameters of the virtual layer of the three-dimensional model image frame, including scaling ratio, rotation angle and transparency, the three-dimensional model image frame superimposed on the mobile terminal display screen is visually overlapped with the real-time image of the patient's surgical site captured on the mobile terminal display screen.

9. A method for visualizing and locating intracranial lesions based on augmented reality using a mobile terminal, as described in claim 4, is characterized in that... In step S7, it is determined whether the matching degree between the three-dimensional model image frame and the real image of the patient's surgical site collected in real time on the mobile terminal display screen meets the alignment accuracy required for clinical surgery. If it meets the requirements, the lesion outline is marked on the patient's body surface based on the visual guidance after overlap. If the matching is still difficult or the accuracy is unsatisfactory after repeated matching, it can return to step S6 for readjustment. When the error originates from the superimposed three-dimensional model image frame or the surgical plan needs to be changed, it returns to step S5 to readjust the three-dimensional model and execute the subsequent steps.

10. A device for visualizing and locating intracranial lesions based on augmented reality using a mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements an augmented reality method for visualizing and locating intracranial lesions based on a mobile terminal, as described in any one of claims 1-9.

Images