A coronary heart disease health education system and method based on augmented reality

Abstract

The application discloses a kind of coronary heart disease health education system and method based on augmented reality, it is related to medical health education field.System includes AR device, VR device, depth camera and intelligent terminal, intelligent terminal linkage hospital database, based on patient CT image, clinical diagnosis and so on Medical data generates personalized three-dimensional cardiovascular model;Depth camera collects human skeleton data, after screening optimization by intelligent terminal, transmission to AR device;AR device is completed coordinate system alignment by PnP algorithm, accurately registers model in patient chest cavity and renders, synchronizes to VR device by TCP / UDP hybrid protocol;Response interaction instruction, dynamically demonstrate myocardial ischemia, chest pain diffusion and other pathological evolution process, combined with resource dynamic loading and space anchor point technology guarantee stability.The application realizes doctor-patient AR visual angle sharing and accurate personalized propaganda and education, solves the pain points of traditional propaganda and education abstraction, interaction deficiency and low participation, significantly improves patient cognitive efficiency, and has wide application prospect.

Description

Technical Field

[0001] This application belongs to the field of medical and health education technology, and in particular relates to an augmented reality-based health education system and method for coronary heart disease. Background Technology

[0002] Coronary heart disease (CHD), a common and complex multifactorial cardiovascular disease, is characterized by high morbidity and mortality, and its incidence has been rising continuously in recent years. It has become one of the major chronic diseases threatening public health, and its trend towards affecting younger people further exacerbates the public health burden. The pathological mechanisms of CHD involve complex physiological processes such as vascular stenosis and myocardial ischemia, which are highly abstract and specialized, posing significant challenges to health education for patients.

[0003] Currently, the mainstream clinical methods for educating patients about coronary artery disease mainly include distributing printed brochures, playing standardized educational videos, and oral explanations by doctors. However, these traditional methods generally have significant limitations: First, the pathophysiological process of coronary artery disease involves complex anatomical structures (such as coronary artery branches) and abstract hemodynamic changes, making it difficult for patients without a medical background to develop an intuitive and accurate understanding based solely on two-dimensional graphics or static descriptions; second, existing educational content is mostly a generic template, unable to be customized according to the specific condition of each patient, such as the location and degree of vascular stenosis and the area of ​​myocardial blood supply, resulting in a disconnect between the educational content and the patient's own situation, affecting the depth of understanding and compliance; third, traditional educational models are mainly based on one-way information transmission, lacking interactivity and immersion, resulting in low patient participation, easy distraction, and poor learning outcomes. Summary of the Invention

[0004] The technical problem to be solved by this application is to provide a coronary heart disease health education system and method based on augmented reality, so as to solve the problems mentioned in the background art.

[0005] To address the aforementioned technical problems, this application provides the following technical solution: Firstly, this application provides an augmented reality-based health education system for coronary heart disease, comprising: The system includes an AR device for doctors, a VR device for patients, a depth camera, and a smart terminal device, wherein the smart terminal device is communicatively connected to the AR device, the VR device, and the depth camera, respectively. The depth camera is configured to acquire first node data of the human skeleton of the target patient in real time and send the first node data of the human skeleton to the smart terminal device. The intelligent terminal device is configured to obtain a three-dimensional cardiovascular model corresponding to the target patient from the database based on the identification information of the target patient, and send the three-dimensional cardiovascular model to the AR device; The intelligent terminal device is also configured to optimize the first node data of the human skeleton to obtain the second node data of the human skeleton, and send the second node data of the human skeleton to the AR device. The AR device is used to register the three-dimensional cardiovascular model at the location of the target patient's chest cavity using coordinate system alignment, based on the second node data of the human skeleton and the three-dimensional cardiovascular model, and then render the registered three-dimensional cardiovascular model and send it to the VR device. The VR device is used to receive and immerse the viewpoint rendered in the AR device.

[0006] Furthermore, the AR device is specifically configured as follows: Based on the preset marker recognition results, the pose of the depth camera in the augmented reality spatial coordinate system is calculated to achieve alignment between the depth camera coordinate system and the augmented reality spatial coordinate system. Based on the aligned coordinate system, the three-dimensional coordinates of the received second node data of the human skeleton are transformed into the augmented reality space coordinate system; The position and orientation of the three-dimensional cardiovascular model are dynamically adjusted according to the transformed coordinates, so that the three-dimensional cardiovascular model is stably registered at the thoracic anatomical position of the target patient. The rendered image containing the registered 3D cardiovascular model is transmitted to the VR device via a low-latency network transmission protocol.

[0007] Furthermore, the preset marker is an artificial visual mark of a known geometric pattern fixed on the housing of the depth camera. The VR device identifies the artificial visual mark in real time through its built-in camera and calculates the six-degree-of-freedom pose of the depth camera based on the PnP algorithm.

[0008] Furthermore, the second node data of the human skeleton includes at least: the hip center node, the spine point, the chest point, the shoulder center point, the left clavicle point, and the right clavicle point.

[0009] Furthermore, the intelligent terminal device is also configured to acquire the health education resource package corresponding to the target patient and send the health education resource package to the VR device.

[0010] Furthermore, the health education resource package is packaged in AssetBundle format using the Unity engine and stored in a database.

[0011] Furthermore, the AR device is also configured to: save the pose of the depth camera in the augmented reality spatial coordinate system as a spatial anchor point; and automatically restore the spatial anchor point when the system restarts in the same physical environment to maintain the continuity of the registered position of the three-dimensional cardiovascular model.

[0012] Secondly, this application also provides an augmented reality-based method for coronary heart disease health education, applied to any of the systems described above, including: Based on the target patient's medical information, a three-dimensional cardiovascular model corresponding to the target patient is generated, and the three-dimensional cardiovascular model is stored in a database; Based on the identification information of the target patient, the three-dimensional cardiovascular model is retrieved from the database and sent to the AR device; A depth camera acquires the first node data of the patient's human skeleton and sends the first node data of the human skeleton to a smart terminal device; The first node data of the human skeleton is optimized to obtain the second node data of the human skeleton, and the second node data of the human skeleton is sent to the AR device; Based on the second node data of the human skeleton and the three-dimensional cardiovascular model, the three-dimensional cardiovascular model is registered at the location of the target patient's chest cavity using coordinate system alignment. After the registered three-dimensional cardiovascular model is rendered, it is sent to the VR device for screen sharing.

[0013] Furthermore, the medical information includes: the patient's personal information, CT images, and clinical diagnostic reports.

[0014] Furthermore, the method also includes: In response to the interactive commands of the AR device, a dynamic demonstration of the pathological evolution of coronary heart disease is shared on the VR device, including color mapping changes in the ischemic area of ​​myocardium and simulation of chest pain diffusion.

[0015] This application provides an augmented reality-based health education system and method for coronary heart disease, the beneficial effects of which are: 1. Significantly improves the intuitiveness and comprehension efficiency of health education, and enhances the depth of patients' understanding. This application uses a patient-specific three-dimensional cardiovascular model as the core carrier and transforms the abstract pathological mechanisms of coronary heart disease (such as vascular stenosis, myocardial ischemia, and disease progression) into visualized and dynamic holographic images through AR technology. Combined with contextualized presentations such as changes in myocardial color (normal → whitish → blackish) and the high-brightness diffusion of chest pain, it solves the problem that traditional static media (manuals, videos) cannot intuitively display complex anatomical structures and pathological processes, enabling patients to quickly understand the nature and development of the disease.

[0016] Leveraging the rendering optimization capabilities of the Unity engine, the system enhances the spatial immersion and realism of virtual scenes by refining the geometric details, lighting effects, and material textures of the models. This significantly improves patients' learning interest and participation, resulting in a substantial increase in knowledge acquisition and memory retention compared to traditional one-way education models.

[0017] 2. Achieve precise adaptation of personalized health education to meet diverse needs. This application establishes a unified resource description framework by parametrically processing multimodal educational resources (personal information, medical examination data, 3D models, educational graphics / animations). Based on the patient's individual characteristics such as medical history, imaging results, and physiological parameters, it can dynamically match the corresponding educational content and accurately map it to the target space, achieving personalized education for each individual and overcoming the limitations of the traditional "one-size-fits-all" approach.

[0018] By adopting the AssetBundle dynamic loading mechanism, the system can obtain personalized resources from the PC server on demand. This avoids the device operating burden caused by resource preloading and supports real-time updates and flexible adjustments of educational content, ensuring that educational resources are synchronized with patients' conditions and the latest clinical guidelines, thereby improving the pertinence and scientific nature of education. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the structure of a coronary heart disease health education system based on augmented reality, as described in an embodiment of this application. Figure 2 This is a schematic diagram of a process for augmented reality-based health education for coronary heart disease, as illustrated in an embodiment of this application.

[0021] Figure label: 101. AR devices; 102. VR devices; 103. Depth cameras; 104. Smart terminal devices. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] Terminology Explanation: AR devices: Augmented Reality devices are smart hardware that uses cameras, sensors, and display technology to overlay and blend virtual digital content (images, text, 3D models, animations, data) into real-world scenes, allowing users to see both the real environment and virtual information at the same time, thus achieving "augmented reality".

[0024] VR devices: Virtual reality devices, through display screens, optical systems, gyroscopes, and positioning sensors, completely shut out the user's vision and hearing, constructing a purely virtual three-dimensional world entirely generated by a computer, giving the user an immersive feeling of "being in a virtual environment" and temporarily isolating them from the real physical world.

[0025] Please see Figure 1 This application provides an augmented reality-based coronary heart disease health education system 100, comprising: The system includes an AR device 101 for doctors, a VR device 102 for patients, a depth camera 103, and a smart terminal device 104, wherein the smart terminal device 104 is communicatively connected to the AR device 101, the VR device 102, and the depth camera 103, respectively. The depth camera 103 is configured to collect first node data of the human skeleton of the target patient in real time and send the first node data of the human skeleton to the smart terminal device 104. The intelligent terminal device 104 is configured to obtain a three-dimensional cardiovascular model corresponding to the target patient from the database based on the identification information of the target patient, and send the three-dimensional cardiovascular model to the AR device 101; The intelligent terminal device 104 is further configured to optimize the first node data of the human skeleton to obtain the second node data of the human skeleton, and send the second node data of the human skeleton to the AR device 101. The AR device 101 is used to register the three-dimensional cardiovascular model at the location of the target patient's chest cavity using coordinate system alignment, based on the second node data of the human skeleton and the three-dimensional cardiovascular model, and then render the registered three-dimensional cardiovascular model and send it to the VR device 102. The VR device 102 is used to receive and immerse the shared viewpoint image rendered in the AR device 101.

[0026] The following is a detailed description of each device in this application: I. Depth Camera 103: The Depth Camera 103 utilizes the Azure Kinect DK sensor, whose core function is to capture real-time 3D pose data of the target patient's skeleton. This is specifically implemented through: Using infrared depth imaging and motion detection algorithms, raw data of 32 skeletal nodes throughout the patient's body are collected simultaneously, covering the three-dimensional spatial coordinates, rotational posture parameters and motion trajectory information of each node, ensuring the comprehensiveness and real-time nature of data collection. The depth camera housing is rigidly fixed with a pre-set artificial visual marker—the marker uses a high-contrast black and white checkerboard pattern, combined with a unique marking area and evenly distributed feature points, which has high recognition and anti-interference, and provides a stable feature reference for subsequent pose calculation; The collected data of the first node of the human skeleton (the original 32 nodes) is transmitted in real time to the smart terminal device 104 via a data cable. During the transmission, a data verification mechanism is used to ensure the integrity and accuracy of the original data, laying the foundation for subsequent data optimization and processing.

[0027] II. Smart Terminal Devices 104: The intelligent terminal device 104 can be a high-performance PC host running an operating system, undertaking core functions such as data integration, resource scheduling, and business processing. Specific functions are as follows: 1. Acquisition and integration of personalized patient data: Based on the unique identification information of the target patients (such as medical record number), the system connects to the hospital's database via wired network to obtain patients' full-dimensional health data in batches, including basic personal information (name, age, height, weight, past medical history, family medical history), medical examination data (CT images, imaging diagnostic reports, physiological parameters), and medical reconstruction data (three-dimensional coronary artery model, heart model, information on the location and degree of vascular blockage). The acquired multi-source heterogeneous data is standardized to establish patient-specific data files, providing data support for personalized education resource matching and accurate registration of virtual models.

[0028] 2. Skeletal data optimization processing: After receiving the first node data of the human skeleton transmitted by the depth camera 103, the system uses a preset skeleton node optimization mapping algorithm to select 19 core nodes (including key anatomical reference nodes such as the hip center node, spine point, chest point, shoulder center point, left clavicle point, and right clavicle point) from 32 original nodes, removes redundant data, and improves the efficiency of subsequent calculations. The filtered core node data is converted into a standardized JSON format, and the data structure is reconstructed and the coordinates are calibrated to generate the second node data of the human skeleton, ensuring that the data can be directly adapted to the skeleton mapping and model-driven requirements of AR device 101.

[0029] 3. Personalized resource scheduling and management: Based on patient health data and disease characteristics, corresponding health education resource packages are dynamically matched from the database. These resource packages include static educational resources (textual and graphic materials and animations related to the prevention, treatment, and surgery of coronary heart disease) and dynamic educational resources (red blood cell models, UV diagrams of the evolution of coronary heart disease, and videos simulating pathological mechanisms). All health education resources are packaged in the AssetBundle format of the Unity engine, supporting management by tag and on-demand access. The smart terminal device 104 can send the resource package to the AR device 101 and VR device 102 in real time according to the education scenario, realizing dynamic loading and accurate distribution of resources. It supports real-time updates of health education resources, and can synchronize the latest prevention and treatment guidelines for coronary heart disease, evidence-based medicine, and typical case data to ensure the scientific accuracy and timeliness of the educational content.

[0030] 4. Cross-device data transfer and protocol adaptation: A hybrid TCP / UDP transport protocol is used to achieve data communication between multiple devices: TCP is used to transmit key data such as interactive instructions, patient health data, and AssetBundle resource packages, ensuring the integrity and orderliness of data transmission; UDP is used to transmit data with high real-time requirements such as skeletal pose data and AR rendering images, ensuring that the transmission latency is controlled within 30ms (average latency 10.64ms). By introducing the object pool design pattern, reusable data message parsing objects are pre-instantiated. When receiving high-frequency data such as skeletal pose, the pre-allocated ProtocolSkele type parsing object is directly called from the object pool to perform data splitting and processing, reducing the performance overhead caused by memory allocation and destruction and improving system operating efficiency.

[0031] III. AR Device 101: The preferred AR device is the HoloLens 2 mixed reality headset, which serves as the core operating platform for doctors, responsible for virtual model registration, rendering, and execution of interactive commands. Specific functionalities include: 1. Coordinate system alignment and pose calibration: The AR device 101 uses its built-in camera to identify artificial visual markers on the depth camera 103 in real time. Based on the PnP (Perspective-n-Point) algorithm, it takes the known 3D world point coordinates on the marker (based on the depth camera coordinate system) and the 2D pixel coordinates extracted from the image as inputs to solve for the six-degree-of-freedom pose (including rotation matrix R and translation matrix T) of the depth camera 103 in the augmented reality space coordinate system of the AR device, thus achieving precise alignment between the depth camera coordinate system and the augmented reality space coordinate system. Based on the solved pose parameters, a coordinate transformation matrix is ​​constructed to transform the second node data of the human skeleton transmitted by the smart terminal device 104 (originally located in the depth camera coordinate system) to the augmented reality space coordinate system, thereby achieving the unification of the skeleton data and the AR space coordinates.

[0032] 2. Precise registration and dynamic driving of 3D cardiovascular models: The system receives a patient-specific three-dimensional cardiovascular model sent by the smart terminal device 104. This model is directly linked to the patient's CT angiography data and accurately reconstructs the patient's coronary artery anatomy and lesion location. Based on the converted second node data of the skeleton, and combined with the medical principle that "there is a fixed anatomical positional relationship between the human skeleton and the cardiovascular system", the three-dimensional cardiovascular model is accurately registered at the corresponding anatomical position of the target patient's chest cavity by taking the hip center node as the root node and locating the coordinates of key nodes such as the chest point and clavicle point. It receives real-time updates of the second node data of the skeleton and realizes dynamic mapping between the skeleton data and the virtual model through the Avatar module of the Unity engine. It drives the 3D cardiovascular model to move synchronously with the changes in the patient's body position (translation, rotation, posture adjustment). At the same time, it dynamically adjusts the model orientation by calculating the normal vector of the plane formed by the left clavicle point, the right clavicle point and the chest point, so as to ensure that the model always maintains a precise fit with the patient's chest cavity without drifting or shaking.

[0033] 3. Model rendering and interactive control: Leveraging the rendering optimization capabilities of the Unity engine, the 3D cardiovascular model is refined, optimizing geometric details, lighting rendering effects, and material textures to enhance the realism and spatial immersion of the virtual model. It supports doctors to operate virtual models through natural interaction methods such as gestures and voice, including model translation, scaling, rotation, internal structure disassembly and display, and simulation of blood vessel stenosis, which makes it easier for doctors to provide targeted explanations for patients' specific conditions; It has a built-in three-stage simulation function for the evolution of coronary heart disease. By adjusting the model material parameters, the color of the myocardium can be dynamically changed (normal → whitish → black). The red highlighted area shows the spread of chest pain as the condition worsens, intuitively presenting the disease development pattern.

[0034] 4. Spatial anchor point persistence and view sharing: The pose data of the depth camera 103 in the augmented reality spatial coordinate system is saved as spatial anchor points, and the anchor point information is persistently stored locally through HoloLens2's Simultaneous Localization and Mapping (SLAM) technology. When the system restarts in the same physical environment, it can automatically identify environmental features and restore the spatial anchor points, maintaining the continuity and stability of the 3D cardiovascular model's registered position and avoiding repeated calibration. The rendered screen, which includes the registered 3D cardiovascular model, the doctor's operation trajectory, and educational annotations, is sent to the VR device 102 in real time via a low-latency network transmission protocol, enabling synchronous sharing of the AR perspective between doctors and patients and laying the foundation for two-way interaction.

[0035] In one specific embodiment of this application, the AR device is specifically configured as follows: Based on the preset marker recognition results, the pose of the depth camera in the augmented reality spatial coordinate system is calculated to achieve alignment between the depth camera coordinate system and the augmented reality spatial coordinate system. Based on the aligned coordinate system, the three-dimensional coordinates of the received second node data of the human skeleton are transformed into the augmented reality space coordinate system; The position and orientation of the three-dimensional cardiovascular model are dynamically adjusted according to the transformed coordinates, so that the three-dimensional cardiovascular model is stably registered at the thoracic anatomical position of the target patient. The rendered image containing the registered 3D cardiovascular model is transmitted to the VR device via a low-latency network transmission protocol.

[0036] IV. VR Device 102: VR device 102 recommends the Pico4 VR headset, whose core function is to provide patients with an immersive health education experience, specifically including: The system receives shared view rendering images transmitted from AR device 101 in real time. Through the immersive display technology of VR headset, patients can intuitively observe the anatomical structure, lesion characteristics and disease progression of the three-dimensional cardiovascular model, and simultaneously perceive the doctor's operation and key points of explanation. It receives health education resource packages in AssetBundle format sent by smart terminal device 104, and supports simultaneous viewing of static text and images, dynamic animations, pathological mechanism videos and other educational content in immersive scenarios, realizing dual education of "visualized model + precise resources"; With the configuration of simple interactive controls, patients can control the education content by pausing, switching, and repeating playback through the handheld device or voice commands. The control commands are transmitted back to the smart terminal device 104 via the TCP protocol, and then synchronized to the AR device 101 to ensure the consistency of doctor-patient interaction. The display parameters and interaction logic have been optimized to suit patients of different ages and educational levels. Users can quickly get started without professional training, thus lowering the barrier to entry.

[0037] The working principle of this application is as follows: System startup phase: Smart terminal device 104 connects to the hospital database to complete patient information configuration and health education resource loading; depth camera 103 and AR device 101 complete initial pose calibration through artificial visual marking; VR device 102 establishes network connection with smart terminal device 104 and AR device 101. Data acquisition and processing stage: Depth camera 103 acquires the first node data of the patient's skeleton in real time and transmits it to intelligent terminal device 104. Intelligent terminal device 104 completes data optimization to obtain the second node data, and at the same time matches a personalized three-dimensional cardiovascular model and education resource package. Model registration and rendering stage: AR device 101 completes coordinate system alignment based on PnP algorithm, accurately registers the 3D cardiovascular model in the patient's chest cavity, and completes model rendering and disease simulation through Unity engine; Immersive education and interaction phase: VR device 102 simultaneously receives the shared view of AR device 101 and the education resources of smart terminal device 104, allowing patients to learn about the disease in an immersive way, and enabling doctors and patients to achieve two-way communication and interaction through device collaboration; System stability assurance phase: Spatial anchoring technology maintains the stability of model registration, hybrid transmission protocol and object pool mechanism ensure the real-time performance of data transmission and the smooth operation of the system, and dynamic resource update mechanism ensures the scientific nature of the propaganda content.

[0038] Through the collaboration of the aforementioned devices and technological innovation, this system transforms abstract pathological information of coronary heart disease into perceptible, interactive, and immersive content, achieving personalized, efficient, and precise health education. This significantly improves patients' cognitive efficiency and participation, while also optimizing the doctor-patient communication experience, providing strong technical support for the prevention and management of coronary heart disease.

[0039] This application provides an augmented reality-based health education system for coronary heart disease, the beneficial effects of which are: 1. Significantly improves the intuitiveness and comprehension efficiency of health education, and enhances the depth of patients' understanding. This application uses a patient-specific three-dimensional cardiovascular model as the core carrier and transforms the abstract pathological mechanisms of coronary heart disease (such as vascular stenosis, myocardial ischemia, and disease progression) into visualized and dynamic holographic images through AR technology. Combined with contextualized presentations such as changes in myocardial color (normal → whitish → blackish) and the high-brightness diffusion of chest pain, it solves the problem that traditional static media (manuals, videos) cannot intuitively display complex anatomical structures and pathological processes, enabling patients to quickly understand the nature and development of the disease.

[0040] Leveraging the rendering optimization capabilities of the Unity engine, the system enhances the spatial immersion and realism of virtual scenes by refining the geometric details, lighting effects, and material textures of the models. This significantly improves patients' learning interest and participation, resulting in a substantial increase in knowledge acquisition and memory retention compared to traditional one-way education models.

[0041] 2. Achieve precise adaptation of personalized health education to meet diverse needs. This application establishes a unified resource description framework by parametrically processing multimodal educational resources (personal information, medical examination data, 3D models, educational graphics / animations). Based on the patient's individual characteristics such as medical history, imaging results, and physiological parameters, it can dynamically match the corresponding educational content and accurately map it to the target space, achieving personalized education for each individual and overcoming the limitations of the traditional "one-size-fits-all" approach.

[0042] By adopting the AssetBundle dynamic loading mechanism, the system can obtain personalized resources from the PC server on demand. This avoids the device operating burden caused by resource preloading and supports real-time updates and flexible adjustments of educational content, ensuring that educational resources are synchronized with patients' conditions and the latest clinical guidelines, thereby improving the pertinence and scientific nature of education.

[0043] Please see Figure 2 This application also provides an augmented reality-based method for coronary heart disease health education, applied to any of the systems described above, comprising at least the following steps: S10. Based on the medical information of the target patient, generate a three-dimensional cardiovascular model corresponding to the target patient, and store the three-dimensional cardiovascular model in a database.

[0044] Specifically, in this step, based on the target patient's comprehensive medical information, a three-dimensional cardiovascular model is generated using medical image reconstruction and 3D modeling technology that highly matches the patient's anatomical structure and disease characteristics. This lays the core foundation for subsequent precise health education, as detailed below: Medical information collection and integration: Collect complete medical information of target patients, including basic personal information (name, age, height, weight, past medical history, family medical history), medical imaging data (coronary artery CT angiography images, tomographic image sequences), and clinical diagnostic reports (key diagnostic conclusions such as the location and degree of stenosis, the extent of myocardial ischemia, and the type of lesion), to ensure that the data source for model generation is comprehensive and accurate.

[0045] Medical image preprocessing and reconstruction: Preprocessing operations such as noise reduction, enhancement, and segmentation are performed on the acquired CT image data to remove redundant background information and accurately extract the contour data of core anatomical structures such as coronary arteries and cardiac chambers. Three-dimensional reconstruction algorithms (such as Marching Cubes algorithm) are used to reconstruct the processed image data to generate a preliminary three-dimensional cardiovascular geometric model, ensuring that the anatomical structure of the model is consistent with the patient's actual condition, and that key features such as vascular branches, lumen diameter, and lesion location are without deviation.

[0046] Model parameterization and personalized optimization: Based on clinical diagnostic reports, the lesion area is accurately marked in the 3D model: by adjusting the model material parameters, the stenotic sites and plaque distribution range are highlighted, and the degree of stenosis (such as mild, moderate, and severe) and the corresponding anatomical location (such as the left anterior descending artery, circumflex artery, and right coronary artery) are marked. The model is optimized for lightweight design, reducing the number of faces and complexity of the model while retaining key anatomical details, ensuring smooth loading and real-time rendering on AR devices; at the same time, the model is associated with the patient's unique identification information (such as medical record number) to generate a personalized 3D cardiovascular model file for the patient.

[0047] Model storage and management: The generated 3D cardiovascular model is stored in the hospital's database in a standardized format (compatible with Unity engine parsing), and is also linked to the patient's medical information and resource tags required for subsequent education, so as to facilitate quick retrieval and access through the identification information and realize the integrated management of model and patient data.

[0048] S20. Based on the identification information of the target patient, obtain the three-dimensional cardiovascular model from the database and send the three-dimensional cardiovascular model to the AR device.

[0049] In this step, based on the unique identifier of the target patient, the personalized 3D cardiovascular model is accurately retrieved and transmitted across devices, ensuring that the model efficiently adapts to the registration and rendering requirements of AR devices. Precise Model Retrieval: Doctors input the target patient's identification information (such as medical record number and ID card number) through a smart terminal device 104. The system automatically connects to the hospital's database and retrieves the corresponding personalized 3D cardiovascular model and related medical information based on the identification information, ensuring that the model and the patient's identity are accurately matched and avoiding confusion.

[0050] Model format adaptation and preprocessing: The smart terminal device 104 performs format conversion and preprocessing on the retrieved 3D cardiovascular model, adapting the model file to a format supported by the Unity engine, and configuring it in association with the system's preset bone mapping rules and rendering parameters to ensure that the model can be directly used for registration and interaction after being transmitted to the AR device.

[0051] Secure and efficient transmission: The preprocessed 3D cardiovascular model is transmitted from the smart terminal device 104 to the AR device 101 via the TCP protocol. Data encryption and verification mechanisms are used during the transmission process to ensure the security and integrity of the model data, avoid data loss or damage during transmission, and ensure that the AR device can quickly receive and load the model.

[0052] S30. The depth camera collects the first node data of the patient's human skeleton and sends the first node data of the human skeleton to the smart terminal device.

[0053] Specifically, in this step, the patient's skeletal pose data is captured in real time using a depth camera to provide a spatial reference for the accurate registration of the 3D cardiovascular model. The specific implementation is as follows: Equipment deployment and calibration: Fix the depth camera 103 (preferably Azure Kinect DK) in a suitable position in the educational scene to ensure that the camera's shooting angle can fully cover the patient's whole body. At the same time, perform initial calibration on the camera to eliminate equipment errors. Fix the preset artificial visual markers (high-contrast black and white checkerboard pattern) rigidly to the depth camera housing to provide a feature reference for subsequent pose calculation.

[0054] Real-time acquisition of skeletal data: After the depth camera is started, infrared depth imaging and human skeleton tracking algorithms are used to capture the whole body skeletal motion data of the target patient in real time. The first node data of the acquired human skeleton covers the three-dimensional spatial coordinates, rotational posture parameters and motion trajectory information of 32 whole body skeletal joints, including node data of key anatomical positions such as hip, spine, chest, shoulder, and limbs. The acquisition frequency is no less than 30Hz to ensure the real-time and continuous nature of the data.

[0055] Real-time data transmission and verification: The raw data of the first node of the human skeleton is transmitted to the smart terminal device 104 in real time via a data cable. During the transmission, a data verification code is embedded. After receiving the data, the smart terminal device automatically verifies the integrity of the data. If data loss or abnormality is detected, the depth camera is triggered to re-acquire the data to ensure the reliability of the raw skeleton data.

[0056] S40. Optimize the first node data of the human skeleton to obtain the second node data of the human skeleton, and send the second node data of the human skeleton to the AR device.

[0057] Specifically, in this step, the intelligent terminal device 104 filters, reconstructs, and standardizes the original skeletal data to generate core node data that meets the model registration requirements, thereby improving the efficiency of subsequent coordinate transformation and model-driven processes. Skeletal node screening and core node extraction: Based on the anatomical needs of coronary heart disease education, 19 core nodes were selected from 32 original skeletal nodes using a preset algorithm. Key nodes related to cardiovascular positioning were retained, including the hip center node (Pelvis), spine point (Spine_Naval), chest point (Spine_Chest), shoulder center point (Neck), head point (Head), left and right clavicle points, left and right shoulder points, left and right elbow points, left and right wrist points, left and right hip points, left and right knee points, and left and right ankle points. Redundant nodes unrelated to thoracic anatomical positioning were removed to reduce data processing overhead.

[0058] Data format standardization and structure reconstruction: The filtered core node data is converted into a standardized JSON format, clarifying the key information such as the identifier, 3D coordinates, and rotation parameters of each node, thus completing the data structure reconstruction. This allows the data to be directly parsed by the Unity engine of AR devices without additional format conversion operations.

[0059] Data calibration and noise filtering: A smoothing filtering algorithm is used to filter noise in the core node data to eliminate data fluctuations caused by slight limb tremors of patients and equipment acquisition errors, ensuring the stability of bone node coordinates; at the same time, based on human anatomical constraints, abnormal node data (such as coordinate values ​​that exceed the normal physiological range of motion) are corrected to ensure the scientific nature and accuracy of the data.

[0060] Optimized data transmission: The processed second-node data of the human skeleton is transmitted to AR device 101 in real time via the UDP protocol. By utilizing the low latency characteristics of the UDP protocol, the data transmission latency of the skeleton is controlled at the millisecond level, which ensures that the model can follow the patient's position changes in real time.

[0061] S50. Based on the second node data of the human skeleton and the three-dimensional cardiovascular model, the three-dimensional cardiovascular model is registered at the location of the target patient's chest cavity using coordinate system alignment. After the registered three-dimensional cardiovascular model is rendered, it is sent to the VR device for screen sharing.

[0062] In this step, AR device 101, based on optimized skeletal data and a personalized 3D cardiovascular model, achieves seamless integration of the virtual model and the patient's real body through operations such as coordinate system alignment, precise registration, and real-time rendering. This is then synchronized to VR device 102, providing an immersive collaborative educational experience for both medical staff and patients, as detailed below: Coordinate system alignment and pose calibration: AR device 101 uses its built-in camera to identify artificial visual markers on the depth camera shell in real time and extract feature point information from the markers. Based on the PnP (Perspective-n-Point) algorithm, it inputs the preset 3D world point coordinates on the markers (based on the depth camera coordinate system) and the 2D pixel coordinates extracted from the image to solve for the six-degree-of-freedom pose (rotation matrix R and translation matrix T) of the depth camera in the augmented reality space coordinate system of the AR device, thus achieving precise alignment between the depth camera coordinate system and the augmented reality space coordinate system.

[0063] Based on the solved pose parameters, a coordinate transformation matrix is ​​constructed to transform the received second node data of the human skeleton (originally in the depth camera coordinate system) to the augmented reality spatial coordinate system, thereby unifying the coordinates of the skeleton data and the AR virtual space and providing a spatial reference for model registration.

[0064] Precise registration of 3D cardiovascular models: After the AR device 101 loads the 3D cardiovascular model, based on the converted second node data of the skeleton and combined with the human anatomical rules (the fixed positional relationship between the skeleton and the cardiovascular system), it uses the center node of the hip as the root node and locates the thoracic anatomical area through the coordinates of the chest point and the left and right clavicle points. The 3D cardiovascular model is then accurately superimposed onto the corresponding position of the target patient's thoracic cavity, ensuring that the position of the heart and the direction of blood vessels in the model are consistent with the patient's actual anatomical structure.

[0065] The system receives updated second-node data of the skeleton in real time and establishes a dynamic mapping relationship between the skeleton data and the virtual model through the Avatar module of the Unity engine. This drives the 3D cardiovascular model to translate and rotate synchronously with changes in the patient's body position (such as standing, sitting, and limb rotation). At the same time, by calculating the normal vector of the plane formed by the left and right clavicle points and the chest point, the model orientation is dynamically adjusted to ensure that the model always fits precisely with the patient's chest cavity without drifting, shaking, or positional deviation.

[0066] Model rendering and detail optimization: Leveraging the rendering capabilities of the Unity engine, the 3D cardiovascular model is rendered with high precision: the geometric details, lighting effects, and material textures of the model are optimized to simulate the realistic texture of the blood vessel walls and the semi-transparent effect of the myocardial tissue, thereby enhancing the realism and spatial immersion of the virtual model. The model highlights the lesion area and distinguishes between normal segments (red), mildly stenotic segments (yellow), moderately stenotic segments (orange), and severely stenotic segments (red) by different colors, intuitively presenting the distribution and severity of the lesion, making it easier for doctors to quickly focus on explaining the key points.

[0067] Rendered screen sharing and VR presentation: The AR device will include a precisely registered 3D cardiovascular model, a synchronized view of the patient's position, and a rendered view of the lesion annotation information. This will be transmitted in real time to the VR device 102 via the UDP protocol, ensuring a transmission latency of less than 30ms (average latency of 10.64ms), thus enabling synchronous sharing of the AR perspective between doctors and patients. After receiving the rendered image, the VR device presents panoramic educational content to the patient through immersive display technology. The patient can intuitively observe the corresponding three-dimensional cardiovascular model, the location of the lesion, and the doctor's subsequent operation explanation, breaking the limitation of "abstract description" in traditional education and improving the patient's cognitive efficiency of the disease.

[0068] In one embodiment of this application, the method further includes: S60. Responding to the interaction command of the AR device, share a dynamic demonstration of the pathological evolution of coronary heart disease on the VR device, including color mapping changes in the ischemic area of ​​myocardium and simulation of chest pain diffusion.

[0069] Specifically, in this step, in response to the doctor's interactive commands on the AR device, a dynamic demonstration of the pathological evolution of coronary heart disease is simultaneously shared on the VR device, achieving dual education through "static model + dynamic process," thereby enhancing patients' understanding of the disease's progression. Interactive command triggering and parsing: Doctors issue interactive commands through gestures, voice or touch operations on AR devices, including "start pathological evolution demonstration", "switch lesion stage", "zoom in ischemic area", "show chest pain range", etc.; AR devices parse the command content in real time and trigger the corresponding pathological demonstration logic.

[0070] Dynamic simulation of three stages of pathological evolution: Healthy state stage: This stage showcases the normal physiological state of the patient's cardiovascular system, with unobstructed blood vessels and sufficient blood supply to the myocardial tissue. The model is presented in normal physiological colors (pink for the myocardium and red for the blood vessels), accompanied by audio explanations of the normal blood circulation pathway and cardiac function mechanisms.

[0071] Mild to moderate ischemia stage: Based on clinical diagnostic data, the process of myocardial ischemia caused by vascular stenosis is dynamically simulated: the lumen diameter of the corresponding vascular segment in the model is gradually adjusted to simulate the increase in the degree of stenosis; at the same time, the color of the myocardium in the ischemic area is gradually changed from pink to off-white, the ischemic range and the affected myocardial area are marked, and the causes of ischemia (such as plaque blockage, vasospasm) and potential symptoms (such as chest tightness, shortness of breath after activity) are explained simultaneously.

[0072] Severe stenosis - myocardial necrosis stage: The pathological simulation is further deepened, and the degree of vascular stenosis reaches severe or complete occlusion. The ischemic area of ​​myocardium further expands and develops into necrosis. The color of the myocardium in the necrotic area changes from whitish to dark black. The chest pain symptoms are simulated by dynamically highlighted red areas, and the red areas gradually spread as the condition worsens (such as spreading from the precordial area to the left shoulder, back, and jaw), which visually presents the risks and hazards of disease progression.

[0073] Multi-terminal synchronization and interactive feedback: The dynamic demonstration of pathological evolution is synchronized to the VR device in real time, allowing patients to immerse themselves in the complete process of disease development. At the same time, patients can use the interactive controls of the VR device (such as controller buttons and voice commands) to make requests for pause, rewind, and repeat explanation. The instructions are transmitted back to the smart terminal device via the TCP protocol and then synchronized to the AR device. Doctors can adjust the demonstration progress or focus on the key points of the explanation according to the patient's needs, realizing two-way interaction between doctors and patients and improving the pertinence and effectiveness of education.

[0074] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.

[0075] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0077] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0078] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0079] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A coronary heart disease health education system based on augmented reality, characterized in that, include: The system includes an AR device for doctors, a VR device for patients, a depth camera, and a smart terminal device, wherein the smart terminal device is communicatively connected to the AR device, the VR device, and the depth camera, respectively. The depth camera is configured to acquire first node data of the human skeleton of the target patient in real time and send the first node data of the human skeleton to the smart terminal device. The intelligent terminal device is configured to obtain a three-dimensional cardiovascular model corresponding to the target patient from the database based on the identification information of the target patient, and send the three-dimensional cardiovascular model to the AR device; The intelligent terminal device is also configured to optimize the first node data of the human skeleton to obtain the second node data of the human skeleton, and send the second node data of the human skeleton to the AR device. The AR device is used to register the three-dimensional cardiovascular model at the location of the target patient's chest cavity using coordinate system alignment, based on the second node data of the human skeleton and the three-dimensional cardiovascular model, and then render the registered three-dimensional cardiovascular model and send it to the VR device. The VR device is used to receive and immerse the viewpoint rendered in the AR device.

2. The system according to claim 1, characterized in that, The AR device is specifically configured as follows: Based on the preset marker recognition results, the pose of the depth camera in the augmented reality spatial coordinate system is calculated to achieve alignment between the depth camera coordinate system and the augmented reality spatial coordinate system. Based on the aligned coordinate system, the three-dimensional coordinates of the received second node data of the human skeleton are transformed into the augmented reality space coordinate system; The position and orientation of the three-dimensional cardiovascular model are dynamically adjusted according to the transformed coordinates, so that the three-dimensional cardiovascular model is stably registered at the thoracic anatomical position of the target patient. The rendered image containing the registered 3D cardiovascular model is transmitted to the VR device via a low-latency network transmission protocol.

3. The system according to claim 2, characterized in that, The preset marker is an artificial visual mark of a known geometric pattern fixed on the housing of the depth camera. The VR device identifies the artificial visual mark in real time through its built-in camera and calculates the six-degree-of-freedom pose of the depth camera based on the PnP algorithm.

4. The system according to claim 1, characterized in that, The second node data of the human skeleton includes at least the following: hip center node, spine point, chest point, shoulder center point, left clavicle point, and right clavicle point.

5. The system according to claim 1, characterized in that, The smart terminal device is also configured to acquire the health education resource package corresponding to the target patient and send the health education resource package to the VR device.

6. The system according to claim 1, characterized in that, The health education resource package is packaged in AssetBundle format using the Unity engine and stored in a database.

7. The system according to claim 1, characterized in that, The AR device is also configured to: save the pose of the depth camera in the augmented reality spatial coordinate system as a spatial anchor point; and automatically restore the spatial anchor point when the system restarts in the same physical environment to maintain the continuity of the registered position of the three-dimensional cardiovascular model.

8. A method for health education on coronary heart disease based on augmented reality, applied to the system of any one of claims 1-7, characterized in that, include: Based on the target patient's medical information, a three-dimensional cardiovascular model corresponding to the target patient is generated, and the three-dimensional cardiovascular model is stored in a database; Based on the identification information of the target patient, the three-dimensional cardiovascular model is retrieved from the database and sent to the AR device; A depth camera acquires the first node data of the patient's human skeleton and sends the first node data of the human skeleton to a smart terminal device; The first node data of the human skeleton is optimized to obtain the second node data of the human skeleton, and the second node data of the human skeleton is sent to the AR device; Based on the second node data of the human skeleton and the three-dimensional cardiovascular model, the three-dimensional cardiovascular model is registered at the location of the target patient's chest cavity using coordinate system alignment. After the registered three-dimensional cardiovascular model is rendered, it is sent to the VR device for screen sharing.

9. The method according to claim 8, characterized in that, The medical information includes: the patient's personal information, CT images, and clinical diagnostic reports.

10. The method according to claim 8, characterized in that, The method further includes: In response to the interactive commands of the AR device, a dynamic demonstration of the pathological evolution of coronary heart disease is shared on the VR device, including color mapping changes in the ischemic area of ​​myocardium and simulation of chest pain diffusion.

Images