High-quality cardio-pulmonary resuscitation and AED self-service training system based on 5G and augmented reality

By introducing 5G and augmented reality technology in CPR training, combining multi-view RGB images with point cloud fusion segmentation and three-dimensional visualization models, the problem that the existing training system cannot improve students' understanding and interest is solved, and high-simulation CPR is achieved Training and evaluation have improved the public's first aid capabilities and health literacy.

CN119049369AInactive Publication Date: 2024-11-29SHANDONG UNIV QILU HOSPITAL

Patent Information

Application Number
CN202411050656.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing CPR training system cannot effectively improve students' understanding and interest in the CPR process, and the training process lacks a sense of reality, making it difficult to meet the public's needs for first aid skills training.

Method used

Adopting high-quality CPR and AED self-service training system based on 5G and augmented reality, the segmentation module of multi-view RGB images and point clouds, three-dimensional visualization model of human chest cavity and neck position, mixed reality display terminal and video understanding-based CPR The autonomous evaluation system combines holographic imaging and virtual reality technology to provide immersive, secure, and highly simulated training and evaluation.

Benefits of technology

Enable students to understand the rescue significance and principles of CPR from an essential basis, improve the training effect, achieve risk-free, safe, high-simulation teaching and evaluation, and enhance the public's ability to rescue and mutual rescue and health literacy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049369A_ABST
    Figure CN119049369A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of cardiopulmonary resuscitation training systems, and discloses a high-quality cardiopulmonary resuscitation and AED self-service training system based on 5G and augmented reality. The method comprises the following steps: collecting holographic images of standardized operation of a teaching teacher, pushing the holographic images to a mixed reality display equipment terminal in real time, then holographic-projecting and superposing a head and neck three-dimensional visual model of a chest and a neck body position of a human body to a real model of a simulated person, and finally, carrying out cardio-pulmonary resuscitation autonomous evaluation system based on video understanding. According to the system, actions of trainees in training and examination are collected and analyzed, results are reproduced in MR equipment, correct actions are fed back to the trainees, the trainees are guided to correct wrong actions, the system is combined with a reality model through image superposition, the students can understand rescue significance and rescue principles of cardiopulmonary resuscitation on sudden cardiac arrest patients essentially, and the rescue effect of the students on sudden cardiac arrest patients is improved. And the first-aid process is learned deeper.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention discloses a high-quality cardiopulmonary resuscitation and AED self-service training system based on 5G and augmented reality, and relates to the technical field of cardiopulmonary resuscitation training systems. Background Art

[0002] Cardiac arrest (CA) refers to the cessation of the mechanical activity of the heart and the disappearance of circulation signs. If the patient receives timely treatment such as cardiopulmonary resuscitation, spontaneous circulation may be restored and the probability of survival is high. Otherwise, biological death is likely to occur, and spontaneous reversal is extremely rare. Out-of-hospital cardiac arrest (OHCA) refers to cardiac arrest that occurs outside the hospital. Cardiopulmonary resuscitation (CPR) is a rescue measure taken for respiratory and cardiac arrest such as artificial respiration and cardiac compression. Cardiac arrest is one of the major public health issues that seriously threatens human life and health.

[0003] Since OHCA can occur at any location, no matter how many professional first aid personnel there are, they cannot meet the requirement of "first aid at the first time and by the first witness". First responders with CPR skills and AED training can improve the survival rate of patients with cardiac arrest. Since most first responders do not have basic medical knowledge, the training should be simple and clear. At present, the training of cardiopulmonary resuscitation and AED mostly relies on local emergency centers (stations), the Red Cross, public hospitals and a few social training institutions to promote basic first aid skills training including CPR and AED. However, due to reasons such as time, venue and lack of training teachers, it is far from meeting the people's growing demand for first aid skills training and learning.

[0004] At present, most of the out-of-hospital CPR training teaching in higher medical schools still uses simulated emergency simulators and continues the concept of model-type electronic signal display in the traditional sense. That is, when practicing and assessing CPR on conventional simulators, an external digital display device is used to determine whether the relevant operation data in the CPR process is correct. As disclosed in CN113223389A, "A self-help training and assessment system for CPR based on AR technology", the problem with this teaching method is that students can only practice mechanically, but cannot understand why CPR is performed from the fundamental principles of first aid. During CPR, students cannot intuitively feel how the internal organs of the human body change, and why pressing and blowing can play a vital role in the internal circulation and respiratory system of the human body. The first aid process is not realistic enough, the gap between teaching and clinical diagnosis and treatment is too large, and students' learning interest cannot be effectively improved. These have affected the teaching and training effects to a certain extent. Therefore, a system is needed that allows students to understand the significance and principles of cardiopulmonary resuscitation for cardiac arrest patients from the essence, rather than simply training the operational movements of cardiopulmonary resuscitation, so as to cultivate students' all-round learning ability of "hands-on, brain-on, and heart-on". Summary of the invention

[0005] In view of the shortcomings of the existing technology, the present invention discloses a high-quality CPR and AED self-service training system based on 5G and augmented reality. The present invention utilizes the system's perfect bionic drive and holographic image overlay technology and rich built-in physiological display data to allow trainees to learn the first aid process and principles in a deeper level, and conduct risk-free, safe, and highly simulated teaching, training, and evaluation.

[0006] The detailed technical scheme of the present invention is as follows:

[0007] A high-quality cardiopulmonary resuscitation and AED self-service training system based on 5G and augmented reality, characterized by including the following components:

[0008] 1) A multi-view RGB image and point cloud fusion segmentation module, which uses a mixed reality first aid interaction method of multi-view RGB image and point cloud fusion segmentation to stream or cache the collected standardized cardiopulmonary resuscitation and AED operation segmentation and reconstructed three-dimensional holographic images to a mixed reality display device terminal in real time;

[0009] 2) A three-dimensional visualization model of the human chest, neck, body and head and neck. The CT tomographic images of the real human chest, neck, body and head and neck are collected, and the CT tomographic images are three-dimensionally reconstructed to obtain a three-dimensional geometric model and a three-dimensional finite element model of the CT tomographic images, and further a three-dimensional visualization model of the digital twin human chest, neck, body and head and neck is obtained;

[0010] 3) Mixed reality display terminal: Using the mixed reality display terminal, the system will push or cache the collected three-dimensional holographic images of the standardized cardiopulmonary resuscitation and AED operation of the teaching teacher to the MR device in real time. By wearing the MR device, the trainees can see the virtual teaching teacher's operation in the real world. At the same time, during the training process, the system will superimpose the three-dimensional visualization model of the digital twin human chest and neck position head and neck on the real model of the simulated person to display realistic human anatomical structure and real-time virtual physiological changes. During the student training process, the patient's human anatomical structure and real-time virtual physiological changes are transmitted to the display system of the mixed display device through Bluetooth technology, and displayed in real time in the real space. At the same time, multi-channel transmission is performed to meet the needs of multiple people to observe and guide;

[0011] In addition, during the training process, the instructor guided the operation of the AED and the location of the electrode stickers through a combination of virtual and real methods. The combination of virtual and real methods refers to the combination of holographic projection and a real model of a simulated person.

[0012] 4) An autonomous cardiopulmonary resuscitation assessment system based on video understanding. The system records and collects video data of the trainees' body shape, movement range, and operation process during training and assessment. It uses machine vision technology to extract each frame of the image, and uses a deep learning algorithm for analysis to calculate and infer the positions and relationships of key points of the human body. The results are displayed to the trainees through an MR device. At the same time, the correct action demonstration is displayed in real time on the MR device, and the areas where the trainees need to improve are marked in a graphical manner to guide and help the trainees correct their incorrect actions.

[0013] Preferably, the mixed reality first aid interaction method of multi-view RGB image and point cloud fusion and segmentation refers to, first, based on multi-view cameras and lidar as visual perception devices, video capture of doctor's standard cardiopulmonary resuscitation and AED operations is performed to obtain image and point cloud data of cardiopulmonary resuscitation and AED operations; further, the camera and lidar are calibrated with the help of a checkerboard as a calibration object, and the camera intrinsic parameter matrix and the camera and lidar extrinsic parameter matrix are solved through the collected image and point cloud data, and the conversion relationship from the lidar coordinate system to the camera coordinate system is constructed to achieve the fusion of image pixels and point cloud data; further, a point cloud segmentation algorithm based on regional growth and clustering is used to perform three-dimensional segmentation on the fused doctor's standard cardiopulmonary resuscitation and AED operation images and point cloud data to obtain a normalized cardiopulmonary resuscitation video three-dimensional model; further, the segmented normalized cardiopulmonary resuscitation video three-dimensional model is reconstructed through VTK technology to generate a three-dimensional visualization scene, which is displayed on a mixed reality or virtual reality display terminal in the form of a video stream.

[0014] Preferably, the point cloud segmentation algorithm based on region growing and clustering refers to using the edge of the target object in the image to extract the edge points of the target point cloud as seed points, and designing a region growing strategy based on the edge properties. At the same time, the center of mass of the target object in the image is used as the center point of the target point cloud, and the correlation between the target point cloud category and other three-dimensional points is calculated, and clustering is performed according to the set correlation criterion.

[0015] Preferably, the three-dimensional geometric model is obtained by performing CT scanning on the chest cavity and neck and body position of an adult male, collecting its CT tomographic images, and performing voxel reconstruction on the collected CT data of the chest cavity and neck and body position; the chest cavity scanning range is the complete chest cavity, from the 1st thoracic vertebra to the 12th thoracic vertebra, including soft tissue, bone tissue, internal organs, and vascular structure;

[0016] The three-dimensional finite element model is formed by using ANSYS to divide the body mesh, assigning material properties to each unit of the body network model, giving the chest cavity corresponding elastic-plastic bone material properties, and optimizing and intelligently repairing the finite element model according to the heart pump mechanism, left heart pump mechanism, chest pump mechanism and chest frequency resonance mechanism of chest compression during cardiopulmonary resuscitation, and finally forming a three-dimensional finite element model of the human chest cavity and neck, body position and head and neck.

[0017] Preferably, the optimization and intelligent repair of the finite element model refers to: using finite element analysis software such as ANSYS for automatic meshing, the software will automatically generate meshes according to the model geometry and material properties, and locally refine the area of ​​interest to improve the accuracy of the model; further, accurately assign material properties according to actual biological tissue properties, select appropriate material models to simulate actual biological tissue behavior, for the thorax, use a nonlinear elastoplastic model to simulate the response of bones, and use a shape optimization algorithm to improve the model to optimize material distribution and structural design.

[0018] Preferably, the simulator real model has a realistic human appearance, and has accurate airway and chest anatomical structures, including basic anatomical structures, and can be displayed in layers. In addition, the simulator has a built-in accurate sensor system, and the collected sensor data covers: CPR operation duration, number of compression interruptions greater than 10 seconds, compression and blowing curve bar graphs, real-time tidal volume calculation, real-time compression position judgment, real-time compression frequency and compression depth judgment, real-time compression number, real-time correct and incorrect compression number, real-time ventilation number, real-time correct and incorrect ventilation number, real-time cycle group number recording, total compression time to total CPR time ratio, total compression interruption time, blowing time and deflation time, and whether the chest cavity rebounds completely.

[0019] Preferably, the video-understanding-based CPR autonomous assessment system installs cameras at appropriate locations on the training and assessment site, which are connected to the system host to record and collect video data of the trainees' body shape, movement range, and operation process during training and assessment. The host uses machine vision technology to extract each frame of the image, and uses a deep learning algorithm to analyze, calculate and infer the positions and relationships of key points of the human body, and accurately judge whether each action is in place. Furthermore, the integrated action information and the sensor data of the cardiopulmonary resuscitation simulator equipment are used to make judgments and feedback to the trainees and training physicians, thereby realizing the organic unity of computer vision technology and simulator sensor technology.

[0020] Preferably, the use of a deep learning algorithm to analyze, calculate and infer the positions and relationships of key points of the human body refers to using a target detection algorithm to locate and identify the body parts of the trainee, and using a posture estimation model to detect key points of the human body, wherein the key points of the human body include the head, shoulders, elbows, wrists, hips, knees and ankles, and then based on the position information of the key points of the human body, analyzing the trainee's movement amplitude, posture and operation process, and comparing the trainee's actual movements with standard movements in real time.

[0021] Beneficial effects of the present invention

[0022] (1) The present invention combines the built-in holographic image of physiological and anatomical structures with a real model of a simulated person to teach students cardiopulmonary resuscitation, so that students can understand the significance and principle of cardiopulmonary resuscitation for patients with cardiac arrest from the essence, and let the trainees learn the first aid process and principles in a deeper level, and conduct risk-free, safe, and highly simulated training and assessment.

[0023] (2) The present invention installs a camera at the examination room location to collect the body shape, movement range, etc. of the trainees during the operation, and uses a deep learning algorithm to understand and evaluate the video, thereby objectively and effectively evaluating the trainees' movements during the operation and timely standardizing the trainees' movements. At the same time, the present invention can further integrate the sensor data of the simulated human equipment, make judgments based on this information and feedback to the trainees and training physicians, thereby realizing the organic unity of computer vision technology and simulated human sensor technology.

[0024] (3) The invention enables the public to receive first aid knowledge and skills training at a time of their choice. The training is simple and easy to conduct and can be easily accepted by the public. It can effectively enhance the self-rescue and mutual rescue capabilities of the entire population and gradually improve the health literacy level of our citizens. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This invention proposes a technical roadmap for a high-quality cardiopulmonary resuscitation and AED self-service training system based on 5G and augmented reality. DETAILED DESCRIPTION

[0026] Embodiment 1,

[0027] like Figure 1 as shown in .

[0028] A high-quality CPR and AED self-service training system based on 5G and augmented reality, including the following components:

[0029] 1) A mixed reality first aid interaction method based on multi-view RGB image and point cloud fusion segmentation is used to stream or cache the collected standardized cardiopulmonary resuscitation and AED operation segmentation and reconstructed three-dimensional holographic images to a mixed reality display device terminal (such as Hololens) in real time, so that students can autonomously and immersively learn cardiopulmonary resuscitation + AED operations.

[0030] 2) A parameterized digital twin human chest and neck body position three-dimensional visualization model holographic projection is superimposed on the real model of the simulated person. During the students' operation practice of cardiopulmonary resuscitation operation training, the human chest anatomical structure, arteries and veins, and heart are simulated based on the three-dimensional finite element model according to the load displacement under chest compression, and real-time virtual physiological changes are performed, so that students can understand the significance and principle of cardiopulmonary resuscitation for patients with cardiac arrest from the essence.

[0031] 3) Mixed reality display terminal: Using the mixed reality display terminal, the system will push or cache the collected three-dimensional holographic images of the standardized cardiopulmonary resuscitation and AED operation of the teaching teacher to the MR device in real time. By wearing the MR device, the trainees can see the virtual teaching teacher's operation in the real world. At the same time, during the training process, the system will superimpose the three-dimensional visualization model of the digital twin human chest and neck position head and neck on the real model of the simulated person to display realistic human anatomical structure and real-time virtual physiological changes. During the student training process, the patient's human anatomical structure and real-time virtual physiological changes are transmitted to the display system of the mixed display device through Bluetooth technology, and displayed in real time in the real space. At the same time, multi-channel transmission is performed to meet the needs of multiple people to observe and guide;

[0032] In addition, during the training process, the instructor guided the operation of the AED and the location of the electrode stickers through a combination of virtual and real methods. The combination of virtual and real methods refers to the combination of holographic projection and a real model of a simulated person.

[0033] 4) Through the video understanding-based CPR autonomous assessment system, the trainees' body shape, movement range, and operation process are recorded and video data is collected throughout the training and assessment. Each frame of the image is extracted using machine vision technology, and analyzed using deep learning algorithms to calculate and infer the positions and relationships of key points of the human body. The standardization of each action can be accurately judged, and the correct action is reproduced in the MR device to provide feedback to the trainees, guiding and helping them to correct incorrect actions. Use target detection algorithms (such as YOLO or Faster R-CNN) to locate and identify the trainees' body parts, and use posture estimation models (such as OpenPose or MediaPipe) to detect key points of the human body. Key points usually include the head, shoulders, elbows, wrists, hips, knees, and ankles. Based on the position information of the key points, analyze the trainees' movement range, posture, and operation process. Compare the trainees' actual movements with the standard movements in real time, and display the results to the trainees through MR devices (such as MR glasses). Display the correct action demonstration in real time in the MR device, and mark the areas where the trainees need to improve in a graphical way.

[0034] The mixed reality first aid interaction method based on multi-view RGB image and point cloud fusion segmentation first uses multi-view cameras and lidar as visual perception devices to perform video capture of doctors' standard cardiopulmonary resuscitation and AED operations to obtain image and point cloud data. Image data has the advantages of high resolution and rich color and texture information, but it is sensitive to external lighting and cannot provide three-dimensional geometric information; lidar is not easily affected by lighting and can obtain high-precision three-dimensional geometric shapes and three-dimensional spatial positions, but the point cloud density will be affected by distance and the distribution of three-dimensional points in the point cloud is disordered, so the two can complement each other's environmental perception capabilities through fusion;

[0035] Furthermore, the camera and lidar are calibrated with the help of a chessboard as a calibration object. The camera intrinsic parameter matrix and the camera and lidar extrinsic parameter matrices are solved through the two types of data collected, and the transformation relationship from the lidar coordinate system to the camera coordinate system is constructed to achieve the fusion of image pixels and point clouds, and realize the complementary advantages of the two modal data.

[0036] Furthermore, the collected videos and point cloud data of doctors’ standard cardiopulmonary resuscitation and AED operations are 3D segmented;

[0037] Furthermore, the above-mentioned segmented and reconstructed three-dimensional model of the standardized cardiopulmonary resuscitation video is used to generate a three-dimensional visualization scene through a three-dimensional graphics engine, and implemented on a mixed reality or virtual reality display terminal (such as Microsoft's Hololens and HTC's Vive). With the help of the 3D field of view inherent in the mixed reality itself, it can provide students with an autonomous and realistic learning environment, and can provide unlimited immersion and virtual experience, thereby accelerating and consolidating the process of students learning knowledge.

[0038] In order to improve the frame rate of video 3D segmentation, the 3D segmentation adopts a segmentation algorithm based on region growing and clustering point cloud. By combining point cloud with image, the image can provide rich color and texture information. The edge of the target object in the image is used to extract the edge points of the target point cloud as seed points, and a region growing strategy is designed according to the edge properties. Since the edge of the target point cloud is determined, the segmentation boundary can be effectively determined, thereby effectively avoiding under-segmentation and over-segmentation. At the same time, the center point of the target point cloud is extracted using the centroid of the target object in the image, and the correlation between the target point cloud category and other 3D points is calculated. Clustering is performed according to the set correlation criterion. Since the cluster center is determined in advance and the category correlation is judged, the category can be effectively divided and randomness can be avoided to a certain extent, thereby improving the segmentation accuracy.

[0039] In addition, remote real-time video streaming can be used to collect multi-camera and lidar data from remote training doctors in real time, segment their operations and reconstruct three-dimensional structures, and allow real-time interaction with locally trained students. On the basis of autonomous training and autonomous learning, a remote mixed reality conference experience can be used to increase the sense of interaction with the teaching teacher.

[0040] The parameterized digital twin human chest and neck body position three-dimensional visualization model is a three-dimensional geometric model and a three-dimensional finite element model holographic projection superimposed on a real human model, and the real-time changes of the human chest structure, arteries and veins, and heart are simulated according to the load displacement of the thorax under external chest compression, so that students can understand the significance and principle of cardiopulmonary resuscitation for patients with cardiac arrest from the essence.

[0041] The three-dimensional geometric model is obtained by performing CT scanning on the chest cavity, neck, torso and head of an adult male, collecting CT tomographic images thereof, and performing voxel reconstruction on the collected CT data of the chest cavity, neck, torso and head of the adult male.

[0042] The chest scan range is the complete chest cavity, from the 1st thoracic vertebra to the 12th thoracic vertebra (including the lower edge of the 12th rib), including soft tissue, bone tissue (sternum, ribs, costal cartilage, vertebrae, clavicle, scapula, and tibia), internal organs (heart, lungs, etc.), and vascular structure.

[0043] The three-dimensional finite element model is formed by using ANSYS to divide the body mesh, assigning material properties to each unit of the body network model, giving the chest cavity corresponding elastic-plastic bone material properties, and optimizing and intelligently repairing the finite element model according to the heart pump mechanism, left heart pump mechanism, chest pump mechanism and chest frequency resonance mechanism of chest compression during cardiopulmonary resuscitation, and finally forming a three-dimensional finite element model of the human chest cavity and neck, body position and head and neck.

[0044] The optimization and intelligent repair of the finite element model are as follows: automatic meshing is performed using finite element analysis software such as ANSYS, and the software automatically generates meshes based on the model geometry and material properties. Local refinement is performed on areas of interest (such as the chest area) to improve the accuracy of the model. Accurately assign material properties based on actual biological tissue properties (such as elastic-plastic bone material properties), and the mechanical properties of each tissue can be determined through references or experimental data. Select appropriate material models (such as linear elasticity, elastic-plasticity, etc.) to simulate actual biological tissue behavior. For the chest cavity, a nonlinear elastic-plastic model can be used to simulate the response of bones, and a shape optimization algorithm (such as topology optimization) can be used to improve the model to optimize material distribution and structural design.

[0045] The load-displacement relationship of the human chest under chest compression conforms to the three-dimensional model, and the two have a good correlation. The sternum displacement increases with the increase of load. Different compression positions during chest compression will result in different sternum displacements.

[0046] The real model of the simulator has a realistic human appearance, and has accurate airway and chest anatomical structures, including basic anatomical structures such as veins, arteries, and bones, which can be displayed in layers. The simulator has a built-in accurate sensor system, and the data that can be collected include: cardio pulmonary resuscitation (CPR) operation time, the number of times the compression is interrupted for more than 10s, the compression and blowing curve bar graph, the calculation of real-time tidal volume, the judgment of real-time compression position, the judgment of real-time compression frequency and compression depth, the statistics of real-time compression times (compression times, correct and wrong times), real-time ventilation times (ventilation times, correct and wrong times), the record of real-time cycle groups, the proportion of total compression time to total CPR time, the total compression interruption time, the blowing time, the deflation time, and whether the chest is completely rebounded. During the cardiopulmonary resuscitation training process, the patient's anatomical structure and real-time virtual physiological changes can be truly transmitted to the image overlay system through Bluetooth technology, and displayed in real time in the physical space. Multi-channel transmission can also be performed to meet the needs of multiple people for observation and guidance.

[0047] The video-understanding-based CPR autonomous assessment system installs cameras at appropriate locations on the training and assessment site, connects to the system host, and records and collects video data of the trainees' body shape, movement range, and operation process during training and assessment. The host uses machine vision technology to extract each frame of the image, and uses a deep learning algorithm for analysis to calculate and infer the position and relationship of each key point of the human body, and can accurately determine whether each action is in place. In addition, the video-understanding-based CPR autonomous assessment system can further integrate the sensor data of the CPR simulator equipment, including compression depth, compression frequency, ventilation volume, etc., and make judgments based on this information and feedback to the trainees and training physicians. Finally, the organic unity of computer vision technology and simulator sensor technology is achieved.

[0048] The deep learning algorithm uses TimeSformer to understand and evaluate cardiopulmonary resuscitation videos. Based on the spatiotemporal joint self-attention mechanism, it captures the spatiotemporal dependencies in the entire cardiopulmonary resuscitation video, extracts the key points and movement processes of the human body that play a key role in the effect of cardiopulmonary resuscitation, and more objectively and effectively evaluates the standardization of cardiopulmonary resuscitation movements and further quantifies them.

[0049] In traditional cardiopulmonary resuscitation (CPR) training, trainees usually use physical simulators for operation practice. These simulators provide a standardized training environment with their fixed anatomical structure and material properties. However, this traditional method has some significant shortcomings. First, the anatomical structure of the physical simulator is static and cannot be adjusted according to feedback from actual operations, thus limiting trainees' understanding of anatomical differences between different individuals. In addition, the physical simulator cannot provide real-time physiological feedback during the operation, making it impossible for trainees to observe the immediate impact of the operation on internal organs, such as changes in intrathoracic pressure or the recovery of heartbeat. Furthermore, since the use of physical simulators is limited by physical equipment and training venues, the time and location of trainees' training are also restricted, which reduces the flexibility of training.

[0050] The present invention proposes a new cardiopulmonary resuscitation training system by introducing virtual reality and augmented reality technology. This system uses a virtual digital twin human three-dimensional model and superimposes it with an actual simulated human model to form a training environment that combines virtual and real. Trainees can see the three-dimensional anatomical structure and physiological changes of the chest, neck, body, head and neck in the virtual model, and observe the immediate impact of the operation on the virtual model. This combination of virtual and real not only improves the understanding of human anatomy, but also provides real-time physiological feedback, allowing trainees to adjust their operation strategies based on the feedback.

[0051] Compared with traditional methods, the system of the present invention has shown obvious advantages in many aspects. The virtual model can be dynamically adjusted according to the needs of actual operation, thereby providing more accurate feedback on physiological changes, so that trainees can better understand and master the operating principles of cardiopulmonary resuscitation. At the same time, training is no longer limited to physical equipment and venues, which greatly improves the flexibility and efficiency of training. Although this system has high requirements for technical equipment, its advantages in accuracy and adaptability significantly improve the training effect.

Claims

1. A high-quality cardiopulmonary resuscitation and AED self-service training system based on 5G and augmented reality, characterized in that: It includes the following components: 1) A multi-view RGB image and point cloud fusion segmentation module, which uses a mixed reality first aid interaction method of multi-view RGB image and point cloud fusion segmentation to stream or cache the collected standardized cardiopulmonary resuscitation and AED operation segmentation and reconstructed three-dimensional holographic images to a mixed reality display device terminal in real time; 2) A three-dimensional visualization model of the human chest, neck, body and head and neck. The CT tomographic images of the real human chest, neck, body and head and neck are collected, and the CT tomographic images are three-dimensionally reconstructed to obtain a three-dimensional geometric model and a three-dimensional finite element model of the CT tomographic images, and further a three-dimensional visualization model of the digital twin human chest, neck, body and head and neck is obtained; 3) Mixed reality display terminal: Using the mixed reality display terminal, the system will push or cache the collected three-dimensional holographic images of the standardized cardiopulmonary resuscitation and AED operation of the teaching teacher to the MR device in real time. By wearing the MR device, the trainees can see the virtual teaching teacher's operation in the real world. At the same time, during the training process, the system will superimpose the three-dimensional visualization model of the digital twin human chest and neck position head and neck on the real model of the simulated person to display realistic human anatomical structure and real-time virtual physiological changes. During the student training process, the patient's human anatomical structure and real-time virtual physiological changes are transmitted to the display system of the mixed display device through Bluetooth technology, and displayed in real time in the real space. At the same time, multi-channel transmission is performed to meet the needs of multiple people to observe and guide; In addition, during the training process, the instructor guided the operation of the AED and the location of the electrode stickers through a combination of virtual and real methods. The combination of virtual and real methods refers to the combination of holographic projection and a real model of a simulated person. 4) An autonomous cardiopulmonary resuscitation assessment system based on video understanding. The system records and collects video data of the trainees' body shape, movement range, and operation process during training and assessment. It uses machine vision technology to extract each frame of the image, and uses a deep learning algorithm for analysis to calculate and infer the positions and relationships of key points of the human body. The results are displayed to the trainees through an MR device. At the same time, the correct action demonstration is displayed in real time on the MR device, and the areas where the trainees need to improve are marked in a graphical manner to guide and help the trainees correct their incorrect actions.

2. According to claim 1, a high-quality cardiopulmonary resuscitation and AED self-service training system based on 5G and augmented reality is characterized in that: The mixed reality first aid interaction method of multi-view RGB image and point cloud fusion and segmentation refers to, first, based on multi-view cameras and laser radars as visual perception devices, video collection of doctor's standard cardiopulmonary resuscitation and AED operations is performed to obtain image and point cloud data of cardiopulmonary resuscitation and AED operations; further, the camera and laser radar are calibrated with the help of a checkerboard as a calibration object, and the camera intrinsic parameter matrix and the camera and laser radar extrinsic parameter matrix are solved through the collected image and point cloud data, and the conversion relationship from the laser radar coordinate system to the camera coordinate system is constructed to achieve the fusion of image pixels and point cloud data; further, a point cloud segmentation algorithm based on regional growth and clustering is used to perform three-dimensional segmentation on the fused doctor's standard cardiopulmonary resuscitation and AED operation images and point cloud data to obtain a cardiopulmonary resuscitation standardized video three-dimensional model; further, the segmented cardiopulmonary resuscitation standardized video three-dimensional model is reconstructed in three dimensions through VTK technology to generate a three-dimensional visualization scene, which is displayed in a mixed reality or virtual reality display terminal in the form of a video stream.

3. A high-quality cardiopulmonary resuscitation and AED self-service training system based on 5G and augmented reality according to claim 2, characterized in that: The point cloud segmentation algorithm based on region growing and clustering refers to extracting the edge points of the target point cloud as seed points using the edge of the target object in the image, designing a region growing strategy based on the edge properties, and taking the centroid of the target object in the image as the center point of the target point cloud, and calculating the correlation between the target point cloud category and other three-dimensional points, and clustering according to the set correlation criterion.

4. According to claim 1, a high-quality cardiopulmonary resuscitation and AED self-service training system based on 5G and augmented reality is characterized in that: The three-dimensional geometric model is obtained by performing CT scanning on the chest cavity and neck and body position of an adult male, collecting its CT tomographic images, and performing voxel reconstruction on the collected CT data of the chest cavity and neck and body position of the human body; the chest cavity scanning range is the complete chest cavity, from the 1st thoracic vertebra to the 12th thoracic vertebra, including soft tissue, bone tissue, internal organs, and vascular structure; The three-dimensional finite element model is formed by using ANSYS to divide the body mesh, assigning material properties to each unit of the body network model, giving the chest cavity corresponding elastic-plastic bone material properties, and optimizing and intelligently repairing the finite element model according to the heart pump mechanism, left heart pump mechanism, chest pump mechanism and chest frequency resonance mechanism of chest compression during cardiopulmonary resuscitation, and finally forming a three-dimensional finite element model of the human chest cavity and neck, body position and head and neck.

5. A high-quality cardiopulmonary resuscitation and AED self-service training system based on 5G and augmented reality according to claim 4, characterized in that: The optimization and intelligent repair of the finite element model refers to: using finite element analysis software such as ANSYS for automatic meshing. The software will automatically generate meshes based on the model geometry and material properties, and locally refine the area of ​​interest to improve the accuracy of the model; further, accurately assign material properties based on actual biological tissue properties, select appropriate material models to simulate actual biological tissue behavior, use a nonlinear elastoplastic model to simulate the response of bones for the thorax, and use a shape optimization algorithm to improve the model to optimize material distribution and structural design.

6. A high-quality cardiopulmonary resuscitation and AED self-service training system based on 5G and augmented reality according to claim 1, characterized in that: The simulator real model has a realistic human appearance and accurate airway and chest anatomical structure, including basic anatomical structure, which can be displayed in layers. In addition, the simulator has a built-in accurate sensor system, and the collected sensor data covers: CPR operation duration, number of compression interruptions greater than 10 seconds, compression and blowing curve bar graph, real-time tidal volume calculation, real-time compression position judgment, real-time compression frequency and compression depth judgment, real-time compression number, real-time correct and incorrect compression number, real-time ventilation number, real-time correct and incorrect ventilation number, real-time cycle group number recording, total compression time as a proportion of total CPR time, total compression interruption time, blowing time and deflation time, and whether the chest cavity has fully rebounded.

7. A high-quality cardiopulmonary resuscitation and AED self-service training system based on 5G and augmented reality according to claim 1, characterized in that: The video-understanding-based CPR autonomous assessment system installs cameras at appropriate locations on the training and assessment site, which are connected to the system host to record and collect video data of the trainees' body shape, movement range, and operation process during training and assessment. The host uses machine vision technology to extract each frame of the image, and uses a deep learning algorithm to analyze, calculate and infer the positions and relationships of key points of the human body, accurately determine whether each action is in place, and further, make judgments based on the comprehensive action information and the sensor data of the CPR simulator equipment and feed back to the trainees and training physicians.

8. A high-quality cardiopulmonary resuscitation and AED self-service training system based on 5G and augmented reality according to claim 7, characterized in that: The use of deep learning algorithms to analyze, calculate and infer the positions and relationships of key points of the human body refers to using a target detection algorithm to locate and identify the body parts of the trainees, and using a posture estimation model to detect key points of the human body, which include the head, shoulders, elbows, wrists, hips, knees and ankles. Based on the position information of the key points of the human body, the trainees' movement amplitude, posture and operation process are analyzed, and the trainees' actual movements are compared with standard movements in real time.

Citation Information

Patent Citations

  • Cardiopulmonary resuscitation training and evaluation method and device, equipment and storage medium

    CN112749684A

  • Implant surgery training system based on mixed reality technology

    CN113554912A

Cited By

  • Multifunctional field hospital communication method and system

    CN119922737A

  • Intelligent oral examination interaction guiding system and method based on augmented reality

    CN120280113A

  • AED-based cardio-pulmonary resuscitation simulation human control method and system

    CN121281359A

  • Medical operation automatic scoring method based on deep learning

    CN121725521A