CPR training system and method with real-time posture control and tactile feedback

Abstract

The invention discloses a CPR VR training system with real-time posture control and adaptive tactile feedback, and relates to the technical field of three-dimensional vision. Compared with the conventional CPR (cardiopulmonary resuscitation) training system, the system disclosed by the invention has the advantages that the problems of high cost, equipment dependence and insufficient whole body posture perception are solved; based on human body motion capture of a monocular camera, accurate hand tracking is achieved by integrating a VR controller, whole body posture data of a user can be obtained in real time, reliable posture feedback is provided, and the user is helped to conduct self-correction in the training process. A flexible tactile feedback module is designed, when the pressing depth reaches the standard, tactile feedback is provided through a flexible actuator, and the sense of reality of training is enhanced. An artificial respiration detection module is further designed, the air blowing amount of a user is monitored through a microphone and compared with airflow data in a baseline database, the accuracy and standardization of artificial respiration are ensured, and the completeness of CPR training is achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of three-dimensional vision, and relates to an advanced CPR (cardiopulmonary resuscitation) training system with real-time posture control and adaptive tactile feedback. Background Art

[0002] CPR (cardiopulmonary resuscitation) is a key first aid procedure that is essential to improving the survival rate of cardiac arrest patients. Convenient CPR training helps more people master first aid skills, thereby providing effective responses in emergency situations and improving the overall emergency response capabilities and survival rates of society.

[0003] However, existing CPR training methods either rely on professional equipment and a large amount of human resources, or lack practice opportunities and real-time feedback, which limits the mastery and maintenance of skills. Traditional CPR training usually requires dedicated coaches and dedicated training facilities. These requirements not only reduce the flexibility of training and increase costs, but also limit the promotion and popularization of CPR training. Although video CPR training methods are low-cost and allow users to learn at their own time and place, they lack practical immersion, tactile feedback and evaluation mechanisms, which affects the learning effect. Some VR-based CPR training systems use VR technology to enhance immersion and interactivity, allowing learners to perform realistic simulation exercises in a virtual environment. Although this method improves CPR performance in realistic scenarios, it often requires additional equipment such as CPR mannequins and dedicated sensors, which increases costs and reduces promotion and popularity. In addition, VR user reconstruction in these systems is usually limited to hand recognition, which limits real-time whole-body posture perception and cannot self-perceive and correct wrong movements. Therefore, current CPR training systems generally have problems such as high cost, incomplete process and lack of whole-body posture perception, making it difficult to promote to individual users. In order to solve the above problems, the present invention proposes an advanced CPR (cardiopulmonary resuscitation) training system with real-time posture control and adaptive tactile feedback.

[0004] In order to achieve flexible and comprehensive CPR training without relying on CPR models and expensive sensors, we proposed FlexCPR, which features real-time full-body motion capture and realistic tactile feedback of chest compressions. The system captures the user's movements in real time through a monocular camera and extracts whole-body joint posture information based on the SMPL-X model. By integrating a VR controller for hand tracking, FlexCPR achieves high-precision hand tracking and ultimately generates accurate real-time full-body posture data. The system provides reliable posture data support required for CPR training and enables users to perceive the movement of the whole body, helping to more accurately evaluate the training effect and make self-corrections in real time.

[0005] In addition, we developed a tactile feedback module based on array-type flexible sensors to enhance the user experience and improve the accuracy of feedback. When the user's compression depth reaches the standard threshold, the feedback system inflates the flexible actuator to provide tactile feedback. The actuator is made of thermoplastic polyurethane (TPU) material, with stable operating characteristics and skin fit, making the training process more realistic. We also integrated an artificial respiration detection module to monitor the user's breathing through a microphone and evaluate whether the artificial respiration meets the standards, thereby ensuring the integrity and compliance of the entire CPR training process. The key to this detection module is to pre-collect and establish a baseline database of airflow data from the airway sensor of the simulation model. During the training process, the system captures the user's ventilation in real time and compares it with the different levels of airflow data in the baseline database to judge the standard and effect of breathing. Based on the data-driven evaluation method, the system can provide more accurate feedback and improve the user's performance in the artificial respiration stage. Summary of the invention

[0006] The purpose of the present invention is to provide a cardiopulmonary resuscitation training system with real-time posture control and adaptive tactile feedback to solve the problems raised in the background technology:

[0007] Existing CPR training systems are unable to provide accurate whole-body posture perception, real-time feedback, and complete CPR process support under low-cost and portable conditions.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A CPR training system with real-time posture control and tactile feedback, the system comprising a device module, a whole-body motion capture module and a flexible controller control module;

[0010] The device module includes a monocular camera, a head-mounted display, a microphone, a VR handle and a flexible brake, which are used to complete real-time posture tracking and tactile feedback, thereby realizing CPR training and evaluation;

[0011] The whole body motion capture module is matched with a monocular camera to capture hand gestures and body gestures to achieve natural interaction and motion mapping;

[0012] The flexible controller control module is matched with the whole body motion capture module, and outputs the acquired cloud data as a control signal to complete the control of the flexible brake.

[0013] A CPR training method with real-time posture control and tactile feedback comprises the following steps:

[0014] S1. Improve the human body reconstruction model trained with public datasets for complex CPR actions, and transform it together with the hand reconstruction model into real-time model so that it can run at a frame rate of 30fps, and obtain and integrate user posture information;

[0015] S2, transmit the integrated user posture information to the Unity environment through Socket, create a real-time controllable virtual human body in Unity based on the SMPL-X model, and combine the position information of the VR handle to achieve accurate mapping of user actions;

[0016] S3. Place a virtual CPR dummy in the Unity virtual environment. When the user is actually operating, his hand movements control the chest compression of the virtual dummy in real time. The system will detect parameters such as compression depth and frequency, and provide feedback when the user does not meet the qualified standards;

[0017] S4. When the user reaches the pressing depth standard, the flexible actuator control module is triggered. The flexible actuator adopts an array-type TPU actuator prepared by a hot pressing process, which is composed of two layers of TPU film to form n×n airbag units. When the user reaches the pressing depth standard, the control unit triggers the solenoid valve, and compressed air enters the airbag to generate tactile pressure feedback; when the user raises his hand, the air in the airbag is released, and the actuator returns to its original state, realizing force feedback synchronized with the pressing action;

[0018] S5. During the pressing process, the user can view the posture feedback of the virtual digital human in real time through the head-mounted display, perceive their own movements through vision, and correct the pressing posture that does not meet the standards in time;

[0019] S6. After the user completes the chest compression stage, the system enters the artificial respiration stage, uses the microphone worn by the user to detect the amount of air blown by the user, compares it with the preset standard, and determines whether the artificial respiration is qualified;

[0020] S7. Generate a comprehensive training profile of the user based on his / her performance in chest compression and artificial respiration stages, including technical accuracy, compression depth, and ventilation volume, for subsequent analysis and improvement.

[0021] Preferably, the specific process of the real-time transformation in S1 is as follows:

[0022] S101. Introduce multiple independent quantization heads in the quantizer module of the human body reconstruction model, corresponding to different body parts (such as torso, limbs and head). Each quantization head has an independent codebook to map the complete posture into distributed codes, so as to achieve precise control and adjustment of the local posture. In addition, when pre-training the quantizer, a CPR action dataset is added to enhance the model's perception and recognition ability of CPR actions. Specifically, we represent the SMPL body parameters as discrete expressions of different parts, namely:

[0023] θ=(θ 头部 ,θ 四肢 ,θ 躯干 )

[0024] Among them, θ represents the joint of each part in R 6 This process encodes and decodes the posture parameters by using the overall and part-autoencoder architecture and its learnable vocabulary. The overall encoder E maps the input posture parameters to a continuous latent representation space to obtain the overall features:

[0025]

[0026] Among them, d c is the feature dimension of the overall encoder output, and M represents the number of generated tokens. At the same time, for different body parts, the corresponding self-attention encoder is used The joint features of the part are locally encoded. For part p (its corresponding joint index set is denoted by J p ), whose input is:

[0027] x (p) ={θ j |j∈J p}

[0028] The local feature representation is obtained through the encoder:

[0029]

[0030] Among them, d p is the part feature dimension, N p The number of tokens generated for this part. In order to obtain a discretized representation, the overall codebook and the codebook for each part are designed separately. Overall codebook:

[0031]

[0032] The codebook corresponding to each part p is:

[0033]

[0034] For each token z of the overall encoder 整体,i (i=1,…,M), and quantize it to the closest discrete code through nearest neighbor search:

[0035]

[0036] Similarly, for tokens in each part The quantization operation is:

[0037]

[0038] The decoder D is responsible for fusing the quantized overall features and local features to reconstruct the final SMPL posture parameters. At the same time, for each part p, the decoder uses the cross-attention module to fuse the quantized local features Interact with the overall feature:

[0039]

[0040] Among them, Cross Attn(.,.) represents a multi-head cross-attention layer, which is used to extract complementary information between local and global. Subsequently, the decoder integrates the features of each part after cross-attention enhancement back into the overall feature representation through residual connection. After the overall features and the enhanced local features undergo multi-layer convolution and nonlinear transformation, the decoder outputs the final reconstructed SMPL posture parameters:

[0041]

[0042] Then the human body and hand reconstruction models are optimized together through TensorRT to ensure that the model runs in real time at a frame rate of 30fps to ensure the real-time data transmission;

[0043] S102, the overall posture parameters of the human body extracted through the monocular video frame are as follows:

[0044]

[0045] in, represents the global rotation of the pelvic joint, θ 人体 represents the trunk rotation posture parameter, β represents the parameter of individual shape change;

[0046] S103, the hand image contained in the monocular video frame is calculated by the hand parameterization model as follows:

[0047]

[0048]

[0049] in, and denotes the global rotation of the wrist joints of the left and right hands, θ 左手旋转 and θ 右手旋转 They represent the rotation posture parameters of the left and right hands respectively, and β represents the parameter of the hand shape change.

[0050] Preferably, the specific process of Unity information transmission and control in S2 is as follows:

[0051] S201, transmitting the real-time posture data to the SMPL-X virtual parametric human body model in Unity through Socket, and uniformly converting all posture parameters into quaternions to meet the real-time posture control requirements of the Unity engine;

[0052] S202, combining the user's posture information and the position information of the VR handle in the Unity environment to achieve accurate position mapping, so that the user's actual movements are synchronized with the movements of the virtual human body; in each iteration, by calculating the error direction (gradient) between the current and target positions of the wrist, the wrist rotation angle is adjusted, and the hand position error is defined as follows:

[0053]

[0054] in, The hand target position provided by the VR handle, FK(θ i ) 手部 Represents the current hand position calculated via forward kinematics.

[0055] Preferably, the specific process of the user interacting with the virtual dummy in S3 is as follows:

[0056] S301, placing a virtual CPR dummy model in the Unity virtual emergency environment, synchronizing the user's hand movements with the compression points on the virtual dummy's chest to generate real-time feedback of chest compression;

[0057] S302, real-time detection of key parameters such as pressing depth, frequency and hand verticality. When it is detected that the user does not meet the set standards, the system will immediately prompt and guide the user to make adjustments;

[0058] S303. In order to ensure the continuity and accuracy of the pressing, a discrete cosine transform (DCT) is applied to the hand historical position sequence, and the formula is as follows:

[0059]

[0060] Among them, p n is the hand position in the nth frame, C k is the kth DCT coefficient.

[0061] Preferably, the specific process of the flexible actuator control system in S4 is as follows:

[0062] S401, when the user presses to a set depth standard, the control unit receives a pressing depth signal, triggers the opening of the solenoid valve, and allows 12 kPa of compressed air to enter the airbag unit of the flexible actuator through an air inlet channel with a diameter of 4.5 mm;

[0063] S402, the flexible actuator is an array-type TPU actuator prepared by a hot pressing process, consisting of two layers of TPU film with a thickness of 0.1 mm, forming 3×4 airbag units, the airbag diameter is 8 mm, the spacing is 2 mm, the airway width is 4 mm, and the air inlet channel width is 4.5 mm;

[0064] S403, after the compressed air enters the airbag, the airbag expands to generate a tactile pressure feedback of about 1.43N, simulating a real pressing force feeling, so that the user can get a force feedback experience during the pressing process;

[0065] S404. When the user raises his hand, the control unit closes the solenoid valve, the air in the airbag is discharged through the release channel, and the actuator returns to its original state, thereby achieving force feedback synchronized with the user's pressing action.

[0066] Preferably, the specific process of user self-action perception in S5 is as follows:

[0067] When the user performs chest compressions, the system detects the relative distance between the user's hands; when the distance between the two hands is less than the preset threshold (indicating the start of the compression action), the virtual digital human's hands are automatically set to the standard chest compression gesture, that is, the hands are crossed and stacked on the chest to ensure the standardization of the compression posture; during this process, the virtual digital human's hands no longer track the user's subtle hand movements in real time, but present a standardized compression posture. When the user stops pressing and the distance between the two hands exceeds the threshold, the system resumes the receiving mode of the real-time data stream; at this time, the virtual digital human's hand posture is updated in real time according to the user's actual hand position and movement, allowing the user to operate freely.

[0068] Preferably, the artificial respiration module in S6 implements the specific process as follows:

[0069] S601. After the user completes the chest compression stage, the system automatically switches to the artificial respiration stage and provides gesture guidance in the virtual environment, prompting the user to perform the correct artificial respiration gesture, that is, pinching the dummy's nose with one hand and lifting the chin with the other hand;

[0070] S602, the Unity engine calls the first-person camera of the VR head mounted display, so that the user can view his / her hand movements in real time to ensure that the standard artificial respiration gestures are completed according to the instructions;

[0071] S603: When the user completes the gesture guidance and the hand position meets the artificial respiration posture standard, the system enters the blowing detection stage. Since the user wears a Bluetooth microphone, the Unity end receives the microphone's audio data in real time;

[0072] S604, the system pre-processes the received audio data, including noise reduction and filtering, to improve signal quality, and then calculates the Mel Frequency Cepstral Coefficient (MFCC, n=13) for each audio sample to extract audio features for analysis;

[0073] S605. Using a pre-trained support vector machine (SVM) multi-classification model, the extracted audio features are classified in real time, and the user's blowing volume is divided into six levels, where a higher level indicates a larger blowing volume.

[0074] S606, the system compares the user's blowing volume level with the preset standard to determine whether the artificial respiration meets the international CPR standard, that is, each blowing lasts for more than one second and the tidal volume is between 500 and 600 mL;

[0075] S607. If it is detected that the blowing volume or duration of the user does not meet the standard, the system provides real-time feedback in the virtual environment, prompting the user to adjust the blowing force and time until the qualified standard is met.

[0076] Preferably, the specific process of generating the user portrait in S7 is as follows:

[0077] The calculation method of pressing frequency and pressing depth in the user portrait indicators is as follows: the system calculates the key indicators during the pressing process by monitoring the position changes of the user's hand in real time. The displacement of the hand in the vertical direction (Y axis) is used to detect the start and end of each press, and the pressing depth, duration and number of presses are calculated.

[0078] Compared with the prior art, the present invention provides a CPR training system and method with real-time posture control and adaptive tactile feedback, which has the following beneficial effects:

[0079] (1) The present invention uses an improved full-body human body reconstruction model to capture the user's full-body dynamic posture data in real time and provide accurate tactile feedback, so that the user can obtain accurate feedback on the whole-body posture during CPR training. The system uses a monocular camera input to provide consistent, real-time and highly accurate feedback when the user performs actions such as compression and artificial respiration, ensuring real-time interactivity and realism of the training process. The constructed system is denoted as FlexCPR. By designing a tactile feedback module, FlexCPR can provide adaptive tactile feedback based on the user's compression depth, helping the user to perceive the intensity of pressure during the compression process, thereby improving the accuracy of the compression; at the same time, the system's integrated artificial respiration detection module can monitor the user's ventilation volume in real time to ensure the standardization and integrity of the entire CPR process.

[0080] (2) The present invention designs a multimodal paradigm for real-time whole-body motion capture and feedback, achieves high-precision hand tracking through an integrated VR controller, and obtains whole-body joint posture information based on the SMPL-X model, so that the system can provide accurate posture detection and feedback under low-cost and portable conditions. In addition, the present invention also designs an adaptive feedback module that can provide instant prompts when the user performs compressions and artificial respiration, helping the user to correct the action in real time. By using a flexible actuator designed with TPU material, the tactile feedback module is both portable and economical while maintaining high sensitivity; at the same time, the human motion prior is trained through a deep learning model, which effectively enhances the accuracy and smoothness of motion capture, achieves the realism and natural transition of the action, and enables users to perform high-quality CPR self-training in a VR environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 This is a framework diagram of the immersive cardiopulmonary resuscitation training system proposed in Example 1 of the present invention;

[0082] Figure 2 It is a framework diagram of the improved human body reconstruction quantizer proposed in Embodiment 1 of the present invention;

[0083] Figure 3 is a qualitative diagram of the improved human body reconstruction quantizer proposed in Example 1 of the present invention;

[0084] Figure 4 This is a component diagram of the training system proposed in Embodiment 2 of the present invention;

[0085] Figure 5 The emergency scenario structure diagram proposed in Embodiment 2 of the present invention;

[0086] Figure 6 This is a structural diagram of the flexible actuator proposed in Example 2 of the present invention. DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0088] Embodiment 1:

[0089] See also Figure 1 The present invention proposes a CPR training method with real-time posture control and tactile feedback, comprising the following steps:

[0090] S1. Please refer to Figure 2 , the human body reconstruction model trained by public datasets is improved for complex CPR actions, and it is transformed into real-time model together with the hand reconstruction model, such as Figure 3 As shown in the figure, the improved human body reconstruction network is more robust for CPR action reconstruction. Finally, it can run at a frame rate of 30fps, and obtain and integrate user posture information through this. The specific implementation process is as follows:

[0091] S101. Introduce multiple independent quantization heads in the quantizer module of the human body reconstruction model, corresponding to different body parts (such as torso, limbs and head). Each quantization head has an independent codebook to map the complete posture into distributed codes, so as to achieve precise control and adjustment of the local posture. In addition, when pre-training the quantizer, a CPR action dataset is added to enhance the model's perception and recognition ability of CPR actions. Specifically, we represent the SMPL body parameters as discrete expressions of different parts, namely:

[0092] θ=(θ 头部 ,θ 四肢 ,θ 躯干 )

[0093] Where θ represents the joint of each part in R 6 This process encodes and decodes the posture parameters by using the overall and part-autoencoder architecture and its learnable vocabulary. The overall encoder E maps the input posture parameters to a continuous latent representation space to obtain the overall features:

[0094]

[0095] Among them, d c is the feature dimension of the overall encoder output, and M represents the number of generated tokens. At the same time, for different body parts, the corresponding self-attention encoder is used The joint features of the part are locally encoded. For part p (its corresponding joint index set is denoted by J p ), whose input is:

[0096] x (p) ={θ j |j∈J p}

[0097] The local feature representation is obtained through the encoder:

[0098]

[0099] Among them, d p is the part feature dimension, N p The number of tokens generated for this part. In order to obtain a discretized representation, the overall codebook and the codebook for each part are designed separately. Overall codebook:

[0100]

[0101] The codebook corresponding to each part p is:

[0102]

[0103] For each token z of the overall encoder 整体,i (i=1,…,M), and quantize it to the closest discrete code through nearest neighbor search:

[0104]

[0105] Similarly, for tokens in each part The quantization operation is:

[0106]

[0107] The decoder D is responsible for fusing the quantized overall features and local features to reconstruct the final SMPL posture parameters. At the same time, for each part p, the decoder uses the cross-attention module to fuse the quantized local features Interact with the overall feature:

[0108]

[0109] Among them, Cross Attn(.,.) represents a multi-head cross-attention layer, which is used to extract complementary information between local and global. Subsequently, the decoder integrates the features of each part after cross-attention enhancement back into the overall feature representation through residual connection. After the overall features and the enhanced local features undergo multi-layer convolution and nonlinear transformation, the decoder outputs the final reconstructed SMPL posture parameters:

[0110]

[0111] The human body and hand reconstruction models are optimized through TensorRT to ensure that the model runs in real time at a frame rate of 30fps, thereby ensuring the real-time nature of data transmission. The real-time network receives the monocular video frames with a resolution of (256, 256) after cropping and extracts the hand and human body parameters.

[0112] S102, the overall posture parameters of the human body extracted through the monocular video frame are as follows:

[0113]

[0114] in, represents the global rotation of the pelvic joint, θ 人体 represents the trunk rotation posture parameter, and β represents the parameter of individual shape variation.

[0115] S103, the hand image contained in the monocular video frame is calculated by the hand parameterization model as follows:

[0116]

[0117] in, and denotes the global rotation of the wrist joints of the left and right hands, θ 左手旋转 and θ 右手旋转 They represent the rotation posture parameters of the left and right hands respectively, and β represents the parameter of the hand shape change.

[0118] S2. The integrated user posture information is transmitted to the Unity environment through Socket. A real-time controllable virtual human body is created in Unity based on the SMPL-X model. The position information of the VR handle is combined to achieve accurate mapping of user actions. The specific implementation process is as follows:

[0119] S201. The real-time posture data is transmitted to the SMPL-X virtual parametric human body model in Unity through Socket, and all posture parameters are uniformly converted into quaternions to meet the real-time posture control requirements of the Unity engine.

[0120] S202. In the Unity environment, the user's posture information and the position information of the VR handle are combined to achieve accurate position mapping, so that the user's actual movements are synchronized with the movements of the virtual human body. In each iteration, the wrist rotation angle is adjusted by calculating the error direction (gradient) between the current and target wrist positions. The hand position error is defined as follows:

[0121]

[0122] in, The hand target position provided by the VR handle, FK(θ i ) 手部 Represents the current hand position calculated via forward kinematics.

[0123] S3. Place a virtual CPR dummy in the Unity virtual environment. When the user is actually operating, his hand movements control the chest compression of the virtual dummy in real time. The system will detect parameters such as compression depth and frequency, and provide feedback when the user does not meet the qualified standards. The specific interaction is implemented as follows:

[0124] S301. A virtual CPR dummy model is placed in the Unity virtual emergency environment. The user's hand movements are synchronized with the compression points on the virtual dummy's chest to generate real-time feedback of chest compression.

[0125] S302, real-time detection of key parameters such as pressing depth, frequency and hand verticality. When it is detected that the user does not meet the set standards, the system will immediately prompt and guide the user to make adjustments.

[0126] S303. In order to ensure the continuity and accuracy of the pressing, a discrete cosine transform (DCT) is applied to the hand historical position sequence, and the formula is as follows:

[0127]

[0128] Among them, p n is the hand position in the nth frame, C k is the kth DCT coefficient.

[0129] S4. When the user reaches the pressing depth standard, the flexible actuator control module is triggered. The flexible actuator uses an array-type TPU actuator prepared by a hot pressing process, which consists of two layers of 0.1mm thick TPU film to form 3×4 airbag units. When the user reaches the pressing depth standard, the control unit triggers the solenoid valve, and 12kPa of compressed air enters the airbag, generating about 1.43N of tactile pressure feedback; when the user raises his hand, the air in the airbag is released, and the actuator returns to its original state, realizing force feedback synchronized with the pressing action. The specific design process is as follows:

[0130] S401. When the user presses to a set depth standard, the control unit receives a pressing depth signal, triggers the opening of the solenoid valve, and allows 12 kPa of compressed air to enter the airbag unit of the flexible actuator through an air inlet channel with a diameter of 4.5 mm.

[0131] S402, the flexible actuator is an array-type TPU actuator prepared by a hot pressing process, which consists of two layers of TPU film with a thickness of 0.1 mm, forming 3×4 airbag units. The airbag diameter is 8 mm, the spacing is 2 mm, the airway width is 4 mm, and the air inlet channel width is 4.5 mm.

[0132] S403. After the compressed air enters the airbag, the airbag expands to generate a tactile pressure feedback of about 1.43N, simulating a real pressing pressure feeling, so that the user can get a force feedback experience during the pressing process.

[0133] S404. When the user raises his hand, the control unit closes the solenoid valve, the air in the airbag is discharged through the release channel, and the actuator returns to its original state, thereby achieving force feedback synchronized with the user's pressing action.

[0134] S5. During the pressing process, the user views the posture feedback of the virtual digital human in real time through the head-mounted display, perceives his own movements through vision, and corrects the pressing posture that does not meet the standards in time. The specific implementation process is as follows:

[0135] When the user performs chest compressions, the system detects the relative distance between the user's hands. When the distance between the two hands is less than the preset threshold (indicating the start of the compression action), the virtual digital human's hands are automatically set to the standard chest compression gesture, that is, the hands are crossed and stacked on the chest to ensure the standardization of the compression posture. During this process, the virtual digital human's hands no longer track the user's subtle hand movements in real time, but present a standardized compression posture. When the user stops pressing and the distance between the two hands exceeds the threshold, the system resumes the real-time data stream receiving mode. At this time, the virtual digital human's hand posture is updated in real time according to the user's actual hand position and movement, allowing the user to operate freely.

[0136] S6. After the user completes the chest compression stage, the system enters the artificial respiration stage, uses the microphone worn by the user to detect the amount of air blown by the user, and compares it with the preset standard to determine whether the artificial respiration is qualified. The specific implementation process is as follows:

[0137] S601. After the user completes the chest compression stage, the system automatically switches to the artificial respiration stage and provides gesture guidance in the virtual environment, prompting the user to perform the correct artificial respiration gesture, that is, pinching the dummy's nose with one hand and lifting the chin with the other hand.

[0138] S602. The Unity engine calls the first-person camera of the VR head mounted display to enable the user to view his or her hand movements in real time to ensure that the standard artificial respiration gestures are completed according to the instructions.

[0139] S603: When the user completes the gesture guidance and the hand position meets the artificial respiration posture standard, the system enters the blowing detection stage. Since the user wears a Bluetooth microphone, the Unity end receives the audio data of the microphone in real time.

[0140] S604: The system preprocesses the received audio data, including noise reduction and filtering, to improve signal quality, and then calculates the Mel Frequency Cepstral Coefficient (MFCC, n=13) for each audio sample to extract audio features for analysis.

[0141] S605. Using a pre-trained support vector machine (SVM) multi-classification model, the extracted audio features are classified in real time, and the user's blowing volume is divided into six levels, where a higher level indicates a larger blowing volume.

[0142] S606. The system compares the user's blowing volume level with the preset standard to determine whether the user's artificial respiration meets the international CPR standard, that is, each blowing lasts for more than one second and the tidal volume is between 500 and 600 mL.

[0143] S607. If it is detected that the blowing volume or duration of the user does not meet the standard, the system provides real-time feedback in the virtual environment, prompting the user to adjust the blowing force and time until the qualified standard is met.

[0144] S7. Generate a comprehensive training profile of the user based on the user's performance in the chest compression and artificial respiration stages, including the performance of technical accuracy, compression depth, and blowing volume, for subsequent analysis and improvement. The specific implementation method is as follows:

[0145] The calculation method of pressing frequency and pressing depth in the user portrait indicators is as follows: the system calculates the key indicators during the pressing process by monitoring the position changes of the user's hand in real time. The displacement of the hand in the vertical direction (Y axis) is used to detect the start and end of each press, and the pressing depth, duration and number of presses are calculated.

[0146] Embodiment 2:

[0147] See also Figure 4-6 , based on Example 1 but different in that the specific implementation process of the CPR training system with real-time posture control and tactile feedback used in the above training method is as follows:

[0148] 1. Overview of cardiopulmonary resuscitation training system:

[0149] This system is an innovative immersive VR platform that provides comprehensive CPR training through real-time posture tracking and tactile feedback. FlexCPR integrates a monocular camera, head-mounted display (HMD), flexible sensors and optional VR controllers to bridge the gap between theoretical knowledge and practical skill acquisition. The system supports both training and assessment modes to accommodate users of different levels.

[0150] In training mode, refer to Figure 4As shown in the figure, the system provides step-by-step guidance on chest compressions and artificial respiration through embedded knowledge cards. These cards provide basic information and practical tips for mastering CPR techniques. After the teaching phase, the user begins actual training. Real-time body posture tracking allows the user's virtual avatar to perform chest compressions on a virtual mannequin, while the system provides instant feedback on key indicators: arm verticality, compression depth, compression frequency, and total number of times. This feedback enables users to adjust their movements in real time to comply with best CPR practices. When incorrect posture or technique is detected, the system provides corrective guidance.

[0151] The system also includes a soft pneumatic actuator that provides realistic tactile feedback. The actuator is made of a thermoplastic polyurethane (TPU) layer, and when the user's pressing depth reaches the standard threshold of 6cm, the feedback system inflates the soft actuator to provide tactile feedback, simulating a realistic sense of resistance.

[0152] In the second stage of training, the user participates in the artificial respiration module, and the microphone monitors the user's exhalation to assess whether the artificial respiration meets the required standards, ensuring the integrity and compliance of the entire CPR training process. If the user's ventilation volume is lower than the recommended level, the system will provide corrective feedback.

[0153] To simulate real CPR rescue scenarios and immerse participants in realistic environments, high-fidelity public scenes were created using Unity (version 6000.0.0f1.c2), replicating real-world settings such as subway stations and hospital wards, as referenced in the paper. Figure 5 In the Flex CPR training system, participants complete the entire CPR training process in a simulated subway environment.

[0154] (ii) Real-time posture tracking:

[0155] To enable natural interaction and real-time motion mapping in a CPR training system, this system implements real-time pose control to estimate the 3D human body (torso and hands) from a monocular video stream. Instead of relying on traditional full-body estimation models that typically suffer from poor accuracy in hand pose estimation, we use a part-level joint estimation approach to improve accuracy.

[0156] The method uses reconstructed models of the human body and hands trained on a large-scale human dataset to generate parameters for the SMPL model and MANO model, respectively. By estimating the torso and hands separately, our method takes advantage of dedicated models for different body parts and produces accurate hand and body pose estimates that are critical for the CPR task. These parameters are then combined in Unity using the SMPL-X model to unify the 3D body and hand poses.

[0157] (III) Tactile feedback module:

[0158] Although tactile feedback cannot convey as much information as visual feedback, it requires less user attention and is often more effective than auditory feedback in many scenarios. In order to enhance user immersion and training effect in chest compression training, this system has developed a brake actuator system controlled by the compression action to provide realistic force feedback. The flexible pneumatic actuator can be customized in shape and structure according to application requirements to provide precise tactile pressure feedback.

[0159] Actuators based on thermoplastic polyurethane (TPU) have skin-like elastomeric properties (Young's modulus of TPU actuators is about 28 MPa, similar to that of human skin), which allows TPU actuators to be used for close, large-area contact with the bottom of the palm to provide uniform pressure feedback. They are widely used in wearable tactile devices. Therefore, we developed a 10 cm × 10 cm array-type TPU actuator fabricated by a hot pressing process, consisting of two layers of TPU film. The shape of TPU can be adjusted by simple heating and pressurization, and then cooled to maintain its shape.

[0160] Specifically, a patterned aluminum alloy plate was heated to 120°C, two layers of TPU films were placed on the plate, and heat pressed for 120 seconds until the films were tightly bonded. The grooves and holes on the aluminum plate did not exert pressure on the TPU film, forming airways and airbag chambers. The TPU actuators prepared by this process showed high elasticity, fatigue resistance, softness, and excellent shape recovery properties. As shown in reference Figure 6 As shown, the TPU actuator has a specific geometry, including 3×4 airbags located in different areas, the airbag diameter is d=8mm, the airbag spacing is l=2mm, the airway width is w=4mm, the air inlet channel width is l=4.5mm, and the thickness of the single-layer TPU film is t=0.1mm. In order to optimize the performance of the array-type TPU actuator in CPR applications, dynamic response tests were performed to verify whether the refresh rate of the control unit meets the required CPR compression frequency and whether the force provided by the airbag meets the feedback requirements. Using a digital bridge equipped with a pressure resistor sensor, a pressure feedback of approximately 1.43±0.05N was measured during activation.

[0161] As the user performs chest compressions, the avatar's hands, controlled by the user's gestures, interact with the virtual mannequin's chest. During this interaction, the system sends signals to the STM32 microcontroller based on the depth of compression. As the user presses down, the microcontroller triggers a solenoid valve, allowing air from a 12kPa compressor to flow into the actuator's airway through a 4.5mm diameter inlet channel. The tube is fixed to the actuator and sealed with an adhesive, and when air enters the actuator, the restrictive hole in the channel expands, producing pressure tactile feedback.

[0162] When the user lifts their hand, the air is released from the airbag and the actuator returns to its original state. This cyclic inflation and deflation, synchronized with the user's pressing action, creates a pressure-based tactile feedback mechanism. Compared to traditional vibrotactile actuators, this approach provides a more comfortable experience and effectively prevents sensory adaptation that can occur with prolonged vibration exposure.

[0163] (IV) Artificial respiration module:

[0164] The artificial respiration module provides real-time assessment of the participant's ventilation during the training and testing phases. According to international CPR standards, each breath must last for more than one second and provide a tidal volume of 500 to 600 mL. To ensure accurate assessment, the module pre-collects data using the airway sensor of the CPR mannequin and creates a baseline database. A support vector machine (SVM) multi-classification model is used to achieve real-time assessment of ventilation.

[0165] To measure ventilation without a human body model, a microphone was used to capture the user's breathing sounds, allowing the system to estimate ventilation directly. This configuration enabled the system to classify ventilation into six different levels, with higher levels corresponding to greater ventilation. In addition, audio data associated with each breath was recorded using the microphone, providing data to assist in the analysis. A labeled audio dataset was constructed by synchronizing the audio data with the ventilation levels detected by the sensor. The system calculated the Mel-frequency cepstral coefficients (MFCCs, n=13) for each audio sample and used these features to train a SVM multi-classification model. The model was then deployed directly in the Unity environment, enhancing performance and processing efficiency by reducing data communication latency.

[0166] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and improved concepts of the present invention within the technical scope disclosed by the present invention, and they should be covered by the protection scope of the present invention.

Claims

1. A CPR training system with real-time posture control and tactile feedback, characterized in that: The system includes an equipment module, a whole-body motion capture module and a flexible controller control module; The device module includes a monocular camera, a head-mounted display, a microphone, a VR handle and a flexible brake, which are used to complete real-time posture tracking and tactile feedback, thereby realizing CPR training and evaluation; The whole body motion capture module is matched with a monocular camera to capture hand gestures and body gestures to achieve natural interaction and motion mapping; The flexible controller control module is matched with the whole body motion capture module, and outputs the acquired cloud data as a control signal to complete the control of the flexible brake.

2. The CPR training method with real-time posture control and tactile feedback applied by the system as claimed in claim 1, characterized in that: The steps include: S1. Improve the human body reconstruction model trained with public datasets for complex CPR movements, and transform it together with the hand reconstruction model into real-time model so that it can run at a frame rate of 30fps, and obtain and integrate user posture information based on it; S2, transmit the integrated user posture information to the Unity environment through Socket, create a real-time controllable virtual human body in Unity based on the SMPL-X model, and combine the position information of the VR handle to achieve accurate mapping of user actions; S3. Place a virtual CPR dummy in the Unity virtual environment. When the user is actually operating, his hand movements control the chest compression of the virtual dummy in real time. The system detects the compression depth and frequency parameters and provides feedback when the user does not meet the qualified standards. S4. When the user reaches the pressing depth standard, the flexible actuator control module is triggered; the flexible actuator adopts an array-type TPU actuator prepared by a hot pressing process, which is composed of two layers of TPU film to form n×n airbag units; when the user reaches the pressing depth standard, the control unit triggers the solenoid valve, and compressed air enters the airbag to generate tactile pressure feedback; when the user raises his hand, the air in the airbag is released, and the actuator returns to its original state, realizing force feedback synchronized with the pressing action; S5. During the pressing process, the user can view the posture feedback of the virtual digital human in real time through the head-mounted display, perceive their own movements through vision, and correct the pressing posture that does not meet the standards in time; S6. After the user completes the chest compression stage, the system enters the artificial respiration stage, uses the microphone worn by the user to detect the amount of air blown by the user, compares it with the preset standard, and determines whether the artificial respiration is qualified; S7. Generate a comprehensive training profile of the user based on his / her performance in chest compression and artificial respiration stages, including technical accuracy, compression depth, and ventilation volume, for subsequent analysis and improvement.

3. A CPR training method with real-time posture control and tactile feedback according to claim 1, characterized in that: The S1 specifically includes the following contents: S101. Introduce multiple independent quantization heads in the quantizer module of the human body reconstruction model, corresponding to different body parts respectively; each quantization head has an independent codebook to map the complete posture into distributed codes, so as to achieve accurate control and adjustment of the local posture; in addition, when pre-training the quantizer, add a CPR action data set to enhance the model's perception and recognition ability of the CPR action; specifically, express the SMPL body parameters as discrete expressions of different parts, namely: θ=(θ 头部 ,i 四肢 ,i 躯干 ) Among them, θ represents the joint of each part in R 6 The process encodes and decodes the posture parameters by using the overall and part-autoencoder architecture and its learnable vocabulary; the overall encoder E maps the input posture parameters to a continuous latent representation space to obtain the overall features: Among them, d c is the feature dimension of the overall encoder output, and M represents the number of generated tokens; For different body parts, use the corresponding self-attention encoder The joint features of the part are locally encoded; for part p, the input is: x (p) ={θ j |j∈J p } Among them, J p Indicates the joint index set corresponding to part p; The local feature representation is obtained through the encoder: Among them, d p is the part feature dimension, N p The number of tokens generated for the part; to obtain a discretized representation, the overall codebook and the codebook for each part are designed separately; The overall codebook is: The codebook corresponding to each part p is: For each token z of the overall encoder 整体,i (i=1,…,M), and quantize it to the closest discrete code through nearest neighbor search: Similarly, for each part The quantization operation is: The decoder D is used to fuse the quantized overall features and local features to reconstruct the final SMPL posture parameters; for each part p, the decoder uses the cross-attention module to fuse the quantized local features Interact with the overall feature: Among them, Cross Attn(.,.) represents a multi-head cross-attention layer, which is used to extract complementary information between local and global. Subsequently, the decoder integrates the features of each part enhanced by cross-attention back to the overall feature representation through residual connection. After the overall features and the enhanced local features are subjected to multi-layer convolution and nonlinear transformation, the decoder outputs the final reconstructed SMPL posture parameters: Then the human body and hand reconstruction models are optimized together through TensorRT to ensure that the model runs in real time at a frame rate of 30fps to ensure the real-time data transmission; S102, the overall posture parameters of the human body extracted through the monocular video frame are as follows: in, represents the global rotation of the pelvic joint; θ 人体 represents the trunk rotation posture parameter; β represents the parameter of individual shape change; S103, the hand image contained in the monocular video frame is calculated by the hand parameterization model as follows: in, and Respectively represent the global rotation of the wrist joints of the left and right hands; θ 左手旋转 and θ 右手旋转 They represent the rotation posture parameters of the left and right hands respectively; β represents the parameter of the hand shape change.

4. A CPR training method with real-time posture control and tactile feedback according to claim 1, characterized in that: The S2 specifically includes the following contents: S201, transmitting the real-time posture data to the SMPL-X virtual parametric human body model in Unity through Socket, and uniformly converting all posture parameters into quaternions to meet the real-time posture control requirements of the Unity engine; S202, combining the user's posture information and the position information of the VR handle in the Unity environment to achieve accurate position mapping, so that the user's actual movements are synchronized with the movements of the virtual human body; in each iteration, by calculating the error direction between the current and target positions of the wrist, the wrist rotation angle is adjusted, and the hand position error is defined as follows: in, Indicates the hand target position provided by the VR handle; FK(θ i ) 手部 Represents the current hand position calculated via forward kinematics.

5. A CPR training method with real-time posture control and tactile feedback according to claim 1, characterized in that: The S3 specifically includes the following contents: S301, placing a virtual CPR dummy model in the Unity virtual emergency environment, synchronizing the user's hand movements with the compression points on the virtual dummy's chest to generate real-time feedback of chest compression; S302, real-time detection of the pressing depth, frequency and hand verticality parameters. When it is detected that the user does not meet the set standards, the system immediately prompts and guides the user to make adjustments; S303, applying discrete cosine transform to the hand historical position sequence to ensure the continuity and accuracy of pressing, the formula is as follows: Among them, p n is the hand position in the nth frame, C k is the kth DCT coefficient.

6. A CPR training method with real-time posture control and tactile feedback according to claim 5, characterized in that: The S4 specifically includes the following contents: S401, when the user presses to a set depth standard, the control unit receives a pressing depth signal, triggers the opening of the solenoid valve, and allows compressed air to enter the airbag unit of the flexible actuator through the air inlet channel; S402, the flexible actuator is an array-type TPU actuator prepared by a hot pressing process, consisting of two layers of TPU film with a thickness of 0.05 to 0.15 mm, forming 3×4 airbag units, the airbag diameter is 6 to 10 mm, the spacing is 1 to 3 mm, the airway width is 3 to 5 mm, and the air inlet channel width is 4 to 5 mm; S403, after the compressed air enters the airbag, the airbag expands to generate 1.4-1.5N tactile pressure feedback, simulating a real pressing force feeling, so that the user can get a force feedback experience during the pressing process; S404. When the user raises his hand, the control unit closes the solenoid valve, the air in the airbag is discharged through the release channel, and the actuator returns to its original state, thereby achieving force feedback synchronized with the user's pressing action.

7. A CPR training method with real-time posture control and tactile feedback according to claim 6, characterized in that: The S5 specifically includes the following contents: When the user performs chest compressions, the system detects the relative distance between the user's hands. When the distance between the two hands is less than the preset threshold, the virtual digital human's hands are automatically set to the standard chest compression gesture to ensure the standardization of the compression posture. During this process, the virtual digital human's hands no longer track the user's subtle hand movements in real time, but present a standardized compression posture. When the user stops pressing and the distance between the two hands exceeds the threshold, the system resumes the real-time data stream receiving mode; at this time, the hand posture of the virtual digital human is updated in real time according to the user's actual hand position and movement, allowing the user to operate freely.

8. A CPR training method with real-time posture control and tactile feedback according to claim 1, characterized in that The S6 specifically includes the following contents: S601, after the user completes the chest compression stage, the system automatically switches to the artificial respiration stage and provides gesture guidance in the virtual environment to prompt the user to perform the correct artificial respiration gesture; S602, the Unity engine calls the first-person camera of the VR head mounted display to enable the user to view his or her hand movements in real time to ensure that the standard artificial respiration gestures are completed according to the instructions; S603: When the user completes the gesture guidance and the hand position meets the artificial respiration posture standard, the system enters the blowing detection stage; the Unity end receives audio data in real time through the Bluetooth microphone worn by the user; S604, the system pre-processes the received audio data, including noise reduction and filtering, to improve signal quality, and then calculates the Mel frequency cepstral coefficient for each audio sample to extract audio features for analysis; S605, using a pre-trained support vector machine multi-classification model, the extracted audio features are classified in real time, and the user's blowing volume is divided into six levels, where a higher level indicates a larger blowing volume; S606, the system compares the user's blowing volume level with the preset standard to determine whether the artificial respiration meets the international CPR standard; S607. If it is detected that the blowing volume or duration of the user does not meet the standard, the system provides real-time feedback in the virtual environment, prompting the user to adjust the blowing force and time until the qualified standard is met.

9. A CPR training method with real-time posture control and tactile feedback according to claim 1, characterized in that: The S7 specifically includes the following contents: The calculation method for pressing frequency and pressing depth in the user portrait indicators is as follows: the system monitors the changes in the position of the user's hand in real time and calculates the key indicators in the pressing process; it uses the vertical displacement of the hand to detect the start and end of each press, and calculates the pressing depth, duration and number of presses.

Images