A ubiquitous training method and system for intelligent cardiopulmonary resuscitation

Abstract

The present invention provides a ubiquitous training method and system for intelligent cardiopulmonary resuscitation, including the steps of deploying monitoring equipment in the training scene of trainees, key limb joints, and the chest area of ​​training dummies to achieve synchronous acquisition of multi-source video stream data; storing and constructing comprehensive feature vectors; constructing a CNN model through expert annotation and standard training feature data extraction; and generating corresponding training result feedback. The present invention combines MSE threshold detection and constraint mechanisms to quickly identify abnormal movements of trainees. The constraint mechanism is introduced in the data preprocessing stage, and a simple threshold comparison is used to quickly filter dangerous or invalid data. A physical reasonable range is set for each training feature data. A compliance label is generated through a binary constraint function and a weighted comprehensive score, avoiding subsequent complex and time-consuming CNN model inference, thereby reducing the computational load and improving the accuracy of trainee assessment.

Description

Technical Field

[0001] The present invention relates to the technical field of cardiopulmonary resuscitation training, and in particular to a cardiopulmonary resuscitation intelligent companion ubiquitous training method and system. Background Art

[0002] There are many significant defects and shortcomings in traditional cardiopulmonary resuscitation (CPR) training methods.

[0003] First, the lack of or inadequate feedback mechanisms is a serious problem. In practice, trainees often only understand whether their movements are correct after training, through subjective evaluations or simple assessments from their coaches. Without timely feedback during training, incorrect movements can be repeatedly practiced, forming muscle memory and making corrections more difficult.

[0004] Secondly, the subjectivity of evaluation criteria is also a prominent issue. Traditional evaluation methods rely primarily on the coach's visual observation and empirical judgment. This approach is not only susceptible to human influence, resulting in inaccurate evaluation results, but also fails to accurately quantify the details and subtle differences in movements, making it difficult to provide targeted improvement suggestions. Furthermore, data integrity and accuracy are difficult to guarantee. During training, data collection and recording are often incomplete and inaccurate, prone to omissions and errors. This not only hinders a comprehensive understanding and analysis of the training process, but also makes data-based evaluation of training effectiveness and method optimization difficult. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a cardiopulmonary resuscitation intelligent companion ubiquitous training method and system to solve the problems raised in the above background technology.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solution: a cardiopulmonary resuscitation intelligent companion ubiquitous training method, comprising the steps of:

[0007] Deploy monitoring equipment in the training scenes, key limb joints, and chest area of ​​the training dummy to achieve simultaneous collection of multi-source video stream data;

[0008] The inter-frame difference threshold recognition method is constructed to obtain representative frame data representing the body movements of the trainees. The clocks of all monitoring devices corresponding to the representative frame data are synchronized through the NTP protocol to establish a consistent time series and construct a comprehensive feature vector F representing the training feature data of the trainees. j =[θ j ,p j ,v j ,d j ],θ j is the limb joint angle, p jis the limb joint position, v j is the velocity of the limb joint, d j is the pressing force;

[0009] Through expert annotation and standard training feature data extraction, the standard action spatiotemporal feature matrix Fstd is generated and the CNN model is constructed;

[0010] The training feature data is evaluated for compliance by building a constraint mechanism to filter invalid training data. If the data is compliant, it is input into the CNN model to extract the feature matrix F. The feature matrix F is compared with the standard action feature matrix Fstd, and the difference d and similarity C between the two are obtained by calculating the weighted Euclidean distance and cosine similarity. sim , generating corresponding training result feedback; the steps of building the constraint mechanism are as follows:

[0011] For each training feature data Xi, calculate whether it exceeds the reasonable range and generate the binary constraint function value fi(Xi). ;

[0012] Assign a weight Wi to each training feature data Xi according to the importance of the training results, and calculate the comprehensive score of the physical constraint ;

[0013] f ph Compare with the preset threshold T to generate a compliance label L ph , if f ph >T then let L ph =1, indicating that the data is compliant and enters the CNN model processing flow; otherwise, let L ph =0, directly trigger feedback, output the over-limit parameter list Ep={Xi|Xi <X imin or Xi>X imax}, no subsequent CNN model processing is performed.

[0014] As a second aspect of the present invention, a cardiopulmonary resuscitation smart companion ubiquitous training system is proposed, including a memory and a processor, wherein the memory includes a cardiopulmonary resuscitation smart companion ubiquitous training program, and when the cardiopulmonary resuscitation smart companion ubiquitous training program is executed by the processor, the above-mentioned cardiopulmonary resuscitation smart companion ubiquitous training method is implemented.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] 1. To address the problem that traditional methods cannot detect abnormal movements in real time, the present invention combines MSE threshold detection and constraint mechanism to quickly identify abnormal movements of trainees. It introduces a constraint mechanism in the data preprocessing stage, uses simple threshold comparison to quickly filter dangerous or invalid data, sets a physical reasonable range for each training feature data (such as pressure, limb joint angle, position and speed), and generates compliance labels through binary constraint functions and weighted comprehensive scores, avoiding subsequent complex and time-consuming CNN model inference, thereby reducing the computational load; at the same time, it improves the accuracy of trainee assessment.

[0017] 2. The present invention proposes to quantify video scene changes by calculating the mean square error (MSE) of adjacent frames: During the data acquisition phase, overlapping window sampling and inter-frame difference analysis are used to capture key scene change points of the trainees' body movements. By setting a threshold Tf, the system is prompted to extract only representative frame data Ri with significant changes, rather than processing all video frames, thereby significantly reducing the amount of data and improving subsequent processing efficiency while ensuring that key action details are not lost. At the same time,

[0018] The present invention synchronizes the clocks of all monitoring devices through the NTP protocol and adds timestamp information to video frames and sensor data. By creating a unified time series, the video frame data and sensor data are aligned to the same time axis, solving the problem of time inconsistency in multi-source data. Even if timestamp overlap or device disconnection occurs, the system can still ensure data integrity by interpolating or marking invalid data, thus providing a reliable basis for subsequent comprehensive analysis.

[0019] 3. This paper proposes a coordinate correction method that combines OpenPose key point detection with neighborhood spatial correlation calculation. In the data preprocessing stage, the two-dimensional coordinates and confidence levels of the key limb movement points of the trainees are extracted through the OpenPose open source library, the joint positions are preliminarily located and the data credibility is quantified. Then, the weights are dynamically adjusted based on the spatial correlation within the neighborhood, and the correction amount of the occurrence point is calculated. Finally, the corrected coordinates are obtained, which effectively suppresses long-distance noise interference, ensures spatial consistency, and improves the accuracy and robustness of the trainees' limb movement posture estimation, facilitating the subsequent construction of an accurate comprehensive feature vector F. j Provide reliable data support to improve the quality of evaluation and feedback.

[0020] 4. The present invention proposes a judgment method that combines weighted Euclidean distance and cosine similarity to quantify the global differences between the feature matrix and the standard action feature matrix, highlight key action patterns (such as pressure and angle deviation), and evaluate the accuracy of its limb movements. At the same time, cosine similarity is used to evaluate the directional consistency between limb movements and standard movements, quantify the standardization of movements, and provide trainees with more targeted correction suggestions. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The disclosure of the present invention is described with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. In the drawings, the same reference numerals are used to refer to the same components. Among them:

[0022] Figure 1 This is a schematic diagram of the overall processing flow of the ubiquitous training method for intelligent cardiopulmonary resuscitation proposed in one embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the entire process of generating training feedback based on the dual-track verification system proposed in one embodiment of the present invention;

[0024] Figure 3 According to one embodiment of the present invention, the CNN model is used to analyze the comprehensive feature vector F j Schematic diagram of feature extraction;

[0025] Figure 4 This is a schematic diagram of correcting the occurrence point (x, y) when each trainee makes a physical movement, as proposed in one embodiment of the present invention. DETAILED DESCRIPTION

[0026] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can propose a variety of interchangeable structural modes and implementation modes. Therefore, the following specific embodiments and drawings are only exemplary descriptions of the technical solution of the present invention and should not be regarded as the entire invention or as a limitation or restriction of the technical solution of the present invention.

[0027] The present invention will be further described in detail below with reference to the accompanying drawings, but this does not limit the present invention.

[0028] As an embodiment of the present invention, Figure 1 - Figure 4 As shown in FIG, a cardiopulmonary resuscitation intelligent companion ubiquitous training method is first proposed, which includes the following steps:

[0029] S1. Deploy monitoring equipment to achieve synchronous data collection from multiple sources:

[0030] Install cameras at the four corners of the training scene to cover the trainees' full body movements;

[0031] Inertial sensors with a sampling frequency of 100 Hz are deployed at the key limb joints of the trainees' wrists, elbows, and shoulders to capture the speed and angular velocity data generated by the trainees' limb movements in real time.

[0032] Integrated into the training dummy's chest, the pressure sensor measures compression force with a range of 0kg-60kg and an accuracy of ±0.5kg. The camera uses a checkerboard calibration to ensure spatial coordinate system alignment, with an error of less than 2mm. The sensor utilizes a six-axis attitude calculation algorithm to eliminate drift errors. After calibration, the static attitude error is less than 0.5°, ensuring compliance with training participants' compression force specifications.

[0033] S2. Obtain characteristic data representing the trainee's CPR operation:

[0034] In the ubiquitous training system for cardiopulmonary resuscitation (CPR), the camera samples the training video in a rolling manner according to the time scale, and uses the set inter-frame difference threshold to identify the changes in the video scene where the trainees perform limb movements. The frames near the scene change point are selected as the representative frame data Ri that characterize the trainees' limb movements. By capturing the key moments when the trainees' movements suddenly change, such as a sudden increase in pressing force, posture deviation or abnormal rhythm, the system can locate possible non-standard movement clips. It is understandable that non-standard movement clips usually correspond to operational errors or nodes that need to be corrected, such as insufficient pressing depth or arm angle deviating from the standard range. For example, a frame shows a participant's arm under-bend, while the associated sensor indicates an elbow angle of 70° (standard: 80°-100°) and a pressing force of 45kg (compliant with the 30-50kg standard). This dataset is fed into the model and subjected to a dual-track discrepancy analysis: First, the CNN compares the video features with the standard action image (e.g., angle deviation); second, it quantifies the degree of deviation between the actual sensor data and the standard parameters (e.g., force fluctuation). If the video shows a non-standard action but the sensor data meets the requirements (e.g., incorrect posture but acceptable pressing force), the system generates layered feedback: "The action angle needs to be adjusted to 90°, and the current pressing force is acceptable." Conversely, if the sensor detects an excessive force (e.g., 55kg), a level 1 alarm is immediately triggered, even if the posture is correct. This approach not only addresses the limitations of a single data source (e.g., video noise or sensor drift) but also provides refined corrections for complex errors (e.g., acceptable force but too fast a tempo), thereby comprehensively improving the scientific nature and effectiveness of training.

[0035] Synchronize the clocks of all monitoring devices through the NTP protocol, align the representative frame data Ri with the synchronously collected monitoring device data Ki (such as the pressure of the pressure sensor, the joint angle and position of the inertial sensor), establish a consistent time series, collect the training feature data of the trainees at different time steps, store and construct a comprehensive feature vector F j =[θ j ,p j ,v j ,d j ],θ jis the limb joint angle of the trainee, p j is the limb joint position, v j is the velocity of the limb joint, d j The pressing force.

[0036] S3. Generate a standard action spatiotemporal feature matrix through expert annotation and standard training feature data extraction, and build a convolutional neural network (CNN) model:

[0037] First, several emergency medical professionals recorded a standard CPR video using the Vicon motion capture system and annotated key parameters: compression depth of 5-6 cm (in this example, based on experiments, the equivalent strength of the pressure monitoring device was estimated to be 30-50 kg), arm angle with the elbows fully extended and both arms perpendicular to the patient's sternum (in this example, the equivalent angle estimated by the monitoring device was 80°-100°), and compression frequency (100-120 times / minute).

[0038] Secondly, the standard CPR operation video data was preprocessed using MATLAB and Python's OpenCV library to extract spatiotemporal feature data, including joint motion trajectories and compression phase timing, and construct a standard action spatiotemporal feature matrix.

[0039] Finally, the TensorFlow framework was used to build a convolutional neural network (CNN) model. The network structure was defined in sequence through the Keras interface: input layer, convolution layer, maximum pooling layer and fully connected layer. The TensorFlow framework was combined to implement batch data enhancement and generate an evaluation model to quantify the standardization of the trainees' movements.

[0040] S4. Use the built convolutional neural network (CNN) model to monitor the comprehensive feature vector F j Perform feature extraction and obtain the feature matrix F:

[0041] First, the acquired training feature data (i.e., containing θ j 、p j 、v j d j Four training feature data at 100 time steps) are processed to meet the input format requirements of the convolutional neural network CNN model;

[0042] Next, we set a convolution operation with a kernel size of 10 (covering a 0.1-second window) and a step size of 5 (sliding every 0.05 seconds) to gradually reduce the original high-dimensional training feature data to semanticize the training feature data. We also set specific action mode detectors for each channel, such as angle mutation and position offset.

[0043] Thirdly, a 2×2 max pooling layer is used to retain key parameters that represent the training feature data, such as the maximum limb joint angle deviation and the farthest limb joint position offset, to reduce the computational complexity of the model.

[0044] Finally, a 32×9 feature matrix F is obtained through the fully connected layer. Each row of this feature matrix is ​​set to represent the action mode corresponding to a channel, and each column represents a time step reflecting the temporal changes of key parameters.

[0045] S5. Compare the extracted feature matrix F with the preset standard action spatiotemporal feature matrix Fstd, and calculate the difference d and similarity C between the two using weighted Euclidean distance and cosine similarity. sim Then, according to the calculated difference d and similarity C sim , perform CPR operation posture deviation feedback.

[0046] In one embodiment of the present invention, based on the above-mentioned S2 technical concept, it can be understood that during the CPR operation, when the trainees make physical movements, the scene in the training area will change, and identifying this scene change is a key step to ensure the integrity and accuracy of the subsequent data. When the trainees are performing CPR operations, if the force or rhythm of the pressing is suddenly changed, this change will be directly reflected in the difference in the pixel values ​​of the sampled video frames. Therefore, in the specific implementation process of the present invention, it is proposed to quantify the video scene change by calculating the mean square error MSE of adjacent frames. The principle is: if the MSE value exceeds the set threshold T f , it indicates that the sampled video scene has changed significantly. At this time, the system will visualize the change results based on the subsequent series of processing, reminding the trainees to adjust their movements and improve the training effect of the trainees. The specific implementation steps of S2 include:

[0047] Select representative frame data Ri:

[0048] S2-1. Capture key scene change points where participants perform physical movements through overlapping window sampling and frame difference analysis. The specific operations are as follows:

[0049] S2-11, set overlapping sampling parameters: in the training video stream, sampling is performed according to a fixed time window, such as sampling 30 frames per second (covering 1 second of video), the sampling window size is set to W frames, the window step is S frames, preferably 15 frames, to achieve 50% overlapping sampling, when the window step S is less than the window size W, by formula , to achieve overlapping sampling, where M is the number of sampling times, N is the total number of frames of the sampled training video stream, and the overlapping sampling window sequence {I t ,I t+1 ,…,I t+W-1}, where the sampling window starting position t increases from 0 to NW with a step size S to achieve continuous sampling of the sampled training video stream.

[0050] S2-12, calculation of inter-frame difference of training video stream: Based on each overlapping sampling window sequence {I t ,I t+1 ,…,I t+W-1}, calculate the adjacent frames I of the training video stream in sequence t and I t+1 Mean squared error MSE: , where and I t and I t+1 In the frame, the pixel value at the point (x, y) when the trainee makes a physical movement, N=640×480=307200 represents the total number of pixels representing the physical movement data (image data) in the training video stream. So far, by obtaining the difference values ​​of all adjacent frame pairs in each overlapping sampling window, basic data is provided for subsequent scene change detection; it is understandable that in the actual CPR training process, the trainee's pressing force suddenly increases from 20kg to 40kg. This change is directly reflected in the training video stream, especially the area showing the pressing depth and force. Therefore, by calculating the pixel value difference in the above area, this change can be detected. At this time, by calculating the mean square error MSE, if the MSE value exceeds the set threshold T f , it is considered that the trainees made physical movements at the first t and I t+1 Scene changes occur between frames.

[0051] S2-13. Extract representative frame data Ri representing the trainee's body movements:

[0052] First, create an empty array D, set its length to the total number of frames of the sampled training video stream minus one, that is, N-1, to compare the differences between consecutive frame pairs, and based on the inter-frame difference calculation in step S2-12, fill the calculated mean square error (MSE) values ​​into the empty array D in sequence, and set the MSE value corresponding to each index i from D[0] to D[N-2] in the empty array D to be the i-th i and I i+1 The difference measure between frames is D={MSE(I0,I1),MSE(I1,I2),…,MSE(I N-2 ,I N-1 )}, where the length of D is N-1.

[0053] Secondly, a threshold T is set based on the actual situation and empirical data of the sampled training video stream. f, and traverse the empty array D, when D [i] >T f When i and I i+1 When a scene changes between frames, the change points in the scene change are recorded and obtained, that is, the points where the trainees make physical movements, forming an array of occurrence points C, C = {i, | D [i] >T f , 0≤i<N-1}. It can be understood that the threshold T f The preferred value is 100. By collecting standard CPR operation videos, calculating the MSE values ​​of adjacent frames, and statistically analyzing their distribution, it is found that the MSE value of normal pressing actions is usually less than 50 (small limb movement or lighting changes). If the pressing force is too large or the arm angle deviates, the calculated MSE value reaches 200-300 (significant pixel value changes). Therefore, in terms of threshold selection, ROC curve analysis is performed using positive sample data, 100 standard CPR operation videos (MSE distribution: 10-80) and negative sample data, 100 standard CPR operation videos (MSE distribution: 10-80), and calculating the balance between false alarm rate and false negative rate. It is found that T f =100, the accuracy is 92% (positive sample recognition rate 90%, negative sample recognition rate 94%), which can achieve a balance between false positive rate (normal actions are misjudged) and false negative rate (abnormal actions are not detected).

[0054] Next, 5 frames before and after each change point i in the occurrence point array C are selected to form a representative frame data Ri containing 11 frames, Ri={I i-5 ,I i-4 ,I i-3 ,I i-2 ,I i-1 ,I i ,I i+1 ,I i+2 ,I i+3 ,I i+4 ,I i+5} to obtain detailed information about scene changes. This detailed information includes at least the position of the trainee's hands, elbow extension, whether the base of the palm is close to the chest during CPR, whether the compression depth reaches 5-6 cm, whether the compression frequency meets the standard of 100-120 times per minute, whether the arm is kept vertical during compression, and whether there are any interruptions during the compression process. By selecting representative frame data Ri near the change point, key details of the trainee's body movements during CPR can be captured, such as the precise changes in body movement position, the amplitude and force of the movement, and the standardization of the operation.

[0055] Example: Assume that the total number of frames of the sampled training video stream is N=1000, and the size of each frame is 640×480 pixels. By calculating the mean square error (MSE) value of each pair of adjacent frames in sequence, we get an empty array D with a length of 999 (1000-1). At this time, D[0] is the mean square error (MSE) value between the first and second frames, and D[1] is the mean square error (MSE) value between the second and third frames. And so on. Set the threshold T f =100, traverse the empty array D, if D [i] >100, then i is recorded as a scene change point to form an occurrence point array C. Assuming that a change point i=100 is detected, the representative frame data Ri selected is: R

[100] ={I 95 ,I 96 ,I 97 ,I 98 ,I 99 ,I 100 ,I 101 ,I 102 ,I 103 ,I 104 ,I 105}.

[0056] In one embodiment of the present invention, since the representative frame data Ri and the monitoring device have different sampling frequencies, it is necessary to align the representative frame data Ri with the synchronously collected monitoring device data Ki to establish a consistent time series.

[0057] That is, execute S2-2 and synchronize the clocks of all monitoring devices using the NTP protocol:

[0058] S2-21. Add timestamp information to the representative frame data Ri and monitoring device data Ki:

[0059] For the timestamp of the representative frame data Ri, let the frame rate of the representative frame data Ri be f fps , preferably f fps is 30, the total number of frames of the representative frame data Ri is N, then the timestamp information T of the representative frame data Ri of the i-th frame is video [i] is: , in seconds.

[0060] For the monitoring device data Ki timestamp, let the sampling frequency of the monitoring device be f s , preferably f s =100Hz, then the timestamp information T of the data point collected by the monitoring device data Ki device [j] is: , M is the total number of data points collected by the monitoring equipment.

[0061] S2-22. Establish a time series and synchronize the timestamp information of the representative frame data Ri and the monitoring device data Ki to the same time axis: Create an empty array T seq , and determine the empty array T according to the maximum sampling frequency of the representative frame data Ri and the monitoring device data Ki seq length, K=max(N×f fps ,M×f s );Time stamp information T of representative frame data Ri video [i] Fill in the empty array T in sequence seq In which, for the i-th frame of the representative frame data Ri, its timestamp information T video [i] Fill to empty array T seq The position is k=i×f fps ; Synchronize the timestamp information T of the monitoring device data Ki device [j] is also filled into the empty array T seq In which, for the jth data point of the monitoring device, its timestamp information T device [j] Fill to empty array T seq The position of is k=j×fs. Thus a time series is formed.

[0062] It should be noted that in the above synchronization process, if timestamp information overlaps, the average value will be taken or interpolation will be performed according to the actual situation. In this process, if the monitoring device is disconnected for more than 200ms, the data for this period will be marked as invalid and the alarm "device connection abnormality" will be triggered.

[0063] In one embodiment of the present invention, step S2 further includes: S2-3, constructing a comprehensive feature vector F j =[θ j ,p j ,v j ,d j ]:

[0064] Based on the established time series, obtain the limb joint angle θ at the occurrence point (x, y) when the trainee makes a limb movement in the time series j , limb joint position p j 、v j Limb joint occurrence point speed, pressing force d j It is understandable that θ j ,p j ,v j ,d j It is the training feature data that represents the physical movements of the trainees.

[0065] In this embodiment, the limb joint angle θ jIndicates the angle change of the limb or the trainee's body sensor at a time step, such as the bending angle of the arm during the pressing process. The unit is radians. By monitoring the changes in joint angles, the system can determine whether the pressing action conforms to the standard posture, such as whether the arm is bent too much or too little, and provide corresponding feedback to the trainee to help them adjust the pressing posture.

[0066] Limb joint position p j Indicates the specific coordinate position of the trainee's joints in the scene, used to describe the absolute position information of the joints at different time steps. By tracking the position changes of the joint occurrence point, the system can monitor whether the pressing point deviates from the target area (standard area), thereby reminding the trainee to adjust the pressing position in the future to ensure the effectiveness of the pressing;

[0067] v j The velocity of the limb joint point indicates the rate of change of the joint position per unit time step, which is used to reflect the speed of joint movement. In CPR training, the system uses joint velocity data to determine whether the compression rhythm meets the requirements (such as the standard frequency of 100-120 times per minute). Based on this, the system provides speed adjustment suggestions to trainees to ensure the effectiveness of the compression action; Compression intensity d j Real-time monitoring feedback through pressure sensors during CPR training.

[0068] In the specific implementation, take the elbow joint of the trainee as an example, for the limb joint angle θ j , calculated using the following formula: , where 、 are the coordinates of the corrected shoulder joint occurrence point extracted from the representative frame data Ri, 、 are the coordinates of the corrected elbow joint occurrence point, 、 They are the coordinates of the corrected wrist joint points, and the limb joint angles θ are calculated by j , reflecting the bending degree and posture of the trainees’ limb joints. j , calculated using the following formula: , where 、 are the corrected (x, y) coordinates of the occurrence point. For the limb joint occurrence point velocity v j , calculated using the following formula: , where is the (x, y) coordinate of the same limb movement occurrence point in two adjacent frames in the occurrence point array C. The movement speed of the trainee is calculated by comparing the changes in these two position coordinates. is the time difference between two adjacent frames, which is used to reflect the speed of the trainee's movement per unit time. j , and read its instantaneous measurement value through the pressure sensor (integrated in the chest cavity of the training dummy).

[0069] Based on the above technical concept, it should be noted that in the CPR Intelligent Companion Ubiquitous Training System, accurately capturing the body movements of trainees is crucial for evaluation and feedback. However, due to the noise and inaccuracy of the sampled training video stream data, it is necessary to correct the position of the point (x, y) when each trainee's body movement occurs in the process of judging whether a scene change has occurred. After improving the estimation accuracy of the trainee's body movement / posture, the comprehensive feature vector F is constructed. j =[θ j ,p j ,v j ,d j ].

[0070] The specific implementation steps for correcting the occurrence point (x, y) are:

[0071] First, based on the representative frame data Ri extracted in step S2-13, Ri={I i-5 ,I i-4 ,I i-3 ,I i-2 ,I i-1 ,I i ,I i+1 ,I i+2 ,I i+3 ,I i+4 ,I i+5}, the OpenPose open source library carried by the system is used to detect the key body movement points of the trainees, and the two-dimensional coordinates (x j ,y j ) and the confidence level P of the coordinate j , to preliminarily locate the elbow, wrist and other joint positions, and quantify the data credibility, where the confidence P j Indicates the reliability of the OpenPose open source library in locating key body movement points (value range 0-1), such as P j =0.9 indicates high confidence, P j =0.3 indicates low confidence, which may be caused by limb occlusion or lighting interference;

[0072] Secondly, based on the neighborhood range ρ defined by the spatial proximity of the representative frame data Ri, preferably ρ = 50 pixels, the two-dimensional coordinates (x j ,y j) and other key limb action points in its neighborhood (x jk ,y jk )’s spatial correlation: quantify the physical distance d between two points using the Euclidean distance formula jk , , combined with other key body movement points (x jk ,y jk ) confidence level P k , dynamically adjust the association weight W ij , set the two-dimensional coordinates (x j ,y j ) has a greater impact, ;

[0073] Again, calculate the correction amount at the occurrence point : , ;

[0074] Finally, the corrected occurrence point (x c j ,y c j ): , It is understandable that, in specific implementation, based on the corrected (two-dimensional coordinates) occurrence point (x c j ,y c j ) extracts four types of feature data (joint angle, position, speed, and pressure), and uses the confidence level P j By weighting the feature data, the final comprehensive feature vector F can be obtained. j， F j =[θ j ,p c j ,v j ,d j ]. The purpose of defining the neighborhood range ρ = 50 pixels for the spatial proximity of the representative frame data Ri is to ensure spatial consistency and suppress the influence of long-distance noise. By setting this range, the system only performs weight calculations on high-confidence key points within the neighborhood, avoiding irrelevant or low-confidence data interfering with coordinate correction, thereby improving the accuracy and robustness of the estimation of the trainees' body movements and postures.

[0075] In one embodiment of the present invention, in step S4, the monitored comprehensive feature vector F j When performing feature extraction, the CNN model needs to be adapted through the following steps to achieve accurate assessment of CPR action quality:

[0076] First, the original training feature data is converted into a three-dimensional tensor in the format of number of samples × time steps × number of features. For example, if the input format of the original training feature data of a single sample is (1, 100, 4), it is converted to (size, time window length, feature dimension). Secondly, Z-score normalization is performed on each feature dimension to ensure dimensional consistency. Thirdly, according to the set convolution kernel parameters (kernel size 10, step length 5), 100 time steps are divided into overlapping windows, and the number of windows is calculated, as shown in the following example: Each window covers 10 time steps with a step size of 5, generating 19 local time window samples. Finally, the four features in each window are treated as channels, resulting in an input format of (number of samples, time steps, number of features) = (19, 10, 4). This converts the raw time series data into spatiotemporal feature maps that can be processed by the CNN model, preserving local limb movement patterns (such as sudden changes in compression angle and rhythm abnormalities) while adapting the model structure (for example, the convolution kernel covers 10 time steps), ultimately achieving accurate assessment of CPR movement quality.

[0077] Based on the above technical concept, it should be noted that based on steps S2 and S4, before inputting the acquired training feature data into the convolutional neural network CNN model and obtaining the feature matrix F, it is necessary to apply a constraint mechanism to conduct a compliance assessment on the training feature data collected at different time steps in step S2, and perform physical over-limit feedback on the CPR operation action. The purpose is to filter out dangerous or invalid data (such as "dangerous data" refers to abnormal parameters that may cause operator injury or complete failure of training; "invalid data" refers to unreliable data generated by sensor errors, environmental interference, or extreme action deviations) to reduce the subsequent computational load of the CNN model. After completion, the training feature data acquired at the time step that meets the constraint mechanism is input into the CNN model. After processing by the convolution layer, pooling layer, and fully connected layer, a 32×9 feature matrix F is obtained, F∈R 32*9 , perform CPR operation posture deviation feedback.

[0078] Understandably, the constraint mechanism operates as an independent, parallel module during the data preprocessing phase, forming a dual-track verification system alongside the CNN model's feature extraction and dissimilarity calculations. Within the overall processing flow, the constraint mechanism is applied immediately after data acquisition, serving as a preliminary physical range check, while the CNN model handles the more complex posture similarity assessment. Its purpose is to prevent dangerous actions (such as excessive compressions) even when the movements are correct, enhance the interpretability of subsequent CPR procedures for trainees, and clearly distinguish between parameter out-of-limit and non-standard posture errors. The verification system's processing flow is as follows: raw data - constraint mechanism check - (compliant) - CNN model processing - posture assessment - feedback; or (non-compliant) - direct feedback.

[0079] In one embodiment of the present invention, the constraint mechanism acts as a low-computational cost primary filter. By using simple threshold comparison, it can quickly intercept dangerous operations that may be caused by abnormal physical parameters, avoiding subsequent complex and time-consuming CNN reasoning. The specific implementation steps are as follows:

[0080] In the data preprocessing stage (before the input format of the convolutional neural network (CNN) model is adjusted), for each feature data (θ j 、p j 、v j d j ) Set the physical reasonable range for safety verification, including the minimum value X imin and the maximum value X imax , the specific thresholds are as follows: pressure intensity d j ∈[30kg,50kg]; arm angle θ j ∈[80 ° ,100°]; pressing position p j Limited to the dummy's chest area; compression speed v j ∈[100 times / minute,120 times / minute].

[0081] For each feature data Xi (Xi represents feature data θ j 、p j 、v j d j ), calculate whether it exceeds the reasonable range, and generate the binary constraint function value f i (Xi), , when X imin ≤Xi≤ X imax When f i (Xi)=1, otherwise f i (Xi)=0;

[0082] At the same time, according to the feature importance (the feature importance is mainly based on the training results that need to be mastered in CPR operation), each feature data is assigned a weight Wi (i=1, 2, 3, 4, corresponding to the feature data θ j 、p j 、v j d j ), calculate its physical constraint comprehensive score: ; Compare the comprehensive score of the physical constraint with the preset threshold T to generate the compliance label Lph: ,in,

[0083] If Lph=0, directly trigger the feedback system and output the over-limit parameter list Ep={Xi|Xi <X imin or Xi>X imax}, if it prompts "Pressure exceeds limit: 55kg>50kg", the subsequent CNN model processing will not be performed at this time, thus saving computing resources. It can be understood that the preset threshold T represents the system's minimum requirement for "overall body movement compliance". Its value is optimized through experimental data to balance safe interception and user experience. During specific implementation, it is necessary to collect a large number of compliant and non-compliant body movement samples and analyze their comprehensive score distribution. If T is set too high, actions that are slightly out of limit but generally safe may be missed (over-interception); if T is set too low, some dangerous operations may be missed (risk of underreporting). Therefore, it is necessary to find the best balance through experiments so that the system can intercept dangerous operations while avoiding frequent false alarms.

[0084] If Lph=1, enter the CNN model processing flow, pass the data to the CNN model for dual-track verification.

[0085] In one embodiment of the present invention, the purpose of calculating the difference d in step S5 is to quantify the global difference between the extracted feature matrix F and the preset standard action spatiotemporal feature matrix Fstd, thereby highlighting key action patterns (such as pressure intensity and angle deviation). The specific calculation formula is: , where the smaller the d value is, the smaller the global difference between the trainees' body movements and the standard movements is. is the training feature data of the i-th channel in the feature matrix F at the j-th time step, is the predefined standard action feature value, is the channel high weight coefficient, which includes the key channel weight coefficient (such as pressure intensity and angle related channels) and non-key channel weight coefficient (such as speed fluctuation channel).

[0086] Calculate the similarity C sim The purpose is to evaluate the consistency between the direction of the trainees' body movements and the standard movements. The specific calculation formula is: , where C sim The closer the value is to 1, the more consistent the direction of the trainee's body movements is with the standard.

[0087] In one embodiment of the present invention, the specific implementation steps of performing CPR operation posture deviation feedback are as follows:

[0088] Combine the set tolerance range to determine whether the trainee's physical movements are standard, and perform corresponding feedback in combination with the following Table 1-CPR Action Quality Judgment Logic Data Table.

[0089] Table 1

[0090]

[0091] It should be noted that in Table 1 above, the tolerance ranges of 0.05 and 0.1 are for different training stages. In subsequent implementation, the tolerances can be dynamically adjusted based on the trainee's historical performance (e.g., loosening for novices and tightening for experienced trainees). Visual reports can then be used to assess CPR action quality and provide real-time feedback, helping trainers adjust their movements in a timely manner.

[0092] Based on the above technical concepts, it can be understood that the present invention not only prevents CPR operation errors caused by physical over-limits, but also improves the training quality of trainees by taking into account both low-latency safety interception and high-precision limb movement recognition optimization. At the same time, it reduces the computing load through phased processing (primary filtering + model evaluation), adapts to real-time training scenarios, and ultimately achieves efficient and safe CPR skill improvement.

[0093] Example: First feedback (constraint mechanism):

[0094] [Data to be processed]: Originally collected time-step feature data, including pressing force, arm angle, pressing position, and pressing speed. Example data value: θ j =85°, p j =(120,200),v j =110 times / min, d j =45kg.

[0095] [Feedback content]: Intercept non-compliant data: If any characteristic data exceeds the physical threshold, such as d j =55kg>50kg; trigger feedback ("pressing force exceeds limit: 55kg>50kg" or "arm angle is insufficient: 75°<80°"); interception basis: based on the comprehensive score of physical constraints f ph and the preset threshold T, if f ph <T, output the out-of-limit parameter list Ep; Data destination: Non-compliant data is intercepted and does not enter the CNN model. An alarm is directly output and the current time step processing is terminated.

[0096] Second feedback (CNN model):

[0097] [Data to be processed]: "Compliant" data that has passed the first constraint mechanism, that is, time series data that has been converted into a spatiotemporal feature matrix. Example data: Compliant comprehensive feature vector F j =[θ j ,p j ,v j ,d j ], which is input into the CNN model after normalization and window segmentation, with the format of (number of samples, time steps, number of features) = (19, 10, 4).

[0098] [Feedback content]: Assessment of participants’ body movement posture deviation: Based on the spatiotemporal features extracted by CNN, the difference d and similarity C with the standard movement are calculated. sim Difference feedback: For example, "The compression angle deviation exceeds 5°; elbow extension needs to be adjusted." Similarity feedback: For example, "The compression speed is consistent with the standard, but the direction is offset; please correct the wrist position." Data destination: Compliant and posture-assessed data is generated into a final report. At this point, through the first feedback channel, real-time security interception based on physical thresholds processes raw feature data and issues an over-limit alert. The second feedback channel, motion assessment based on the CNN model, processes compliant spatiotemporal data and issues posture optimization suggestions.

[0099] As a second aspect of the present invention, a cardiopulmonary resuscitation smart companion ubiquitous training system is proposed, including a memory and a processor, wherein the memory includes a cardiopulmonary resuscitation smart companion ubiquitous training program, and when the cardiopulmonary resuscitation smart companion ubiquitous training program is executed by the processor, the above-mentioned cardiopulmonary resuscitation smart companion ubiquitous training method is implemented.

[0100] The technical scope of the present invention is not limited to the contents of the above description. Those skilled in the art can make various deformations and modifications to the above embodiments without departing from the technical idea of ​​the present invention, and these deformations and modifications should all fall within the protection scope of the present invention.

Claims

1. A ubiquitous cardiopulmonary resuscitation training method, characterized by: Including steps: Deploy monitoring equipment in the training scenes, key limb joints, and chest area of ​​the training dummy to achieve simultaneous collection of multi-source video stream data; Calculate the mean square error between adjacent frames in multi-source video stream data, , where and I t and I t+1 In the frame, the pixel value at the point (x, y) when the trainee makes a physical movement, N represents the total number of pixels representing the physical movement in the training video stream; fill the created empty array D with the mean square error MSE in turn, and set a threshold T f , and traverse the empty array D, when D [i] >T f When i and I i+1 When a scene changes between frames, the change points in the scene change are recorded and obtained to form representative frame data representing the body movements of the trainees, so as to obtain detailed information about the scene change; Synchronize the clocks of all monitoring devices corresponding to the representative frame data through the NTP protocol, establish a consistent time series, and construct a comprehensive feature vector F that represents the training feature data of the trainees. j =[θ j ,p j ,v j ,d j ],θ j is the limb joint angle, p j is the limb joint position, v j is the velocity of the limb joint point, d j is the pressing force; Through expert annotation and standard training feature data extraction, the standard action spatiotemporal feature matrix Fstd is generated and the CNN model is constructed; A constraint mechanism is constructed to evaluate the compliance of the training feature data to filter out invalid training data. If the data is compliant, it is input into the CNN model to extract the feature matrix F. Compare the feature matrix F with the standard action feature matrix Fstd, and calculate the weighted Euclidean distance and cosine similarity to obtain the difference d and similarity C between the two. sim , generating corresponding training result feedback; the steps of building the constraint mechanism are as follows: For each training feature data Xi, calculate whether it exceeds the reasonable range and generate a binary constraint function value f i (Xi), ; Assign weight Wi to each training feature data Xi according to the importance of the training results, and calculate the comprehensive score of the physical constraint ; f ph Compare with the preset threshold T to generate a compliance label L ph , if f ph >T then let L ph =1, indicating that the data is compliant and enters the CNN model processing flow; otherwise, let L ph =0, directly trigger feedback, output the over-limit parameter list Ep={Xi|Xi <X imin or Xi>X imax }, no subsequent CNN model processing is performed; Among them, before the compliance evaluation of the training feature data is performed through the constraint mechanism, it is necessary to correct the occurrence point (x, y) of each trainee's body movement based on the collected training feature data to improve the subsequent estimation accuracy of the trainee's body movement, and then construct the comprehensive feature vector F j , the specific steps are: First, the OpenPose open source library installed in the system is used to detect the key body movement points of the trainees, and the two-dimensional coordinates (x, y, and y) of each point where the trainees make body movements are directly extracted in the 640×480 pixel coordinate system. j ,y j ) and the confidence level P of the coordinate j , to preliminarily locate the elbow and wrist joint positions and quantify the data credibility; Secondly, define a neighborhood range ρ and calculate the two-dimensional coordinates (x j ,y j ) and other key limb action points in its neighborhood (x jk ,y jk )’s spatial correlation: quantify the physical distance d between two points using the Euclidean distance formula jk , , combined with other key body movement points (x jk ,y jk ) confidence level P k , dynamically adjust the association weight W ij , set the two-dimensional coordinates (x j ,y j ) has a greater impact, ; Again, calculate the correction amount of the occurrence point , ; Finally, the corrected occurrence point (x c j ,y c j ): .

2. The ubiquitous cardiopulmonary resuscitation training method according to claim 1, characterized in that: Construct comprehensive feature vector F j In , the feature vector of each training feature data is calculated as follows: Limb joint angle θ j : , where 、 are the coordinates of the corrected shoulder joint occurrence point extracted from the representative frame data Ri, 、 are the coordinates of the corrected elbow joint occurrence point, 、 are the coordinates of the wrist joint occurrence point after correction; Limb joint position p j : , where 、 are the corrected coordinates of the occurrence point; Limb joint velocity v j : , where is the coordinate of the occurrence point of the same limb action in two adjacent frames in the occurrence point array C, is the time difference between two adjacent frames; Pressing force d j : Read the instantaneous measurement value through the monitoring device.

3. The ubiquitous cardiopulmonary resuscitation training method according to claim 1, characterized in that: The steps of obtaining the representative frame data are as follows: First, in the multi-source video stream data, sampling is performed according to a fixed time window. Assume that the sampling window size is W frames and the window step is S frames. When the window step S is less than the window size W, the formula , to achieve overlapping sampling, where M is the number of sampling times, N is the total number of sampled video stream frames, and the overlapping sampling window sequence {I t ,I t+1 ,…,I t+W-1 }; Secondly, based on each overlapping sampling window sequence {I t ,I t+1 ,…,I t+W-1 }, calculate the adjacent frames I of the video stream in sequence t and I t+1 Mean square error MSE; Finally, create an empty array D, set its length to the total number of frames of the sampled training video stream minus one, to compare the differences between consecutive frame pairs, and fill the empty array D with the mean square error MSE in sequence, and set the empty array D so that the mean square error MSE value corresponding to each index i from D[0] to D[N-2] is the i-th i and I i+1 The difference measure between frames is D={MSE(I0,I1),MSE(I1,I2),…,MSE(I N−2 ,I N-1 )}; Set a threshold T f , and traverse the empty array D, when D [i] >T f When i and I i+1 When a scene changes between frames, the change points in the scene change are recorded and obtained to form an occurrence point array C, C={i,| D [i] >T f , 0≤i<N-1}; select 5 frames before and after each change point i in the occurrence point array C to form a representative frame data Ri containing 11 frames, Ri={I i-5 ,I i-4 ,I i-3 ,I i-2 ,I i-1 ,I i ,I i+1 ,I i+2 ,I i+3 ,I i+4 ,I i+5 } to obtain scene change details.

4. A ubiquitous cardiopulmonary resuscitation intelligent training method according to claim 1 or 3, characterized in that: The steps to establish a consistent time series using the NTP protocol are as follows: First, add timestamp information to the representative frame data Ri and monitoring device data Ki: For the timestamp of the representative frame data Ri, let the frame rate of the representative frame data Ri be f fps , the total number of frames of the representative frame data Ri is N, then the timestamp information T of the representative frame data Ri of the i-th frame video [i] is: , in seconds; For the monitoring device data Ki timestamp, let the sampling frequency of the monitoring device be f s , then the timestamp information T of the data point collected by the monitoring device data Ki device [j] is: , M is the total number of data points collected by the monitoring equipment; Next, create a time series: Create an empty array T seq , and determine the empty array T according to the maximum sampling frequency of the representative frame data Ri and the monitoring device data Ki seq length, K=max(N×f fps ,M×f s ), the timestamp information T video [i] Fill in the empty array T in sequence seq In which, for the i-th frame of the representative frame data Ri, its timestamp information T video [i] Fill to empty array T seq The position is k=i×f fps ; Synchronize the timestamp information T device [j] is also filled into the empty array T seq In which, for the jth data point of the monitoring device, its timestamp information T device [j] Fill to empty array T seq The position is k=j×fs, completing the construction of consistent time series; If the monitoring device is disconnected for more than 200ms, the data for that period will be marked as invalid.

5. The ubiquitous cardiopulmonary resuscitation training method according to claim 1, characterized in that: The monitoring equipment includes at least: cameras installed at the four corners of the trainee's training scene to cover the trainee's whole body movements; inertial sensors installed at the trainee's key limb joints of the wrist, elbow, and shoulder to capture the speed and angular velocity data generated by the trainee's limb movements in real time; and a pressure sensor integrated in the chest area of ​​the training dummy to measure the trainee's pressing force.

6. The ubiquitous cardiopulmonary resuscitation training method according to claim 1, characterized in that: The monitored comprehensive feature vector F j When performing feature extraction, the CNN model needs to be adapted through the following steps to achieve accurate assessment of CPR action quality: First, convert the original training feature data into a three-dimensional tensor in the format of number of samples × time steps × number of features. If the original training feature data input format of a single sample is: [1, 100, 4], then the corresponding conversion is: [size, time window length, feature dimension]; Secondly, Z-score standardization is performed on each feature dimension to ensure dimensional consistency; Again, according to the set convolution kernel parameters, kernel size 10, step size 5, 100 time steps are divided into overlapping windows, and the number of windows is calculated. , each window covers 10 time steps with a step size of 5, generating 19 local time window samples; Finally, the four features of each window are regarded as channels, and the final input format is: [number of samples, time steps, number of features] = [19, 10, 4].

7. The ubiquitous cardiopulmonary resuscitation training method according to claim 1, characterized in that: Difference d and similarity C sim The calculation formulas are as follows: , In the formula, the smaller the d value is, the smaller the global difference between the trainees' body movements and the standard movements is. is the training feature data of the i-th channel in the feature matrix F at the j-th time step, is the predefined standard action feature value, is the channel high weight coefficient, which includes the key channel weight coefficient and the non-key channel weight coefficient; , where C sim The closer the value is to 1, the more consistent the direction of the trainee's body movements is with the standard.

8. The ubiquitous cardiopulmonary resuscitation training method according to claim 1, characterized in that: The specific steps to generate the standard action spatiotemporal feature matrix Fstd and build the CNN model are as follows: First, several emergency medical professionals recorded a standard CPR video using the Vicon motion capture system and annotated key parameters: compression depth of 5-6 cm, elbows fully extended, arms perpendicular to the patient's sternum, and compression frequency of 100-120 times per minute. Secondly, the data of the standard CPR operation video was preprocessed using the OpenCV library of MATLAB and Python to extract the spatiotemporal feature data including joint motion trajectory and compression phase sequence, and construct the standard action spatiotemporal feature matrix Fstd. Finally, the TensorFlow framework was used to build a CNN model, and the network structure was defined in sequence through the Keras interface: input layer, convolution layer, maximum pooling layer and fully connected layer. The TensorFlow framework was combined to implement batch data enhancement and generate an evaluation model to quantify the standardization of the trainees' movements.

9. A ubiquitous cardiopulmonary resuscitation intelligent training system, characterized by: include: A memory and a processor, wherein the memory includes a cardiopulmonary resuscitation smart companion ubiquitous training program, and when the cardiopulmonary resuscitation smart companion ubiquitous training program is executed by the processor, the cardiopulmonary resuscitation smart companion ubiquitous training method described in any one of claims 1 to 8 is implemented.

Images