Visual analysis method and system for assisting CPR training

Abstract

The invention provides a visual analysis method and system for assisting CPR training, and relates to the technical field of CPR training.The visual analysis method for assisting CPR training comprises the steps that the posture of a user is obtained; cPR posture detection is carried out on the user posture through the constructed target auto-encoder model; when the user posture is a CPR posture, performing CPR preparation posture detection on the CPR posture; when the CPR posture is a CPR preparation posture, entering a real-time training mode; in the real-time training mode, three-dimensional coordinates of skeleton key points of the user are extracted, and palm root key point coordinates and corresponding body key angles are obtained; and according to the palm root key point coordinates, obtaining a hand motion track and range within a preset time. According to the visual analysis method and system provided by the invention, the defects in training precision, real-time performance and universality can be effectively solved, the interactivity, scientificity and user experience in the CPR training process are remarkably improved, and a technical foundation is laid for popularization and promotion of CPR first-aid skills in the whole society.

Description

Technical Field

[0001] The present invention relates to the technical field of CPR (Cardiopulmonary Resuscitation) training, and particularly to a visual analysis method and system for assisting CPR training. Background Art

[0002] With the development of technology and the increasing demand for training accuracy, numerous studies have been dedicated to the real-time guidance during first aid training, aiming to provide suggestive guidance for beginners without first aid experience. These guidances mainly include three methods: feedback devices based on Internet of Things functions, mixed reality, and machine learning.

[0003] During the existing CPR training assistance, the hardware devices relied on are expensive and difficult to be popularized to ordinary users, which limits the promotion scope. Moreover, the action guidance mainly focuses on static display, lacking real-time performance and interactivity, unable to dynamically capture users' actions and give effective feedback. There is a lack of accurate quantitative criteria for the evaluation of CPR actions, resulting in difficulty for trainees to clarify the improvement direction. When guiding users to correct actions, it is difficult to comprehensively consider the consistency of spatial actions and the coherence in time, and unable to provide users with comprehensive and scientific guidance. Summary of the Invention

[0004] In view of the above-mentioned disadvantages of the prior art, the purpose of the present invention is to provide a visual analysis method and system for assisting CPR training, which is used to solve the problems in the existing CPR training assistance process, such as the expensive hardware devices relied on, difficult to be popularized to ordinary users, limiting the promotion scope; the action guidance mainly focuses on static display, lacking real-time performance and interactivity, unable to dynamically capture users' actions and give effective feedback; the lack of accurate quantitative criteria for the evaluation of CPR actions, resulting in difficulty for trainees to clarify the improvement direction; and when guiding users to correct actions, it is difficult to comprehensively consider the consistency of spatial actions and the coherence in time, and unable to provide users with comprehensive and scientific guidance.

[0005] To achieve the above and other related objectives, the present invention provides a visual analysis method for assisting CPR training, including: obtaining the user's posture; detecting the CPR posture of the user's posture through the constructed target autoencoder model; when the user's posture is the CPR posture, detecting the CPR preparation posture of the CPR posture; when the CPR posture is the CPR preparation posture, entering the real-time training mode; in the real-time training mode, extracting the three-dimensional coordinates of the user's skeletal key points to obtain the coordinates of the heel of the palm key point and the corresponding key body angles; according to the coordinates of the heel of the palm key point, obtaining the hand movement trajectory and range within a preset time; according to the hand movement trajectory and range, obtaining the corresponding compression frequency; according to the key body angles and the compression frequency, respectively calculating the specific scores corresponding to each moment of the key body angles and the compression frequency; according to the specific scores corresponding to each moment of the key body angles and the compression frequency, calculating the consistency relationship data between the key body angles and between the key body angles and the compression frequency; according to the consistency relationship data, generating an auxiliary CPR training view with the compression time displayed in the clockwise direction to guide the user to coordinate the CPR actions.

[0006] In an embodiment of the present invention, the target loss function corresponding to the target autoencoder model includes a cross-entropy loss function, a spatial consistency loss function, and a temporal coherence loss function.

[0007] In an embodiment of the present invention, the calculation formula of the spatial consistency loss function is: ; the calculation formula of the temporal coherence loss function is: ; where represents the time step, represents the number of skeletal key points, represents the number of time steps, represents the number of key points at each time step, represents the predicted key point coordinates, represents the standard key point coordinates, represents the Euclidean distance, represents the total number of time steps, represents the change in the key point position between two consecutive frames.

[0008] In an embodiment of the present invention, when the user's posture is the CPR posture, detecting the CPR preparation posture of the CPR posture includes: when the user's posture is the CPR posture, obtaining the three-dimensional coordinates of the skeletal key points through the Mediapipe recognition algorithm; according to the three-dimensional coordinates of the skeletal key points, obtaining the similarity between the CPR posture and the standard posture; monitoring the similarity for a threshold; when the similarity meets the standard, the CPR posture is the CPR preparation posture.

[0009] In an embodiment of the present invention, the key body angles include: a first included angle between the upper arm and the body, a second included angle between the upper arm and the lower arm, and a third included angle between the body and the thigh.

[0010] In an embodiment of the present invention, based on the coordinates of the metacarpal key point, the hand movement trajectory and range within a preset time are obtained, including: drawing the hand movement trajectory according to the coordinates of the metacarpal key point; calculating the hand movement range corresponding to the hand movement trajectory within the preset time through the convex hull algorithm.

[0011] In an embodiment of the present invention, according to the key body angles and the pressing frequency, the specific scores corresponding to each moment of the key body angles and the pressing frequency are calculated respectively, including: obtaining the first mean value and the first standard deviation within the standard angle range according to the upper limit value and the lower limit value of the standard angle range corresponding to the key body angles; wherein, the upper limit value and the lower limit value of the standard angle range are respectively extracted and calculated from the medical CPR standard video; obtaining the first normal distribution score according to the first mean value, the first standard deviation and the corresponding key body angles; performing normalization processing on the first normal distribution score to obtain the specific score corresponding to each moment of the key body angles; obtaining the second mean value and the second standard deviation within the standard pressing frequency range according to the upper limit value and the lower limit value of the standard pressing frequency range specified by the medical CPR standard corresponding to the pressing frequency; obtaining the second normal distribution score according to the second mean value, the second standard deviation and the corresponding pressing frequency; performing normalization processing on the second normal distribution score to obtain the specific score corresponding to each moment of the pressing frequency.

[0012] In an embodiment of the present invention, according to the specific scores corresponding to each moment of the key body angles and the pressing frequency, the consistency relationship data between the key body angles and between the key body angles and the pressing frequency are calculated, including: performing consistency calculation between different key body angles according to the specific scores corresponding to each moment of the key body angles and the pressing frequency to obtain the CPR synchronization index and the CPR stability index; obtaining the first quantization scoring standard as the consistency relationship data according to the CPR synchronization index and the CPR stability index; performing consistency calculation between each key body angle and the pressing frequency according to the specific scores corresponding to each moment of the key body angles and the pressing frequency to obtain the rhythm matching index and the correlation index; obtaining the second quantization scoring standard as the consistency relationship data according to the rhythm matching index and the correlation index.

[0013] In an embodiment of the present invention, according to the consistency relationship data, an auxiliary CPR training view with the pressing time displayed in the clockwise circumferential direction is generated to guide the user to coordinate the CPR actions, including: respectively performing correlation detection on the consistency relationship data corresponding between different body key angles and between each body key angle and the pressing frequency; when the consistency relationship data is greater than the corresponding target setting value, a strong consistency association of the consistency relationship data is obtained; generating corresponding display features corresponding to the auxiliary CPR training view according to the specific scores; generating a connection curve between the display features with strong consistency associations in the auxiliary CPR training view according to the display features with strong consistency associations corresponding to each moment; and generating an auxiliary CPR training view with the pressing time displayed in the clockwise circumferential direction according to the display features and the connection curve to guide the user to coordinate the CPR actions.

[0014] To achieve the above object and other related objects, the present invention further provides a visual analysis system for auxiliary CPR training, including: an acquisition module for acquiring the user's posture; a first detection module for performing CPR posture detection on the user's posture through a constructed target autoencoder model; a second detection module for performing CPR preparation posture detection on the CPR posture when the user's posture is a CPR posture; a mode switching module for entering the real-time training mode when the CPR posture is a CPR preparation posture; a key angle calculation module for extracting the three-dimensional coordinates of the user's skeletal key points in the real-time training mode to obtain the coordinates of the heel of the palm key point and the corresponding body key angles; a hand movement calculation module for obtaining the hand movement trajectory and range within a preset time according to the coordinates of the heel of the palm key point; a pressing parameter calculation module for obtaining the corresponding pressing frequency according to the hand movement trajectory and range; a score calculation module for respectively calculating the specific scores corresponding to each moment of the body key angles and the pressing frequency according to the body key angles and the pressing frequency; a consistency calculation module for calculating the consistency relationship data between each body key angle and between the body key angle and the pressing frequency according to the specific scores corresponding to each moment of the body key angles and the pressing frequency; and a view generation module for generating an auxiliary CPR training view with the pressing time displayed in the clockwise circumferential direction according to the consistency relationship data to guide the user to coordinate the CPR actions.

[0015] As described above, a visual analysis method and system for assisting CPR training according to the present invention has the following beneficial effects: By means of a pre-trained autoencoder model, combining the characteristics such as the spatial consistency and temporal coherence of CPR actions to optimize the loss function, it can effectively guide users to complete a roughly accurate CPR posture, which is more real-time and targeted than traditional static-template-based methods; By analyzing in real time and generating various visual cue features and pressing consistency relationship diagrams of CPR actions, and providing multi-dimensional visual feedback (including visual cues, charts, and voices), users can clearly understand the deficiencies and improvement directions of the current actions, significantly improving the efficiency and effect of training; By analyzing the consistency relationship between the three key body angles and frequency changes based on the consistency quantification standard of CPR action points, users can better specifically understand the inaccurate and uncoordinated parts in the actions, solving the problems of rough evaluation criteria and inability to accurately quantify in the process of assisting CPR training. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic flowchart of a visual analysis method for assisting CPR training provided by an embodiment of the present invention.

[0017] Figure 2 It is a schematic diagram showing a view of assisting CPR training provided by an embodiment of the present invention.

[0018] Figure 3 It shows a provided by an embodiment of the present invention Figure 2 An enlarged schematic diagram of areas c1, c2, and c3 in it.

[0019] Figure 4 It is a block diagram showing the structure of a visual analysis system for assisting CPR training provided by an embodiment of the present invention.

[0020] Figure 5 It is a schematic diagram showing the structure of an electronic device provided by an embodiment of the present invention.

[0021] ELEMENT LABEL DESCRIPTION Electronic device 1; Visual analysis system 11; Memory 12; Processor 13; Acquisition module 111; First detection module 112; Second detection module 113; Mode switching module 114; Key angle calculation module 115; Hand movement calculation module 116; Pressing parameter calculation module 117; Score calculation module 118; Consistency calculation module 119; View generation module 120. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0023] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Therefore, only the components related to the present invention are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0024] In the following description, a large number of details are explored to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0025] Please refer to Figure 1 , the present invention provides a visual analysis method for assisting CPR training, including: Step S10: Obtain the user's posture; Step S20: Detect the CPR posture of the user's posture through the constructed target autoencoder model; Step S30: When the user's posture is the CPR posture, detect the CPR preparation posture for the CPR posture; Step S40: When the CPR posture is the CPR preparation posture, enter the real-time training mode; Step S50: In the real-time training mode, extract the three-dimensional coordinates of the user's skeletal key points to obtain the coordinates of the heel of the palm key point and the corresponding key body angles; Step S60: According to the coordinates of the heel of the palm key point, obtain the hand movement trajectory and range within a preset time; Step S70: According to the hand movement trajectory and range, obtain the corresponding compression frequency; Step S80: According to the key body angles and the compression frequency, calculate the specific scores corresponding to each moment of the key body angles and the compression frequency respectively; Step S90: According to the specific scores corresponding to each moment of the key body angles and the compression frequency, calculate the consistency relationship data between the key body angles and between the key body angles and the compression frequency; Step S100: Generate an auxiliary CPR training view in which the pressing time is displayed in the clockwise direction according to the consistency relationship data to guide the user to coordinate the CPR actions.

[0026] It is not difficult to find from the above steps that in order to implement auxiliary CPR training, the user's posture can be obtained in advance by using a camera, and further, it can be determined whether the current user's posture belongs to the CPR posture through the target autoencoder model; when the user's posture is the CPR posture, further judgment is made on the CPR posture to determine whether the current CPR posture is the CPR preparation posture. If it is not the CPR preparation posture, the user can be guided to assume the correct CPR preparation posture. After the CPR preparation posture is completed, the real-time training mode is entered. By extracting the three-dimensional coordinates of the skeletal key points in the CPR action picture captured by the camera, and based on the three-dimensional coordinates of the skeletal key points, the coordinates of the heel of the palm key point are obtained, and then the corresponding body key angles are obtained through the coordinates of the heel of the palm key point. And through the coordinates of the heel of the palm key point, the hand movement trajectory and range within a preset time are further calculated to determine the corresponding pressing frequency according to the hand movement trajectory and range, so as to calculate the respective specific scores according to the body key angles and the pressing frequency, and determine the consistency relationship data between the body key angles and between the body key angles and the pressing frequency through the specific scores, and establish an auxiliary CPR training view according to the consistency relationship data, and guide the user to coordinate the CPR actions in a visual way. Specifically, when each key frame in the auxiliary CPR training view is generated in the order of pressing time, the generated view is displayed in the clockwise direction indicated by the clock, so as to effectively highlight the urgency and criticality of time in the first aid process. Through the above visual analysis method, multi-dimensional visual feedback can be provided for auxiliary CPR training, so that the user can clearly understand the deficiencies of the current actions and the further improvement directions, so as to significantly improve the training efficiency and effect; and based on the consistency quantification standard of the CPR action points, the consistency relationship between the body key angles and between the body key angles and the frequency change is analyzed to better enable the user to specifically understand the inaccurate and uncoordinated parts of the actions, thus solving the problem of rough evaluation criteria and inability to accurately quantify in the process of auxiliary CPR training.

[0027] Figure 1 The flowchart of the visual analysis method for auxiliary CPR training in an exemplary embodiment of the present application is shown, including steps S10 - step 100. The technical solution of the present application will be elaborated in detail below in conjunction with Figure 1 to elaborate the technical solution of the present application in detail.

[0028] First, execute steps S10 and S20 to obtain the user's posture; perform CPR posture detection on the user's posture through the constructed target autoencoder model.

[0029] When obtaining the user's pose by camera shooting, in order to determine the nature of the user's pose, it is also necessary to judge whether the user's pose is a CPR pose according to the user's pose. During the recognition and guidance process, the user's pose can be detected by the target autoencoder model, and when the user's pose is a CPR pose, the current CPR pose can be further detected for the CPR preparation pose, so as to start the real-time training mode when the user's CPR pose is a CPR preparation pose.

[0030] In an embodiment of the present invention, the target loss function corresponding to the target autoencoder model includes a cross-entropy loss function, a spatial consistency loss function, and a temporal coherence loss function.

[0031] Specifically, the calculation formula of the spatial consistency loss function is: ; the calculation formula of the temporal coherence loss function is: ; where represents the time step, represents the number of skeletal key points, represents the number of time steps, represents the number of key points at each time step, represents the predicted key point coordinates, represents the standard key point coordinates, represents the Euclidean distance, represents the total number of time steps, represents the change in the position of key points between two consecutive frames.

[0032] During the training process of the autoencoder model, considering the standardization and seriousness of the CPR action different from general sports actions, as well as its inherent characteristics of spatial consistency and temporal coherence, the cross-entropy loss function of the target autoencoder model can be optimized to significantly improve the action recognition accuracy of the autoencoder model and its adaptability to the CPR action.

[0033] For spatial consistency, its corresponding spatial consistency loss function is defined as , and it is necessary to calculate the distance between the predicted skeletal key points and the standard key point coordinates corresponding to the standard pose, as well as the stability of the relative positions of these key points. Assume that at time step , the predicted coordinate of the th key point is , and the standard coordinate of the th key point is , the spatial consistency loss can be defined as: . Where represents the number of time steps, Denoted as the number of key points for each time step, Denoted as the predicted key points, Denoted as the standard key points, Denoted as the Euclidean distance.

[0034] For temporal coherence, its corresponding temporal coherence loss function is defined as , and it is necessary to calculate the time step (i.e., the difference between consecutive frames) to ensure the smoothness and fluency of the action. Assume that at time step , the predicted key point coordinates are , then the temporal coherence loss can be defined as: . Among them, Denoted as the total number of time steps, Calculates the change in the position of key points between two consecutive frames, penalizes large changes to ensure the coherence of the action.

[0035] According to the basic formula of the cross-entropy loss function: ; among them, is the standard label, is the probability predicted by the model. By adding the spatial consistency loss function and the temporal coherence loss function , the objective loss function can be obtained as: ; among them, Denoted as the weight of the spatial loss function during training, Denoted as the weight of the temporal loss function during training. Thus, by optimizing the objective loss function L, the accuracy of action recognition of the target autoencoder model and its adaptability to CPR actions can be significantly improved.

[0036] Next, execute step S30 and step S40. When the user's posture is a CPR posture, perform CPR preparation posture detection on the CPR posture; when the CPR posture is a CPR preparation posture, enter the real-time training mode.

[0037] After performing CPR posture detection on the user's posture through the autoencoder model and determining that the user's posture is a CPR posture, then perform CPR preparation posture detection on the current CPR posture, so as to enable the real-time training mode when the CPR posture is a CPR preparation posture and perform visual analysis on the user's CPR posture.

[0038] In step S30, when the user's posture is a CPR posture, performing CPR preparation posture detection on the CPR posture includes: Step S301: When the user's posture is a CPR posture, obtain the three-dimensional coordinates of the skeletal key points through the Mediapipe recognition algorithm; Step S302: Obtain the similarity between the CPR posture and the standard posture based on the three-dimensional coordinates of the skeletal key points; Step S303: Monitor the similarity against a threshold; Step S304: When the similarity meets the standard, the CPR posture is the CPR preparation posture.

[0039] In this embodiment, when detecting the CPR preparation posture of the CPR posture and determining that the CPR posture is the CPR preparation posture, the Mediapipe recognition algorithm is used to recognize the CPR posture to obtain the three-dimensional coordinates of the skeletal key points. Based on the obtained three-dimensional coordinates of the skeletal key points, the similarity between the current CPR posture and the standard posture can be further calculated. Thus, it is possible to compare the similarity between the CPR posture and the standard posture against a threshold, and when the similarity meets the standard, the CPR posture is the CPR preparation posture. Additionally, when the similarity does not meet the standard, the user can be prompted to adjust the action through voice and text.

[0040] Among them, the MediaPipe gesture recognition algorithm is an efficient solution based on deep learning and computer vision. It uses a pre-trained model provided by Google to achieve real-time hand detection and gesture recognition.

[0041] In a preferred embodiment of the present invention, by extracting the three-dimensional coordinates of the skeletal key points when the user's posture is the CPR posture, the three-dimensional coordinates of 33 skeletal key points of the user are used as input and input into the calculation model to calculate the similarity S between the user's current CPR posture and the standard posture. When the system determines that the user's posture meets the standard and allows entry into the real-time training mode; otherwise, the user is prompted to adjust the action through voice and text. Among them, represents the three-dimensional coordinates of the skeletal key points corresponding to the standard posture; represents the three-dimensional coordinates of the skeletal key points of the current user; represents the similarity threshold. represents the error deviation floating value between the user's posture and the actual standard posture.

[0042] Next, step S50 is executed. In the real-time training mode, the three-dimensional coordinates of the skeletal key points of the user are extracted to obtain the coordinates of the palm root key points and the corresponding key body angles.

[0043] In the real-time training mode, by obtaining the CPR posture captured by the camera, the three-dimensional coordinates of the user's skeletal key points can be extracted through, for example, the Mediapipe recognition algorithm. And based on the three-dimensional coordinates of the user's skeletal key points, the coordinates of the palm root key point and the coordinates of other skeletal key points are determined. Further, through inverse trigonometric function calculation, the respective body key angles corresponding to the coordinates of the palm root key point are obtained.

[0044] Preferably, the body key angles include: the first included angle between the upper arm and the body, the second included angle between the upper arm and the forearm, and the third included angle between the body and the thigh.

[0045] Next, steps S60 and S70 are executed. According to the coordinates of the palm root key point, the hand movement trajectory and range within a preset time are obtained; according to the hand movement trajectory and range, the corresponding pressing frequency is obtained.

[0046] In step S60, obtaining the hand movement trajectory and range within a preset time according to the coordinates of the palm root key point includes: Step S601: Draw the hand movement trajectory according to the coordinates of the palm root key point; Step S602: Calculate the hand movement range corresponding to the hand movement trajectory within a preset time through the convex hull algorithm.

[0047] In this embodiment, after obtaining the coordinates of the palm root key point, the hand movement trajectory can be drawn according to the coordinates of the palm root key point. For the hand movement trajectory, the convex hull algorithm is further used for calculation to obtain the hand movement range corresponding to the hand movement trajectory within a preset time. In addition, according to the hand movement trajectory and range corresponding to the coordinates of the palm root key point, the pressing and rebounding sufficiency can be further calculated and displayed in the auxiliary CPR training view.

[0048] The calculation formula for the pressing and rebounding sufficiency is: ; where represents the maximum pressing and rebounding height, represents the minimum pressing and rebounding height, represents the pressing and rebounding sufficiency.

[0049] Next, step S80 is executed. According to the body key angles and the pressing frequency, the specific scores corresponding to each moment of the body key angles and the pressing frequency are calculated respectively.

[0050] After obtaining the body key angles and the pressing frequency, the specific scores corresponding to each moment of the respective body key angles can be calculated, and the specific scores corresponding to each moment of the pressing frequency can be calculated to obtain their respective specific scores.

[0051] In step S80, according to the body key angles and the pressing frequency, calculate the specific scores corresponding to each moment of the body key angles and the pressing frequency respectively, including: Step S801: Obtain the first mean value and the first standard deviation within the standard angle range according to the upper limit value and the lower limit value of the standard angle range corresponding to the body key angles; wherein, the upper limit value and the lower limit value of the standard angle range are respectively extracted and calculated from the medical CPR standard video; Step S802: Obtain the first normal distribution score according to the first mean value, the first standard deviation and the corresponding body key angles; Step S803: Perform normalization processing on the first normal distribution score to obtain the specific score corresponding to each moment of the body key angles; Step S804: Obtain the second mean value and the second standard deviation within the standard pressing frequency range according to the pressing frequency and the upper limit value and the lower limit value of the standard pressing frequency range specified by the medical CPR standard; Step S805: Obtain the second normal distribution score according to the second mean value, the second standard deviation and the corresponding pressing frequency; Step S806: Perform normalization processing on the second normal distribution score to obtain the specific score corresponding to each moment of the pressing frequency.

[0052] In this embodiment, before calculating the consistency relationship data, it is necessary to obtain the specific scores corresponding to each moment of each body key angle and the pressing frequency respectively. For each key index (such as the first included angle Angle1 between the upper arm and the body, the second included angle Angle2 between the upper arm and the lower arm, the third included angle Angle3 between the body and the thigh, the pressing frequency Frequency), according to the statistical law, each index follows the normal distribution. Therefore, according to the difference between its actual value and the standard range, the probability density function of the normal distribution can be used to estimate the score corresponding to the value .

[0053] Specifically, first calculate the mean value within the standard range , and the formula is: ; wherein, and respectively represent the upper limit value and the lower limit value of the standard angle range extracted and calculated from the medical CPR standard video. For example: The AHA stipulates that the standard range of the pressing frequency Frequency is 100 - 120 times per minute. Therefore, for Frequency, is 100, is 120, and further estimate the standard deviation , and the formula is: , that is, the estimated standard deviation is half of the standard value range. Then calculate the score corresponding to the value , and the score under the normal distribution is obtained as: . Then, normalize the normal distribution scoring criteria. Since the deviation degrees of each index may be different, in order to make the scores comparable, the scores can be normalized. Let the normalized score be , and the value range is specified between 0 and 100, which is expressed by the formula: ; where and represent the minimum and maximum values of all index scores respectively. The normalized score can reflect the deviation degree of each index and has a unified scale.

[0054] Next, execute step S90, and calculate the consistency relationship data between each body key angle and between the body key angle and the pressing frequency according to the specific scores corresponding to each moment of the body key angle and the pressing frequency.

[0055] In step S90, according to the specific scores corresponding to each moment of the body key angle and the pressing frequency, calculate the consistency relationship data between each body key angle and between the body key angle and the pressing frequency, including: Step S901: Calculate the consistency between different body key angles according to the specific scores corresponding to each moment of the body key angle and the pressing frequency, and obtain the CPR synchronization index and the CPR stability index; Step S902: Use the CPR synchronization index and the CPR stability index to obtain the first quantization scoring criteria as the consistency relationship data; Step S903: Calculate the consistency between each body key angle and the pressing frequency according to the specific scores corresponding to each moment of the body key angle and the pressing frequency, and obtain the rhythm matching index and the correlation index; Step S904: Use the rhythm matching index and the correlation index to obtain the second quantization scoring criteria as the consistency relationship data.

[0056] In this embodiment, when the specific scores corresponding to each moment of the body key angle and the pressing frequency are obtained, when calculating the consistency relationship data between each body key angle and between the body key angle and the pressing frequency, the consistency between the body key angles and the consistency between the body key angle and the pressing frequency can be obtained.

[0057] When calculating the consistency between key body angles, it is necessary to calculate two parts: the synchronization index SI and the stability index STI. The calculation formula for the CPR synchronization index SI is: ; where and represent the scores of angle and angle at time respectively. and represent the average scores of angle and angle respectively. is the length of the time window. If is closer to 1, it indicates that the change trends of the two angles are more consistent; the closer it is to -1, the more opposite the change trends are; and close to 0 indicates no significant correlation. The calculation formula for the CPR stability index STI is: ; where: is the standard deviation of angle , is the theoretically reasonable range of angle (for example, the angle between the upper arm and the forearm Angle2 is 170 - 180 degrees, Range = 10). If is closer to 1, it indicates that the angle change is more stable; the closer it is to 0, the greater the fluctuation. By synthesizing the synchronization index SI and the stability index STI, the first quantitative scoring criterion for the consistency between key body angles can be obtained . If is closer to 1, it indicates a higher consistency between key body angles. The formula is expressed as: .

[0058] When calculating the consistency between key body angles and the compression frequency, it is necessary to calculate two parts: the rhythm matching index RMI and the correlation index CI. The calculation formula for the CPR rhythm matching index RMI is: ; where the change phase of a certain angle at time and the change phase of the frequency at time can be extracted through Fourier transform. If is closer to 1, it indicates that the angle change matches the frequency rhythm better. The calculation formula for the CPR correlation index CI is: ; where is the score of the frequency at time , The average score for frequency. If the CI value is closer to 1, it indicates that the angle change is more correlated with the frequency change. By synthesizing the above two indicators of the key body angles and the pressing frequency, the second quantitative scoring standard for the consistency between the angle and the frequency can be obtained. , if the value is closer to 1, it indicates a higher consistency between the angles. The formula is expressed as: .

[0059] Next, perform step S100 to generate an auxiliary CPR training view with the pressing time displayed in the clockwise circumferential direction according to the consistency relationship data to guide the user to coordinate the CPR actions.

[0060] In step S100, according to the consistency relationship data, generate an auxiliary CPR training view with the pressing time displayed in the clockwise circumferential direction to guide the user to coordinate the CPR actions, including: Step S1001: Perform correlation detection on the corresponding consistency relationship data between different key body angles and between each key body angle and the pressing frequency respectively; Step S1002: When the consistency relationship data is greater than the corresponding target setting value, then the consistency relationship data is a strong consistency association; Step S1003: Generate corresponding display features corresponding to the auxiliary CPR training view according to the specific scores; Step S1004: Generate a connection curve between the display features with strong consistency associations in the auxiliary CPR training view according to the display features with strong consistency associations corresponding to each moment; Step S1005: Generate an auxiliary CPR training view with the pressing time displayed in the clockwise circumferential direction according to the display features and the connection curve to guide the user to coordinate the CPR actions.

[0061] In this embodiment, after obtaining the consistency relationship data of the first quantitative scoring standard and the second quantitative scoring standard for the pressing consistency in the quantitative CPR process, it is also necessary to perform correlation detection on the consistency relationship data. Specifically, for example, when ACS≥0.5 or AFCS≥0.5, it can be expressed as a strong consistency association, and the display feature can adopt a hollow graph, and the strong consistency association is represented by a curve connection. When ACS<0.5 or AFCS<0.5, it can be expressed as a weak consistency association. Since the consistency association strength between the two indicators is low, there is no curve connection between the two indicators. After determining the display features and the connection curve, output them to the auxiliary CPR training view, realizing the visualization of the consistency relationship, and thus guiding the user to coordinate the CPR actions according to the connection situation of the consistency relationship.

[0062] Please refer to Figure 2 , Figure 2 In an embodiment shown, an auxiliary CPR training view is displayed. The auxiliary CPR training view adopts a circular structure and draws on the concept of a clock to accurately calibrate each key frame in the CPR process, aiming to highlight the urgency and criticality of time in the first aid process. This view consists of an outer circle and an inner circle, and the two circles respectively divide different sub-views: the outer circle shows two major categories of data related to the external curve and pressing consistency, reflecting the core indicators of pressing depth and frequency in the CPR process and the coherence of pressing actions; the inner circle shows a graph of the degree of chest wall recoil during pressing, used to simulate the synchronization between the change trajectory of pressing depth and the curve of cardiac rhythm change. Specifically, a snapshot of the current CPR operation is displayed in the central area of the auxiliary CPR training view; in area a1, the hand movement trajectory and its movement range during this period are drawn; in area a2, the current pressing depth and pressing frequency data are drawn, and corresponding data progress bars are drawn, and the current time is displayed in conjunction with the time pointer. In area b, that is, the outermost area of the view, an external pressing data curve is drawn. Among them, the distance between the key point position of the external curve and the outer circle is proportional. The greater the pressing depth, the farther the key point is from the outer circle; the thickness of the curve is proportional to the pressing frequency, and the higher the pressing frequency, the thicker the curve. Areas c1, c2, and c3 are located between the outer circle and the inner circle and are pressing consistency curves. This curve reflects the consistency of CPR actions based on CPR medical standards and the spatio-temporal coherence characteristics of CPR actions, combined with three key angles and four data points of pressing frequency. Area d, which is the middle part of the auxiliary CPR training view, is a drawn curve of the degree of internal chest wall recoil. According to CPR medical standards, the degree of chest wall recoil during CPR pressing is calculated to reflect the sufficiency of pressing, thereby characterizing the effectiveness of the CPR process during this period. The distance of the key point of this internal curve from the inner circle is inversely proportional to the degree of recoil. The closer the distance is to the inner circle, the higher the degree of recoil. When the degree of recoil reaches 100%, it means that the key point is located at the inner circle. Area e can be the calculation result based on the above various types of data, identifying the core indicators affecting the current CPR score and pointing out the most posture or action that needs improvement. For example: providing feedback to the user through intuitive guidance text and supporting voice announcements for multi-dimensional real-time feedback.

[0063] Through Figure 4 It can be seen from area d in

[0064] Please refer to Figure 3 , Figure 3In one embodiment, four types of data, namely, the first angle between the upper arm and the body, the second angle between the upper arm and the lower arm, the third angle between the body and the thigh, and the pressing frequency, are sequentially displayed along the radial direction of the auxiliary CPR training view. And these four types of data are abstracted into display features according to the specific score of the index. Among them, the display features are divided into three shapes: square, triangle, and circle. According to the CPR medical standard, it can be defined that: the square represents the index with the highest score, the triangle represents the index with the lowest score, and the remaining circles represent the other two indexes with moderate scores. Moreover, the graphics corresponding to the display features are divided into two modes: solid and hollow. The solid graphic indicates that the performance of the current index fails to form a consistent association with other indexes, while the hollow graphic indicates that there is a consistent association between the current index and other indexes, highlighting the action coherence during the CPR process. For two hollow graphics, if there is a consistent association between the two indexes, that is, the two indexes can reflect the characteristics such as action coherence and posture coordination during the CPR process, then the two graphics are connected by an arc, and the protruding direction of the arc can be used to represent the direction of time advancement. In addition, between two consecutive key frames of the same type of data, the change of the index with the best or worst score is reflected by a connecting curve. That is, the connection between square graphics represents the change of the index with the highest score, and the connection between triangle graphics represents the change of the index with the lowest score.

[0065] Please refer to FIG. 4. The present invention also provides a visual analysis system 11 for assisting CPR training, including: an acquisition module 111 for acquiring the user's posture; a first detection module 112 for detecting the CPR posture of the user's posture through a constructed target autoencoder model; a second detection module 113 for detecting the CPR preparation posture of the CPR posture when the user's posture is the CPR posture; a mode switching module 114 for entering the real-time training mode when the CPR posture is the CPR preparation posture; a key angle calculation module 115 for extracting the three-dimensional coordinates of the user's skeletal key points in the real-time training mode to obtain the coordinates of the palm root key points and the corresponding body key angles; a hand movement calculation module 116 for obtaining the hand movement trajectory and range within a preset time according to the coordinates of the palm root key points; a pressing parameter calculation module 117 for obtaining the corresponding pressing frequency according to the hand movement trajectory and range; a score calculation module 118 for respectively calculating the specific scores corresponding to each moment of the body key angles and the pressing frequency according to the body key angles and the pressing frequency; a consistency calculation module 119 for calculating the consistency relationship data between the body key angles and between the body key angles and the pressing frequency according to the specific scores corresponding to each moment of the body key angles and the pressing frequency; and a view generation module 120 for generating an auxiliary CPR training view with the pressing time displayed in the clockwise circumferential indication direction according to the consistency relationship data to guide the user to coordinate the CPR actions.

[0066] It should be noted that the visual analysis system 11 for assisting CPR training provided in the above embodiments and the visual analysis method for assisting CPR training provided in the above embodiments belong to the same concept. The specific manners in which each module and unit perform operations have been described in detail in the method embodiments, and will not be elaborated here. In practical applications, the visual analysis system 11 for assisting CPR training provided in the above embodiments can, as needed, allocate the above functions to different functional modules, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above. This is not limited here either.

[0067] Please refer to Figure 5 , the electronic device 1 may include a memory 12, a processor 13, and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a visual analysis program for assisting CPR training.

[0068] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as: SD or DX memory, etc.), magnetic memory, magnetic disk, optical disc, etc. The memory 12 may be an internal storage unit of the electronic device 1 in some embodiments, such as the mobile hard disk of the electronic device 1. The memory 12 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 1. Further, the memory 12 may include both the internal storage unit and the external storage device of the electronic device 1. The memory 12 can be used not only to store application software installed on the electronic device 1 and various types of data, such as the code for visual analysis of assisting CPR training, but also to temporarily store data that has been output or will be output.

[0069] In some embodiments, the processor 13 may be composed of an integrated circuit. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control core of the electronic device 1. It uses various interfaces and circuits to connect all components of the entire electronic device 1. By running or executing programs or modules stored in the memory 12 (such as a visual analysis program for assisting CPR training, etc.), and by calling the data stored in the memory 12, it executes various functions of the electronic device 1 and processes data.

[0070] The processor 13 executes the operating system of the electronic device 1 and various installed application programs. The processor 13 executes the application programs to implement the steps in the above-mentioned visual analysis method for assisting CPR training.

[0071] Exemplarily, the computer program may be divided into one or more modules. The one or more modules are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and these instruction segments are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into an acquisition module 111, a first detection module 112, a second detection module 113, a mode switching module 114, a key angle calculation module 115, a hand movement calculation module 116, a pressing parameter calculation module 117, a score calculation module 118, a consistency calculation module 119, and a view generation module 120.

[0072] The above-mentioned integrated units implemented in the form of software function modules can be stored in a computer-readable storage medium. The computer-readable storage medium may be non-volatile or volatile. The above-mentioned software function modules are stored in a storage medium and include several instructions to enable a computer device (which may be a personal computer, a computer device, or a network device, etc.) or a processor to execute some functions of the visual analysis method for assisting CPR training described in various embodiments of this application.

[0073] In summary, a visual analysis method and system for assisting CPR training disclosed by the present invention can effectively guide users to complete a roughly accurate CPR posture by means of a pre-trained autoencoder model and optimizing the loss function in combination with the characteristics such as the spatial consistency and temporal coherence of CPR actions, which is more real-time and targeted than the traditional static-template-based method. By analyzing in real time and generating various visual cue features and pressing consistency relationship diagrams of CPR actions, and providing multi-dimensional visual feedback (including visual cues, charts and voices), users can clearly understand the deficiencies and improvement directions of the current actions, significantly improving the efficiency and effect of training. By analyzing the consistency relationship between the three key body angles and the frequency changes based on the consistency quantification standard of CPR action points, users can better specifically understand the inaccurate and uncoordinated parts of the actions, solving the problem of rough evaluation criteria and inability to accurately quantify in the process of assisting CPR training. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0074] The above embodiments are only illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes completed by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by the present invention should still be covered by the claims of the present invention.

Claims

1. A visual analysis method for assisting CPR training, characterized in that: include: Get user posture; Performing CPR posture detection on the user posture through the constructed target autoencoder model; When the user posture is a CPR posture, performing CPR preparation posture detection on the CPR posture; When the CPR posture is the CPR preparation posture, the real-time training mode is entered; In real-time training mode, the three-dimensional coordinates of the user's skeletal key points are extracted to obtain the palm base key point coordinates and the corresponding body key angles; According to the palm base key point coordinates, the hand movement trajectory and range within a preset time are obtained; According to the hand movement trajectory and range, a corresponding pressing frequency is obtained; According to the key body angle and the pressing frequency, respectively calculating specific scores corresponding to the key body angle and the pressing frequency at each moment; According to the specific scores corresponding to the key body angles and the pressing frequency at each moment, the consistency relationship data between the key body angles and between the key body angles and the pressing frequency are calculated; Based on the consistency relationship data, an auxiliary CPR training view is generated in which the compression time is displayed in the direction indicated by the clock circumference, so as to guide the user to coordinate CPR actions.

2. The visual analysis method for assisting CPR training according to claim 1, characterized in that: The target loss functions corresponding to the target autoencoder model include a cross entropy loss function, a spatial consistency loss function, and a temporal coherence loss function.

3. The visual analysis method for assisting CPR training according to claim 2, characterized in that: The calculation formula of the spatial consistency loss function is: ; The calculation formula of the temporal coherence loss function is: ; in, is represented as a time step, Expressed as the number of bone key points, Expressed as the number of time steps, Expressed as the number of key points at each time step, Expressed as the predicted key point coordinates, Expressed as standard keypoint coordinates, Expressed as the Euclidean distance, Expressed as the total number of time steps, It is expressed as the change in the position of the key point between two consecutive frames.

4. The visual analysis method for assisting CPR training according to claim 1, characterized in that: When the user posture is a CPR posture, performing CPR preparation posture detection on the CPR posture includes: When the user's posture is the CPR posture, the three-dimensional coordinates of the skeleton key points are obtained through the Mediapipe recognition algorithm; Obtaining the similarity between the CPR posture and the standard posture according to the three-dimensional coordinates of the skeleton key points; Performing threshold monitoring on the similarity; When the similarity reaches the standard, the CPR posture is the CPR preparation posture.

5. The visual analysis method for assisting CPR training according to claim 1, characterized in that: The key body angles include: a first angle between the upper arm and the body, a second angle between the upper arm and the forearm, and a third angle between the body and the thigh.

6. The visual analysis method for assisting CPR training according to claim 1, characterized in that: According to the palm base key point coordinates, the hand movement trajectory and range within a preset time are obtained, including: Drawing a hand motion trajectory according to the palm base key point coordinates; The hand motion range corresponding to the hand motion trajectory within the preset time is calculated through the convex hull algorithm.

7. The visual analysis method for assisting CPR training according to claim 1, characterized in that: According to the key body angle and the pressing frequency, specific scores corresponding to the key body angle and the pressing frequency at each moment are calculated respectively, including: According to the key body angle and the corresponding upper limit and lower limit of the standard angle range, a first mean and a first standard deviation within the standard angle range are obtained; wherein the upper limit and the lower limit of the standard angle range are respectively extracted and calculated from a medical CPR standard video; Obtaining a first normal distribution score according to the first mean, the first standard deviation and the corresponding key body angles; Normalizing the first normal distribution score to obtain a specific score corresponding to each moment of the key body angle; Obtaining a second mean and a second standard deviation within the standard compression frequency range according to the compression frequency and an upper limit and a lower limit of a standard compression frequency range specified by a corresponding medical CPR standard; Obtaining a second normal distribution score according to the second mean, the second standard deviation and the corresponding pressing frequency; The second normal distribution score is normalized to obtain a specific score corresponding to the pressing frequency at each moment.

8. The visual analysis method for assisting CPR training according to claim 1, characterized in that: According to the specific scores corresponding to the key body angles and the pressing frequency at each moment, the consistency relationship data between the key body angles and between the key body angles and the pressing frequency are calculated, including: According to the specific scores corresponding to the key body angles and the compression frequency at each moment, the consistency between the different key body angles is calculated to obtain the CPR synchronization index and the CPR stability index; According to the CPR synchronization index and the CPR stability index, a first quantitative scoring standard is obtained as the consistency relationship data; According to the specific scores corresponding to the key body angles and the pressing frequencies at each moment, the consistency between each key body angle and the pressing frequency is calculated to obtain a rhythm matching index and a correlation index; According to the rhythm matching index and the correlation index, a second quantitative scoring standard is obtained as the consistency relationship data.

9. The visual analysis method for assisting CPR training according to claim 1, characterized in that: According to the consistency relationship data, an auxiliary CPR training view is generated in which the compression time is displayed in the direction indicated by the clock circumference to guide the user to coordinate CPR actions, including: Respectively performing correlation detection on the consistency relationship data corresponding to different key body angles and between each key body angle and the pressing frequency; When the consistency relationship data is greater than the corresponding target setting value, the consistency relationship data is obtained as a strong consistency association; generating, according to the specific score, a corresponding display feature corresponding to the auxiliary CPR training view; generating, according to the display features having the strong consistency association corresponding to each moment, a connection curve between the display features having the strong consistency association in the auxiliary CPR training view; Based on the display characteristics and the connection curve, an auxiliary CPR training view is generated in which the compression time is displayed in the direction indicated by the clock circumference, so as to guide the user to coordinate CPR actions.

10. A visual analysis system for assisting CPR training, characterized in that: include: An acquisition module, used to acquire user posture; A first detection module, configured to perform CPR posture detection on the user posture through a constructed target autoencoder model; A second detection module is used to perform CPR preparation posture detection on the CPR posture when the user posture is the CPR posture; A mode switching module, used for entering a real-time training mode when the CPR posture is a CPR preparation posture; The key angle calculation module is used to extract the three-dimensional coordinates of the user's skeletal key points in real-time training mode, and obtain the coordinates of the palm base key points and the corresponding body key angles; A hand motion calculation module, used to obtain the hand motion trajectory and range within a preset time according to the palm base key point coordinates; A pressing parameter calculation module, used to obtain a corresponding pressing frequency according to the hand movement trajectory and range; A score calculation module, used to calculate the specific scores corresponding to the key body angles and the pressing frequency at each moment according to the key body angles and the pressing frequency; A consistency calculation module, for calculating the consistency relationship data between the key body angles and between the key body angles and the pressing frequency according to the specific scores corresponding to the key body angles and the pressing frequency at each moment; as well as The view generation module is used to generate an auxiliary CPR training view in which the compression time is displayed in a clock circumferential direction according to the consistency relationship data, so as to guide the user to coordinate CPR actions.

Images