External chest compression real-time guiding method, system and device based on structured light and medium

Through structured light camera technology, the problem that a monocular camera cannot estimate the pixel depth of field is solved, the accuracy and reliability of chest compression is achieved, and more accurate compression depth and frequency indicators are provided, which improves the effect of first aid.

CN118948606APending Publication Date: 2024-11-15JIANGSU YUYUE MEDICAL EQUIP&SUPPLY CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411025585.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Traditional monocular camera methods cannot accurately estimate the pixel depth of field, resulting in reduced accuracy and reliability of machine vision technology assisting chest compression guidance, and the inability to directly estimate the depth of chest compression of the rescuer, affecting the effect of cardiopulmonary resuscitation.

Method used

The structured light camera is used to acquire video and obtain RGB images and depth of field images. Three-dimensional coordinates are obtained through alignment and processing, the foreground and background are separated, and the chest compression quality indicators are calculated, including the compression depth, posture and frequency, providing real-time guidance.

Benefits of technology

It improves the accuracy and reliability of chest compression guidance, ensures the accuracy of the depth of the rescuer's compression, provides faster and more accurate guidance, and improves the effect of first aid training and actual rescue.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118948606A_ABST
    Figure CN118948606A_ABST
Patent Text Reader

Abstract

The invention discloses a structured light-based external chest compression real-time guiding method, system and device and a medium. The method comprises the following steps: establishing a human body posture point physical model by using a posture detection model; obtaining an RGB image and a depth-of-field image by the structured light camera; obtaining depth-of-field data, and calculating three-dimensional coordinates of a human body posture point; separating and filtering the foreground and the background; enhancing according to a specific local area in the human body posture point physical model so as to track a pressing action track; and calculating the external chest compression quality index according to a result of tracking the compression action track. The system comprises a visual acquisition module, a main control module, a calculation module and a feedback module. According to the scheme, accurate pixel depth-of-field data can be obtained, so that the external chest compression depth of the rescuer in the external chest compression quality indexes obtained through subsequent calculation can be more accurately estimated, an accurate and reliable index is provided for real-time guidance of external chest compression, and quicker and more accurate guidance auxiliary work is provided for the rescuer.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of cardiopulmonary resuscitation, and in particular to a real-time guidance method, system, device and medium for chest compression based on structured light. Background Art

[0002] The statements in this section merely provide background technical information related to the present application and do not necessarily constitute prior art.

[0003] Chest compression is of irreplaceable importance in cardiopulmonary resuscitation. In the case of cardiac arrest, chest compression is the only available means to maintain cardiac circulation and ensure that oxygen and nutrients are delivered to critical organs such as the brain and heart. Through chest compression, a certain hemodynamics can be maintained, the survival window of cardiac arrest patients can be extended, and precious time can be bought for other first aid measures and further medical intervention. In addition, chest compression can also promote the heart to re-establish autonomous heart rhythm and improve the patient's survival rate. Therefore, chest compression is regarded as a vital step in the first aid process of cardiac arrest, and its timely and effective implementation can greatly improve the patient's prognosis and survival rate.

[0004] Using machine vision technology to assist chest compression is a potential method. By integrating cameras and image processing systems, it can effectively combine posture recognition technology to capture the rescuer's behavior trajectory, including the compression posture, frequency and number of chest compressions. The application of this technology can provide personalized guidance for the automatic external defibrillator (AED) system, through voice prompts and screen display of specific operating steps, thereby improving the accuracy and efficiency of rescue.

[0005] However, the traditional monocular camera method has some limitations that affect the application of machine vision technology-assisted chest compression guidance methods in AED systems. First, the monocular camera cannot accurately estimate the pixel depth of field, which has a negative impact on the separation effect of foreground and background, and may cause the AED system to be unable to accurately identify the rescuer's actions in complex environments or low light conditions, thereby reducing the accuracy and reliability of guidance. Second, the monocular camera is also unable to directly estimate the depth of the rescuer's chest compression, which is a key parameter that will directly affect the effectiveness of cardiopulmonary resuscitation.

[0006] In view of this, how to solve the problem that the traditional monocular camera method cannot accurately estimate the pixel depth of field, thus reducing the accuracy and reliability of guidance, and cannot directly estimate the depth of the rescuer's chest compression, thus directly affecting the effect of cardiopulmonary resuscitation, has become the subject to be studied and solved by the present invention. Summary of the invention

[0007] The purpose of the present invention is to provide a real-time guidance method, system, device and medium for chest compression based on structured light.

[0008] To achieve the above object, a first aspect of the present invention provides a real-time guidance method for chest compression based on structured light, the method comprising:

[0009] Use the posture detection model to establish a physical model of human posture points;

[0010] Construct a chest compression scene based on a structured light camera. The structured light camera collects the video of the rescuer performing chest compressions on the patient and obtains RGB images and depth of field images.

[0011] Align the RGB image and the depth image to obtain the depth data corresponding to the RGB image, and calculate the three-dimensional coordinates of the human body posture points;

[0012] Estimate the two-dimensional vector field of pixel displacement and depth change of RGB image and depth image respectively, and separate and filter the foreground and background of three-dimensional coordinates of human posture points;

[0013] Based on the results of foreground and background separation and filtering combined with the human body posture point physical model, the human body posture points during chest compression are tracked in real time, and specific local areas in the human body posture point physical model are enhanced to track the compression action trajectory;

[0014] Calculating a chest compression quality index based on the result of tracking the compression action trajectory, where the chest compression quality index includes at least compression depth and one or more of the following: compression posture, compression frequency, and number of compressions;

[0015] Provide text, image and / or voice prompt guidance based on chest compression quality indicators.

[0016] The second aspect of the present invention proposes a real-time guidance system for chest compression based on structured light, the real-time guidance system comprising a visual acquisition module, a main control module, a calculation module, and a feedback module;

[0017] The visual acquisition module uses a structured light camera, which includes an infrared camera and an RGB camera. The structured light camera collects a video of the rescuer performing chest compressions on the patient and obtains an RGB image and a depth of field image.

[0018] The main control module is used to obtain the RGB image and the depth image collected by the visual acquisition module, transmit the RGB image and the depth image to the calculation module, and output the control instruction of the result calculated by the calculation module to the feedback module;

[0019] A physical model of human posture points is established in the calculation module, and the RGB image and the depth of field image are aligned to obtain the depth of field data corresponding to the RGB image, and the three-dimensional coordinates of the human posture points are calculated; the two-dimensional vector fields of the RGB image and the depth of field image in pixel displacement and depth change are estimated respectively, and the foreground and background of the three-dimensional coordinates of the human posture points are separated and filtered; according to the results of the separation and filtering of the foreground and background combined with the physical model of the human posture points, the human posture points in the process of chest compression are tracked in real time, and the specific local areas in the physical model of the human posture points are enhanced to track the compression action trajectory; the chest compression quality index is calculated according to the result of tracking the compression action trajectory; and the CPR guidance index is sent to the main control module;

[0020] The feedback module includes an audio feedback module and a video feedback module, which are used to perform audio feedback and video feedback according to the control instructions of the main control module.

[0021] A third aspect of the present invention provides an emergency treatment device, which is the real-time chest compression guidance system based on structured light as described in the second aspect of the present invention.

[0022] A fourth aspect of the present invention provides a readable storage medium having a control program stored thereon. When the control program is executed by a processor, the processor executes the steps of the method described in the first aspect.

[0023] The relevant contents of the present invention are explained as follows:

[0024] 1. Through the implementation of the above technical solutions of the present invention, in order to solve the problem that the traditional monocular camera method cannot accurately estimate the pixel depth of field, thus reducing the accuracy and reliability of guidance, and cannot directly estimate the depth of chest compression by the rescuer, thus directly affecting the effect of cardiopulmonary resuscitation, an innovative method, system, device and medium for real-time guidance of chest compression based on structured light are designed; in particular, in the method, a chest compression scene based on a structured light camera is constructed, and the structured light camera is used to collect the video of the rescuer applying chest compression to the patient and obtain the RGB image and the depth of field image, and then a series of processing is performed on the RGB image and the depth of field image, including aligning the RGB image and the depth of field image, obtaining the depth of field data corresponding to the RGB image, and calculating the three-dimensional coordinates of the human body posture points, so as to obtain accurate pixel depth of field data for the subsequent real-time chest compression. The accuracy and reliability of guidance are guaranteed by basic data; the three-dimensional coordinates of the human body posture points are calculated after obtaining the human body posture points, and then the two-dimensional vector fields of pixel displacement and depth change of RGB images and depth of field images are estimated respectively, and the foreground and background of the three-dimensional coordinates of the human body posture points are separated and filtered, so that the depth of chest compression by the rescuer in the subsequent calculated chest compression quality indicators can be estimated more accurately, thereby providing accurate and reliable indicators for real-time guidance of chest compression, and providing rescuers with faster and more accurate guidance assistance; and considering the cumulative error in evaluating the CPR compression motion trajectory, the specific local area in the physical model of the human body posture points is enhanced to track the compression action trajectory, so as to ensure high correlation, eliminate cumulative errors, and more accurately calculate chest compression quality indicators such as compression depth.

[0025] 2. In the first aspect of the above technical solution, in the process of using the posture detection model to establish the human posture point physical model, a pre-trained weighted skeleton point detection model is used to detect the human posture points, and the human posture point physical model at least includes the joints in the rescuer's head, shoulders, elbows, palms, torso and legs. In this way, the joints of the key moving parts of the human body when the rescuer performs CPR chest compressions are covered, so as to improve the accuracy of the calculation of CPR chest compression indicators.

[0026] 3. In the first aspect of the above technical solution, in the process of constructing a chest compression scene based on a structured light camera, the structured light camera used includes an infrared camera and an RGB camera, which form a cross-structured light source projection and image sensor capture, and the cross-structured light source is projected to the rescuer being detected. The infrared camera and the RGB camera collect the video of the rescuer applying chest compression to the patient, capture the morphological changes of the light spot projected on the rescuer, and obtain the RGB image and the depth of field image. In this way, the composition of the structured light camera is improved to ensure that the subsequent operations such as merging, depth of field calculation, foreground and background filtering, and real-time tracking are obtained, providing the necessary scene conditions for the fact of the real-time guidance method of chest compression in the present invention.

[0027] 4. In the first aspect of the above technical solution, in the process of aligning the RGB image and the depth image to obtain the depth data corresponding to the RGB image, the following steps are included:

[0028] By analyzing the speckle structure reflected by the rescuer's surface, the distance between the rescuer and the light source is calculated based on the speckle ranging and the grayscale value of the pixel point of the digitized speckle pattern is obtained, that is, the width and depth of the speckle pattern;

[0029] The actual distance is estimated for the depth image, and the RGB image and the depth image are aligned to obtain the depth data of the RGB image.

[0030] The above steps are used to obtain more accurate depth of field data, which will help calculate the compression depth in the subsequent real-time tracking process, ensure the accuracy and reliability of the compression depth data, and further provide more standardized and reasonable guidance operations for the real-time guidance process of chest compressions.

[0031] 5. In the first aspect of the above technical solution, the process of calculating the distance between the rescuer and the light source according to the speckle ranging and obtaining the grayscale value of the pixel point of the digitized speckle pattern includes the following steps:

[0032] The grayscale function of the RGB image and the depth of field image is expressed as follows:

[0033] Among them, s represents the number of speckles, a represents the size of the speckles, I0 represents the image background intensity, usually I0 is 1, and the one-dimensional random variable (x k ,y k ) represents the center position of the kth speckle particle, which can be obtained by the uniform random distribution function rand, expressed as:

[0034] Where M and N represent the number of pixels in the X and Y directions, that is, the width and depth of the speckle pattern, respectively, and the one-dimensional random variable is spatially discretized, expressed as:

[0035] Where i=0, 1, ..., M-1, j=0, 1, ..., N-1.

[0036] This makes the depth of field data obtained more accurate.

[0037] 6. In the first aspect of the above technical solution, the process of estimating the actual distance of the depth image, aligning the RGB image and the depth image, and obtaining the depth data of the RGB image includes the following steps:

[0038] The actual distance is estimated for the depth image. The depth image pixels correspond to the width and depth of the speckle pattern, that is, M×N pixels, where each pixel (i, j) corresponds to an integer value d between 0 and the set resolution, d represents the depth digital value of the pixel relative to the center of the structured light camera, and the actual depth value z i,j Use the measured numeric depth value d i,j Convert according to the following equation:

[0039] The constants α, β, and μ in the formula are manually calibrated and adjusted in reality by using the undetermined coefficient method to calibrate the parameters in the structured light camera;

[0040] For the alignment of the RGB image and the depth image, given the depth value in the metric space and the intrinsic parameters obtained from the calibration process, the 3D coordinates of the point (x i,j ,y i,j , z i,j ) is associated with each pixel according to the following equation:

[0041] where f x and f y , respectively represent the focal lengths in the X and Y directions, and x0 and y0 represent the center pixel positions of the structured light camera in the X and Y directions, respectively.

[0042] 7. In the first aspect of the above technical solution, in the process of respectively estimating the two-dimensional vector field of pixel displacement and depth change of the RGB image and the depth image, and separating and filtering the foreground and background of the three-dimensional coordinates of the human posture point, the following steps are included:

[0043] The optical flow method is used to estimate the two-dimensional vector field of pixel displacement and depth change of RGB image and depth image respectively. The two-dimensional vector field reflects the change trend of the grayscale of each point on the image. The two-dimensional vector field is regarded as the instantaneous velocity field generated by the movement of grayscale pixels on the image plane. The information contained in the two-dimensional vector field is the instantaneous motion velocity vector information of each image point.

[0044] The optical flow vector is obtained by derivation of the image data of the two-dimensional vector field. The continuous movement vector of the foreground skeleton point on the RGB image is found based on the chest compression scene, and the foreground and background results obtained by RGB image recognition and the foreground and background results obtained by depth of field image recognition are obtained in the area where the depth of field movement vector remains unchanged.

[0045] By implementing this step, the process of separating and filtering the foreground and background of the three-dimensional coordinates of the human body posture points can be faster and the results can be more accurate. In this way, the foreground and background of the RGB image and the foreground and background of the depth of field image can be separated and filtered. The foreground and background of the RGB image and the foreground and background of the depth of field image after separation and filtering can make the subsequent real-time tracking of the human body posture points and the results more consistent with the actual compression depth of the rescuer during actual chest compression.

[0046] 8. In the first aspect of the above technical solution, in the process of respectively estimating the two-dimensional vector field of pixel displacement and depth change of the RGB image and the depth image, and separating and filtering the foreground and background of the three-dimensional coordinates of the human posture point, the following steps are included:

[0047] Consider the light intensity of a pixel I(x, y, t) in the first frame, moving a distance (dx, dy) to the next frame, using a time dt, expressed as: I(x, y, t) = I(x+dt, y+dt, t+dt), and Taylor expansion of this formula where v represents a second-order infinitesimal term, and after removing dt we obtain

[0048] Let u and v be the velocity vectors of the optical flow along the X-axis and Y-axis respectively,

[0049] make Respectively represent the partial derivatives of the grayscale of the image pixel along the X, Y, and T directions;

[0050] And it is deduced as I x u+I y , v+I t =0, where I x , I y , I t Both can be derived from the image data, and u and v are the required optical flow vectors;

[0051] Based on the chest compression scene, we search for the continuous movement vector of the foreground skeleton points on the RGB image, and the area where the depth of field movement vector remains unchanged, as follows:

[0052]

[0053] in, Represents the foreground and background results obtained by RGB image recognition, and Respectively represent the foreground and background recognized by the RGB image, u RGB and u RGB Represents the motion vector in the X-axis direction and the Y-axis direction, ||u RGB , v RGB ||2 represents the second normal form of the vector, thr RGB is the manually calibrated threshold; Represents the foreground and background results obtained by depth of field image recognition, Respectively represent the foreground and background recognized by the depth of field image, u Dep and v Dep Represents the motion vector in the X-axis direction and the Y-axis direction, ||u Dep , v Dep ||2 represents the second normal form of the vector, thr De p is the manually calibrated threshold.

[0054] 9. In the first aspect of the above technical solution, in the process of real-time tracking of human posture points during chest compression based on the results of separation and filtering of foreground and background combined with the physical model of human posture points, and enhancing according to specific local areas in the physical model of human posture points to track the trajectory of the pressing action, the rescuer's cyclic arm pressing action is used as a specific local area in the physical model of human posture points, and the arm pressing action is respectively for the rescuer's arm and torso, respectively based on the RGB image space and the depth of field image space, and the forward correlation of the action in the CPR process is established using Kalman filtering for enhancement to track the trajectory of the pressing action. Considering that in the CPR scenario, the recognition focus is on the cyclic arm pressing movement, therefore, the information across frames shows a high degree of correlation, and the action mainly occurs in a specific local area, and the position of the human body is relatively fixed. This feature helps to enhance the local recognition accuracy of the posture recognition model specially customized for the CPR environment.

[0055] 10. In the first aspect of the above technical solution, the variables defining the Kalman filter include the following:

[0056] Lock the target area: First, use a rectangular box to lock the arm area Box aand the torso area Box b , this area is locked in the same position area indicated by the RGB image space and the depth image space;

[0057] Define the observation space O: define the ten-dimensional state space (x, y, z, w, h, dx, dy, dz, dw, dh), the region matrix center (x, y), the region center depth (z), the region rectangle width and height (w, h), and the speed of each component in the RGB image and depth image coordinate system;

[0058] Define the state space S: Use an explicit way to define it, so the spatial state is the same as the observation state, which is a ten-dimensional state space (x, y, z, w, h, dx, dy, dz, dw, dh);

[0059] Define the prior process noise covariance matrix Q: where Q is a diagonal matrix, and the diagonal vector set is the predetermined prior parameters;

[0060] Define the projection matrix H from the measurement space to the state space: For the explicit transformation from the observation space to the state space, and assuming that each velocity component is a constant velocity model, it is:

[0061]

[0062] Define the state transfer matrix F as follows:

[0063]

[0064] Here, dt represents the rate of change, which is related to the FPS of the video captured by the structured light camera.

[0065] 11. In the first aspect of the above technical solution, the state update process of the Kalman filter is as follows:

[0066] Initialization parameters: For the state space S, a constant velocity model is used to estimate the change process of the motion trajectory. Therefore, each element of the state space is initialized to 0;

[0067] Estimate the prior state space: in represents the prior estimate of the current spatial state, represents the prior estimate of the previous state space;

[0068] Estimate the prior covariance: Among them, F T represents the transpose operation of the state transfer matrix F;

[0069] Update the Kalman gain:

[0070] Update the posterior state space: Where y represents the current observation error, S k represents the posterior estimate of the current state space;

[0071] Update the posterior covariance: Where I represents a diagonal matrix.

[0072] 12. In the first aspect of the above technical solution, in the CPR chest compression index:

[0073] For pressing posture: evaluate and define the standard pressing posture. The evaluation standard is determined based on the angle θ1 perpendicular to the horizontal plane, especially 85°≤θ1≤90°. When detecting the human body posture points, the average arm angle θ2 is estimated. If the average arm angle θ2 is θ2, the arm is considered to be straight and it is considered to be a standard pressing posture.

[0074] For compression frequency: evaluate and define the standard compression frequency, use the human body posture point physical model to identify and record the rescuer's key bone points, which include the shoulder, arm and palm positions during compression. These bone points start to move downward together, and then the downward displacement continues until it reaches the lowest point and returns to the original height; the moment when the continuous downward displacement starts is marked as t1 and t2, and Δt represents the interval between these two moments, which constitutes a compression cycle, measured in seconds. The compression frequency is defined as 60seconds / Δt, which represents the number of compressions per minute. The evaluation standard stipulates that the compression frequency is in the range of 100 to 120 times per minute;

[0075] For the number of compressions: in the process of identifying the compression cycle of recording the compression frequency, each compression cycle is also counted and recorded as the number of compressions;

[0076] For compression depth: evaluate and define the standard compression depth, use the physical model of human posture points to identify and record the positions of the rescuer's shoulders, arms and palms during compression, and after the operator's palms are aligned with the simulated individual, the moment when the center points of the shoulders, arms and palms begin to move downward is marked as the highest point. As the movement proceeds downward to the lowest point and is about to start moving upward, this moment is recorded as the lowest point. The displacement is projected and scaled using the depth of field scale in the physical model of the marker and is defined as the difference ΔA between the highest point and the lowest point. The standard for compression depth assessment is 5cm≤ΔA≤6cm.

[0077] 13. In the present invention, unless otherwise clearly defined and specified, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0078] 14. In the present invention, the orientation or position relationship indicated by the terms "center", "upper", "lower", "axial", "bottom", "inner", "outer", etc. is based on the orientation or position assembly relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0079] 15. In addition, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0080] 16. Also, the term “and / or” in this application means three parallel schemes, taking “obtaining the current measurement result information of the current user’s blood pressure and / or pulse” as an example, including the scheme of “obtaining the current measurement result information of the current user’s blood pressure”, or the scheme of “obtaining the current measurement result information of the current user’s pulse”, or the scheme of “obtaining the current measurement result information of the current user’s blood pressure and pulse” that are satisfied at the same time.

[0081] Due to the application of the above scheme, the present invention has the following advantages and effects compared with the prior art:

[0082] In the present invention, a chest compression scene based on a structured light camera is constructed, and the structured light camera collects a video of the rescuer applying chest compression to the patient and obtains an RGB image and a depth of field image, and then a series of processing is performed on the RGB image and the depth of field image, including aligning the RGB image and the depth of field image, obtaining the depth of field data corresponding to the RGB image, and calculating the three-dimensional coordinates of the human body posture point, so as to obtain accurate pixel depth of field data, and provide basic data guarantee for the accuracy and reliability of the subsequent real-time guidance of chest compression; and after obtaining the human body posture point to calculate the three-dimensional coordinates of the human body posture point, and then respectively estimate the two-dimensional vector field of the RGB image and the depth of field image in pixel displacement and depth change, separate and filter the foreground and background of the three-dimensional coordinates of the human body posture point, so that the depth of the rescuer's chest compression in the subsequently calculated chest compression quality index can be more accurately estimated; and considering the cumulative error in evaluating the CPR compression motion trajectory, enhancement is performed according to the specific local area in the physical model of the human body posture point to track the compression action trajectory, so as to ensure high correlation, eliminate the cumulative error, and more accurately calculate the chest compression quality indicators such as the compression depth. This can provide accurate and reliable indicators for real-time guidance of chest compressions and provide rescuers with faster and more accurate guidance assistance. It can not only enable rescuers to receive accurate guidance during first aid training and improve their operational level, but also help rescuers to provide accurate and rapid rescue during chest compressions on patients, thereby ensuring that patients' productivity is improved and that the prognosis after first aid is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Attached Figure 1 It is a flow chart of a method for real-time guidance of chest compression based on structured light according to an embodiment of the present invention;

[0084] Attached Figure 2 is a schematic diagram of a real-time chest compression guidance system based on structured light in an embodiment of the present invention;

[0085] Attached Figure 3 It is a schematic diagram of re-projecting a rectangular physical model using affine transformation in an embodiment of the present invention;

[0086] Attached Figure 4 The figure is a schematic diagram of the actual operation flow of an embodiment of the present invention. DETAILED DESCRIPTION

[0087] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0088] The present invention aims to address the problems that the traditional monocular camera method cannot accurately estimate the pixel depth of field, which reduces the accuracy and reliability of guidance, and cannot directly estimate the depth of the rescuer's chest compression, which directly affects the effect of cardiopulmonary resuscitation. The present invention innovatively designs a real-time guidance method, system, device, and medium for chest compression based on structured light. In particular, in the present invention, a chest compression scene based on a structured light camera is constructed.

[0089] Embodiment 1, as Figure 1 As shown, the first embodiment of the present invention proposes a real-time guidance method for chest compression based on structured light, and the method includes:

[0090] Use the posture detection model to establish a physical model of human posture points;

[0091] Construct a chest compression scene based on a structured light camera. The structured light camera collects the video of the rescuer performing chest compressions on the patient and obtains RGB images and depth of field images.

[0092] Align the RGB image and the depth image to obtain the depth data corresponding to the RGB image, and calculate the three-dimensional coordinates of the human body posture points;

[0093] Estimate the two-dimensional vector field of pixel displacement and depth change of RGB image and depth image respectively, and separate and filter the foreground and background of three-dimensional coordinates of human posture points;

[0094] Based on the results of foreground and background separation and filtering combined with the human body posture point physical model, the human body posture points during chest compression are tracked in real time, and specific local areas in the human body posture point physical model are enhanced to track the compression action trajectory;

[0095] Calculating a chest compression quality index based on the result of tracking the compression action trajectory, where the chest compression quality index includes at least compression depth and one or more of the following: compression posture, compression frequency, and number of compressions;

[0096] Provide text, image and / or voice prompt guidance based on chest compression quality indicators.

[0097] Through the implementation of the first embodiment of the present invention, a chest compression scene based on a structured light camera is constructed, and the structured light camera is used to collect a video of a rescuer applying chest compression to a patient and obtain an RGB image and a depth-of-field image, and a series of subsequent processing of the RGB image and the depth-of-field image, including aligning the RGB image and the depth-of-field image, obtaining depth-of-field data corresponding to the RGB image, and calculating the three-dimensional coordinates of human body posture points, thereby obtaining accurate pixel depth-of-field data, providing basic data guarantee for the accuracy and reliability of subsequent real-time guidance of chest compression; and after obtaining the human body posture points and calculating the three-dimensional coordinates of the human body posture points, the RGB image and the depth-of-field image are estimated in pixels respectively. The two-dimensional vector field of displacement and depth change is used to separate and filter the foreground and background of the three-dimensional coordinates of the human posture points, so that the depth of the rescuer's chest compression in the subsequently calculated chest compression quality index can be estimated more accurately, thereby providing accurate and reliable indicators for real-time guidance of chest compressions and providing rescuers with faster and more accurate guidance assistance; and considering the cumulative error in evaluating the CPR compression motion trajectory, the specific local area in the physical model of the human posture points is enhanced to track the compression action trajectory, thereby ensuring high correlation and eliminating cumulative errors, and more accurately calculating chest compression quality indicators such as compression depth.

[0098] In one implementation of the first embodiment of the present invention, in the process of using the posture detection model to establish a physical model of human posture points, a pre-trained weighted skeleton point detection model is used to detect human posture points, and the physical model of human posture points at least includes joints in the rescuer's head, shoulders, elbows, palms, torso and legs. In this way, the joints of the key moving parts of the human body when the rescuer performs CPR chest compressions are covered, thereby improving the accuracy of the calculation of CPR chest compression indicators.

[0099] In another implementation of Example 1 of the present invention, in the process of constructing a chest compression scene based on a structured light camera, the structured light camera used includes an infrared camera and an RGB camera, which form a cross-structured light source projection and image sensor capture, and the cross-structured light source is projected to the rescuer being detected. The infrared camera and the RGB camera collect the video of the rescuer applying chest compression to the patient, and capture the morphological changes of the light spot projected on the rescuer to obtain an RGB image and a depth of field image. In this way, the composition of the structured light camera is improved to ensure that the acquisition can be subsequently merged, depth of field calculated, foreground and background filtered, real-time tracking and other operations, providing the necessary scene conditions for the fact of the real-time guidance method of chest compression in the present invention.

[0100] In another implementation of the first embodiment of the present invention, in the process of aligning the RGB image and the depth image to obtain the depth data corresponding to the RGB image, the following steps are included:

[0101] By analyzing the speckle structure reflected by the rescuer's surface, the distance between the rescuer and the light source is calculated based on the speckle ranging and the grayscale value of the pixel point of the digitized speckle pattern is obtained, that is, the width and depth of the speckle pattern;

[0102] The actual distance is estimated for the depth image, and the RGB image and the depth image are aligned to obtain the depth data of the RGB image.

[0103] The above steps are used to obtain more accurate depth of field data, which will help calculate the compression depth in the subsequent real-time tracking process, ensure the accuracy and reliability of the compression depth data, and further provide more standardized and reasonable guidance operations for the real-time guidance process of chest compressions.

[0104] Specifically, the process of calculating the distance between the rescuer and the light source according to the speckle ranging and obtaining the grayscale value of the pixel point of the digitized speckle pattern includes the following steps:

[0105] The grayscale function of the RGB image and the depth of field image is expressed as follows:

[0106] Among them, s represents the number of speckles, a represents the size of the speckles, I0 represents the image background intensity, usually I0 is 1, and the one-dimensional random variable (x k ,y k ) represents the center position of the kth speckle particle, which can be obtained by the uniform random distribution function rand, expressed as:

[0107] Where M and N represent the number of pixels in the X and Y directions, that is, the width and depth of the speckle pattern, respectively, and the one-dimensional random variable is spatially discretized, expressed as:

[0108] Where i=0, 1, ..., M-1, j=0, 1, ..., N-1.

[0109] This makes the depth of field data obtained more accurate.

[0110] Furthermore, the process of estimating the actual distance of the depth image, aligning the RGB image and the depth image, and obtaining the depth data of the RGB image includes the following steps:

[0111] The actual distance is estimated for the depth image. The depth image pixels correspond to the width and depth of the speckle pattern, that is, M×N pixels, where each pixel (i, j) corresponds to an integer value d between 0 and the set resolution, d represents the depth digital value of the pixel relative to the center of the structured light camera, and the actual depth value z i,j Use the measured numeric depth value d i,j Convert according to the following equation:

[0112] The constants α, β, and μ in the formula are manually calibrated and adjusted in reality by using the undetermined coefficient method to calibrate the parameters in the structured light camera;

[0113] For the alignment of the RGB image and the depth image, given the depth value in the metric space and the intrinsic parameters obtained from the calibration process, the 3D coordinates of the point (x i,j ,y i,j , z i,j ) is associated with each pixel according to the following equation:

[0114] where f x and f y Respectively represent the focal length in the X and Y directions, x o and y0 represent the center pixel position of the structured light camera in the X and Y directions respectively.

[0115] In one implementation of the first embodiment of the present invention, in the process of respectively estimating the two-dimensional vector field of pixel displacement and depth change of the RGB image and the depth image, and performing separation and filtering of the foreground and background of the three-dimensional coordinates of the human posture point, the following steps are included:

[0116] The optical flow method is used to estimate the two-dimensional vector field of pixel displacement and depth change of RGB image and depth image respectively. The two-dimensional vector field reflects the change trend of the grayscale of each point on the image. The two-dimensional vector field is regarded as the instantaneous velocity field generated by the movement of grayscale pixels on the image plane. The information contained in the two-dimensional vector field is the instantaneous motion velocity vector information of each image point.

[0117] The optical flow vector is obtained by derivation of the image data of the two-dimensional vector field. The continuous movement vector of the foreground skeleton point on the RGB image is found based on the chest compression scene, and the foreground and background results obtained by RGB image recognition and the foreground and background results obtained by depth of field image recognition are obtained in the area where the depth of field movement vector remains unchanged.

[0118] By implementing this step, the process of separating and filtering the foreground and background of the three-dimensional coordinates of the human body posture points can be faster and the results can be more accurate. In this way, the foreground and background of the RGB image and the foreground and background of the depth of field image can be separated and filtered. The foreground and background of the RGB image and the foreground and background of the depth of field image after separation and filtering can make the subsequent real-time tracking of the human body posture points and the results more consistent with the actual compression depth of the rescuer during actual chest compression.

[0119] Specifically, in the process of respectively estimating the two-dimensional vector field of pixel displacement and depth change of the RGB image and the depth image, and separating and filtering the foreground and background of the three-dimensional coordinates of the human posture point, the following steps are included:

[0120] Consider the light intensity of a pixel I(x, y, t) in the first frame, moving a distance (dx, dy) to the next frame, using a time dt, expressed as: I(x, y, t) = I(x+dt, y+dt, t+dt), and Taylor expansion of this formula where ε represents a second-order infinitesimal term, and when dt is removed, we obtain

[0121] Let u and v be the velocity vectors of the optical flow along the X-axis and Y-axis respectively,

[0122] make Respectively represent the partial derivatives of the grayscale of the image pixel along the X, Y, and T directions;

[0123] And it is deduced as I x u+I y v+I t =0, where I x , I y , I t Both can be derived from the image data, and u and v are the required optical flow vectors;

[0124] Based on the chest compression scene, we search for the continuous movement vector of the foreground skeleton points on the RGB image, and the area where the depth of field movement vector remains unchanged, as follows:

[0125]

[0126] in, Represents the foreground and background results obtained by RGB image recognition, and Respectively represent the foreground and background recognized by the RGB image, u RGB and u RGB Represents the motion vector in the X-axis direction and the Y-axis direction, ||u RGB , vRGB ||2 represents the second normal form of the vector, thr RGB is the manually calibrated threshold; Represents the foreground and background results obtained by depth of field image recognition, Respectively represent the foreground and background recognized by the depth of field image, u Dep and v Dep Represents the motion vector in the X-axis direction and the Y-axis direction, ||u Dep , v Dep ||2 represents the second normal form of the vector, thr D ep is the manually calibrated threshold.

[0127] In another implementation of the first embodiment of the present invention, in the process of real-time tracking of human body posture points during chest compression according to the results of separation and filtering of foreground and background combined with the physical model of human body posture points, and enhancing according to specific local areas in the physical model of human body posture points to track the trajectory of the pressing action, the rescuer's cyclic arm pressing action is used as a specific local area in the physical model of human body posture points, and the arm pressing action is respectively for the rescuer's arm and torso, respectively based on the RGB image space and the depth of field image space, and the forward correlation of the action in the CPR process is established using Kalman filtering for enhancement to track the trajectory of the pressing action. Considering that in the CPR scenario, the recognition focus is on the cyclic arm pressing movement, therefore, the information across frames shows a high degree of correlation, and the action mainly occurs in a specific local area, and the position of the human body is relatively fixed. This feature helps to enhance the local recognition accuracy of the posture recognition model specially customized for the CPR environment.

[0128] Specifically, the variables defining the Kalman filter include the following:

[0129] Lock the target area: First, use a rectangular box to lock the arm area Box a and the torso area Box b , this area is locked in the same position area indicated by the RGB image space and the depth image space;

[0130] Define the observation space O: define the ten-dimensional state space (x, y, z, w, h, dx, dy, dz, dw, dh), the region matrix center (x, y), the region center depth (z), the region rectangle width and height (w, h), and the speed of each component in the RGB image and depth image coordinate system;

[0131] Define the state space S: Use an explicit way to define it, so the spatial state is the same as the observation state, which is a ten-dimensional state space (x, y, z, w, h, dx, dy, dz, dw, dh);

[0132] Define the prior process noise covariance matrix Q: where Q is a diagonal matrix, and the diagonal vector set is the predetermined prior parameters;

[0133] Define the projection matrix H from the measurement space to the state space: For the explicit transformation from the observation space to the state space, and assuming that each velocity component is a constant velocity model, it is:

[0134]

[0135] Define the state transfer matrix F as follows:

[0136]

[0137] Here, dt represents the rate of change, which is related to the FPS of the video captured by the structured light camera.

[0138] The state update process of Kalman filter is as follows:

[0139] Initialization parameters: For the state space S, a constant velocity model is used to estimate the change process of the motion trajectory. Therefore, each element of the state space is initialized to 0;

[0140] Estimate the prior state space: in represents the prior estimate of the current spatial state, represents the prior estimate of the previous state space;

[0141] Estimate the prior covariance: Among them, F T represents the transpose operation of the state transfer matrix F;

[0142] Update the Kalman gain:

[0143] Update the posterior state space: Where y represents the current observation error, S k represents the posterior estimate of the current state space;

[0144] Update the posterior covariance: Where I represents a diagonal matrix.

[0145] Embodiment 2, as Figure 2 As shown, the second embodiment of the present invention proposes a real-time guidance system for chest compression based on structured light, and the real-time guidance system includes a visual acquisition module, a main control module, a calculation module, and a feedback module;

[0146] The visual acquisition module uses a structured light camera, which includes an infrared camera and an RGB camera. The structured light camera collects a video of the rescuer performing chest compressions on the patient and obtains an RGB image and a depth of field image.

[0147] The main control module is used to obtain the RGB image and the depth image collected by the visual acquisition module, transmit the RGB image and the depth image to the calculation module, and output the control instruction of the result calculated by the calculation module to the feedback module;

[0148] A physical model of human posture points is established in the calculation module, and the RGB image and the depth of field image are aligned to obtain the depth of field data corresponding to the RGB image, and the three-dimensional coordinates of the human posture points are calculated; the two-dimensional vector fields of the RGB image and the depth of field image in pixel displacement and depth change are estimated respectively, and the foreground and background of the three-dimensional coordinates of the human posture points are separated and filtered; according to the results of the separation and filtering of the foreground and background combined with the physical model of the human posture points, the human posture points in the process of chest compression are tracked in real time, and the specific local areas in the physical model of the human posture points are enhanced to track the compression action trajectory; the chest compression quality index is calculated according to the result of tracking the compression action trajectory; and the CPR guidance index is sent to the main control module;

[0149] The feedback module includes an audio feedback module and a video feedback module, which are used to perform audio feedback and video feedback according to the control instructions of the main control module.

[0150] Embodiment 3: Embodiment 3 of the present invention discloses a first aid device, and the first aid device is the real-time chest compression guidance system based on structured light as described in Embodiment 2 of the present invention.

[0151] Embodiment 4: Embodiment 4 of the present invention further discloses a readable storage medium, on which a control program is stored. When the control program is executed by a processor, the processor executes the steps of the method described in Embodiment 1.

[0152] Those skilled in the art can understand that the operation of the main control unit in the above-mentioned embodiment method can be completed by instructing the relevant hardware (blood pressure detection unit) through a program, and the program is stored in a readable storage medium, including a number of instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform the operation performed by the main control unit in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0153] The following is an example of the actual application process of the above embodiment of the present invention.

[0154] In the present invention, a structured light optical sensor is used to collect chest compression video images during CPR cardiopulmonary resuscitation, and machine vision technology is used to track the rescuer's action trajectory. By real-time calculation of CPR quality indicators, including compression posture, compression depth, compression frequency, and compression times, the rescuer's chest compression status is monitored and real-time guidance is provided.

[0155] The specific algorithm process of the design includes:

[0156] 1. Use the posture detection model to establish a physical model of human posture points;

[0157] 2. Calculate the depth of field based on the speckle structured light technology, align the depth of field data and the RGB image, and obtain the three-dimensional coordinates of the physical model;

[0158] 3. Separate and filter foreground and background based on RGB image and depth image;

[0159] 4. Tracking fast-changing dynamic trajectory states based on Hidden Markov Hypothesis;

[0160] 5. Calculate CPR chest compression indicators based on tracking results, including compression posture, compression depth, compression frequency, and number of compressions.

[0161] The more specific process can be found below:

[0162] 1. Use the posture detection model to establish a physical model of human posture points:

[0163] A public pre-trained weighted skeleton point detection model is used to detect key points of the human body. For public models with different key point detections, such as 15 points, 18 points, and 33 points, it is ensured that the posture recognition model for chest compression recognition contains at least the joint points in the rescuer's head, shoulder, elbow, palm, torso, and leg areas.

[0164] When calculating the angle between the main trunks of the human body, the cosine angle is used, and the calculation method is as follows:

[0165]

[0166] in Represents two vectors that form an angle.

[0167] 2. Calculate the depth of field based on the speckle structured light technology, align the depth of field data and the RGB image, and obtain the three-dimensional coordinates of the physical model:

[0168] The cross-structured light camera is a device that uses structured light sources and special image sensors to capture scene depth information. It is mainly composed of an infrared camera and an RGB camera. The chest compression scene constructed based on the structured light camera is as follows: Figure 3As shown:

[0169] It projects a known pattern of light beams, such as stripes or grids, onto the object being measured, and uses a camera to capture and analyze the changes in the shape of these light spots, thereby inferring the three-dimensional shape of the target object. Speckle ranging uses the interference or refraction of light to analyze the speckle structure reflected from the target surface to calculate the distance between the target and the light source. Its basic principles are as follows:

[0170] Digital speckle superimposes a certain number of randomly distributed Gaussian spots. Assuming that the background light intensity of the image is uniform, the image grayscale function can be expressed as:

[0171]

[0172] Among them, s represents the number of speckles, a represents the size of the speckles, I0 represents the image background intensity, usually I0 is 1, and the one-dimensional random variable (x k ,y k ) represents the center position of the kth speckle particle, which can be obtained by the uniform random distribution function rand and can be expressed as:

[0173]

[0174] Where M and N represent the number of pixels in the X and Y directions, respectively, that is, the width and depth of the speckle pattern.

[0175] Since the speckle pattern is obtained by the gray value of the pixel point, it is digital. The formula (3) is spatially discretized and can be expressed as:

[0176]

[0177] In the formula, i=0, 1, ....M-1, j=0, 1, ..., N-1.

[0178] Based on the above speckle structured light principle, the camera further aligns the speckle infrared image with the color image to obtain the depth map corresponding to the color image. The specific process is as follows:

[0179] (1) Actual distance estimation for depth image: The pixels of the depth image correspond to the width and depth of the speckle pattern, that is, M×N pixels, where each pixel (i, j) corresponds to an integer value d between 0 and the set resolution, representing the digital depth value of the pixel relative to the center of the camera. Actual depth value z i,j Use the measured numeric depth value d i,j Convert to millimeters using the following equation:

[0180]

[0181] The constants α, β, and μ in the equation are manually calibrated and adjusted in reality using the undetermined coefficient method to calibrate the camera intrinsic parameters.

[0182] (2) For the alignment of the color RGB image and the depth image, given the depth value in the metric space and the intrinsic parameters obtained from the calibration process, the 3D point (x i,j ,y i,j , z i,j ) (in millimeters) can be associated with each pixel according to the following equation:

[0183]

[0184] where f x and f y They represent the focal lengths in the X and Y directions respectively, and x0 and y0 represent the camera center pixel positions in the X and Y directions respectively.

[0185] 3. Separate and filter foreground and background based on RGB image and depth image;

[0186] The optical flow method is used to estimate the depth map and the two-dimensional vector field of the RGB image in terms of pixel displacement and depth change to separate the foreground or background.

[0187] The optical flow field is a two-dimensional vector field that reflects the trend of grayscale changes at each point on the image. It can be seen as the instantaneous velocity field generated by the movement of grayscale pixels on the image plane. The information it contains is the instantaneous motion velocity vector information of each pixel. The specific process is as follows:

[0188] Consider the light intensity of a pixel I(x, y, t) in the first frame (where t represents the time dimension). Moving a distance (dx, dy) to the next frame, using a time dt, is expressed as:

[0189] I(x,y,t)=I(x+dt,y+dt,t+dt) (7)

[0190] Taylor expansion of the above equation yields:

[0191]

[0192] in ε represents the second-order infinitesimal term, which can be ignored. Substituting equation (8) into equation (7) and dividing by dt, we can get:

[0193]

[0194] Let u and v be the velocity vectors of the optical flow along the X-axis and Y-axis respectively, and we get:

[0195]

[0196] make Respectively represent the partial derivatives of the grayscale of the image pixel along the X, Y, and T directions. In summary, formula (3) can be written as:

[0197] I x u+I y v+I t =0 (11)

[0198] Among them I x , I y , I t Both can be derived from the image data, and u and v are the required optical flow vectors.

[0199] Based on the CPR scenario, the goal is to find the continuous movement vector of the foreground skeleton points on the color RGB image, and the area where the depth of field movement vector remains unchanged, as follows:

[0200]

[0201]

[0202] in, Represents the foreground and background results obtained by RGB image recognition, and Respectively represent the foreground and background recognized by the RGB image, u RGB and u RGB Represents the motion vector in the X-axis direction and the Y-axis direction, ||u RGB , v RGB ||2 represents the second normal form of the vector, thr RGB is the manually calibrated threshold; similarly, the superscript Dep represents the foreground and background related results of the depth of field image recognition.

[0203] 4. Tracking fast-changing dynamic trajectory states based on Hidden Markov Hypothesis

[0204] Chest compressions are a series of rapid, cyclical movements in which the rescuer holds his arms vertically with his shoulders above his palms and applies pressure to the victim's chest using his shoulders, elbows, and wrists simultaneously.

[0205] When using machine vision, especially for bone point detection, executing the bone recognition algorithm on consecutive frames with rapidly changing motion trajectories may lead to cumulative errors in evaluating the CPR compression motion trajectory due to problems such as motion blur and error propagation in the heat map features.

[0206] Considering that in the CPR scenario, the recognition focus is on the cyclic arm compression movement. Therefore, the information across frames shows a high degree of correlation, and the action mainly occurs in a specific local area, and the position of the human body is relatively fixed. This feature helps to specifically enhance the local recognition accuracy of the posture recognition model customized for the CPR environment.

[0207] For the local motion of the action, the Kalman filter is used to establish the forward correlation of the action during CPR based on the RGB space and the depth image space for the arm and the torso respectively. The variables of the Kalman filter are defined as follows:

[0208] (1) Lock the target area: First, use a rectangular box to lock the arm area Box a and the torso area Box b This area is locked to the same location indicated in the RGB space and the depth image space.

[0209] (2) Define the observation space O: Define the ten-dimensional state space (x, y, z, w, h, dx, dy, dz, dw, dh), the region matrix center (x, y), the region center depth (z), the region rectangle width and height (w, h), and the velocity of each component in the RGB image and depth image coordinate systems.

[0210] (3) Define the state space s: In the model, it is defined explicitly, so the spatial state is the same as the observation state, which is also a ten-dimensional state space (x, y, z, w, h, dx, dy, dz, dw, dh).

[0211] (4) Define the prior process noise covariance matrix Q, where Q is a diagonal matrix and the diagonal vector set is the predetermined prior parameters.

[0212] (5) Define the projection matrix H from the measurement space to the state space. For the explicit transformation from the observation space to the state space, and assuming that each velocity component is a constant velocity model, it is:

[0213]

[0214] (6) Define the state transfer matrix F as follows

[0215]

[0216] Here, dt represents the rate of change, which is related to the FPS of image acquisition.

[0217] The state update process of Kalman filter is as follows:

[0218] (1) Initialization parameters: For the state space S, a constant velocity model is used to estimate the change process of the motion trajectory. Therefore, each element of the state space is initialized to 0.

[0219] (2) Estimate the prior state space:

[0220]

[0221] in represents the prior estimate of the current spatial state, represents the prior estimate of the previous state space.

[0222] (3) and estimate the prior covariance:

[0223]

[0224] Among them, F T Represents the transpose operation of the state transfer matrix F.

[0225] (4) Update Kalman gain:

[0226]

[0227] (5) Update the posterior state space:

[0228]

[0229]

[0230] Where y represents the current observation error, S k represents the posterior estimate of the current state space.

[0231] (6) and update the posterior covariance:

[0232]

[0233] Where I represents a diagonal matrix.

[0234] 5. Calculate CPR chest compression indicators based on tracking results, including compression posture, compression depth, compression frequency, and number of compressions.

[0235] Compression posture: The standard CPR procedure requires the rescuer to position both arms vertically with the shoulders above the palms and apply pressure to the patient's chest using the shoulders, elbows, and wrists simultaneously. The operator's shoulder skeleton, arm skeleton points, and the position of the palm immersed in the patient's chest were tracked using a human posture model. The evaluation criteria were determined based on the angle θ1 perpendicular to the horizontal plane, specifically 85°≤θ1≤90°. Because effective CPR requires deep and repetitive compression movements, the arms should be straight, that is, 85° but θ1≤90°, to avoid premature fatigue, which is detrimental to performance. In posture estimation, the arms were considered straight if the average arm angle θ2 was above 170°, with a standard deviation below 5° to account for anatomical differences between participants.

[0236] Compression frequency: The human posture model is used to identify and record the operator's key skeletal points, including the position of the shoulders, arms, and palms during compression. The moment when these skeletal points begin to move downward together is designated as t1. Subsequently, the downward displacement continues until it reaches the lowest point and returns to the original height. The moments when the downward displacement begins are marked as t1 and t2. Δt represents the interval between these two moments, which constitutes a compression cycle, measured in seconds. The compression frequency is defined as 60seconds / Δt, which represents the number of compressions per minute. The evaluation criteria stipulate that the compression frequency should be in the range of 100 to 120 times per minute.

[0237] Number of presses: In the above-mentioned recognition process of the press cycle, in addition to recording the press frequency, each press cycle is also counted.

[0238] Compression depth: The position of the operator's shoulder, arm, and palm during compression is identified and recorded using a human posture model. After the operator's palm is aligned with the simulated individual, the moment when the center point of the shoulder, arm, and palm begins to move downward is marked as the starting point. As the movement progresses downward to the lowest point and is about to begin moving upward, this moment is recorded as the lowest point. The displacement is projected and scaled using a depth of field scale and is defined as the difference ΔA between the highest and lowest points. The standard for CPR compression depth assessment has been established as 5cm≤ΔA≤6cm.

[0239] When the rescuer performs actual CPR chest compressions, he or she may refer to the following: Figure 4 The process shown is carried out.

[0240] A simulated dummy is used for testing. The simulated dummy is equipped with pressure sensors, displacement sensors and other sensors for recording CPR chest compression indicators in the embodiment. The pressure and compression depth applied to the chest can be measured in real time, and the number of compressions and compression frequency can be calculated synchronously. By comparing the measured values ​​of the simulated dummy sensors with the results of the method of the present invention, the comparison results are as follows:

[0241] For the compression depth index, the mean error was less than 5% (P<0.01);

[0242] For the index of compression depth times, the average number of errors was less than 1 (P<0.01);

[0243] For compression frequency, the mean error was less than 10% (P < 0.01);

[0244] The judgment results of pressing posture were compared with the observation and evaluation of experts, and the average error was less than 1 time / person (P<0.01).

[0245] Through the implementation of the above embodiments, through the deployment and implementation of the real-time guidance method, system, device, and medium for chest compression based on structured light, accurate pixel depth of field data can be obtained, so that the depth of the rescuer's chest compression in the subsequently calculated chest compression quality index can be more accurately estimated, providing accurate and reliable indicators for real-time guidance of chest compression, and providing rescuers with faster and more accurate guidance assistance, which can not only enable rescuers to receive accurate guidance during first aid training and improve their operating level, but also provide assistance to rescuers in accurate and rapid rescue during chest compression of patients, ensuring that the patient's productivity is improved and the prognosis after first aid can also be better.

[0246] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A real-time guidance method for chest compression based on structured light, characterized in that: The method comprises: Use the posture detection model to establish a physical model of human posture points; Construct a chest compression scene based on a structured light camera. The structured light camera collects the video of the rescuer performing chest compressions on the patient and obtains RGB images and depth of field images. Align the RGB image and the depth image to obtain the depth data corresponding to the RGB image, and calculate the three-dimensional coordinates of the human body posture points; Estimate the two-dimensional vector field of pixel displacement and depth change of RGB image and depth image respectively, and separate and filter the foreground and background of three-dimensional coordinates of human posture points; Based on the results of foreground and background separation and filtering combined with the human body posture point physical model, the human body posture points during chest compression are tracked in real time, and specific local areas in the human body posture point physical model are enhanced to track the compression action trajectory; Calculating a chest compression quality index based on the result of tracking the compression action trajectory, where the chest compression quality index includes at least compression depth and one or more of the following: compression posture, compression frequency, and number of compressions; Provide text, image and / or voice prompt guidance based on chest compression quality indicators.

2. The real-time guidance method for chest compression based on structured light according to claim 1, characterized in that: In the process of using the posture detection model to establish a physical model of human posture points, a pre-trained weighted skeleton point detection model is used to detect human posture points, and the human posture point physical model at least includes joint points in the rescuer's head, shoulders, elbows, palms, torso and legs.

3. The real-time guidance method for chest compression based on structured light according to claim 2, characterized in that: In the process of constructing a chest compression scene based on a structured light camera, the structured light camera used includes an infrared camera and an RGB camera. The infrared camera and the RGB camera constitute a cross-structured light source projection and image sensor capture. The cross-structured light source is projected onto the rescuer being detected. The infrared camera and the RGB camera collect the video of the rescuer applying chest compression to the patient, capture the morphological changes of the light spot projected onto the rescuer, and obtain the RGB image and depth of field image.

4. The real-time guidance method for chest compression based on structured light according to claim 3, characterized in that: In the process of aligning the RGB image and the depth image to obtain the depth data corresponding to the RGB image, the following steps are included: By analyzing the speckle structure reflected by the rescuer's surface, the distance between the rescuer and the light source is calculated based on the speckle ranging and the grayscale value of the pixel point of the digitized speckle pattern is obtained, that is, the width and depth of the speckle pattern; The actual distance is estimated for the depth image, and the RGB image and the depth image are aligned to obtain the depth data of the RGB image.

5. The real-time guidance method for chest compression based on structured light according to claim 4, characterized in that: The process of calculating the distance between the rescuer and the light source based on the speckle ranging and obtaining the grayscale value of the pixel point of the digitized speckle pattern includes the following steps: The grayscale function of the RGB image and the depth of field image is expressed as follows: Among them, s represents the number of speckles, a represents the size of the speckles, I0 represents the image background intensity, usually I0 is 1, and the one-dimensional random variable (x k ,y k ) represents the center position of the kth speckle particle, which can be obtained by the uniform random distribution function rand, expressed as: Where M and N represent the number of pixels in the X and Y directions, that is, the width and depth of the speckle pattern, respectively, and the one-dimensional random variable is spatially discretized, expressed as: Where i=0, 1, ..., M-1, j=0, 1, ..., N-1.

6. The real-time guidance method for chest compression based on structured light according to claim 5, characterized in that: The process of estimating the actual distance of the depth image, aligning the RGB image and the depth image, and obtaining the depth data of the RGB image includes the following steps: The actual distance is estimated for the depth image. The depth image pixels correspond to the width and depth of the speckle pattern, that is, M×N pixels, where each pixel (i, j) corresponds to an integer value d between 0 and the set resolution, d represents the depth digital value of the pixel relative to the center of the structured light camera, and the actual depth value z i,j Use the measured numeric depth value d i,j Convert according to the following equation: The constants α, β, and μ in the formula are manually calibrated and adjusted in reality by using the undetermined coefficient method to calibrate the parameters in the structured light camera; For the alignment of the RGB image and the depth image, given the depth value in the metric space and the intrinsic parameters obtained from the calibration process, the 3D coordinates of the point (x i,j ,y i,j , z i,j ) is associated with each pixel according to the following equation: where f x and f y Respectively represent the focal length in the X direction and the Y direction, and x0 and y0 represent the center pixel position of the structured light camera in the X direction and the Y direction respectively.

7. The real-time guidance method for chest compression based on structured light according to claim 6, characterized in that: In the process of estimating the two-dimensional vector field of pixel displacement and depth change of the RGB image and the depth image, and separating and filtering the foreground and background of the three-dimensional coordinates of the human posture point, the following steps are included: The optical flow method is used to estimate the two-dimensional vector field of pixel displacement and depth change of RGB image and depth image respectively. The two-dimensional vector field reflects the change trend of the grayscale of each point on the image. The two-dimensional vector field is regarded as the instantaneous velocity field generated by the movement of grayscale pixels on the image plane. The information contained in the two-dimensional vector field is the instantaneous motion velocity vector information of each image point. The optical flow vector is obtained by derivation of the image data of the two-dimensional vector field. The continuous movement vector of the foreground skeleton point on the RGB image is found based on the chest compression scene, and the foreground and background results obtained by RGB image recognition and the foreground and background results obtained by depth of field image recognition are obtained in the area where the depth of field movement vector remains unchanged.

8. The real-time guidance method for chest compression based on structured light according to claim 7, characterized in that: In the process of estimating the two-dimensional vector field of pixel displacement and depth change of the RGB image and the depth image, and separating and filtering the foreground and background of the three-dimensional coordinates of the human posture point, the following steps are included: Consider the light intensity of a pixel I(x, y, t) in the first frame, moving a distance (dx, dy) to the next frame, using a time dt, expressed as: I(x, y, t) = I(x+dt, y+dt, t+dt), and Taylor expansion of this formula where ε represents a second-order infinitesimal term, and when dt is removed, we obtain Let u and v be the velocity vectors of the optical flow along the X-axis and Y-axis respectively, make Respectively represent the partial derivatives of the grayscale of the image pixel along the X, Y, and T directions; And it is deduced as I x u+I y v+I t =0, where I x , I y ,I t Both can be derived from the image data, and u and v are the required optical flow vectors; Based on the chest compression scene, we search for the continuous movement vector of the foreground skeleton points on the RGB image, and the area where the depth of field movement vector remains unchanged, as follows: in, Represents the foreground and background results obtained by RGB image recognition, and Respectively represent the foreground and background recognized by the RGB image, u RGB and u RGB Represents the motion vector in the X-axis direction and the Y-axis direction, ||u RGB , v RGB ||2 represents the second normal form of the vector, thr RGB is the manually calibrated threshold; Represents the foreground and background results obtained by depth of field image recognition, Respectively represent the foreground and background recognized by the depth of field image, u Dep and v Dep Represents the motion vector in the X-axis direction and the Y-axis direction, ||u Dep , v Dep ||2 represents the second normal form of the vector, thr Dep is the manually calibrated threshold.

9. The real-time guidance method for chest compression based on structured light according to claim 8, characterized in that: In the process of real-time tracking of human posture points during chest compression based on the results of separation and filtering of foreground and background and combining with the physical model of human posture points, and enhancing according to specific local areas in the physical model of human posture points to track the trajectory of compression action, the rescuer's cyclic arm compression action is used as the specific local area in the physical model of human posture points. For the arm compression action, the rescuer's arm and torso are respectively targeted, and the forward correlation of the action in the CPR process is established based on the RGB image space and the depth of field image space for enhancement to track the compression action trajectory.

10. The real-time guidance method for chest compression based on structured light according to any one of claim 9, characterized in that: The variables that define the Kalman filter are as follows: Lock the target area: First, use a rectangular box to lock the arm area Box a and the torso area Box b , this area is locked in the same position area indicated by the RGB image space and the depth image space; Define the observation space O: define the ten-dimensional state space (x, y, z, w, h, dx, dy, dz, dw, dh), the region matrix center (x, y), the region center depth (z), the region rectangle width and height (w, h), and the speed of each component in the RGB image and depth image coordinate system; Define the state space S: Use an explicit way to define it, so the spatial state is the same as the observation state, which is a ten-dimensional state space (x, y, z, w, h, dx, dy, dz, dw, dh); Define the prior process noise covariance matrix Q: where Q is a diagonal matrix, and the diagonal vector set is the predetermined prior parameters; Define the projection matrix H from the measurement space to the state space: For the explicit transformation from the observation space to the state space, and assuming that each velocity component is a constant velocity model, it is: Define the state transfer matrix F as follows: Here, dt represents the rate of change, which is related to the FPS of the video captured by the structured light camera.

11. The real-time guidance method for chest compression based on structured light according to any one of claim 10, characterized in that: The state update process of Kalman filter is as follows: Initialization parameters: For the state space S, a constant velocity model is used to estimate the change process of the motion trajectory. Therefore, each element of the state space is initialized to 0; Estimate the prior state space: in represents the prior estimate of the current spatial state, represents the prior estimate of the previous state space; Estimate the prior covariance: where F T represents the transpose operation of the state transfer matrix F; Update the Kalman gain: Update the posterior state space: Where y represents the current observation error, S k represents the posterior estimate of the current state space; Update the posterior covariance: Where I represents a diagonal matrix.

12. The real-time chest compression guidance method based on structured light according to any one of claims 1 to 11, characterized in that: In the CPR chest compression indicators: For pressing posture: evaluate and define the standard pressing posture. The evaluation standard is determined based on the angle θ1 perpendicular to the horizontal plane, especially 85°≤θ1≤90°. When detecting the human body posture points, the average arm angle θ2 is estimated. If the average arm angle θ2 is θ2, the arm is considered to be straight and it is considered to be a standard pressing posture. For compression frequency: evaluate and define the standard compression frequency, use the human body posture point physical model to identify and record the rescuer's key bone points, which include the shoulder, arm and palm positions during compression. These bone points start to move downward together, and then the downward displacement continues until it reaches the lowest point and returns to the original height; the moment when the continuous downward displacement starts is marked as t1 and t2, and Δt represents the interval between these two moments, which constitutes a compression cycle, measured in seconds. The compression frequency is defined as 60seconds / Δt, which represents the number of compressions per minute. The evaluation standard stipulates that the compression frequency is in the range of 100 to 120 times per minute; For the number of compressions: in the process of identifying the compression cycle of recording the compression frequency, each compression cycle is also counted and recorded as the number of compressions; For compression depth: evaluate and define the standard compression depth, use the physical model of human posture points to identify and record the positions of the rescuer's shoulders, arms and palms during compression, and after the operator's palms are aligned with the simulated individual, the moment when the center points of the shoulders, arms and palms begin to move downward is marked as the highest point. As the movement proceeds downward to the lowest point and is about to start moving upward, this moment is recorded as the lowest point. The displacement is projected and scaled using the depth of field scale in the physical model of the marker and is defined as the difference ΔA between the highest point and the lowest point. The standard for compression depth assessment is 5cm≤ΔA≤6cm.

13. A real-time guidance system for chest compression based on structured light, used for real-time guidance for chest compression based on structured light in the method according to any one of claims 1 to 12, characterized in that: The real-time guidance system includes a visual acquisition module, a main control module, a calculation module, and a feedback module; The visual acquisition module uses a structured light camera, which includes an infrared camera and an RGB camera. The structured light camera collects a video of the rescuer performing chest compressions on the patient and obtains an RGB image and a depth of field image. The main control module is used to obtain the RGB image and the depth image collected by the visual acquisition module, transmit the RGB image and the depth image to the calculation module, and output the control instruction of the result calculated by the calculation module to the feedback module; A physical model of human posture points is established in the calculation module, and the RGB image and the depth of field image are aligned to obtain the depth of field data corresponding to the RGB image, and the three-dimensional coordinates of the human posture points are calculated; the two-dimensional vector fields of the RGB image and the depth of field image in pixel displacement and depth change are estimated respectively, and the foreground and background of the three-dimensional coordinates of the human posture points are separated and filtered; according to the results of the separation and filtering of the foreground and background combined with the physical model of the human posture points, the human posture points in the process of chest compression are tracked in real time, and the specific local areas in the physical model of the human posture points are enhanced to track the compression action trajectory; the chest compression quality index is calculated according to the result of tracking the compression action trajectory; and the CPR guidance index is sent to the main control module; The feedback module includes an audio feedback module and a video feedback module, which are used to perform audio feedback and video feedback according to the control instructions of the main control module.

14. A first aid device, characterized in that: The emergency rescue device includes the structured light-based real-time chest compression guidance system as described in claim 13.

15. A readable storage medium, characterized in that: The readable storage medium stores a control program, and when the control program is executed by a processor, the processor executes the steps of the method according to any one of claims 1 to 12.