An automated assessment device and method for cardiopulmonary resuscitation chest compression performance

Abstract

The application provides a cardiopulmonary resuscitation heart compression operation automatic evaluation method and device, and belongs to the technical field of behavior operation automatic evaluation; solves the problems of real-time monitoring and evaluation of the standardization and accuracy of cardiopulmonary resuscitation compression operation, and comprises the following steps: analyzing standard operation and typical error operation in cardiopulmonary resuscitation manikin compression operation; collecting sensor data of standard operation and typical error operation of the cardiopulmonary resuscitation manikin through a sensor array and a camera device, performing feature extraction on the data collected by the sensor, and establishing a compression operation sensor evaluation model by using sensor feature data; obtaining a deep learning network operation part and action recognition model by deep learning training according to a manikin operation picture data set, and establishing a compression operation video evaluation model according to the recognition result; and evaluating the operation of an operator in combination with the compression operation video evaluation model and the sensor evaluation model; the application is applied to cardiopulmonary resuscitation compression operation automatic evaluation.

Description

TECHNICAL FIELD

[0001] The application provides a kind of cardiopulmonary resuscitation heart compression operation automatic evaluation device and method, belong to behavior operation automatic evaluation technical field. BACKGROUND

[0002] When cardiac arrest occurs, whether timely and effective cardiopulmonary resuscitation can be received within the golden four minutes of first aid is a key factor affecting the survival rate of patients with cardiac arrest, and the tragedy of losing life due to the failure to receive timely and effective cardiopulmonary resuscitation occurs frequently.

[0003] The effective implementation of cardiopulmonary resuscitation requires correct knowledge and skills, and most of the public cannot correctly respond to emergencies and perform effective cardiopulmonary resuscitation, so how to improve the popularization rate of cardiopulmonary resuscitation for the whole people is of great significance for the implementation of the national health strategy. However, many people still have limited understanding and knowledge of cardiopulmonary resuscitation, which leads to the fact that most of the public cannot correctly respond to emergencies and perform effective cardiopulmonary resuscitation.

[0004] Sensors can be used to monitor physiological parameters of patients during cardiopulmonary resuscitation and provide necessary feedback to ensure effective cardiopulmonary resuscitation. Modern sensor technology usually has high precision and can accurately measure the operation accuracy of the operator. In cardiopulmonary resuscitation, accurate data are crucial for judging the patient's condition and adjusting first aid measures. Sensors can provide real-time feedback to guide first aid personnel to adjust compression force, frequency, and ventilation depth, etc. to ensure the quality and effect of cardiopulmonary resuscitation. However, if the sensor fails or is accidentally disconnected, it may cause data collection to be interrupted, affecting the operation of the first aid personnel; the data reliability of the sensor is crucial for cardiopulmonary resuscitation, and some sensors may be affected by environmental interference, equipment failure, or improper placement, etc., resulting in problems in the accuracy and stability of the data; for some operations of cardiopulmonary resuscitation, it is difficult for the sensor to accurately collect data or to increase the difficulty and complexity of the operator's operation.

[0005] Deep learning image processing algorithms can analyze and learn a large amount of cardiopulmonary resuscitation data to extract valuable models and features, which can provide more scientific decision-making basis for the operator, and deep learning image processing algorithms provide real-time feedback and guidance to help the operator adjust the operation mode to ensure the correctness and effectiveness of cardiopulmonary resuscitation. However, deep learning image algorithms may be very sensitive to changes in input data, which may affect their accuracy and robustness in cardiopulmonary resuscitation operations; at the same time, deep learning algorithms are difficult to accurately provide judgment data when the operation accuracy requirement is high, such as compression depth. SUMMARY

[0006] The application discloses a device and a method for automatically evaluating cardiopulmonary resuscitation (CPR) heart compression operation.

[0007] In order to solve the above technical problems, the application adopts the technical scheme of a device for automatically evaluating CPR heart compression operation, which comprises a simulation man and a sensor array and a camera device for collecting standard operation and typical error operation of the simulation man.

[0008] The upper computer is provided with a compression position sensor evaluation model, a compression method sensor evaluation model, a compression depth sensor evaluation model, a compression frequency sensor evaluation model, a compression operation posture video evaluation model, a compression operation preparation video evaluation model and a compression operation process video evaluation model.

[0009] The data collected by the tactile sensor array, the pressure sensor array and the camera device are synchronously transmitted to the upper computer.

[0010] The sensor data is collected by a fixed baud rate and transmitted to the host computer through a serial port, the video data is collected by a fixed frame rate and transmitted to the host computer through a wireless network, and the host computer issues a start collection command; after collection, the sensor data and video data are added with time attributes for synchronization, the sensor data time attribute records the number of data, and the video data time attribute records the number of frame data, and the synchronization is realized by converting the fixed baud rate and fixed frame rate into time for synchronization judgment.

[0011] An automatic evaluation method for cardiopulmonary resuscitation heart compression operation, using an automatic evaluation device for cardiopulmonary resuscitation heart compression operation, comprising the following steps:

[0012] Step one, analyze the standard operation and typical error operation in the cardiopulmonary resuscitation mannequin compression operation, the standard operation includes accurate compression site, accurate compression technique, accurate compression depth and correct compression frequency, the typical error operation includes deviation of compression site, finger not raised, curved arm compression, compression depth too large or too small and compression frequency too large or too small;

[0013] Step two, collect sensor data of the standard operation and typical error operation of the cardiopulmonary resuscitation mannequin through a sensor array, collect video data of the standard operation and typical error operation of the cardiopulmonary resuscitation mannequin through a camera device, and synchronize the sensor data and video data during collection;

[0014] Step three, feature extraction is performed on the data collected by the sensor to obtain standard operation and typical error operation sensor feature data, and a cardiopulmonary resuscitation mannequin compression operation sensor evaluation model is established using the sensor feature data, the compression operation sensor evaluation model includes a compression position sensor evaluation model, a compression technique sensor evaluation model, a compression depth sensor evaluation model and a compression frequency sensor evaluation model;

[0015] Step four, sparse equal time interval sampling is performed on the video data of the standard operation and typical error operation of the cardiopulmonary resuscitation mannequin to obtain a series of operation pictures as a cardiopulmonary resuscitation mannequin operation picture data set;

[0016] Step five, the cardiopulmonary resuscitation mannequin deep learning network operation part and operation action recognition model is trained through the cardiopulmonary resuscitation mannequin operation picture data set to obtain the cardiopulmonary resuscitation mannequin deep learning network operation part and operation action recognition model;

[0017] Step six, the operator's operation result picture is recognized by the cardiopulmonary resuscitation mannequin deep learning network operation part and operation action recognition model that has been trained, the cardiopulmonary resuscitation mannequin operation part and operation part space position of the operator are recognized, and the operation time point is recorded;

[0018] Step seven, according to the identification of the operator's cardiopulmonary resuscitation simulation person operation part and the spatial position of the operation part, the operation action, the compression operation is summarized and induced, and the compression operation video evaluation model is established, the compression operation video evaluation model includes the compression operation preparation video evaluation model, the compression operation posture video evaluation model and the compression operation process video evaluation model;

[0019] Step eight, according to the cardiopulmonary resuscitation simulation person compression operation video evaluation model, the operation of the operator is evaluated, according to the cardiopulmonary resuscitation simulation person compression operation sensor evaluation model, the operation of the operator is evaluated, according to the operation time synchronization situation, combined with the cardiopulmonary resuscitation simulation person compression operation video evaluation model evaluation result and sensor evaluation model evaluation result, the operation evaluation result of the operator is given, whether the operation is standard, if the operation is not standard, the corresponding reason and suggestion are given.

[0020] The compression position sensor evaluation model in step three is centered on the correct compression position of the mannequin chest, and a NxN rectangular compression position area is established according to the installation position of the tactile sensor array in a 1:1 ratio. If the tactile sensor data at a certain position is greater than the contact threshold, the corresponding compression position area is assigned a value of 1, otherwise a value of 0. The area with a value of 1 is a polygon, and the centroid of the polygon is calculated.

[0021] According to the accurate operation of different operators on the compression site, the data collected by the sensor array when the compression site is operated to the left, right, up and down, the centroid of the polygon of each operation is calculated, and the centroid of the operation polygon is clustered according to the k-means clustering method. According to the clustering result, the area is divided into three sub-areas, namely the core area, the overlapping area and the deviation area. The core area refers to the area where the polygon centroid falls only when the compression site is accurately operated. The overlapping area refers to the area where the polygon centroid falls when the compression site is accurately operated and the polygon centroid falls when the compression site is deviated. The deviation area refers to the area where the polygon centroid falls only when the compression site is deviated.

[0022] The polygon whose centroid falls into the overlapping area is calculated. The area of the core area and the area of the deviation area in the overlapping area are calculated, and the ratio of the two is calculated. If the ratio is greater than the threshold, the compression site operation is accurate, otherwise the compression site is not accurate and the operation deviates.

[0023] The step three pressing method sensor evaluation model, according to the installation position of the tactile sensor array, establishes an NxN rectangular pressing position area, processes the area with value 1 using morphological closing operation and opening operation respectively to obtain areas A and B, calculates the area ratio R of areas A and B, and requires the palm root to be close to the patient's chest wall and the fingers to be raised during correct operation. The fingers of the typical incorrect operator are not raised. The R value is larger during correct operation and smaller during incorrect operation. The R value is used to evaluate whether the fingers are raised during pressing.

[0024] The step three pressing depth sensor evaluation model, according to the installation position of the pressure sensor array, establishes an NxN rectangular pressing position area in a 1:1 ratio, and according to the core area obtained by the pressing position sensor evaluation model during correct operation, the average value of the pressure sensor in the core area during correct operation is taken as the pressing depth value Si of the corresponding operation pressure sensor. The maximum value of the pressing depth of all operators during correct operation is taken as the upper limit of the interval, and the minimum value is taken as the lower limit, i.e. the correct operation value interval of the pressing depth is [min{Si}, max{Si}].

[0025] The step three pressing frequency sensor evaluation model, according to the correct operation pressure sensor pressing depth value Si obtained by the pressing depth sensor evaluation model, the pressure sensor pressing depth value Si within 1 second is plotted into a curve, the number of wave peaks appearing within 1 second is calculated, which is the pressing frequency Fi, the maximum value of the pressing frequency of all operators during correct operation is taken as the upper limit of the interval, and the minimum value is taken as the lower limit, i.e. the correct operation value interval of the pressing frequency is [min{Fi}, max{Fi}].

[0026] The establishment and evaluation criteria of the pressing operation preparation video evaluation model in step seven are as follows:

[0027] The standard operation involves simulating the state change of the operating part, as well as the operation action, and the state change of the operating part has spatial logical relationship and time sequence relationship. The pressing operation preparation stage evaluation model is established by correctly identifying the operating part, the spatial position of the operating part and the operation action in a certain time sequence;

[0028] A certain period of operation involves operating part A, operating part B and operation action C. The Euclidean distance of operating part A and operating part B needs to meet the spatial position relationship, and the time needs to meet the condition that operating part B appears after operating part A. Whether the spatial position relationship of operating part A and operating part B meets the requirements is judged by whether operating part A and operating part B can be correctly identified in the operation picture, and whether operating part B appears after operating part A is judged. Whether there is operation action C is judged. If the above conditions are met, the operation is correct, otherwise the operation is incorrect.

[0029] The establishment and evaluation criteria of the pressing operation posture video evaluation model in step seven are as follows:

[0030] The pressing posture involves multiple pressing positions, and the pressing position locations have spatial logical relationships. The pressing operation posture video evaluation model is established by correctly identifying the pressing positions and judging their spatial position relationships.

[0031] The standard operation of cardiopulmonary resuscitation cardiac compression requires the left leg and the patient's shoulder to be on a straight line. The operator's left leg A(x, y, w, h) and the patient's shoulder B(x, y, w, h) can be correctly identified in the operation picture, and then the vertical position relationship between the operator's left leg A and the patient's shoulder B is judged. The operator's left leg A(x, y, w, h) and the patient's shoulder B(x, y, w, h) are identified in the picture, and the spatial position relationship between the operator's left leg A and the patient's shoulder B satisfies By>Ay and ABS(Bx-Ax)<THx, ABS is the absolute value operation, THx is the horizontal distance judgment threshold value of the operator's left leg A and the patient's shoulder B. If the above conditions are met, the operation posture is standard, otherwise the operation is wrong.

[0032] The establishment and evaluation criteria of the pressing operation process video evaluation model in step seven are as follows:

[0033] The operation involves multiple operation positions, and the operation position locations have spatial logical relationships and time sequence relationships. The pressing operation process video evaluation model is established by correctly identifying the operation positions in a certain time sequence, the spatial position relationships of the operation positions, and the time sequence relationships of the operation positions.

[0034] The standard operation of cardiopulmonary resuscitation cardiac compression requires the palms of both hands to overlap, the fingers to cross and rise away from the thoracic cavity, and the arms to be straight. This operation involves the operator's overlapping hands A, the exposed manikin torso B at different times, the operator's straight arms C, and the operator's curved arms D. The Euclidean distance between the operator's overlapping hands A and the exposed manikin torso B needs to satisfy the spatial position relationship, and the distance between the operator's straight arms C at different times needs to satisfy the spatial position relationship.

[0035] The operation picture recognizes a surgeon's overlapping hand A (x, y, w, h, t1) at a certain moment, an exposed mannequin torso B (x, y, w, h, t1), a surgeon's straight arm C (x, y, w, h, t1) and a surgeon's curved arm D (x, y, w, h, t2), the operation picture recognizes a surgeon's overlapping hand A (x, y, w, h, t2) at another moment, an exposed mannequin torso B (x, y, w, h, t2), a surgeon's straight arm C (x, y, w, h, t2) and a surgeon's curved arm D (x, y, w, h, t2); the spatial position relationship between the exposed mannequin torso B and the surgeon's overlapping hand at the moment before t2 satisfies distance_A_B < THAB, wherein distance_A_B = ABS (A [x] - B [x]) + ABS (A [y] - B [y]), ABS is an absolute value operation, and THAB is a Euclidean distance decision threshold of the surgeon's overlapping hand A and the exposed mannequin torso B; the spatial position relationship of the surgeon's straight arm C at different moments satisfies THC1 < distance_Cy < THC2, wherein distance_Cy = ABS (Ct2 [y] - Ct1 [y]), ABS is an absolute value operation, THC1 and THC2 are vertical distance decision thresholds of the surgeon's straight arm C at different moments, Ct2 [y] is the vertical coordinate of the center of the surgeon's straight arm at t2, Ct1 [y] is the vertical coordinate of the center of the surgeon's straight arm at t1, and the surgeon's curved arm D is not recognized in this process, which satisfies the above conditions, and the pressing method step is standard; otherwise, the process recognizes the surgeon's curved arm D, which means that the pressing method operation is wrong, and the error reason is the curved arm operation.

[0036] The present application has the beneficial effects relative to the prior art: the present application combines sensor technology and deep learning image processing technology, uses a sensor array, avoids the influence of factors such as environmental interference, equipment failure or improper placement of a single sensor, which may cause problems in the accuracy and stability of data, maps the standard operation and typical error operation in cardiopulmonary resuscitation mannequin pressing operation to a cardiopulmonary resuscitation mannequin pressing operation video evaluation model and a pressing operation sensor evaluation model, monitors and evaluates the standardization and accuracy of the cardiopulmonary resuscitation operation of the operator in real time and provides personalized operation feedback and improvement suggestions, helps the operator to improve the operation skill, gradually improves the quality of the cardiopulmonary resuscitation operation, improves the cardiopulmonary resuscitation skill level of the general public, improves the success rate of rescue, and promotes the popularization and application of cardiopulmonary resuscitation skills. BRIEF DESCRIPTION OF DRAWINGS

[0037] The present application will be further described below in conjunction with the drawings:

[0038] Figure 1 The present application is a flowchart of the method;

[0039] Figure 2The overall block diagram of the device of the present application;

[0040] Figure 3 The installation position diagram of the sensor array S installed on the chest of the cardiopulmonary resuscitation simulator of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described below taking the cardiopulmonary resuscitation cardiac compression operation as an example.

[0042] The standard operation of the cardiac compression is as follows: (1) the operator stands or kneels on the side of the patient, with the left leg and the shoulder of the patient on a straight line; (2) unbutton the collar and unfasten the belt to expose the chest and abdomen of the patient; (3) the compression site: the intersection of the middle and lower 1 / 3 of the sternum, i.e. the intersection of the middle line of the sternum and the line connecting the two nipples; (4) the compression method: the root of the two palms overlaps, the fingers of the two hands cross and rise away from the chest; the arms are straight; the shoulders are above the sternum of the patient, and are vertically pressed downward; (5) the compression depth: the sternum is sunken, and the adult is 5-6 cm; (6) the compression frequency: 100-120 times / min. The typical incorrect operations include: (1) the compression site is not accurate when compressing; (2) the arm is curved when compressing; (3) the fingers are not raised when compressing; (4) the compression depth is too large or too small when compressing; (5) the compression frequency is not accurate when compressing. The steps of the automatic evaluation method of the cardiopulmonary resuscitation cardiac compression operation are as shown in Figure 1 , and the specific implementation steps are as follows:

[0043] The first step: analyze the standard operation and the typical incorrect operation in the compression operation of the cardiopulmonary resuscitation simulator, the standard operation includes accurate compression site, accurate compression depth, correct compression frequency, and the typical incorrect operation includes deviation of the compression site, fingers not raised, curved arm compression, compression depth too large or too small, and inaccurate compression frequency. Map the standard operation and the typical incorrect operation of this cardiopulmonary resuscitation operation to the operation sensor data set and the operation image data set of the standard operation and the typical incorrect operation. The operation sensor data set includes the tactile sensor array data set and the pressure sensor array data set. The operation image data set includes the operation site and the operation action, the operation site includes the hand of the operator (excluding the overlapped hand), the overlapped hand of the operator, the head of the operator, the torso of the operator, the leg of the operator, the straight arm of the operator, the curved arm of the operator, the head of the simulator, the neck of the simulator, the shoulder of the simulator, the unexposed torso of the simulator, the exposed torso of the simulator, etc., and the operation action includes the unbuttoning action, the straight arm compression action, the curved arm compression action and the palm contacting the chest wall action.

[0044] The second step: build a sensor array data acquisition and transmission module, take the correct compression position of the chest of the simulator as the center, and install a 13x13 sensor array S, as shown in Figure 2As shown, wherein the sensor array S includes a tactile sensor array and a pressure sensor array. The tactile sensor array and the pressure sensor array analog data are converted into digital data by an analog-to-digital conversion circuit, transmitted back to the MSP430 single-chip microcomputer through the I / O port of the MSP430 single-chip microcomputer, and the MSP430 single-chip microcomputer sorts the sensor array in sequence and transmits the tactile sensor array and the pressure sensor array data to the host computer through the RS232 serial port.

[0045] Third step: Through the sensor array data acquisition and transmission module built in the second step, the operator (the operator is representative, covering different age groups and different genders, not less than 20 people) respectively performs standard operation and typical error operation, and the sensor array collects the data of the standard operation and the typical error operation of the cardiopulmonary resuscitation manikin. The standard operation data includes accurate data of the pressing position, data of the finger lifting when pressing, accurate data of the pressing depth, and accurate data of the pressing frequency; the typical error operation data includes data of the pressing position deviating to the left, right, up and down (mainly collecting data of the distance from the accurate pressing position, which is less than the standard operation, and belongs to the deviation of the pressing position), data of the finger not lifting when pressing, data of the pressing depth being too large, data of the pressing depth being too small, data of the pressing frequency being too large, and data of the pressing frequency being too small. The above data are transmitted to the host computer through the serial port, and the overall block diagram of the device as shown in Figure 2

[0046] ​Step 4: The host computer extracts features from the data collected by the sensors and establishes evaluation models for the pressure position sensor and the pressure technique sensor. A 13x13 rectangular pressure position area is established according to the installation position of the tactile sensor array, at a 1:1 scale. The center coordinates of the 13x13 rectangular area are (x, y) (0, 0). Offsets to the right and downwards are positive, and offsets to the left and upwards are negative, establishing a Cartesian coordinate system. If the tactile sensor data at a certain position is greater than the contact threshold, the corresponding pressure position area is assigned a value of 1; otherwise, it is assigned a value of 0. The area assigned a value of 1 results in polygon M. The centroid Zxy of this polygon is then calculated. Based on the data collected by the sensor array when different operators press the correct area, and when the pressing area is slightly to the left, right, top, or bottom, the centroid Zxy of the polygon is calculated for each operation. Zxy is then clustered using k-means clustering, dividing the region into three sub-regions: the core region, the overlapping region, and the deviation region. The core region refers to the area where the polygon's centroid falls only when the pressing area is accurate. The overlapping region refers to the area where the polygon's centroid falls both when the pressing area is accurate and when the pressing area is deviated. The deviation region refers to the area where the polygon's centroid falls only when the pressing area is deviated. For polygons whose centroid falls within the overlapping region, the area of ​​the polygon falling within the core region and the area of ​​the polygon falling within the non-core region (overlapping and deviation regions) are calculated. The ratio of these two ratios is calculated. If the ratio is greater than a threshold, the pressing area is considered accurate; otherwise, the pressing area is considered inaccurate, and the operation is considered deviated. The regions assigned a value of 1 are processed using morphological closing and opening operations to obtain regions A and B, respectively. The ratio R of the areas of regions A and B is calculated. The correct operation requires the palm heel to be pressed firmly against the patient's chest wall with the fingers raised. In the incorrect operation, the fingers are not raised. The R value is larger when the operation is correct and smaller when the operation is incorrect. The R value is used to evaluate whether the operator's fingers are raised when pressing.

[0047] The fifth step: the host computer extracts the data collected by the sensor, establishes a pressing depth sensor evaluation model and a pressing frequency sensor evaluation model. According to the 1:1 ratio, 13x13 rectangular pressing position areas are established according to the installation position of the pressure sensor array, the center coordinates (x, y) of the 13x13 rectangular area are (0, 0), the right and downward offset is positive, and the left and upward offset is negative, and a rectangular coordinate system is established. According to the core area obtained when the correct operation in step three, the average value of the pressure sensor in the core area when the correct operation is obtained as the operation corresponding pressure sensor pressing depth value Si, the maximum value of the pressing depth number of all operators during correct operation as the upper limit of the interval, and the minimum value as the lower limit of the interval, that is, the correct operation value interval of the pressing depth is [min{Si}, max{Si}]. The pressure sensor pressing depth value Si within 1 second is plotted as a curve. Calculate the number of peaks that appear within 1 second, that is, the pressing frequency Fi, and the maximum value of the pressing frequency number of all operators during correct operation as the upper limit of the interval, and the minimum value as the lower limit of the interval, that is, the correct operation value interval of the pressing frequency is [min{Fi}, max{Fi}]. At the same time, the difference between adjacent peaks DFj is calculated, and the maximum and minimum values of DFj within 1 second are counted. The difference between the maximum and minimum values of DFj during correct operation of all operators is used as the threshold for judging smooth operation.

[0048] The sixth step: the operator (the operator is representative, covering different age groups and different genders, not less than 20 people) performs standard operation and typical error operation respectively. The present application collects video through the camera device, and samples the standard operation and typical error operation video at a sparse time interval, such as sampling the standard operation and typical error operation video at a sampling rate of 2 frames per second, to obtain pictures containing each operation part and operation action. The pictures containing all operation parts and operation actions obtained by sampling constitute the pressing operation picture data set. Use the image calibration tool software LabelImg to label the pressing operation picture data set, mainly label the operation parts and operation actions of the standard operation and typical error operation, and the labeling content includes operation part name, operation part center coordinates, operation part length, width and operation action, to obtain the pressing operation label data set. Use the deep learning network YOLOv7 to build a pressing operation automatic recognition model, train the pressing operation automatic recognition model through the pressing operation picture training data set and the pressing operation training label data set, verify the trained pressing operation automatic recognition model through the pressing operation picture verification data set and the pressing operation verification label data set, and the obtained pressing operation automatic recognition model can correctly identify the pressing part, the center coordinates of the pressing part, the length and width of the pressing part and the pressing action.

[0049] Step 7: Establish the pressing operation posture video evaluation model. The pressing posture involves multiple pressing parts, and the pressing part positions have spatial logical relationships. The pressing operation posture video evaluation model is established by correctly identifying the pressing parts and judging their spatial position relationships, including horizontal distance, vertical distance, and Euclidean distance, etc. Assume that the pressing operation requires the left leg and the patient's shoulder to be on a straight line. The operator's left leg A(x, y, w, h) and the patient's shoulder B(x, y, w, h) can be correctly identified in the operation picture, and then the vertical position relationship between the operator's left leg A and the patient's shoulder B is judged. The operator's left leg A(x, y, w, h) and the patient's shoulder B(x, y, w, h) are identified in the operation picture, and the spatial position relationship between the operator's left leg A and the patient's shoulder B satisfies By > Ay and ABS(Bx-Ax) < THx, ABS is the absolute value operation, and THx is the horizontal distance judgment threshold of the operator's left leg A and the patient's shoulder B. If the above conditions are met, the operation posture is standard, otherwise the operation is wrong.

[0050] Step 8: Establish the pressing operation preparation video evaluation model. The standard operation involves changes in the state of the manikin operation part, as well as operation actions, and the operation part state changes have spatial logical relationships and time sequence relationships. The pressing operation preparation stage evaluation model is established by correctly identifying the operation part, operation part spatial position, and operation action in a certain time sequence. Assume that a certain time operation involves operation part A, operation part B, and operation action C, and the Euclidean distance of operation part A and operation part B needs to satisfy the spatial position relationship, and the time needs to satisfy that operation part B appears after operation part A. By correctly identifying operation part A and operation part B in the operation picture, it is judged whether the spatial position relationship of operation part A and operation part B meets the requirements, whether operation part B appears after operation part A, and whether there is operation action C. Operation part A(x, y, w, h, t1), operation part B(x, y, w, h, t2), and operation action C(x, y, w, h, t3) can be identified in the operation picture; the spatial position relationship between operation part B and operation part A satisfies distance_A_B < THAB, where distance_A_B = ABS(Ax-Bx) + ABS(Ay-By), ABS is the absolute value operation, and THAB is the Euclidean distance judgment threshold of operation part A and operation part B; it is judged that t2 > t1, and t2 > t3 > t1. If the above conditions are met, the operation is standard, otherwise the operation is wrong.

[0051] The ninth step is to press the operation process video evaluation model. The operation involves multiple operation parts, and the operation part position has spatial logical relationship and time sequence relationship. The press operation process video evaluation model is established by correctly identifying the operation part of a time sequence, the spatial position relationship of the operation part and the time sequence relationship of the operation part. For example, the palms of both hands need to be overlapped during the pressing operation, and the fingers of both hands need to be crossed and raised away from the chest; the arms are straight, which involves the overlapped hands A of the operator, the exposed torso B of the manikin, the straight arms C of the operator and the curved arms D of the operator at different times, and the Euclidean distance between the overlapped hands A of the operator and the exposed torso B of the manikin needs to meet the spatial position relationship, and the distance between the straight arms C of the operator at different times needs to meet the spatial position relationship. The operation picture identifies the overlapped hands A of the operator (x, y, w, h, t1), the exposed torso B of the manikin (x, y, w, h, t1), the straight arms C of the operator (x, y, w, h, t1) and the curved arms D of the operator (x, y, w, h, t2) at a certain time, and the operation picture identifies the overlapped hands A of the operator (x, y, w, h, t2), the exposed torso B of the manikin (x, y, w, h, t2), the straight arms C of the operator (x, y, w, h, t2) and the curved arms D of the operator (x, y, w, h, t2) at another time; the spatial position relationship between the exposed torso B of the manikin and the overlapped hands of the operator before t2 meets distance_A_B < THAB, where distance_A_B = ABS(A[x]-B[x])+ABS(A[y]-B[y]), ABS is an absolute value operation, and THAB is the Euclidean distance decision threshold of the overlapped hands A of the operator and the exposed torso B of the manikin; the spatial position relationship of the straight arms C of the operator at different times meets THC1 < distance_Cy < THC2, where distance_Cy = ABS(Ct2[y]-Ct1[y]), ABS is an absolute value operation, THC1 and THC2 are the vertical distance decision thresholds of the straight arms C of the operator at different times, Ct2[y] is the vertical coordinate of the center of the straight arms of the operator at t2, Ct1[y] is the vertical coordinate of the center of the straight arms of the operator at t1, and the curved arms D of the operator are not identified in this process. If the above conditions are met, the pressing technique step is standard. Otherwise, the process identifies the curved arms D of the operator, which means that the pressing technique operation is wrong, and the error is the curved arm operation.

[0052] The tenth step is to collect sensor data and video data in real time. The sensor data and video data are synchronized during collection. The sensor data is collected at a fixed baud rate and transmitted to the host computer through the serial port. The video data is collected at a fixed frame rate and transmitted to the host computer through a wireless network. The host computer issues a start collection command. After collection, the sensor data and video data are given time attributes for synchronization. The sensor data time attribute records the number of data, and the video data time attribute records the number of frame data. The synchronization is judged by converting the fixed baud rate and the fixed frame rate to time.

[0053] Eleventh step: Perform the heart compression standard operation (1) that the operator stands or kneels on the side of the patient, the left leg and the shoulder of the patient are in a straight line, and the automatic evaluation is performed. This operation uses the compression operation posture video evaluation model established in the seventh step to evaluate, and the left leg A(x, y, w, h) of the operator and the shoulder B(x, y, w, h) of the patient can be correctly identified in the operation picture, and then the vertical position relationship between the left leg A of the operator and the shoulder B of the patient is judged. The operation picture identifies the left leg A(x, y, w, h) of the operator and the shoulder B(x, y, w, h) of the patient, the spatial position relationship between the left leg A of the operator and the shoulder B of the patient satisfies By>Ay and ABS(Bx-Ax)<THx, ABS is an absolute value operation, THx is the horizontal distance judgment threshold value of the left leg A of the operator and the shoulder B of the patient, if ABS(Bx-Ax)>THx, the value BAx of Bx-Ax is given. If the above conditions are met, the operation posture is standard, otherwise the operation is wrong, the reason is that the left leg of the operator and the shoulder of the patient are not in a straight line, the deviation degree is BAx, and it is suggested to adjust the left leg of the operator by about -BAx.

[0054] Twelfth step: Perform the heart compression standard operation (2) that the collar and the belt are unfastened to expose the chest and abdomen of the patient, and the automatic evaluation is performed. This operation uses the compression operation preparation video evaluation model established in the eighth step, which involves a period of time without exposing the simulated person's torso A, exposing the simulated person's torso B, and unfastening the clothes action C. The Euclidean distance between the unexposed simulated person's torso A and the exposed simulated person's torso B needs to satisfy the spatial position relationship, and the time needs to satisfy that the exposed simulated person's torso B appears after the unexposed simulated person's torso A. Through the operation picture, the unexposed simulated person's torso A and the exposed simulated person's torso B can be correctly identified, and the spatial position relationship between the unexposed simulated person's torso A and the exposed simulated person's torso B is judged to meet the requirements, and whether the exposed simulated person's torso B appears after the unexposed simulated person's torso A is judged, and whether there is an unfastening clothes action C is judged. The operation picture can identify the unexposed simulated person's torso A(x, y, w, h, t1), the exposed simulated person's torso B(x, y, w, h, t2) and the unfastening clothes action C(x, y, w, h, t3); the spatial position relationship between the exposed simulated person's torso B and the unexposed simulated person's torso A satisfies distance_A_B< THAB, wherein distance_A_B=ABS(Ax-Bx)+ABS(Ay-By), ABS is an absolute value operation, THAB is the Euclidean distance judgment threshold value of the unexposed simulated person's torso A and the exposed simulated person's torso B; judge t2>t1, and t2>t3>t1. If the above conditions are met, the operation is standard, otherwise the operation is wrong, if the exposed simulated person's torso B and the unfastening clothes action C are not identified at all, the error reason is that the patient's collar and belt are not unfastened, and it is suggested to unfasten the patient's collar and belt to expose the patient's chest and abdomen.

[0055] Thirteenth step: carry out the heart compression standard operation (3), i.e. the automatic evaluation of the compression site operation. This operation uses the compression position sensor evaluation model established in the fourth step to evaluate. In the operation, the tactile sensor array data is transmitted to the upper computer through the RS232 serial port, and the upper computer extracts the features of the tactile sensor array data. According to the 1:1 ratio, the 13x13 rectangular compression position area is established according to the installation position of the tactile sensor array, the center coordinates (x, y) of the 13x13 rectangular area are (0, 0), the right and downward offset is positive, and the left and upward offset is negative, and a rectangular coordinate system is established. If the data of a certain position tactile sensor is greater than the contact threshold, the corresponding compression position area is assigned a value of 1, otherwise a value of 0. The area assigned a value of 1 is a polygon M, and the center of gravity Z (x, y) of the polygon is calculated. If the center of gravity Z falls into the core area established by the compression position sensor evaluation model, the compression site operation is accurate; if the center of gravity Z falls into the deviation area established by the compression position sensor evaluation model, the compression site operation is inaccurate, and the operation deviates; if the center of gravity Z falls into the overlapping area established by the compression position sensor evaluation model, the area of the polygon falling into the core area and the area of the polygon falling into the non-core area (overlapping area and deviation area) are calculated, and the ratio of the two is calculated. If the ratio is greater than the threshold, the compression site operation is accurate, otherwise the compression site is inaccurate and the operation deviates.

[0056] Fourteenth step: Perform the heart compression standard operation (4), i.e., the compression method operation automatic evaluation. This operation uses the compression method sensor evaluation model established in the fourth step to evaluate. In operation, the tactile sensor array data is transmitted to the upper computer through the RS232 serial port, and the upper computer extracts the features of the tactile sensor array data. According to the 1:1 ratio, 13x13 rectangular compression position areas are established according to the installation position of the tactile sensor array, the center coordinates (x, y) of the 13x13 rectangular area are (0, 0), the right and downward offset is positive, and the left and upward offset is negative, and a rectangular coordinate system is established. If the data of a certain position tactile sensor is greater than the contact threshold, the corresponding compression position area is assigned a value of 1, otherwise a value of 0, and the area assigned a value of 1 is obtained as a polygon M. The area assigned a value of 1 is processed using morphological closing and opening operations respectively to obtain areas A and B, and the area ratio R of areas A and B is calculated. R is greater than the threshold value THR, the palm root is in close contact with the patient's chest wall, the finger is raised, and the operation is correct. R is less than the threshold value THR, the palm root is in close contact with the patient's chest wall, the finger is not raised, and the operation is incorrect. At the same time, video data is collected synchronously, and the ninth step compression operation process video evaluation model is used for automatic evaluation. This operation involves the superimposed hand A of the operator at different times, the exposed simulated human torso B, the straight arm C of the operator, and the spatial position relationship of the Euclidean distance between the superimposed hand A of the operator and the exposed simulated human torso B. The spatial position relationship of the straight arm C of the operator at different times needs to be satisfied. The operation picture recognizes the superimposed hand A (x, y, w, h, t1) of the operator at a certain time, the exposed simulated human torso B (x, y, w, h, t1), the straight arm C (x, y, w, h, t1) of the operator, and the curved arm D (x, y, w, h, t2) of the operator. The operation picture recognizes the superimposed hand A (x, y, w, h, t2) of the operator at another time, the exposed simulated human torso B (x, y, w, h, t2), the straight arm C (x, y, w, h, t2) of the operator, and the curved arm D (x, y, w, h, t2) of the operator. The spatial position relationship between the exposed simulated human torso B and the superimposed hand of the operator at t2 before satisfies distance_A_B < THAB, where distance_A_B = ABS(A[x]-B[x])+ABS(A[y]-B[y]), ABS is an absolute value operation, and THAB is a Euclidean distance decision threshold for the superimposed hand A of the operator and the exposed simulated human torso B. The spatial position relationship of the straight arm C of the operator at different times satisfies THC1 < distance_Cy < THC2, where distance_Cy = ABS(Ct2[y]-Ct1[y]), ABS is an absolute value operation, THC1 and THC2 are vertical distance decision thresholds for the straight arm C of the operator at different times, Ct2[y] is the vertical coordinate of the center of the straight arm of the operator at t2, Ct1[y] is the vertical coordinate of the center of the straight arm of the operator at t1, and the curved arm D of the operator is not recognized in this process. Satisfy the above conditions, this compression method step is standard.Otherwise, the process identifies the surgeon's elbow D, and the pressing method operation error is pressed, and the error reason is the elbow operation. The evaluation result given by the pressing method sensor evaluation model is combined with the pressing operation process video evaluation model to give the evaluation result of the pressing method operation. If the pressing method sensor evaluation model and the pressing operation process video evaluation model are both normal according to the evaluation result of the pressing method sensor evaluation model, a normal conclusion of the pressing method operation is given, if the evaluation result of the pressing method sensor evaluation model or the pressing operation process video evaluation model has an abnormal operation, an abnormal conclusion of the pressing method operation is given, and the corresponding reason and suggestion are given.

[0057] Fifteenth step: the operation of the standard operation of the cardiac compression (5) is automatically evaluated, that is, the operation of the compression depth. The operation uses the compression depth sensor evaluation model established in the fifth step to evaluate. In the operation, the pressure sensor array data is transmitted to the upper computer through the RS232 serial port, and the upper computer extracts the features of the pressure sensor array data. According to the 1:1 ratio, 13x13 rectangular compression position areas are established according to the installation position of the pressure sensor array, the center coordinates (x, y) of the 13x13 rectangular area are (0, 0), the right and downward offset is positive, the left and upward offset is negative, and a rectangular coordinate system is established. According to the core area obtained in the correct operation in the third step, the average value of the pressure sensor in the core area during the operation is obtained, which is the compression depth value Si of the corresponding pressure sensor of the operation. If the Si value is greater than max{Si}, the compression depth is wrong, and the compression depth is too large; if the Si value is less than min{Si}, the compression depth is wrong, and the compression depth is too small; if the Si value is in the interval [min{Si}, max{Si}], the compression depth operation is correct. At the same time, the video data is synchronously collected, and the ninth step is used to automatically evaluate the video evaluation model of the compression operation process. This operation involves the superimposed hand A of the operator at different times, the exposed simulated human torso B, the straight arm C of the operator, and the Euclidean distance between the superimposed hand A of the operator and the exposed simulated human torso B needs to satisfy the spatial position relationship, and the distance between the straight arm C of the operator at different times needs to satisfy the spatial position relationship. The operation picture recognizes the superimposed hand A (x, y, w, h, t1) of the operator at a certain time, the exposed simulated human torso B (x, y, w, h, t1), the straight arm C (x, y, w, h, t1) of the operator, and the curved arm D (x, y, w, h, t2) of the operator. The operation picture recognizes the superimposed hand A (x, y, w, h, t2) of the operator at another time, the exposed simulated human torso B (x, y, w, h, t2), the straight arm C (x, y, w, h, t2) of the operator, and the curved arm D (x, y, w, h, t2) of the operator. The spatial position relationship between the exposed simulated human torso B and the superimposed hand of the operator before t2 satisfies distance_A_B < THAB, where distance_A_B = ABS(A[x]-B[x])+ABS(A[y]-B[y]), ABS is an absolute value operation, and THAB is the Euclidean distance decision threshold of the superimposed hand A of the operator and the exposed simulated human torso B. The spatial position relationship of the straight arm C of the operator at different times satisfies THC1 < distance_Cy < THC2, where distance_Cy = ABS(Ct2[y]-Ct1[y]), ABS is an absolute value operation, THC1 and THC2 are the vertical distance decision thresholds of the straight arm C of the operator at different times, Ct2[y] is the vertical coordinate of the center of the straight arm of the operator at t2, Ct1[y] is the vertical coordinate of the center of the straight arm of the operator at t1, and the curved arm D of the operator is not recognized in this process. If the above conditions are met, the compression technique step is standard.Otherwise, the process identifies the operator elbow D, and the pressing manipulation operation error is pressed. The error is caused by the elbow operation. The evaluation results given by the pressing depth sensor evaluation model and the pressing operation process video evaluation model are combined to give the pressing depth operation evaluation result. If the evaluation results of the pressing depth sensor evaluation model and the pressing operation process video evaluation model are both normal, the pressing depth operation normal conclusion is given, if the pressing depth sensor evaluation model evaluation or the pressing operation process video evaluation model has non-standard operation, the pressing depth operation non-standard conclusion is given, and the corresponding reason and suggestion are given.

[0058] Sixteenth step: the operation of the standard operation of the heart compression (6) is the automatic evaluation of the compression frequency operation. This operation uses the compression frequency sensor evaluation model established in the fifth step to evaluate. When operating, the pressure sensor array data is transmitted to the upper computer through the RS232 serial port, and the upper computer extracts the features of the pressure sensor array data. According to the 1:1 ratio, 13x13 rectangular compression position area is established according to the installation position of the pressure sensor array, the center coordinates (x, y) of the 13x13 rectangular area are (0, 0), the right and downward offset is positive, the left and upward offset is negative, and a rectangular coordinate system is established. According to the core area obtained in the third step, the average value of the pressure sensor in the core area during operation is obtained, which is the pressure sensor compression depth value Si corresponding to the operation. The pressure sensor compression depth value Si in 1 second is drawn as a curve, the number of wave peaks appearing in 1 second is calculated, that is, the compression frequency Fi, and the difference DF between adjacent wave peaks is calculated. The maximum and minimum values of DFj in 1 second are counted, and the difference between the maximum and minimum values of DFj when all operators operate correctly is the operation stability judgment threshold. If the value of Fi is greater than max{Fi}, the compression frequency is wrong, and the compression frequency is too large; if the value of Fi is less than min{Fi}, the compression frequency is wrong, and the compression frequency is too small; if the value of Fi is in the interval [min{Fi}, max{Fi}], the compression frequency operation is correct. At the same time, the video data is collected synchronously, and the ninth step is used to evaluate the video evaluation model of the compression operation process. This operation involves the superimposed hand A of the operator at different times, the exposed simulated human torso B, the straight arm C of the operator, and the Euclidean distance between the superimposed hand A of the operator and the exposed simulated human torso B needs to meet the spatial position relationship, and the distance between the straight arm C of the operator at different times needs to meet the spatial position relationship.The operation picture recognizes the surgeon's overlapping hand A (x, y, w, h, t1), the exposed mannequin torso B (x, y, w, h, t1), the surgeon's straight arm C (x, y, w, h, t1) and the surgeon's curved arm D (x, y, w, h, t2) at a certain moment, and recognizes the surgeon's overlapping hand A (x, y, w, h, t2), the exposed mannequin torso B (x, y, w, h, t2), the surgeon's straight arm C (x, y, w, h, t2) and the surgeon's curved arm D (x, y, w, h, t2) at another moment; the spatial position relationship between the exposed mannequin torso B and the surgeon's overlapping hand at the moment before t2 satisfies distance_A_B < THAB, where distance_A_B = ABS (A [x] - B [x]) + ABS (A [y] - B [y]), ABS is an absolute value operation, and THAB is the Euclidean distance decision threshold of the surgeon's overlapping hand A and the exposed mannequin torso B; the spatial position relationship of the surgeon's straight arm C at different moments satisfies THC1 < distance_Cy < THC2, where distance_Cy = ABS (Ct2 [y] - Ct1 [y]), ABS is an absolute value operation, THC1 and THC2 are the vertical distance decision thresholds of the surgeon's straight arm C at different moments, Ct2 [y] is the vertical coordinate of the center of the surgeon's straight arm at t2, Ct1 [y] is the vertical coordinate of the center of the surgeon's straight arm at t1, and the surgeon's curved arm D is not recognized in this process. If the above conditions are met, this pressing method step is standard. Otherwise, the process recognizes the surgeon's curved arm D, and the pressing method operation is wrong, and the error reason is the curved arm operation. The evaluation results given by the pressing frequency sensor evaluation model and the pressing operation process video evaluation model are combined to give the evaluation result of the pressing frequency operation. If the evaluation results of the pressing frequency sensor evaluation model and the pressing operation process video evaluation model are both standard, the conclusion of the standard pressing frequency operation is given, if there is non-standard operation in the evaluation of the pressing frequency sensor evaluation model or the pressing operation process video evaluation model, the conclusion of the non-standard pressing frequency operation is given, and the corresponding reasons and suggestions are given.

[0059] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A device for automated assessment of cardiopulmonary resuscitation (CPR) chest compression performance, comprising a manikin and a sensor array and a video camera for capturing CPR manikin standard performance and typical error performance, characterized in that: The sensor array includes a tactile sensor array and a pressure sensor array, the tactile sensor array is used for collecting press position data when the operator performs cardiopulmonary resuscitation pressing operation, and the pressure sensor array is used for collecting press depth data when the operator performs cardiopulmonary resuscitation pressing operation, the tactile sensor array and the pressure sensor array are installed on the chest of the simulation man and centered on the correct heart pressing position, a camera device is further arranged on one side of the simulation man, and the camera device is used for collecting video data of preparation action, pressing posture and operation process of the operator before the operator performs cardiopulmonary resuscitation pressing operation, and the data collected by the above tactile sensor array, pressure sensor array and camera device is synchronously transmitted to the upper computer; The upper computer is provided with a press position sensor evaluation model, a press method sensor evaluation model, a press depth sensor evaluation model, a press frequency sensor evaluation model, a press operation posture video evaluation model, a press operation preparation video evaluation model and a press operation process video evaluation model, wherein the press position sensor evaluation model judges whether the pressing position of the operator is accurate through the contact threshold value of the tactile sensor array, the press method sensor evaluation model calculates whether the gesture of the operator in the pressing operation area is correct through the tactile sensor array, the press depth sensor evaluation model calculates the press depth according to the average value of the pressure sensor in the correct operation area collected by the pressure sensor array and judges whether it conforms to the specification, the press frequency sensor evaluation model calculates the press frequency according to the press depth data and compares it with the press frequency of the standard operation to make a judgment, the press operation posture video evaluation model judges whether the operation posture is standard according to the press position and its spatial logical relationship collected by the camera device, the press operation preparation video evaluation model judges whether the operation is standard according to the simulation man operation position, the operation position spatial position and the operation action of the operator collected by the camera device, and the press operation process video evaluation model judges whether the operation is standard according to the operation position, the spatial position relationship of the operation position and the time sequence relationship of the operation position collected by the camera device.

2. The automatic evaluation device for chest compression of cardiopulmonary resuscitation according to claim 1, characterized in that: The synchronous operation of the data collected by the tactile sensor array, the pressure sensor array and the camera device and transmitted to the upper computer is as follows: The sensor data is collected at a fixed baud rate and transmitted to the upper computer through a serial port, the video data is collected at a fixed frame rate and transmitted to the upper computer through a wireless network, and a start collection command is issued by the upper computer; after collection, the sensor data and the video data are added with time attributes for synchronization of the sensor data and the video data, the sensor data time attribute records the number of data, the video data time attribute records the number of frame data, and the synchronization is judged by converting the fixed baud rate and the fixed frame rate into time.

3. A method of automatically evaluating a chest compression operation in cardiopulmonary resuscitation, using the automatic evaluation device for a chest compression operation in cardiopulmonary resuscitation according to claim 1 or 2, characterized by: The method comprises the following steps: Step one, analyze the standard operation and typical error operation in the cardiopulmonary resuscitation simulation man pressing operation, the standard operation includes accurate pressing position, accurate pressing method, accurate pressing depth and correct pressing frequency, and the typical error operation includes deviation of pressing position, fingers not raised, curved arm pressing, too large or too small pressing depth and too large or too small pressing frequency. Step two, collect sensor data of standard operation and typical error operation of cardiopulmonary resuscitation simulator through sensor array, collect video data of standard operation and typical error operation of cardiopulmonary resuscitation simulator through camera, and synchronize sensor data and video data during collection; Step three, extract features from the data collected by the sensor to obtain sensor feature data of standard operation and typical error operation, and establish a compression operation sensor evaluation model of cardiopulmonary resuscitation simulator using the sensor feature data, wherein the compression operation sensor evaluation model includes a compression position sensor evaluation model, a compression method sensor evaluation model, a compression depth sensor evaluation model and a compression frequency sensor evaluation model; Step four, sparse and equal time interval sampling is performed on the video data of standard operation and typical error operation of cardiopulmonary resuscitation simulator to obtain a series of operation pictures as the operation picture data set of cardiopulmonary resuscitation simulator; Step five, train the operation part and operation action recognition model of the deep learning network of cardiopulmonary resuscitation simulator through the operation picture data set of cardiopulmonary resuscitation simulator to obtain the operation part and operation action recognition model of the deep learning network of cardiopulmonary resuscitation simulator; Step six, identify the operation result picture of the operator through the trained operation part and operation action recognition model of the deep learning network of cardiopulmonary resuscitation simulator to identify the operation part of the operator and the spatial position of the operation part, the operation action, and record the operation time point; Step seven, according to the operation part of the operator and the spatial position of the operation part, the operation action, summarize and induce the compression operation to establish a compression operation video evaluation model, wherein the compression operation video evaluation model includes a compression operation preparation video evaluation model, a compression operation posture video evaluation model and a compression operation process video evaluation model; Step eight, evaluate the operation of the operator according to the compression operation video evaluation model of the cardiopulmonary resuscitation simulator, and give the operation evaluation result of the operator according to the operation time synchronization condition, the evaluation result of the compression operation video evaluation model of the cardiopulmonary resuscitation simulator and the sensor evaluation model, whether the operation is standard, and give the corresponding reason and suggestion if the operation is not standard.

4. The method of claim 3, wherein the method further comprises: In the step three, the compression position sensor evaluation model is established according to the installation position of the tactile sensor array with the correct compression position of the simulator chest as the center and in a 1:1 ratio, the tactile sensor data of a certain position is greater than the contact threshold, then the corresponding compression position area is assigned as 1, otherwise as 0, the area with value 1 is a polygon, and the center of gravity of the polygon is calculated. According to the data collected by the sensor array when the pressing position is accurately operated, the data collected by the sensor array when the pressing position is operated to the left, to the right, to the upper and to the lower, the center of gravity of the polygon is calculated for each operation, and the center of gravity of the polygon is clustered according to the k-means clustering method. According to the clustering results, the area is divided into three sub-areas, namely the core area, the overlapping area and the deviation area. The core area refers to the area where the center of gravity of the polygon falls when the pressing position is accurately operated. The overlapping area refers to the area where the center of gravity of the polygon falls when the pressing position is accurately operated and the center of gravity of the polygon falls when the pressing position is deviated. The deviation area refers to the area where the center of gravity of the polygon falls only when the pressing position is deviated. If the center of gravity of the polygon falls in the overlapping area, the area of the core area and the area of the deviation area in the overlapping area are calculated, and the ratio of the two areas is calculated. If the ratio is greater than a threshold value, the pressing position is accurately operated, otherwise the pressing position is not accurate and the operation is deviated.

5. The method of claim 4, wherein the method further comprises: In step three, the pressing method sensor evaluation model is established according to the installation position of the tactile sensor array to establish an NxN rectangular pressing position area. The area with the value of 1 is processed by morphological closing operation and opening operation respectively to obtain areas A and B. The ratio R of the areas of A and B is calculated. The correct operation of pressing requires the palm root to be close to the patient's chest wall and the fingers to be raised. The typical incorrect operator does not raise his fingers. The R value is larger for correct operation and smaller for incorrect operation. The R value is used to evaluate whether the fingers are raised during pressing.

6. The method of claim 4, wherein the method further comprises: In step three, the pressing depth sensor evaluation model is established according to the installation position of the pressure sensor array to establish an NxN rectangular pressing position area. According to the core area obtained by the pressing position sensor evaluation model during correct operation, the average value of the pressure sensor in the core area during correct operation is taken as the pressing depth value Si of the corresponding pressure sensor. The maximum value of the pressing depth value Si of all operators during correct operation is taken as the upper limit of the interval, and the minimum value is taken as the lower limit of the interval, i.e. the correct operation value interval of the pressing depth is [min{Si}, max{Si}].

7. The method of claim 6, wherein the method further comprises: In step three, the pressing frequency sensor evaluation model is established according to the correct operation pressure sensor pressing depth value Si obtained by the pressing depth sensor evaluation model. The pressure sensor pressing depth value Si within 1 second is plotted into a curve, and the number of wave peaks appearing within 1 second is calculated, which is the pressing frequency Fi. The maximum value of the pressing frequency of all operators during correct operation is taken as the upper limit of the interval, and the minimum value is taken as the lower limit of the interval, i.e. the correct operation value interval of the pressing frequency is [min{Fi}, max{Fi}].

8. The method of claim 3, wherein the method further comprises: In step seven, the establishment and evaluation criteria of the pressing operation preparation video evaluation model are as follows: The standard operation involves the change of the state of the simulated human operation position, and involves the operation action. There is a spatial logical relationship and a time sequence relationship between the change of the state of the operation position. The pressing operation preparation stage evaluation model is established by correctly identifying the operation position, the spatial position of the operation position and the operation action in a certain time sequence. The operation of a certain period of time involves operation site A, operation site B and operation action C. The Euclidean distance of operation site A and operation site B needs to meet the spatial position relationship, and the time needs to meet the condition that operation site B appears after operation site A. Whether the spatial position relationship of operation site A and operation site B meets the requirement is determined by whether operation site A and operation site B can be correctly recognized in the operation picture, and whether operation site B appears after operation site A is determined. Whether there is operation action C is determined. If the above conditions are met, the operation is correct, otherwise the operation is incorrect.

9. The method of claim 3, wherein the method further comprises: The establishment of the pressing operation posture video evaluation model in step seven and the evaluation criteria are as follows: The pressing posture involves multiple pressing sites, and the pressing site positions have spatial logical relationship. The pressing operation posture video evaluation model is established by correctly identifying the pressing sites and judging their spatial position relationship. The standard operation of cardiopulmonary resuscitation heart compression requires that the left leg and the shoulder of the patient are on a straight line. The vertical position relationship between the surgeon's left leg A(x, y, w, h) and the patient's shoulder B(x, y, w, h) is determined by correctly recognizing the surgeon's left leg A(x, y, w, h) and the patient's shoulder B(x, y, w, h) in the operation picture. The surgeon's left leg A(x, y, w, h) and the patient's shoulder B(x, y, w, h) are recognized in the picture. The spatial position relationship between the surgeon's left leg A and the patient's shoulder B satisfies By>Ay and ABS(Bx-Ax)<THx. ABS is the absolute value operation, and THx is the horizontal distance judgment threshold value of the surgeon's left leg A and the patient's shoulder B. If the above conditions are met, the operation posture is correct, otherwise the operation is incorrect.

10. The method of claim 3, wherein the method further comprises: The establishment of the pressing operation process video evaluation model in step seven and the evaluation criteria are as follows: The operation involves multiple operation sites, and the operation site positions have spatial logical relationship and time sequence relationship. The pressing operation process video evaluation model is established by correctly identifying the operation site of a certain period of time, the spatial position relationship of the operation site and the time sequence relationship of the operation site. The standard operation of cardiopulmonary resuscitation heart compression requires that the two palms are overlapped, the fingers are crossed and raised away from the thoracic cavity, and the arms are straight. This operation involves the surgeon's overlapped hands A, the exposed manikin torso B at different times, the surgeon's straight arm C and the surgeon's curved arm D. The Euclidean distance of the surgeon's overlapped hands A and the exposed manikin torso B needs to meet the spatial position relationship, and the distance of the surgeon's straight arm C at different times needs to meet the spatial position relationship. The operation picture recognizes the surgeon's overlapping hand A (x, y, w, h, t1), the exposed mannequin torso B (x, y, w, h, t1), the surgeon's straight arm C (x, y, w, h, t1) and the surgeon's curved arm D (x, y, w, h, t2) at a certain moment, recognizes the surgeon's overlapping hand A (x, y, w, h, t2), the exposed mannequin torso B (x, y, w, h, t2), the surgeon's straight arm C (x, y, w, h, t2) and the surgeon's curved arm D (x, y, w, h, t2) at another moment; the spatial position relationship between the exposed mannequin torso B and the surgeon's overlapping hand at the moment before t2 satisfies distance_A_B < THAB, wherein distance_A_B = ABS(A[x]-B[x])+ABS(A[y]-B[y]), ABS is an absolute value operation, and THAB is the Euclidean distance decision threshold of the surgeon's overlapping hand A and the exposed mannequin torso B; the spatial position relationship of the surgeon's straight arm C at different moments satisfies THC1 < distance_Cy < THC2, wherein distance_Cy = ABS(Ct2[y]-Ct1[y]), ABS is an absolute value operation, THC1 and THC2 are the vertical distance decision thresholds of the surgeon's straight arm C at different moments, Ct2[y] is the vertical coordinate of the center of the surgeon's straight arm at t2, Ct1[y] is the vertical coordinate of the center of the surgeon's straight arm at t1, and the surgeon's curved arm D is not recognized in this process; if the above conditions are met, the pressing method step is standardized; otherwise, the process recognizes the surgeon's curved arm D, which means that the pressing method operation is wrong, and the error reason is the curved arm operation.

Images