Abnormal behavior detection method and device, electronic equipment and storage medium

Abstract

The invention discloses an abnormal behavior detection method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining an image frame sequence of a patient in response to triggering of an abnormal behavior detection event; wherein the image frame sequence comprises at least two image frames; for each image frame in the image frame sequence, determining a human body key point of the image frame, determining a head neighborhood in the image frame according to the human body key point, and determining a relative position relationship between a wrist key point in the human body key point and the head neighborhood; determining a human body key point change degree and an optical flow field change degree between any two target image frames spaced by a preset number of image frames in the image frame sequence; according to the relative position relation, the human body key point change degree and the optical flow field change degree, whether the patient has abnormal behaviors or not is detected. According to the scheme, whether the patient has the abnormal behavior or not can be rapidly detected, the workload of medical staff can be greatly reduced, and the patient treatment efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an abnormal behavior detection method, device, electronic equipment and storage medium. Background Art

[0002] The effective treatment of diseases such as severe illness depends not only on the severity of the patient's condition and the treatment plan, but also on the close care and timely decision-making of medical staff. When bedridden patients encounter emergencies, medical staff need to discover and deal with them in time. At present, hospital wards mainly rely on monitors, ventilators and other equipment to collect data such as patients' physiological indicators in real time, and use changes in indicator data to warn of emergencies. However, the abnormal limb behavior of critically ill patients also requires real-time attention from doctors and patients, and currently mainly relies on nurses to check the patient's condition regularly. At present, with the construction of hospital information systems, the signals of bedside equipment such as monitors and ventilators can be collected and transmitted in real time, enabling medical staff to remotely monitor patient vital signs. However, there are no equipment and means to detect abnormal behavior of patients. Summary of the invention

[0003] The present invention provides an abnormal behavior detection method, device, electronic device and storage medium, which can quickly detect whether a patient has abnormal behavior, greatly reduce the workload of medical staff and improve the efficiency of patient treatment.

[0004] According to one aspect of the present invention, there is provided a method for detecting abnormal behavior, comprising:

[0005] In response to an abnormal behavior detection event being triggered, acquiring an image frame sequence of the patient; wherein the image frame sequence includes at least two image frames;

[0006] For each image frame in the image frame sequence, determine a human body key point of the image frame, determine a head neighborhood in the image frame according to the human body key point, and determine a relative positional relationship between a wrist key point in the human body key point and the head neighborhood;

[0007] Determine the degree of change of key points of the human body and the degree of change of the optical flow field between any two target image frames separated by a preset number of image frames in the image frame sequence;

[0008] Whether the patient has abnormal behavior is detected based on the relative position relationship, the degree of change of the key points of the human body and the degree of change of the optical flow field.

[0009] According to another aspect of the present invention, there is provided an abnormal behavior detection device, comprising:

[0010] An image frame sequence acquisition module, configured to acquire an image frame sequence of a patient in response to an abnormal behavior detection event being triggered; wherein the image frame sequence includes at least two image frames;

[0011] A relative position relationship determination module, used for determining, for each image frame in the image frame sequence, a human body key point of the image frame, determining a head neighborhood in the image frame according to the human body key point, and determining a relative position relationship between a wrist key point in the human body key point and the head neighborhood;

[0012] A change degree determination module, used to determine the change degree of human key points and the change degree of optical flow field between any two target image frames separated by a preset number of image frames in the image frame sequence;

[0013] The abnormal behavior detection module is used to detect whether the patient has abnormal behavior according to the relative position relationship, the degree of change of the key points of the human body and the degree of change of the optical flow field.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the abnormal behavior detection method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the abnormal behavior detection method described in any embodiment of the present invention when executed.

[0019] The abnormal behavior detection scheme of the embodiment of the present invention, in response to the abnormal behavior detection event being triggered, obtains the patient's image frame sequence; wherein the image frame sequence includes at least two image frames; for each image frame in the image frame sequence, determines the human body key points of the image frame, determines the head neighborhood in the image frame based on the human body key points, and determines the relative position relationship between the wrist key points in the human body key points and the head neighborhood; determines the degree of change of the human body key points and the degree of change of the optical flow field between any two target image frames separated by a preset number of image frames in the image frame sequence; detects whether the patient has abnormal behavior based on the relative position relationship, the degree of change of the human body key points and the degree of change of the optical flow field. Through the technical scheme provided by the embodiment of the present invention, it is possible to quickly detect whether the patient has abnormal behavior, which can greatly reduce the workload of medical staff and improve the efficiency of patient treatment.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of an abnormal behavior detection method provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the distribution of key points of a human body provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a head neighborhood provided by an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of dividing the optical flow attention area when a patient is lying horizontally provided in an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of dividing the optical flow attention area when a patient is in a vertical lying position provided by an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of a process for determining whether a patient has abnormal behavior provided by an embodiment of the present invention;

[0028] Figure 7 A schematic diagram of the structure of an abnormal behavior detection device provided by an embodiment of the present invention;

[0029] Figure 8 A schematic diagram of the structure of an electronic device for implementing the abnormal behavior detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Figure 1 This is a flowchart of an abnormal behavior detection method provided by an embodiment of the present invention. This embodiment can be applied to the case of detecting whether a patient has abnormal behavior. The method can be performed by an abnormal behavior detection device. The abnormal behavior detection device can be implemented in the form of hardware and / or software. The abnormal behavior detection device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0033] S110 . In response to an abnormal behavior detection event being triggered, acquiring an image frame sequence of the patient; wherein the image frame sequence includes at least two image frames.

[0034] The abnormal behavior detection method provided by the embodiment of the present invention is applied to the application scenario of abnormal behavior detection for critically ill patients in a hospital. Exemplarily, when an abnormal behavior detection instruction input by a medical staff is detected, it is determined that an abnormal behavior detection event is triggered; another exemplary method is to set an abnormal behavior detection cycle, and when the abnormal behavior detection cycle is reached, it is determined that an abnormal behavior detection event is triggered. In response to the abnormal behavior detection event being triggered, an image frame sequence of the patient is acquired, wherein the image frame sequence includes at least two image frames containing the patient.

[0035] S120. For each image frame in the image frame sequence, determine a human body key point of the image frame, determine a head neighborhood in the image frame according to the human body key point, and determine a relative positional relationship between a wrist key point in the human body key point and the head neighborhood.

[0036] In an embodiment of the present invention, for each image frame in the image frame sequence, the human key points in the image frame are determined based on a preset human key point detection algorithm. Exemplarily, for each image frame in the image frame sequence, the image frame is input into a human target detector based on YOLOX to obtain a human target detection frame in the image frame, and the human body region is cropped out from the image frame according to the human target detection frame, and the human body region is input into a human posture estimator based on RTMPose (i.e., a human key point detector) to obtain the human body key points in the image frame. Optionally, the confidence of each human key point in the image frame can be calculated according to a preset confidence calculation method, and the key points whose confidence is less than a preset confidence threshold are filtered out. Exemplarily, Figure 2 A schematic diagram of the distribution of key points of a human body provided by an embodiment of the present invention. Figure 2 As shown, each image frame can include 17 human body key points.

[0037] In an embodiment of the present invention, for each image frame in an image frame sequence, a head neighborhood in the image frame is determined based on the human body key points in the image frame, wherein the head neighborhood can be understood as an area containing the patient's head. For example, the minimum circumscribed rectangular area containing the patient's head is determined based on the human body key points, and the minimum circumscribed rectangular area containing the patient's head is used as the head neighborhood in the image frame. Optionally, determining the head neighborhood in the image frame based on the human body key points includes: determining two upper arm midpoints and two eye points of the patient in the image frame based on the human body key points; and using a quadrilateral area formed by side lines that pass through each upper arm midpoint and are perpendicular to the line connecting the two upper arm midpoints, the line connecting the two upper arm midpoints, and the line connecting the two eye points as the head neighborhood in the image frame. Exemplarily, the two upper arm midpoints of the patient in the image frame are determined based on the human body key points, such as Figure 2As shown, the midpoint A of the line segment formed by human key point No. 6 and human key point No. 8 and the midpoint B of the line segment formed by human key point No. 5 and human key point No. 7 are respectively used as the two midpoints of the patient's upper arms. Human key point No. 1 and human key point No. 2 are the two eye points of the patient. A sideline AC is made perpendicular to the line AB connecting the two midpoints of the upper arms through the midpoint A of the upper arm, and a sideline BD is made perpendicular to the line AB connecting the two midpoints of the upper arms through the midpoint B of the upper arm, and the quadrilateral area formed by ABCD is used as the head neighborhood in the image frame, wherein the line segment CD passes through the two eye points of the patient (human key point No. 1 and human key point No. 2). Exemplarily, Figure 3 A schematic diagram of a head neighborhood provided for an embodiment of the present invention.

[0038] In an embodiment of the present invention, the head neighborhood in each image frame in the image frame sequence can be determined according to the above method, and the relative position relationship between the wrist key point in the human body key points in the image frame and the corresponding head neighborhood can be determined. Among them, the relative position relationship between the wrist key point and the head neighborhood includes the wrist key point being in the head neighborhood and the wrist key point being outside the head neighborhood. Since the human body key points include two wrist key points (human body key point No. 9 and human body key point No. 10), as long as there is a wrist key point in the head neighborhood, it is considered that the wrist key point is in the head neighborhood. Exemplarily, the relative position relationship between the wrist key point and the head neighborhood can be determined based on the following vector method: 1. The four vertices of the head neighborhood are represented as A(x1, y1), B(x2, y2), C(x3, y3), and D(x4, y4), and the wrist key point to be determined is represented as P(x, y); 2. Calculate the vectors from each vertex of the head neighborhood to the wrist key point. For example, the vector from A to P can be represented as 3. Calculate the cross product: Calculate the following cross product: Among them, in two-dimensional space, the vector With vector The cross product of is defined as u1v2-u2v1; 4. Determine the sign of the cross product: If the signs of the four cross products are all negative, it can be determined that P is in the head neighborhood. According to the above method, the relative position relationship between the wrist key point and the head neighborhood in each image frame in the image frame sequence can be determined.

[0039] S130, determining the degree of change of key points of the human body and the degree of change of the optical flow field between any two target image frames separated by a preset number of image frames in the image frame sequence.

[0040] In an embodiment of the present invention, the image frame sequence includes a plurality of image frames, and any two target image frames separated by a preset number of image frames are determined from the image frame sequence. For example, the image frame sequence includes 16 image frames, and the preset number is 10. Then the first image frame and the 11th image frame in the image frame sequence can be used as target image frames, or the second image frame and the 12th image frame in the image frame sequence can be used as target image frames, or the sixth image frame and the 16th image frame in the image frame sequence can be used as target image frames. Optionally, the image frame sequence of the patient is obtained according to a preset abnormal behavior detection cycle, and the first image frame and the last image frame in the image frame sequence can be used as target image frames. The degree of change of the human body key points and the degree of change of the optical flow field between the two target image frames are determined, wherein the degree of change of the human body key points is used to reflect whether the change of the human body key points between the two target image frames is drastic, and the degree of change of the optical flow field is used to reflect whether the change of the optical flow field between the two target image frames is drastic.

[0041] Optionally, determining the degree of change of the key points of the human body between any two target image frames separated by a preset number of image frames in the image frame sequence includes: determining the key point vector modulus between each corresponding human body key point between any two target image frames separated by a preset number of image frames in the image frame sequence; and determining the degree of change of the key points of the human body between the two target image frames according to the key point vector modulus. Exemplarily, the key point vector modulus of each corresponding human body key point between two target image frames is calculated, for example, the key point vector modulus between the left eye key point in the first target image frame and the left eye key point in the second target image frame is calculated, and the key point vector modulus between the left wrist key point in the first target image frame and the left wrist key point in the second target image frame is calculated. In the above manner, the key point vector modulus of 17 corresponding human body key points between two target image frames can be calculated. The degree of change of the human body key points between the two target image frames is determined based on the 17 key point vector moduli. For example, the average of the 17 key point vector moduli can be used as the degree of change of the human body key points between the two target image frames, or the number of key point vector moduli greater than a preset moduli can be determined from the 17 key point vector moduli, and the number of key point vector moduli greater than the preset moduli can be used as the degree of change of the human body key points between the two target image frames.

[0042] Optionally, determining the degree of change in the optical flow field between any two target image frames separated by a preset number of image frames in the image frame sequence includes: determining, for each image frame in the image frame sequence, each optical flow attention area in the image frame; calculating the average modulus of the cumulative change in the optical flow field within the optical flow attention area at a corresponding position in all image frames between the two target image frames; and determining the degree of change in the optical flow field between the two target image frames based on the average modulus of the cumulative change in the optical flow field.

[0043] Exemplarily, for each image frame in the image frame sequence, the current image frame and the previous adjacent image frame are input into an optical flow detection model (such as a LiteFlowNet2 model) to determine the optical flow field of the current image frame, wherein the optical flow field of the current image frame can be represented by an optical flow vector matrix. The average modulus of the cumulative change of the optical flow field in the optical flow attention area at the corresponding position in all image frames between the two target image frames is calculated. For example, the image frame sequence contains 16 image frames, and the two target image frames are the 1st image frame and the 16th image frame in the image frame sequence, respectively. Then, the average modulus of the cumulative change of the optical flow field in the optical flow attention area of ​​the 2nd to 15th image frames is calculated, and the average modulus of the cumulative change of the optical flow field is used as the degree of change of the optical flow field between the two target image frames.

[0044] Optionally, for each image frame in the image frame sequence, determining each optical flow attention area in the image frame includes: for each image frame in the image frame sequence, determining the presentation angle of the patient in the image frame; and determining each optical flow attention area in the image frame according to the presentation angle. In an embodiment of the present invention, the image frame sequence of the patient is collected at different angles, and the presentation angle of the patient in the image frame may be different. Since the patient may present two perspectives of lying horizontally or lying vertically, the presentation angle of the patient in the image frame includes a horizontally lying perspective and a vertically lying perspective. Exemplarily, for each image frame in the image frame sequence, it is determined whether the slope of the line connecting the two shoulder key points (human key point No. 5 and human key point No. 6) in the image frame exceeds a preset slope threshold. If so, it is determined that the presentation of the patient in the image frame is regarded as a horizontally lying perspective, otherwise, it is determined that the presentation of the patient in the image frame is regarded as a vertically lying perspective. Determine each optical flow attention area in the image frame according to the presentation angle of the patient in the image frame.

[0045] Optionally, the presentation viewing angle includes a horizontal viewing angle and a vertical viewing angle; determining each optical flow attention area in the image frame according to the presentation viewing angle, including: when the presentation viewing angle is a horizontal viewing angle, determining the midpoints of the two shoulders and the midpoint of the hip of the patient in the image frame according to the human body key points; determining each first human body sub-area formed by dividing the human body area in the image frame by a vertical line passing through the midpoints of the two shoulders and a vertical line passing through the midpoint of the hip, and using each of the first human body sub-area as each optical flow attention area in the image frame; when the presentation viewing angle is a vertical viewing angle, determining the high shoulder of the two shoulder points of the patient in the image frame according to the human body key points. point and the high hip joint point among the two hip joint points; determine the second human body sub-regions formed by dividing the human body area in the image frame by a horizontal line passing through the high shoulder point and a horizontal line passing through the high hip joint point; respectively determine the human body enclosing area in each of the second human body sub-regions, and use each of the human body enclosing area as a respective optical flow attention area in the image frame; wherein, the height of the human body enclosing area is the same as the height of the corresponding second human body sub-region, the width of the human body enclosing area is smaller than the width of the corresponding second human body sub-region, and the width of the human body enclosing area is the product of the maximum human body width in the corresponding second human body sub-region and a preset proportional coefficient.

[0046] In an embodiment of the present invention, when the presentation angle of the patient in the image frame is a horizontal lying angle of view, the midpoints of the two shoulders (that is, the midpoint of the line connecting the No. 5 human key point and the No. 6 human key point) and the midpoint of the hip (the midpoint of the line connecting the No. 11 human key point and the No. 12 human key point) of the patient in the image frame are determined according to the human key points in the image frame. A first vertical line is made through the midpoints of the two shoulders, and a second vertical line is made through the midpoint of the hip. The first vertical line and the second vertical line divide the human body area in the image frame into three human body sub-areas, and these three human body sub-areas are used as the respective optical flow attention areas in the image frame. Exemplarily, Figure 4 A schematic diagram of dividing the optical flow attention area when a patient is lying horizontally is provided in an embodiment of the present invention. Figure 4As shown, the optical flow attention area No. 1 is the patient's head area, the optical flow attention area No. 2 is the patient's torso area, and the optical flow attention area No. 3 is the patient's leg area. When the patient's presentation perspective in the image frame is a vertical lying perspective, the relatively higher high shoulder point of the two shoulder points of the patient in the image frame and the relatively higher high hip joint point of the two hip joint points are determined according to the key points of the human body in the image frame. A first horizontal line is made through the high shoulder point, and a second horizontal line is made through the high hip joint point. The first horizontal line and the second horizontal line divide the human body area in the image frame into three human body sub-areas (i.e., the second human body sub-area). For each second human body sub-area, a human body enclosing area in the second human body sub-area is determined, wherein the height of the human body enclosing area is the same as the height of the second human body sub-area, the width of the human body enclosing area is less than the width of the corresponding second human body sub-area, and the width of the human body enclosing area is the product of the maximum human body width in the second human body sub-area and the preset proportional coefficient. It should be noted that the preset proportional coefficients corresponding to each second human body sub-area are greater than or equal to 1, wherein the preset proportional coefficients corresponding to each second human body sub-area can be the same or different. It can be understood that the human body enclosed area is a sub-area of ​​the second human body sub-area. The human body enclosed areas in the three human body sub-areas are used as the respective optical flow attention areas in the image frame. Exemplarily, Figure 5 A schematic diagram of the division of the optical flow attention area when the patient is in a vertical lying position provided by an embodiment of the present invention. Figure 5 As shown, optical flow attention area No. 1 is the patient's head area, optical flow attention area No. 2 is the patient's torso area, and optical flow attention area No. 3 is the patient's leg area.

[0047] Optional, when Figure 4 or Figure 5 When the three human body sub-regions shown are used as optical flow attention regions, the average modulus of the cumulative change of the optical flow field in each optical flow attention region at the corresponding position in the two target image frames can be determined, and the number of the average modulus of the cumulative change of the optical flow greater than the preset optical flow change threshold is counted, and the number of the average modulus of the cumulative change of the optical flow greater than the preset optical flow change threshold is used as the degree of change of the optical flow field.

[0048] S140, detecting whether the patient has abnormal behavior according to the relative position relationship, the degree of change of the key points of the human body and the degree of change of the optical flow field.

[0049] In an embodiment of the present invention, whether a patient has abnormal behavior is detected based on the relative position relationship between the wrist key point and the head neighborhood in each image frame in the image frame sequence, the degree of change of the human body key points and the degree of change of the optical flow field between any two target image frames separated by a preset number of image frames in the image frame sequence. Optionally, the relative position relationship includes that the wrist key point is within the head neighborhood and that the wrist key point is outside the head neighborhood; based on the relative position relationship, the degree of change of the human body key points and the degree of change of the optical flow field, detecting whether the patient has abnormal behavior, including: if there is at least one image frame in the image frame sequence whose corresponding relative position relationship is that the wrist key point is within the head neighborhood, determining that the patient has abnormal behavior suspected of extubation; if the relative position relationship corresponding to each image frame in the image frame sequence is that the wrist key point is outside the head neighborhood, and the degree of change of the optical flow field is greater than a preset optical flow field change threshold, determining that the patient has abnormal behavior of severe agitation; if the relative position relationship corresponding to each image frame in the image frame sequence is that the wrist key point is outside the head neighborhood, and the degree of change of the optical flow field is less than the preset optical flow field change threshold and the degree of change of the human body key points is greater than the preset human body key point change threshold, determining that the patient has abnormal behavior of mild agitation.

[0050] For example, Figure 6 A schematic diagram of a process for determining whether a patient has abnormal behavior provided by an embodiment of the present invention. Figure 6 As shown, it is determined whether there is at least one image frame in the image frame sequence in which the wrist key point is in the vicinity of the head. If so, it can be determined that the patient has abnormal behavior suspected of extubation. If there is no image frame in the image frame sequence in which the wrist key point is in the vicinity of the head, that is, the wrist key point in each image frame in the image frame sequence is outside the vicinity of the head, it is determined whether the degree of change of the optical flow field between any two target image frames separated by a preset number of image frames in the image frame sequence is greater than the preset optical flow field change threshold. If so, it is determined that the patient has abnormal behavior of severe agitation. If not, it is further determined whether the degree of change of the human body key points between any two target image frames separated by a preset number of image frames in the image frame sequence is greater than the preset human body key point change threshold. If so, it is determined that the patient has abnormal behavior of mild agitation. Otherwise, it is determined that the patient is in a normal state, that is, there is no abnormal behavior.

[0051] The abnormal behavior detection method of the embodiment of the present invention obtains a patient's image frame sequence in response to an abnormal behavior detection event being triggered; wherein the image frame sequence includes at least two image frames; for each image frame in the image frame sequence, the human body key points of the image frame are determined, the head neighborhood in the image frame is determined based on the human body key points, and the relative position relationship between the wrist key points in the human body key points and the head neighborhood is determined; the degree of change of the human body key points and the degree of change of the optical flow field between any two target image frames separated by a preset number of image frames in the image frame sequence are determined; based on the relative position relationship, the degree of change of the human body key points and the degree of change of the optical flow field, the patient is detected to have abnormal behavior. Through the technical solution provided by the embodiment of the present invention, it is possible to quickly detect whether a patient has abnormal behavior, which can greatly reduce the workload of medical staff and improve the efficiency of patient treatment.

[0052] Figure 7 Schematic diagram of the structure of an abnormal behavior detection device provided by an embodiment of the present invention. Figure 7 As shown, the device comprises:

[0053] The image frame sequence acquisition module 710 is used to acquire an image frame sequence of the patient in response to the abnormal behavior detection event being triggered; wherein the image frame sequence includes at least two image frames;

[0054] A relative position relationship determination module 720 is used to determine, for each image frame in the image frame sequence, a human body key point of the image frame, determine a head neighborhood in the image frame according to the human body key point, and determine a relative position relationship between a wrist key point in the human body key point and the head neighborhood;

[0055] A change degree determination module 730 is used to determine the change degree of human key points and the change degree of the optical flow field between any two target image frames separated by a preset number of image frames in the image frame sequence;

[0056] The abnormal behavior detection module 740 is used to detect whether the patient has abnormal behavior according to the relative position relationship, the degree of change of the key points of the human body and the degree of change of the optical flow field.

[0057] Optionally, the relative position relationship determination module is used to:

[0058] Determine two upper arm midpoints and two eye points of the patient in the image frame according to the human body key points;

[0059] A quadrilateral area formed by a sideline passing through each of the upper arm midpoints and perpendicular to the line connecting the two upper arm midpoints, a line connecting the two upper arm midpoints, and a line connecting the two eye points is used as a head neighborhood in the image frame.

[0060] Optionally, the variation degree determination module is used to:

[0061] Determine the key point vector modulus between each corresponding human key point between any two target image frames separated by a preset number of image frames in the image frame sequence;

[0062] The degree of change of the key points of the human body between the two target image frames is determined according to the modulus of the key point vector.

[0063] Optionally, the change degree determination module includes:

[0064] an optical flow attention region determining unit, configured to determine, for each image frame in the image frame sequence, respective optical flow attention regions in the image frame;

[0065] A variation average modulus calculation unit, used to calculate the cumulative variation average modulus of the optical flow field in the optical flow attention area at the corresponding position in all image frames between the two target image frames;

[0066] The optical flow field change degree determining unit is used to determine the optical flow field change degree between two target image frames according to the average modulus of the accumulated change of the optical flow field.

[0067] Optionally, the optical flow attention area determination unit includes:

[0068] a presentation viewing angle determination subunit, configured to determine, for each image frame in the image frame sequence, a presentation viewing angle of the patient in the image frame;

[0069] The optical flow attention area determination subunit is used to determine each optical flow attention area in the image frame according to the presentation perspective.

[0070] Optionally, the presentation viewing angle includes a horizontal viewing angle and a vertical viewing angle;

[0071] The optical flow attention area determination subunit is used to:

[0072] When the presentation viewing angle is a horizontally lying viewing angle, determining the midpoints of the two shoulders and the midpoint of the hip of the patient in the image frame according to the key points of the human body;

[0073] Determine first human body sub-regions into which a human body region in the image frame is divided by a vertical line passing through the midpoints of the two shoulders and a vertical line passing through the midpoint of the hip, and use the first human body sub-regions as optical flow attention regions in the image frame;

[0074] When the presentation viewing angle is a vertical lying viewing angle, determining a high shoulder point among two shoulder points and a high hip joint point among two hip joint points of the patient in the image frame according to the human body key points;

[0075] Determine respective second human body sub-regions formed by dividing the human body region in the image frame by a horizontal line passing through the high shoulder point and a horizontal line passing through the high hip joint point;

[0076] Determine the human body enclosing area in each of the second human body sub-areas respectively, and use each of the human body enclosing areas as each optical flow attention area in the image frame; wherein the height of the human body enclosing area is the same as the height of the corresponding second human body sub-area, the width of the human body enclosing area is smaller than the width of the corresponding second human body sub-area, and the width of the human body enclosing area is the product of the maximum human body width in the corresponding second human body sub-area and a preset proportional coefficient.

[0077] Optionally, the relative position relationship includes that the wrist key point is within the head neighborhood and that the wrist key point is outside the head neighborhood;

[0078] The abnormal behavior detection module is used to:

[0079] If there is at least one image frame in the image frame sequence whose corresponding relative position relationship is that the wrist key point is within the head neighborhood, it is determined that the patient has abnormal behavior suspected of extubation;

[0080] If the relative position relationship corresponding to each image frame in the image frame sequence is that the wrist key point is outside the head neighborhood, and the degree of change of the optical flow field is greater than a preset optical flow field change threshold, it is determined that the patient has severe agitated abnormal behavior;

[0081] If the relative position relationship corresponding to each image frame in the image frame sequence is that the wrist key point is outside the head neighborhood, and the degree of change of the optical flow field is less than the preset optical flow field change threshold and the degree of change of the human body key point is greater than the preset human body key point change threshold, it is determined that the patient has abnormal behavior of mild agitation.

[0082] The abnormal behavior detection device provided in the embodiment of the present invention can execute the abnormal behavior detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0083] Figure 8A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0084] like Figure 8 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0085] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0086] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as an abnormal behavior detection method.

[0087] In some embodiments, the abnormal behavior detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the abnormal behavior detection method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the abnormal behavior detection method in any other appropriate manner (e.g., by means of firmware).

[0088] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0089] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0090] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0091] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0092] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0093] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0094] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0095] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting abnormal behavior, characterized in that: include: In response to an abnormal behavior detection event being triggered, acquiring an image frame sequence of the patient; wherein the image frame sequence includes at least two image frames; For each image frame in the image frame sequence, determine a human body key point of the image frame, determine a head neighborhood in the image frame according to the human body key point, and determine a relative positional relationship between a wrist key point in the human body key point and the head neighborhood; Determine the degree of change of key points of the human body and the degree of change of the optical flow field between any two target image frames separated by a preset number of image frames in the image frame sequence; Whether the patient has abnormal behavior is detected based on the relative position relationship, the degree of change of the key points of the human body and the degree of change of the optical flow field.

2. The method according to claim 1, characterized in that Determining a head neighborhood in the image frame according to the human body key points includes: Determine two upper arm midpoints and two eye points of the patient in the image frame according to the human body key points; A quadrilateral area formed by a sideline passing through each of the upper arm midpoints and perpendicular to the line connecting the two upper arm midpoints, a line connecting the two upper arm midpoints, and a line connecting the two eye points is used as a head neighborhood in the image frame.

3. The method according to claim 1, characterized in that Determining the degree of change of key points of a human body between any two target image frames separated by a preset number of image frames in the image frame sequence comprises: Determine the key point vector modulus between each corresponding human key point between any two target image frames separated by a preset number of image frames in the image frame sequence; The degree of change of the key points of the human body between the two target image frames is determined according to the modulus of the key point vector.

4. The method according to claim 1, characterized in that: Determining the degree of change of the optical flow field between any two target image frames separated by a preset number of image frames in the image frame sequence comprises: For each image frame in the image frame sequence, determining each optical flow attention region in the image frame; Calculate the average modulus of the cumulative change of the optical flow field in the optical flow attention area at the corresponding position in all image frames between the two target image frames; The degree of change of the optical flow field between the two target image frames is determined according to the average modulus length of the accumulated change of the optical flow field.

5. The method according to claim 4, characterized in that For each image frame in the image frame sequence, determining each optical flow attention area in the image frame includes: For each image frame in the sequence of image frames, determining a presentation viewing angle of the patient in the image frame; Each optical flow attention area in the image frame is determined according to the presentation perspective.

6. The method according to claim 5, characterized in that The presentation viewing angle includes a horizontal viewing angle and a vertical viewing angle; Determining each optical flow attention area in the image frame according to the presentation perspective includes: When the presentation viewing angle is a horizontally lying viewing angle, determining the midpoints of the two shoulders and the midpoint of the hip of the patient in the image frame according to the key points of the human body; Determine first human body sub-regions into which a human body region in the image frame is divided by a vertical line passing through the midpoints of the two shoulders and a vertical line passing through the midpoint of the hip, and use the first human body sub-regions as optical flow attention regions in the image frame; When the presentation viewing angle is a vertical lying viewing angle, determining a high shoulder point among two shoulder points and a high hip joint point among two hip joint points of the patient in the image frame according to the human body key points; Determine respective second human body sub-regions formed by dividing the human body region in the image frame by a horizontal line passing through the high shoulder point and a horizontal line passing through the high hip joint point; Determine the human body enclosing area in each of the second human body sub-areas respectively, and use each of the human body enclosing areas as each optical flow attention area in the image frame; wherein the height of the human body enclosing area is the same as the height of the corresponding second human body sub-area, the width of the human body enclosing area is smaller than the width of the corresponding second human body sub-area, and the width of the human body enclosing area is the product of the maximum human body width in the corresponding second human body sub-area and a preset proportional coefficient.

7. The method according to claim 1, characterized in that The relative position relationship includes that the wrist key point is within the head neighborhood and the wrist key point is outside the head neighborhood; Detecting whether the patient has abnormal behavior according to the relative position relationship, the degree of change of the key points of the human body, and the degree of change of the optical flow field includes: If there is at least one image frame in the image frame sequence whose corresponding relative position relationship is that the wrist key point is within the head neighborhood, it is determined that the patient has abnormal behavior suspected of extubation; If the relative position relationship corresponding to each image frame in the image frame sequence is that the wrist key point is outside the head neighborhood, and the degree of change of the optical flow field is greater than a preset optical flow field change threshold, it is determined that the patient has severe agitated abnormal behavior; If the relative position relationship corresponding to each image frame in the image frame sequence is that the wrist key point is outside the head neighborhood, and the degree of change of the optical flow field is less than the preset optical flow field change threshold and the degree of change of the human body key point is greater than the preset human body key point change threshold, it is determined that the patient has abnormal behavior of mild agitation.

8. An abnormal behavior detection device, characterized in that: include: An image frame sequence acquisition module, configured to acquire an image frame sequence of a patient in response to an abnormal behavior detection event being triggered; wherein the image frame sequence includes at least two image frames; A relative position relationship determination module, used for determining, for each image frame in the image frame sequence, a human body key point of the image frame, determining a head neighborhood in the image frame according to the human body key point, and determining a relative position relationship between a wrist key point in the human body key point and the head neighborhood; A change degree determination module, used to determine the change degree of human key points and the change degree of optical flow field between any two target image frames separated by a preset number of image frames in the image frame sequence; The abnormal behavior detection module is used to detect whether the patient has abnormal behavior according to the relative position relationship, the degree of change of the key points of the human body and the degree of change of the optical flow field.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the abnormal behavior detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the abnormal behavior detection method according to any one of claims 1 to 7 when executed.

Images