Human behavior monitoring method, system and storage medium
The video frames are obtained through the camera, semantic segmentation and data analysis are performed, which solves the problem of inconvenience in detection of wearable sensors, and accurately monitors falls and convulsions, improving the timeliness and comfort of monitoring.
Patent Information
- Application Number
- CN202210185713.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-28
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-02-28
AI Technical Summary
In the prior art, wearable sensors are used to detect dangerous behaviors in humans, and there are problems of discomfort and inconvenience in detection, making it difficult to monitor falls or convulsions in a timely manner.
The video frame is obtained through the camera, semantic segmentation is performed to obtain ground and human position data, calculate the displacement standard deviation and height difference of limb data, and determine whether an alarm is issued, avoiding the discomfort and detection difficulties of traditional sensors.
It realizes accurate judgment of falling or twitching without wearing sensors, improves the timeliness and comfort of monitoring, and solves the inconvenience problems in traditional methods.
Smart Images

Figure CN114694269B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of human behavior detection, and in particular to a human behavior monitoring method, system and storage medium. Background Art
[0002] With the development of smart medicine, more and more artificial intelligence technologies have been integrated into the medical field, such as remote diagnosis, remote surgery, remote emergency, remote monitoring, remote teaching, smart robots, mobile medical care and other medical fields. Among them, smart medicine has achieved relatively mature development in the fields of preoperative diagnosis, intraoperative surgery, emergency treatment, and medical imaging diagnosis. Remote monitoring of postoperative personnel has also achieved certain development.
[0003] Remote monitoring generally refers to the monitoring of patients in wards or elderly people living alone. The monitored person may experience falls, limb convulsions, facial convulsions, coughs and other discomforts indoors, but because there is no one to monitor, when discomfort occurs, they cannot receive timely treatment and their lives may be in danger.
[0004] In the prior art, sensors are usually worn on the human body to detect whether the person has fallen, convulsed or other dangerous features, and then an alarm is issued to achieve the effect of monitoring. However, wearing sensor equipment will make the human body feel uncomfortable and bring inconvenience to behavior detection. Summary of the invention
[0005] Based on this, the purpose of the present invention is to provide a human behavior monitoring method, system and storage medium to solve the problem of inconvenience in detecting dangerous behaviors with wearable sensors in the background technology.
[0006] One aspect of the present invention provides a method for monitoring human behavior, comprising:
[0007] Acquire a target video at a first preset frequency, the target video including a plurality of video frames, perform semantic segmentation on the video frames, and obtain ground position data and human body position data;
[0008] Acquire characteristic data of human body position data, the characteristic data including central position data of two hips and limb data of the human body;
[0009] Acquire multiple consecutive video frames of the target video according to the limb data, and determine whether the standard deviation of the displacement of the limb data in the multiple consecutive video frames is greater than a preset standard deviation;
[0010] and obtaining the current height data between the center position data of the two hips of the human body and the ground position data, and determining whether the current height data is lower than the preset height. If so, obtaining the historical video frames in the last preset time period, and calculating the height difference between the historical height data of the corresponding historical video frames and the current height data, and determining whether the height difference is greater than the preset height difference;
[0011] If the standard deviation of the limb data displacement in multiple consecutive video frames is greater than the preset standard deviation, it is determined that the human body is in a convulsion state and an alarm is issued; if the height difference is greater than the preset height difference, it is determined that the human body is in a fall state and an alarm is issued.
[0012] The human behavior monitoring method of the present invention obtains the video frame of the ward through the camera, and determines whether the ward falls or convulses by analyzing the human behavior characteristics of the video frame, thereby avoiding the discomfort and inconvenience of detection of traditional wearable sensors. Specifically, by semantically segmenting the video frame, the ground position data and the human position data are obtained, and then the two-span center position data and limb data in the human position data are obtained. When the current height of the two-span center position and the ground position is lower than the preset height, the previous historical video frame in the adjacent time period is obtained again, and the historical height in the corresponding historical video frame is obtained. It is further determined whether the height difference between the historical height and the current height is greater than the preset height difference, so as to determine that the ward is in a state of falling; and by calculating whether the standard deviation of the displacement of the limb data in multiple consecutive video frames is lower than the preset standard deviation, it is determined whether the human body is convulsing, thereby solving the problem of the inconvenience of wearing sensors to detect dangerous behaviors in the background technology.
[0013] Furthermore, the target video is acquired at a first preset frequency, the target video includes a plurality of video frames, and the video frames are semantically segmented to obtain ground position data and human body position data, including the following steps:
[0014] A target video is obtained, where the target video includes several video frames, and the ground area and the human body area in the video frame are marked with different colors, and the position data corresponding to the ground area color are summarized as ground position data, and the position data corresponding to the human body area color are summarized as human body position data.
[0015] Furthermore, if the standard deviation of the displacement of the limb data in the continuous multiple video frames is greater than the preset standard deviation, the method further includes:
[0016] Obtaining facial data in the video frame, comparing the facial data with the pre-trained facial expression model, and determining whether the facial data conforms to a preset facial expression, wherein the preset facial expressions include twitching, fright, and painful facial expressions;
[0017] If so, it is determined that the human body is in a convulsion state and an alarm is sounded.
[0018] Furthermore, the step of obtaining current height data between the center position data of the two hips of the human body and the ground position data includes:
[0019] According to the human body position data and the ground position data, target position data within a preset distance associated with the human body position data is determined in the ground position data, and current height data between the center position data of the two hips of the human body and the target ground position data is determined.
[0020] Furthermore, the limb data includes limb key point position data, and the step of determining whether the standard deviation of the limb data displacement in a plurality of consecutive video frames is greater than a preset standard deviation includes:
[0021] The position data of the limb key points of the continuous multiple video frames are obtained, the standard deviation of the displacement of the limb key points in the continuous multiple video frames is calculated according to the position data of the limb key points, and it is determined whether the standard deviation of the displacement of the limb key points is less than the preset standard deviation.
[0022] Furthermore, the method also includes:
[0023] Acquire the key point data of the hand in the human body position data according to the human body position data, and determine whether the key point data of the hand meets the preset help gesture;
[0024] If so, it meets the characteristics of help-seeking behavior and an alarm is issued.
[0025] Furthermore, the method also includes:
[0026] Acquire the speech data in the target video, detect the coughing speech data in the speech data, and extract the frequency information and intensity information in the coughing speech data.
[0027] Determine whether the frequency information and intensity information reach a preset threshold;
[0028] If so, the first preset frequency is increased to a second preset frequency to obtain the target video according to the second preset frequency.
[0029] Furthermore, before the step of extracting the frequency information and intensity information from the cough voice data, the step further includes:
[0030] Pre-emphasize the cough speech data to remove the lip radiation effect in the cough speech data;
[0031] Performing windowing and framing processing on the coughing voice data to decompose the coughing voice data into multiple frames of slice data;
[0032] Endpoint detection is performed on the cough voice data to extract valid data from the cough voice data.
[0033] Another aspect of the present invention provides a human behavior monitoring system, comprising:
[0034] A semantic segmentation module is used to obtain a target video at a first preset frequency, where the target video includes a plurality of video frames, and perform semantic segmentation on the video frames to obtain ground position data and human body position data;
[0035] A special data extraction module is used to obtain characteristic data of human body position data, the characteristic data including the center position data of the two hips of the human body and the limb data;
[0036] A twitching behavior judgment module is used to obtain multiple consecutive video frames of the target video according to the limb data, and judge whether the standard deviation of the displacement of the limb data in the multiple consecutive video frames is greater than a preset standard deviation;
[0037] The fall behavior judgment module is used to obtain the current height data between the center position data of the two hips of the human body and the ground position data, and judge whether the current height data is lower than the preset height. If so, obtain the historical video frame in the previous preset time period, and calculate the height difference between the historical height data of the corresponding historical video frame and the current height data, and judge whether the height difference is greater than the preset height difference;
[0038] The alarm module is used to determine that the human body is in a convulsion state and issue an alarm if the standard deviation of the limb data displacement in multiple consecutive video frames is greater than a preset standard deviation; if the height difference is greater than a preset height difference, it is used to determine that the human body is in a fall state and issue an alarm.
[0039] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the human behavior monitoring method as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 This is a flow chart of the human behavior monitoring method in the first embodiment of the present invention;
[0041] Figure 2 This is a flow chart of a method for monitoring human behavior in a second embodiment of the present invention;
[0042] Figure 3 A block diagram of a human behavior monitoring system in a third embodiment of the present invention;
[0043] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION
[0044] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.
[0045] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.
[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.
[0047] Embodiment 1
[0048] See also Figure 1 , shown is a human behavior monitoring method in the first embodiment of the present invention, including steps S11-S16.
[0049] S11. Acquire a target video at a first preset frequency, where the target video includes a plurality of video frames, and perform semantic segmentation on the video frames to obtain ground position data and human body position data.
[0050] The human behavior monitoring method in this embodiment can be applied to the monitoring of patients in hospitals and the monitoring of the elderly at home. The real-time behavior feature video of the ward is obtained by the camera according to the first preset frequency, and it is judged whether the real-time behavior feature of the ward meets the behavior feature of falling, convulsion or alarm, so as to alarm and improve the safety of the ward. First, the target video of the ward is obtained at the first preset frequency. The target video includes the ward and the surrounding environment. The first preset frequency can be 1-2min / time. The frequency of video processing can be adjusted, and the behavior features of the person in the video are detected according to the adjusted frequency. The target video is divided into several video frames, and the video frames are semantically segmented by Mask2Former, wherein semantic segmentation is to label each pixel in the image with a category label, and each category in the image is marked with a color. In this embodiment, the human position area and the ground position area in the video frame are marked with different colors by semantic segmentation, so as to obtain the human position data and the ground position data.
[0051] S12, acquiring characteristic data of the human body position data, wherein the characteristic data includes the center position data of the two hips of the human body and the limb data.
[0052] Then, the characteristic data of the human body is obtained according to the human body position data, and the characteristic data includes the center position data of the two spans of the human body and the limb data. The center position data of the two spans of the human body can be approximately determined as the center of gravity data of the human body, which is used to determine whether the human body falls.
[0053] Limb data is the key points of the human body's limbs, which are used to determine whether the human body is twitching. OpenPose (body language recognition system) is used to detect the key points of the limbs in the video frame. Assume that the key points of the left arm of the human body are g[0] , the key point of the left elbow is g[1] , the key points of the left arm and shoulder are g[2] The coordinates of the detected limb key points are [gx0,gy0], [gx1,gy1], [gx2,gy2].
[0054] Furthermore, the ground position data includes a large ground area. When judging the behavioral characteristics of the ward, in order to reduce the target detection range, the human body position data is used as the center and the target ground position data that is within a preset distance from the human body position data is determined in the ground position data. Optionally, the preset distance can be set to a ground area of 2m around the ward.
[0055] S13, determining whether the standard deviation of the displacement of the limb data in the continuous multiple video frames is greater than a preset standard deviation;
[0056] If the standard deviation of the displacement of the limb data in multiple consecutive video frames is greater than the preset standard deviation, it is determined that the human body is in a twitching state, and step S15 is executed.
[0057] After obtaining the limb data, further obtain multiple consecutive video frames in the target video. Assume that A[i] is the key point of the limbs detected in each video frame, such as the endpoints of the limbs, i is greater than or equal to 0 and less than or equal to 3, and its position is [xi,yi]. The formula for calculating the standard deviation of the limb data displacement is as follows:
[0058]
[0059] Among them, n represents n consecutive video frames, and di represents the standard deviation of the displacement change of the limb key point labeled i in n consecutive images. By comparing the size of di with a custom threshold, it can be determined that the limb key point moves back and forth in multiple consecutive video frames, thereby determining that the limb position labeled i has twitching behavior, that is, the human body is in a twitching state.
[0060] Optionally, in order to improve the accuracy of judging the twitching state of a human body, when it is judged that the standard deviation of the displacement of limb data in a plurality of consecutive video frames is greater than a preset standard deviation, facial expressions can be further recognized.
[0061] Specifically, a facial expression data set of dangerous states such as twitching, fright, and pain is used to pre-train a facial expression model, and the facial data of the guardian is obtained in the video frame. The facial data is input into the pre-trained facial expression model to determine whether the facial expression conforms to the preset facial expression in the above model. If so, it can be determined that the human body is in a twitching state based on the guardian's facial expression being in a painful state and the continuous reciprocating movement of key points of the limbs.
[0062] S14, determining whether the current height data between the center position data of the two hips of the human body and the ground position data is lower than a preset height;
[0063] If the current altitude data is lower than the preset altitude, step S16 is executed.
[0064] The center position data of the two spans of the human body is recorded in the video frame, and the height is detected with the target ground position data, where the target ground position data can be approximated as a horizontal plane, and then the current height data between the center position data of the two spans of the human body and the target ground position data, and the distance between the center position of the two spans and the ground are calculated. When the current height data is lower than the preset height, the center position of the two spans is close to the ground, it can be determined that the human body may be in a falling state, and further judgment of the falling state is required.
[0065] Furthermore, the position data of the foot (the end point of the foot) in the human body position data is obtained, and the foot data and the center position data of the two spans are connected into a straight line in a horizontal position. The slope of the straight line is obtained through the two points and the ground position data. When the slope of the straight line is lower than the preset slope, it can be determined that the human body may be in a falling state.
[0066] Similarly, obtain the head position data (center of the head) in the human body position data, connect the key points of the head and the center position data of the two spans into a straight line, and calculate the slope of the straight line. When the slope of the straight line is lower than the preset slope, it can be determined that the human body may be in a falling state.
[0067] S16, determining whether the height difference between the historical height data in the historical video frame and the current height data is greater than a preset height difference;
[0068] If the height difference is greater than the preset height difference, step S15 is executed.
[0069] When it is determined that the center position data of the two spans of the human body is close to the ground position data, historical data is extracted from the historical video frame, and the previous video frame in the adjacent preset time period is obtained. For example, the historical video frame before 1S-2S is obtained, and the historical height data of the center position data of the two spans of the human body and the target ground position data in the historical video frame are correspondingly extracted, and the height difference between the historical height data and the current height data is calculated. When the height difference is greater than the preset height difference, it means that the center position data of the two spans of the human body has dropped significantly in a short period of time, so it can be determined that the human body is in a state of falling.
[0070] S15. Sound an alarm.
[0071] When it is determined that a person is in a state of falling, a fall alarm is issued. The specific alarm method can be set according to the needs of the guardian, such as calling a distress number, contacting an emergency contact, or sounding a sound alarm to provide timely assistance to the guardian.
[0072] If the calculated standard deviations of the displacements of multiple limb positions in multiple consecutive video frames are all lower than the preset standard deviation, it is determined that multiple limb key point positions are twitching, that is, it is determined that the human body is twitching. When it is determined that the human body is in a twitching state, an alarm is issued, and the specific alarm is similar to the fall alarm.
[0073] In some other optional embodiments, the method for triggering an automatic alarm further includes:
[0074] The hand key point data in the human body position data is obtained, and the obtained hand key point data is compared with the preset help gesture to determine whether it meets the preset help gesture, wherein the preset help gesture can be customized. When the ward wants to send a distress signal, he or she can make a preset help gesture to the camera. When it is determined that the hand key point data meets the preset help gesture, an alarm is issued, such as dialing a distress call or contacting an emergency contact.
[0075] In some other optional embodiments, the method for triggering an automatic alarm further includes:
[0076] Get the target video, which includes multiple video frames. Use the face detection algorithm to get the face image from the video frame, and perform facial expression recognition on the acquired face image. Facial expression recognition includes recognizing the expressions of twitching, frightening, and pain on the face of the monitored person. Use the twitching, frightening, and painful expression data set to train the facial expression classification model. Read the video frame from the camera, get the face image, and then send it to the facial expression classification model to get the recognition result of the face. If the twitching, frightening, or painful expressions can be recognized within the preset time, an alarm will be issued.
[0077] In summary, the human behavior monitoring method in the above-mentioned embodiment of the present invention obtains the video frame of the ward through the camera, and determines whether the ward falls or convulses by analyzing the human behavior characteristics of the video frame, thereby avoiding the discomfort and inconvenience of detection of traditional wearable sensors. Specifically, by semantically segmenting the video frame, the ground position data and the human position data are obtained, and then the two-span center position data and limb data in the human position data are obtained. When the current height of the two-span center position and the ground position is lower than the preset height, the previous historical video frame in the adjacent time period is obtained again, and the historical height in the corresponding historical video frame is obtained. It is further determined whether the height difference between the historical height and the current height is greater than the preset height difference, so as to determine that the ward is in a falling state; and by calculating whether the standard deviation of the displacement of the limb data in multiple consecutive video frames is lower than the preset standard deviation, it is determined whether the human body is convulsing, thereby solving the problem of the inconvenience of wearing sensors to detect dangerous behaviors in the background technology.
[0078] Embodiment 2
[0079] See also Figure 2 , which shows a human behavior monitoring method in a second embodiment of the present invention. The behavior monitoring method in this embodiment is based on the method in the first embodiment, and detects coughing voice in the video voice, and then adjusts the preset frequency of obtaining the video in the first embodiment to adjust the frequency of detecting the behavioral characteristics of the person in the video. The method in this embodiment includes steps S21-S24.
[0080] S21. Acquire voice data in the target video, and detect cough voice data in the voice data.
[0081] The voice module set in the camera can be used to obtain the voice data of the guardian, obtain the target video through the camera, the target video includes the voice data of the guardian, and detect whether there is a cough in the voice data through an algorithm. For example, the cough voice data in the voice is quickly and accurately identified through the Hidden Markov Model (HMM). When the cough voice data of the guardian is identified, the cough voice data of the guardian is extracted.
[0082] S22: pre-process the cough voice data to extract frequency information and intensity information from the cough voice data.
[0083] The extracted cough speech data is preprocessed to obtain effective cough speech data, wherein the preprocessing can maximize certain information in the cough speech data to achieve the extraction of the best feature parameters.
[0084] Preprocessing includes pre-emphasis, windowing and framing, and endpoint detection. Pre-emphasis is to remove the impact of lip radiation on audio and increase the high-frequency resolution of speech. A first-order FIR high-pass filter is often used to implement pre-emphasis. The transfer function of pre-emphasis is:
[0085] H(z)=1-a*z -1
[0086] Among them, H is the output after filtering, z is the waveform data, a is the pre-emphasis coefficient, 0.9 <a<1.0。
[0087] Windowing and framing is to customize a pane size, and slide the pane to divide long audio data into pieces of data frame by frame.
[0088] Speech endpoint detection is to detect valid speech segments from a continuous speech stream. It includes two aspects: detecting the starting point of valid speech, i.e. the front end point, and detecting the end point of valid speech, i.e. the back end point. Endpoint detection is used to identify the starting point and end point of cough speech data, thus forming a complete cough speech data segment.
[0089] Feature extraction includes extracting the frequency and audio intensity of coughs in speech data.
[0090] The audio intensity of a cough, defined here as d, represents the average of the peak and trough of the audio within a certain time t, and the formula is as follows:
[0091]
[0092] In the formula, x represents the value of the audio wave, max represents the maximum value, and min represents the minimum value.
[0093] Within time t, use max(x) and min(x) to find a median horizontal line with a slope of (max(x)+min(x)) / 2. Assuming that the number of intersections between the audio wave and the horizontal line is m, the frequency of coughing is approximately defined as m / (3t).
[0094] S23: Determine whether the frequency information and the intensity information reach a preset threshold.
[0095] To determine whether the frequency information and intensity information of the cough voice data reach a preset threshold, it can be understood that the frequency information and intensity information can be compared with the preset frequency and preset intensity respectively. When any one of the frequency information and the intensity information exceeds the preset frequency and the preset intensity, it is deemed to have reached the preset threshold; the frequency information and the intensity information can also be compared with the preset threshold at the same time. When both the frequency information and the intensity information exceed the preset frequency and the preset intensity, it is deemed to have reached the preset threshold.
[0096] S24: If the frequency information and the intensity information reach a preset threshold, the first preset frequency is increased to a second preset frequency to obtain a target video according to the second preset frequency.
[0097] After judging each valid cough voice data, the judgment results of all cough voice data are integrated together through dynamic time warping, and repeated or incomplete detection results are removed. Finally, the judgment result is output to determine whether the threshold is reached.
[0098] After reaching the threshold, the first preset frequency is increased to the second preset frequency, and the second preset frequency is higher than the first preset frequency, for example 30s-1min / time, so as to speed up the frequency of video analysis and processing. By monitoring whether the cough of the person reaches the threshold, the possible dangerous situation of the person under guardianship is preset, and the frequency of video analysis and processing is adjusted accordingly, the timeliness of monitoring can be improved and the untimely discovery after a person falls or convulses can be prevented.
[0099] In summary, the human behavior monitoring method in the above embodiments of the present invention, by acquiring cough voice data in the target video, extracting the frequency and intensity information of the cough voice data, and judging whether the frequency and intensity information reaches a preset threshold and increasing the first preset frequency to the second preset frequency, can improve the timeliness of the behavior analysis and processing of the monitored person in the video clock, thereby improving the efficiency of monitoring.
[0100] Embodiment 3
[0101] Another aspect of the present invention is to provide a human behavior monitoring system. Figure 3 , as shown in the block diagram of the human behavior monitoring system in this embodiment, includes:
[0102] A semantic segmentation module is used to obtain a target video at a first preset frequency, wherein the target video includes a plurality of video frames, and perform semantic segmentation on the video frames to obtain ground position data and human body position data;
[0103] A special data extraction module, used to obtain characteristic data of the human body position data, wherein the characteristic data includes central position data of the two hips of the human body and limb data;
[0104] A twitching behavior judgment module, used for obtaining a plurality of continuous video frames of the target video according to the limb data, and judging whether a standard deviation of the displacement of the limb data in the plurality of continuous video frames is greater than a preset standard deviation;
[0105] A fall behavior judgment module is used to obtain the current height data between the center position data of the two hips of the human body and the ground position data, and judge whether the current height data is lower than the preset height. If so, obtain the historical video frame within the last preset time period, and calculate the height difference between the historical height data corresponding to the historical video frame and the current height data, and judge whether the height difference is greater than the preset height difference;
[0106] The alarm module is used to determine that the human body is in a convulsion state and issue an alarm if the standard deviation of the limb data displacement in the continuous multiple frames of video frames is greater than a preset standard deviation; if the height difference is greater than a preset height difference, it is used to determine that the human body is in a falling state and issue an alarm.
[0107] Further, in some other optional embodiments, the semantic segmentation module includes:
[0108] A target video acquisition unit, used to acquire a target video, where the target video includes a plurality of video frames;
[0109] The semantic segmentation unit is used to mark the ground area and the human body area in the video frame with different colors, summarize the position data corresponding to the ground area color into ground position data, and summarize the position data corresponding to the human body area color into human body position data.
[0110] Further, in some other optional embodiments, the fall behavior judgment module includes:
[0111] The target position data determining unit is used to
[0112] According to the human body position data and the ground position data, target position data within a preset distance associated with the human body position data is determined in the ground position data, and current height data between the center position data of the two hips of the human body and the target ground position data is determined.
[0113] Further, in some other optional embodiments, the twitching behavior determination module includes:
[0114] A facial expression recognition unit, used to obtain facial data in the video frame, compare the facial data with a pre-trained facial expression model, and determine whether the facial data conforms to a preset facial expression, wherein the preset facial expression includes twitching, fright, and painful facial expressions;
[0115] If the facial data matches the preset facial expression, it is determined that the human body is in a convulsion state and an alarm is issued.
[0116] Further, in some other optional embodiments, the fall behavior determination module includes:
[0117] The current height data acquisition unit is used to determine the target position data within a preset distance associated with the human body position data in the ground position data according to the human body position data and the ground position data, and to determine the current height data between the center position data of the two hips of the human body and the target ground position data.
[0118] Further, in some other optional embodiments, the twitching behavior determination module includes:
[0119] The displacement standard deviation judgment unit is used to obtain the limb key point position data of the continuous multiple video frames, calculate the standard deviation of the limb key point displacement in the continuous multiple video frames according to the limb key point position data, and judge whether the standard deviation of the limb key point displacement is less than the preset standard deviation.
[0120] Further, in some other optional embodiments, the device also includes:
[0121] A help-seeking gesture judgment module is used to obtain the hand key point data in the human body position data according to the human body position data, and judge whether the hand key point data conforms to the preset help-seeking gesture;
[0122] If so, it meets the characteristics of help-seeking behavior and an alarm is issued.
[0123] Further, in some other optional embodiments, the device also includes:
[0124] A preset frequency adjustment module is used to obtain voice data in the target video, detect cough voice data in the voice data, and extract frequency information and intensity information from the cough voice data.
[0125] Determine whether the frequency information and intensity information reach a preset threshold;
[0126] If the frequency information and the intensity information reach a preset threshold, the first preset frequency is increased to a second preset frequency to obtain the target video according to the second preset frequency.
[0127] Further, in some other optional embodiments, the preset frequency adjustment module includes:
[0128] A speech preprocessing unit, used for performing pre-emphasis processing on the cough speech data to remove the influence of lip radiation in the cough speech data;
[0129] Performing windowing and framing processing on the coughing voice data to decompose the coughing voice data into multiple frames of slice data;
[0130] Endpoint detection is performed on the cough voice data to extract valid data from the cough voice data.
[0131] The functions or operation steps implemented when the above modules and units are executed are generally the same as those in the above method embodiments, and will not be repeated here.
[0132] In summary, the human behavior monitoring system in the above-mentioned embodiment of the present invention obtains the video frame of the ward through the camera, and determines whether the ward falls or convulses by analyzing the human behavior characteristics of the video frame, thereby avoiding the discomfort and inconvenience of detection of traditional wearable sensors. Specifically, by semantically segmenting the video frame, the ground position data and the human position data are obtained, and then the two-span center position data and limb data in the human position data are obtained. When the current height of the two-span center position and the ground position is lower than the preset height, the previous historical video frame in the adjacent time period is obtained again, and the historical height in the corresponding historical video frame is obtained. It is further determined whether the height difference between the historical height and the current height is greater than the preset height difference, thereby determining that the ward is in a falling state; and by calculating whether the standard deviation of the displacement of the limb data in multiple consecutive video frames is lower than the preset standard deviation, it is determined whether the human body is convulsing, thereby solving the problem of the inconvenience of wearing sensors to detect dangerous behaviors in the background technology.
[0133] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the human behavior monitoring method in the above embodiment are implemented.
[0134] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0135] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.
Claims
1. A human behavior monitoring method, characterized in that: The method comprises: Acquire a target video at a first preset frequency, the target video including a plurality of video frames, and perform semantic segmentation on the video frames to obtain ground position data and human body position data; Acquire characteristic data of the human body position data, wherein the characteristic data includes central position data of two hips and limb data of the human body; Acquire a plurality of continuous video frames of the target video according to the limb data, and determine whether a standard deviation of the displacement of the limb data in the plurality of continuous video frames is greater than a preset standard deviation; and obtaining current height data between the center position data of the two hips of the human body and the ground position data, and determining whether the current height data is lower than a preset height. If so, obtaining a historical video frame within a previous preset time period, and calculating a height difference between the historical height data corresponding to the historical video frame and the current height data, and determining whether the height difference is greater than a preset height difference; If the standard deviation of the displacement of the limb data in the continuous multiple video frames is greater than the preset standard deviation, it is determined that the human body is in a convulsion state and an alarm is issued; if the height difference is greater than the preset height difference, it is determined that the human body is in a fall state and an alarm is issued; The method further comprises: Acquire voice data in the target video, detect cough voice data in the voice data, and extract frequency information and intensity information from the cough voice data, Determining whether the frequency information and the intensity information reach a preset threshold; If yes, increasing the first preset frequency to a second preset frequency to obtain the target video according to the second preset frequency; Wherein, before the step of extracting the frequency information and intensity information from the cough voice data, the step further includes: Performing pre-emphasis processing on the cough voice data to remove the influence of lip radiation in the cough voice data; Performing windowing and framing processing on the coughing voice data to decompose the coughing voice data into multiple frames of slice data; Performing endpoint detection processing on the cough voice data to extract valid data from the cough voice data; Among them, the transfer function of pre-emphasis is: H(z)=1-a*z -1 ; Where H is the output after filtering, z is the waveform data, a is the pre-emphasis coefficient, 0.9 <a<1.0; In the cough frequency information, the calculation formula of audio intensity is: ; In the formula, x represents the value of the audio wave, max represents the maximum value, and min represents the minimum value. In time t, use max(x) and min(x) to find a median horizontal line with a slope of (max(x)+min(x)) / 2. Assuming that the number of intersections between the audio wave and the horizontal line is m, the frequency of coughing is approximately defined as m / (3t). The standard deviation of limb data displacement is calculated as: ; Among them, n represents n consecutive video frames, di represents the standard deviation of the displacement change of the limb key point labeled i in n consecutive images; by comparing the size of di with a custom threshold, it can be judged that the limb key point moves back and forth in multiple consecutive video frames, thereby judging that the limb position labeled i has twitching behavior, that is, the human body is in a twitching state.
2. The human behavior monitoring method according to claim 1, characterized in that: The step of acquiring a target video at a first preset frequency, wherein the target video includes a plurality of video frames, and performing semantic segmentation on the video frames to obtain ground position data and human body position data includes: A target video is obtained, wherein the target video includes several video frames, and a ground area and a human body area in the video frames are marked with different colors, and the position data corresponding to the color of the ground area is summarized as ground position data, and the position data corresponding to the color of the human body area is summarized as human body position data.
3. The human behavior monitoring method according to claim 1, characterized in that: After the step of if the standard deviation of the displacement of the limb data in the continuous multiple video frames is greater than the preset standard deviation, the following step further comprises: Acquire the facial data in the video frame, compare the facial data with a pre-trained facial expression model, and determine whether the facial data conforms to a preset facial expression, wherein the preset facial expression includes twitching, frightening, and painful facial expressions; If so, it is determined that the human body is in a convulsion state and an alarm is sounded.
4. The human behavior monitoring method according to claim 1, characterized in that: The step of obtaining the current height data between the center position data of the two hips of the human body and the ground position data comprises: According to the human body position data and the ground position data, target position data within a preset distance associated with the human body position data is determined in the ground position data, and current height data between the center position data of the two hips of the human body and the target ground position data is determined.
5. The human behavior monitoring method according to claim 1, characterized in that: The limb data includes limb key point position data, and the step of determining whether the standard deviation of the limb data displacement in the continuous multiple video frames is greater than a preset standard deviation includes: The limb key point position data of the continuous multiple video frames are obtained, the standard deviation of the limb key point displacement in the continuous multiple video frames is calculated according to the limb key point position data, and it is determined whether the standard deviation of the limb key point displacement is less than a preset standard deviation.
6. The human behavior monitoring method according to claim 1, characterized in that: The method further comprises: Acquire hand key point data in the human body position data according to the human body position data, and determine whether the hand key point data conforms to a preset help gesture; If so, it meets the characteristics of help-seeking behavior and an alarm is issued.
7. A human behavior monitoring system, characterized in that: include: A semantic segmentation module is used to obtain a target video at a first preset frequency, wherein the target video includes a plurality of video frames, and perform semantic segmentation on the video frames to obtain ground position data and human body position data; A special data extraction module, used to obtain characteristic data of the human body position data, wherein the characteristic data includes central position data of the two hips of the human body and limb data; A twitching behavior judgment module, used for obtaining a plurality of continuous video frames of the target video according to the limb data, and judging whether a standard deviation of the displacement of the limb data in the plurality of continuous video frames is greater than a preset standard deviation; A fall behavior judgment module is used to obtain the current height data between the center position data of the two hips of the human body and the ground position data, and judge whether the current height data is lower than the preset height. If so, obtain the historical video frame within the last preset time period, and calculate the height difference between the historical height data corresponding to the historical video frame and the current height data, and judge whether the height difference is greater than the preset height difference; An alarm module, configured to determine that the human body is in a convulsion state and issue an alarm if the standard deviation of the displacement of the limb data in the continuous multiple video frames is greater than a preset standard deviation; If the height difference is greater than the preset height difference, it is determined that the human body is in a falling state and an alarm is issued; The system further comprises: Acquire voice data in the target video, detect cough voice data in the voice data, and extract frequency information and intensity information from the cough voice data, Determining whether the frequency information and the intensity information reach a preset threshold; If yes, increasing the first preset frequency to a second preset frequency to obtain the target video according to the second preset frequency; Wherein, before the step of extracting the frequency information and intensity information from the cough voice data, the step further includes: Performing pre-emphasis processing on the cough voice data to remove the influence of lip radiation in the cough voice data; Performing windowing and framing processing on the coughing voice data to decompose the coughing voice data into multiple frames of slice data; Performing endpoint detection processing on the cough voice data to extract valid data from the cough voice data; Among them, the transfer function of pre-emphasis is: H(z)=1-a*z -1 ; Where H is the output after filtering, z is the waveform data, a is the pre-emphasis coefficient, 0.9 <a<1.0; In the cough frequency information, the calculation formula of audio intensity is: ; In the formula, x represents the value of the audio wave, max represents the maximum value, and min represents the minimum value. In time t, use max(x) and min(x) to find a median horizontal line with a slope of (max(x)+min(x)) / 2. Assuming that the number of intersections between the audio wave and the horizontal line is m, the frequency of coughing is approximately defined as m / (3t). The standard deviation of limb data displacement is calculated as: ; Among them, n represents n consecutive video frames, di represents the standard deviation of the displacement change of the limb key point labeled i in n consecutive images; by comparing the size of di with a custom threshold, it can be judged that the limb key point moves back and forth in multiple consecutive video frames, thereby judging that the limb position labeled i has twitching behavior, that is, the human body is in a twitching state.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the human behavior monitoring method as described in any one of claims 1 to 6 is implemented.
