A real-time monitoring method for human behavior recognition and abnormal behavior detection

Abstract

The application provides a real-time monitoring method for human behavior recognition and abnormal behavior detection, characterized by building a multi-process pipeline system for behavior recognition and fall detection, wherein the multi-process pipeline system comprises three stages of monitoring data reading and human target detection, human multi-target tracking, human behavior recognition and abnormal behavior detection; the three stages are processed in parallel in three processes. Meanwhile, the application also provides a real-time monitoring method for fall and getting-up intention based on infrared vision AI, which is used to solve the problem that indoor personnel fall detection in the current nursing home completely depends on manual experience due to privacy risks and low-density scheduling of nursing staff, so that the nursing home cannot guarantee timely and effective detection of the getting-up and fall of the elderly and cannot provide timely and effective help.

Description

Technical Field

[0001] This invention belongs to the field of information technology, specifically relating to a real-time monitoring method for human behavior recognition and abnormal behavior detection. In particular, this invention can solve the problem of elderly people in nursing homes not being able to receive timely assistance when they fall or need help getting up. Background Technology

[0002] With economic development and social progress, people's living standards and awareness are gradually changing. They are becoming more alert to abnormal behaviors in daily life, and the requirements for timely detection and response to such behaviors are increasing, especially in cities where nursing homes have become a mainstream mode of care. Modern nursing homes not only require caregivers to provide care for the elderly and use more intelligent equipment for efficient 24-hour safety monitoring, but also need real-time alarms to detect whether elderly residents want to get up or have fallen. Due to considerations such as privacy, limited staff, and low shift schedules, it is generally difficult to entrust caregivers with 24-hour monitoring of elderly residents. Furthermore, privacy concerns prevent the installation of conventional cameras and the use of conventional artificial intelligence algorithms for monitoring, making it impossible to arrange for caregivers to effectively assist elderly residents who have difficulty getting up or have fallen.

[0003] Existing technologies for behavior recognition and abnormal behavior detection, such as fall detection, mostly rely on monitoring equipment or sensors for human behavior detection. These technologies cannot simultaneously achieve privacy protection and real-time detection of falls or intentions to get up. Installing conventional cameras in nursing home bedrooms for this purpose often raises privacy concerns among the elderly, placing a greater risk on the nursing home.

[0004] For example, document CN102646320A discloses an intelligent care host and a plurality of sensors, a vital signs monitoring device, and a one-button call device that communicate with the intelligent care host. The intelligent care host receives data from the sensors and the vital signs monitoring device, calculates the baseline of each alarm parameter using a multi-parameter baseline algorithm, compares it with a threshold set by the system, and then classifies the comparison results into alarm categories before informing the designated user. However, most elderly people are unwilling to wear this type of sensor, and the designated user usually does not live with the elderly, making a rapid response impossible. In other words, this mode can only monitor but cannot provide a quick and effective response. More importantly, it cannot monitor whether the elderly person has fallen or gotten out of bed. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a real-time monitoring method for human behavior recognition and abnormal behavior detection, and establishes a multi-process pipeline system for behavior recognition and abnormal behavior detection. The multi-process pipeline system includes three stages: monitoring data reading and human target detection, multi-target human tracking, and human behavior recognition and abnormal behavior detection; the three stages are processed in parallel by three processes.

[0006] The monitoring data reading and human target detection are performed by detecting human targets in the collected data, and the detection results are pushed into the multi-process queue 1;

[0007] The data in the multi-process queue 1 is processed by the human multi-target tracking process to extract target detection box data;

[0008] The target detection box data is used to identify human behavior and detect abnormal behavior through the static behavior recognition model in the human behavior recognition and abnormal behavior detection.

[0009] Furthermore, the specific methods for reading the monitoring data and detecting human targets are as follows:

[0010] 1) After reading a frame of data through the monitoring equipment, a target detection model is used to detect human targets;

[0011] 2) Assuming the target is successfully detected using the target detection model, the detected human bounding box visual data is initialized to form a multi-target tracker, which is then used simultaneously for human target detection in subsequent sequences;

[0012] 3) If the target detection model fails to detect a target in a future frame, the human bounding box name from the multi-target tracker will be used to supplement it;

[0013] 4) After obtaining the human body bounding box for each frame, put it into the multi-process queue 1.

[0014] Furthermore, the specific method for multi-target human body tracking is as follows:

[0015] 1) In order to identify different human bodies in the same frame and match them with the previous human body bounding boxes, id1, id2...idn are used to give different people in the same frame image aliases, and an ID library is used to store the target detection bounding box data of the current id.

[0016] 2) In the ID database, each ID has a fixed-length double-ended queue used to store the detection box data belonging to that ID;

[0017] 3) Retrieve all the detection box data from the multi-process queue 1, and select the detection box that is the most recent in time;

[0018] 4) Extract the current bounding boxes for all IDs in the current ID database, perform IoU calculation with the latest bounding box, and obtain the cost matrix; the IoU formula is as follows:

[0019]

[0020] Wherein, ID box is the detection box, ∩ is the intersection operator, ∪ is the union operator, and the latest detection box is the latest detection box;

[0021] 5) After obtaining the cost matrix, optimize the allocation and remove values ​​less than IoU. min The allocation process updates the ID database with the latest detection boxes belonging to the current ID and reclaims IDs that have not appeared for a long time.

[0022] Furthermore, the specific methods for human behavior recognition and abnormal behavior detection are as follows:

[0023] Using a trained static behavior recognition model, static behavior recognition is performed on the corresponding content of the latest detection box in the double-ended queue of each ID in the original image, and the static behavior recognition result of that ID is pushed into the static state queue belonging to that ID; abnormal behavior detection is performed on the static state queue according to a certain logical relationship.

[0024] This invention also provides a real-time monitoring method for falls and intentions to get up based on infrared visual AI. This method can effectively protect the privacy of the elderly and perform real-time monitoring. Upon detecting an elderly person getting up or falling, it promptly alerts relevant personnel in the nursing home to provide assistance. Specifically, it includes the following steps:

[0025] 1) Set static behaviors, collect data on different categories of static behaviors in the target scene, and form a fusion dataset; the static behaviors are divided into five situations: lying down, standing, sitting, bending over, and squatting.

[0026] 2) Create a static behavior recognition model and train it automatically on the data;

[0027] 3) Build a multi-process pipeline system for monitoring falls and intention to get up, and monitor falls and / or intention to get up in real time;

[0028] 4) If the system detects that the elderly person intends to get up, it sends the information to the caregiver; if the system detects that the elderly person has fallen, it issues an alarm.

[0029] Furthermore, the multi-process pipeline system for fall and get-up intention monitoring includes three stages: infrared thermal sensor interface reading and human target detection, multi-target human tracking, and determining whether the human target is on the bed and detecting get-up intention or fall detection; the three stages are processed in parallel by three processes.

[0030] Further, in step 1), the specific method for setting static behaviors and collecting data on different categories of static behaviors in the target scenario to form a fused dataset is as follows:

[0031] 1) Video Acquisition: Data acquisition is performed on the target scene, and videos are taken from different people, different angles, and different distances using an infrared thermal imager;

[0032] 2) Filtering video frames: Select frames from different times in the captured video to ensure that human targets are present in the extracted frames;

[0033] 3) Second round of screening: In the images selected in the previous step, when the human target is occluded by less than 30%, use screenshot software to crop the determined human target according to its outer contour and save it as an image;

[0034] 4) Data augmentation: Traverse all images, flip them horizontally, and save them in the original image path to multiply the acquired images.

[0035] Furthermore, the specific method for automatically training the static behavior recognition model using data in step 2) is as follows:

[0036] 2-1) Dataset expansion: After processing the newly collected data according to step 1), fill the old dataset with the labels corresponding to the new and old datasets to complete the dataset expansion;

[0037] 2-2) Import the static behavior recognition model. Import the index of all images in the dataset and classify them according to their filenames; the static behavior recognition model is ResNet18.

[0038] 2-3) Import the Adam optimizer and set the initial learning rate to 0.0001, using cross-entropy as the loss function; after each training round on the training set, the static behavior recognition model will automatically save the model and test it on the test set, saving the confusion matrix image to help trainers judge the recall and precision.

[0039] 2-4) Once the expected recall and precision are achieved, interrupt the training process directly.

[0040] Furthermore, the specific method for reading the infrared thermal sensor interface and detecting human targets is as follows:

[0041] 1) Obtain a complete thermal image by reading the infrared thermal sensor interface; the pixel value of each pixel represents the relative temperature of that point.

[0042] 2) After reading a frame of data from the infrared thermal sensor interface, use YOLOv5 to detect human targets;

[0043] 3) Assuming YOLOv5 successfully detects the target, initialize OpenCV's multi-target tracker using the detected human bounding box, and then use the multi-target tracker for human target detection in subsequent sequences.

[0044] 4) If YOLOv5 fails to detect a target in a future frame, the human bounding box from the tracker will be used to fill in the gap;

[0045] 5) After obtaining the human body bounding box of each frame, put it into the multi-process queue 1.

[0046] Furthermore, the method for determining whether a human target is on the bed is as follows:

[0047] First, foreground separation is performed to obtain a foreground mask;

[0048] Secondly, retrieve the latest detection bounding boxes for each ID from the ID database, and use the intersection of the detection bounding box and the foreground mask for each ID as the human segmentation result for that ID.

[0049] Next, based on the human segmentation results obtained for each ID, determine whether the human target is in bed;

[0050] The specific methods for detecting the intention to wake up or the fall detection are as follows:

[0051] Using the trained static behavior recognition model, static behavior recognition is performed on the corresponding content of the latest detection box in the double-ended queue of each ID in the original infrared image, and the static behavior recognition result of that ID is pushed into the static state queue belonging to that ID.

[0052] If the first two states of a certain ID's static state queue are any of the actions of squatting, bending over, sitting, or standing, and at least two of the three states are lying down, then it is considered that the person represented by that ID has fallen down.

[0053] The present invention has the following beneficial effects:

[0054] 1. The overall system design uses a multi-threaded pipeline design model, which can divide the overall task into several processing stages. The result of the previous stage is left for the next stage to process. At the same time, all processes are processed in parallel, which can make full use of computing resources and ensure the real-time operation of the system.

[0055] 2. The invented real-time infrared visual AI method for detecting falls and intention to get up uses infrared cameras to analyze the daily life of elderly people in nursing homes while respecting their privacy.

[0056] 3. Automated operations effectively reduce the workload of nursing home caregivers. Using more accurate real-time detection algorithms, alarms can be triggered promptly for elderly people getting up and falling, enabling caregivers to assist them in critical moments.

[0057] 4. The present invention provides a multi-process pipeline intelligent wake-up intention detection and fall detection algorithm, which utilizes traditional rule matching and feature extraction methods for wake-up intention detection, as well as the deep learning object detection framework YOLOv5 and the static behavior classification model ResNet18, and reasonably integrates many pattern matching algorithms.

[0058] 5. In the wake-up intention detection, the matching of multiple rules ensures the accuracy of this part of the algorithm; at the same time, in the fall detection, the setting of static behavior state queue and fall judgment queue improves the stability and applicability of the algorithm; the overall multi-process pipeline framework design is simple enough to reduce the complexity of the algorithm and ensure the real-time performance of the system as a whole.

[0059] 6. This invention fully utilizes the advantages of infrared cameras in terms of good privacy protection, fast computation speed of multi-threaded pipeline design, and high accuracy and stability of the fusion of deep networks and traditional algorithms to design a simple training method, high detection accuracy, excellent real-time performance, and strong reliability for detecting intention to get up and fall in nursing homes. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a logical schematic diagram of the real-time monitoring method for human behavior recognition and abnormal behavior detection involved in the present invention;

[0062] Figure 2 This invention relates to a multi-process pipeline system for an infrared visual AI fall and get-up intention detection system;

[0063] Figure 3 This invention relates to an infrared visual AI fall and get-up intention detection system, which includes an infrared thermal sensor interface reading and human target detection flowchart.

[0064] Figure 4 The present invention relates to a multi-target human body tracking flowchart of an infrared visual AI fall and get-up intention detection system.

[0065] Figure 5 This invention relates to a static behavioral state transition diagram of the infrared visual AI fall and get-up intention detection system.

[0066] Figure 6 The output image shows the result of the wake-up intention detection of the infrared visual AI fall and wake-up intention detection system involved in this invention.

[0067] Figure 7 This is an output image of the fall detection results of the infrared vision AI fall and get-up intention detection system involved in this invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] To achieve the above objectives, this invention provides a real-time monitoring method for human behavior recognition and abnormal behavior detection, which establishes a multi-process pipeline system for behavior recognition and abnormal behavior detection. Specifically, as follows... Figure 1 As shown, the multi-process pipeline system includes three stages: monitoring data reading and human target detection, multi-target human tracking, and human behavior recognition and abnormal behavior detection; the three stages are processed in parallel by three processes.

[0070] The specific methods for reading monitoring data and detecting human targets are as follows:

[0071] ①After reading a frame of data through the monitoring equipment, a target detection model is used to detect human targets;

[0072] ②Assuming the target detection model successfully detects the target, the detected human bounding box visual data is initialized to form a multi-target tracker. The multi-target tracker is then used to detect human targets in subsequent sequences. In particular, if the target detection model fails to detect the target in a future frame, the human bounding box name from the multi-target tracker is used to supplement it.

[0073] ③ After obtaining the human body bounding box of each frame, put it into the multi-process queue 1.

[0074] In the above method, the monitoring equipment can be a camera, infrared sensor, monitor, or other device capable of recording video and audio. The target detection model can be the YOLOv5 target detection framework or other target detection models capable of capturing the target object in the currently recorded image. Specifically, in this embodiment, the target object is a human body; in other application scenarios, the target object can be changed according to actual needs.

[0075] Furthermore, such as Figure 4 As shown, the specific method for multi-target human body tracking is as follows:

[0076] ① In order to identify different human bodies in the same frame and match them with previous human body bounding boxes, id1, id2...idn are used to give different people in the same frame image aliases, and an ID library is used to store the target detection bounding box data of the current id;

[0077] ② In the ID database, each ID has a fixed-length dequeue used to store the detection box data belonging to that ID; specifically, the ID dequeue stores the historical images corresponding to each ID.

[0078] ③ Retrieve all the detection box data from the multi-process queue 1, and select the detection box that is the most recent in time;

[0079] ④ Extract the current bounding boxes for all IDs in the current ID database, perform IoU calculation with the latest bounding box, and obtain the cost matrix; the IoU formula is as follows:

[0080]

[0081] Wherein, ID box is the detection box, ∩ is the intersection operator, ∪ is the union operator, and the latest detection box is the latest detection box;

[0082] ⑤ After obtaining the cost matrix, the Hungarian algorithm is used to optimize the allocation, removing values ​​less than IoU. min The allocation process updates the ID database with the latest detection boxes belonging to the current ID and reclaims IDs that have not appeared for a long time.

[0083] Furthermore, the specific methods for human behavior recognition and abnormal behavior detection are as follows:

[0084] ① Using the pre-trained static behavior recognition model - ResNet18 automatic training framework, static behavior recognition is performed on the corresponding content of the latest detection box in the double queue of detection boxes for each ID in the original image;

[0085] ② Push the static behavior recognition result of the ID into the static state queue belonging to the ID;

[0086] ③ Detect abnormal behavior in the static state queue according to a certain logical relationship.

[0087] Specifically, the logical relationship upon which abnormal behavior detection relies can be determined by forming abnormal behavior engine rules based on the relationship between abnormal behavior and normal behavior in different application scenarios.

[0088] Based on the above embodiments of the present invention, in scenarios for achieving efficient and safe 24 / 7 monitoring of elderly people in nursing homes, the monitoring equipment uses an infrared thermal sensor interface to read image data, thereby forming a real-time monitoring method for falls and intentions to get up based on infrared visual AI, such as... Figure 2 As shown, the specific implementation steps are as follows:

[0089] (1) Design static behaviors, collect data on different categories of static behaviors, and create a fusion dataset. Based on the purpose of this application scenario—real-time monitoring of the target object's intention to fall and get up—before training the static behavior recognition model in this system, corresponding static behaviors are designed according to the behavioral rules that form the intention to fall and get up. Data on different static behaviors are collected by category, and a fusion dataset is created to improve the accuracy of the static behavior recognition model training. The specific data collection method is as follows:

[0090] ① Video Collection: Data collection will be conducted in target settings such as nursing homes, capturing videos of different people, from different angles, and at different distances using infrared thermal imaging cameras. The actions of the people in the videos should include lying down, squatting, bending over, sitting, and standing.

[0091] ② Filter video frames: Select frames from different times in the captured video, with each frame lasting 2-8 seconds, to ensure that human targets are present in the captured frames.

[0092] ③ Second round of filtering: Further filtering is performed on the images selected in the previous step. When the human subject is less than 30% occluded, screenshot software is used to crop the identified human subject along its outer contour and save the image. Specifically, in...

[0093] ④ Human Image Classification: Based on the human images selected in the previous step, divide them into five categories and set labels as: Lying, Standing, Sitting, Bending, and Squatting; representing lying down, standing, sitting, bending over, and squatting respectively, and place them in five folders.

[0094] ⑤ Data augmentation: Traverse all images, flip them horizontally, and save them in the original image path to multiply the acquired images.

[0095] (2) Building a Static Behavior Recognition Model - ResNet18 Automatic Training Framework. After collecting and processing data on different categories of static behaviors in step (1), a static behavior recognition model is built. In this embodiment, ResNet18 is used for automatic training of static behavior recognition. The specific method is as follows:

[0096] (2-1) Dataset expansion: After processing the newly collected data according to step (1), fill the old dataset with the labels corresponding to the new and old datasets to complete the dataset expansion.

[0097] (2-2) Import the static behavior recognition model ResNet18, import the index of all images in the dataset and classify them according to filename (label). Randomly select 20% of the images in the dataset as the test set, and the rest as the training set.

[0098] (2-3) Import the Adam optimizer and set the initial learning rate to 0.0001; use cross-entropy as the loss function. The static behavior recognition model ResNet18 automatically saves the model after each training round on the training set and tests it on the test set, saving the confusion matrix image to help trainers judge recall and precision.

[0099] (2-4) Once the expected recall and precision are achieved, interrupt the training directly.

[0100] (3) A multi-process pipeline system for detecting falls and intention to get up is constructed to monitor intention to get up and / or falls in real time. After collecting and training the static behavior recognition model through steps (1) and (2), the trained static behavior recognition model is applied to build a multi-process pipeline system for detecting falls and intention to get up. Specifically, the entire system is divided into three stages: infrared thermal sensor interface reading and human target detection, multi-target human tracking, and determining whether the human target is on the bed and detecting intention to get up or fall. These three stages are processed in parallel by three processes. In particular, before the system is fully operational, the user needs to manually mark the location of the bed in the room. This step will not disclose the privacy information of the elderly in the room.

[0101] (3-1) Infrared thermal sensor interface reading and human target detection, such as Figure 3As shown: By reading from this interface, a complete thermal image can be obtained. The pixel value of each point represents the relative temperature of that point, ranging from 0 to 255. The higher the temperature, the whiter the pixel appears. Generally, the temperature of a human body is higher than the ambient temperature, resulting in a white silhouette. To more quickly and accurately identify all human targets within a frame, the YOLOv5 target detection framework and a multi-target tracker from OpenCV are used. After reading a frame of data from the infrared thermal sensor interface, YOLOv5 is used for human target detection. If YOLOv5 successfully detects a target, the detected human bounding box is used to initialize the OpenCV multi-target tracker. In subsequent sequences, the multi-target tracker will be used simultaneously for human target detection. If YOLOv5 fails to detect a target in a future frame, the tracker's human bounding box will be used to supplement it. After obtaining the human bounding box for each frame, it is placed into a multi-process queue 1. This queue has a limited length; when it reaches its maximum length, the oldest data is discarded, and new data is pushed in, preventing the process from being blocked.

[0102] (3-2) Multi-target human tracking, the implementation method is as follows: Figure 4 As shown, the specific steps have been described in detail above and will not be repeated here. It should be noted that in this embodiment, ID recycling is for cases where a target character appears, is assigned an ID0, but then does not reappear. In such cases, the content stored in ID0 will be cleared, and ID0 will no longer be used for a certain period. Generally, this period of non-reappearance is between 20 and 30 frames.

[0103] (3-3-1) Determining if a human target is in bed: Accurate determination of the target's lying position is crucial for both intention-to-get-up and fall detection. The level of danger varies significantly depending on whether a person is lying in bed. Therefore, determining whether a human target is in bed is particularly important in the process of human behavior recognition and abnormal behavior detection. The specific implementation method is as follows:

[0104] To more accurately determine whether a human target is on a bed, a finer-grained pixel-level human segmentation method was used. Since there is a significant temperature difference between the human body and the environment in infrared images (higher temperatures result in higher pixel values ​​for the human body, which is white; lower temperatures result in lower pixel values ​​for the environment, which is black), foreground separation was performed using formula (1-2) to obtain a foreground mask. The latest detection bounding boxes for each ID were retrieved from the ID database, and the intersection of the detection bounding box and the foreground mask for each ID was taken as the human segmentation result for that ID.

[0105]

[0106] T is typically between 0.6 and 0.7; in this embodiment, T of 0.7 is a better solution, as it prevents the human body from being completely covered while minimizing the amount of environmental data that would otherwise be considered as human.

[0107] human mask=(foreground mask)∪(ID box) (1-3)

[0108] After obtaining the human body segmentation results for each ID, the following steps are required. If the human body represented by a given ID meets the following conditions, then that human body is considered to be in bed.

[0109] Condition 1. Calculate IoH using formula (1-4):

[0110]

[0111] The Count() function is used to calculate the number of 1s in the matrix. Beds represents the bed positions marked before the overall system starts, and is a polygonal label. When the IoH value of a certain ID is greater than 0.5, the human body represented by that ID is considered to meet condition 1, and that bed is set as the target bed for that ID. At the same time, for the convenience of subsequent intent detection, a fixed-length double-ended queue is added to each bed. The length of the double-ended queue is 10, and the bright spot changes of the bed in the current frame are calculated and saved according to formula (1-5).

[0112] High lights of bed=Count(Humanmask)∩Count(Bed) (1-5)

[0113] Condition 2. The lowest point of the human body segmentation result of the ID that satisfies Condition 1 is considered to be one-third higher than the lowest point of the target bed surface if the point is higher than the lowest point of the target bed surface. The human body represented by the ID and the target bed surface are considered to satisfy Condition 2.

[0114] Condition 3. If the target human body and target bed surface that satisfy conditions 1 and 2 are such that Count(Human mask) < Count(Beds), then the human body and target bed surface represented by that ID are considered to satisfy condition 3.

[0115] If all three conditions above are met, the target human body is considered to be on the target bed surface, and the system proceeds to the intention to get up branch. Otherwise, the system proceeds to the fall detection branch.

[0116] (3-3-2) Real-time detection of intention to get up: Further, the following conditions need to be added to determine whether the target human body has the intention to get up.

[0117] Condition 1. The ID of the target human body is a newly added and assigned ID in the current frame. The newly added and assigned ID means that the human body represented by this ID is likely to be a newly added human body, rather than a human body with a misassigned ID, thus satisfying Condition 1.

[0118] Condition 2. Provided that Condition 1 is satisfied, further extract all bright spot data from the double-ended queue corresponding to the target bed surface. If the current number of bright spot increases is greater than 6, or the number of bed surface bright spots in the current frame is significantly greater than the number of bed surface bright spots in the tenth frame from the end, then Condition 2 is satisfied.

[0119] When both of the above conditions are met, it is assumed that the target person intends to get up. The system then sends the elderly person's intention to get up to the caregiver and displays it on the interface. Figure 6 As shown.

[0120] (3-3-3) Real-time fall detection, such as Figure 5 As shown: To ensure more stable fall alarm performance (stabilizing the output state), two double-ended queues were designed for each ID in the ID database (these queues are also reclaimed if the current ID is reclaimed), named the static state queue and the fall judgment queue, with lengths of 5 and 3 respectively. To obtain the static behavior recognition results for each ID, a trained ResNet18 model was used to perform static behavior recognition on the corresponding content of the latest detection box in the double-ended queue of each ID's detection boxes in the original infrared image, and the static behavior recognition result for that ID was pushed into the static state queue belonging to that ID. If the first two states of a certain ID's static state queue are any of the actions of squatting, bending over, sitting, or standing, and at least two of the last three states are lying down, it is considered that the person represented by that ID may have fallen. Furthermore, the fall label (True) is pushed into the fall judgment queue for that ID; otherwise, False is pushed into the fall judgment queue.

[0121] If two or more True values ​​appear in the fall detection queue for a given ID, it is assumed that the person represented by that ID has fallen, and a fall detection alarm is triggered. Figure 7 As shown.

[0122] In particular, for nursing homes and other care facilities, it is especially important to detect the elderly’s intention to get up and fall detection, because these two behaviors are abnormal behaviors that require close attention from caregivers. When the corresponding intention to get up or fall behavior is detected by the above methods, the corresponding alarm information is issued in a timely manner. Caregivers can receive the corresponding instructions and react in a timely manner. In practical applications, this can greatly improve the safety of the elderly and the response speed of caregivers.

[0123] Applying the model of this invention, actual testing showed a classification accuracy of 93.2%, an improvement of 12.7% compared to before using this invention. Specifically, the regression rate for the label "lying down" reached 98.2%, an improvement of 11.2%. Another significant improvement is in speed. On a computer with a Telsa V80, the system can achieve 15fps for up to three users, compared to the previous 12fps; for more than three users, it can achieve 10fps, compared to the previous 7fps, representing a substantial increase. In practical applications, compared to previous offline alarm systems, the system using this invention can effectively function as a real-time alarm system.

[0124] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0125] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0126] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0127] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A real-time monitoring method of human behavior recognition and abnormal behavior detection, characterized in that, A multi-process pipeline system for behavior recognition and fall detection is built, which includes three stages of monitoring data reading and human target detection, human multi-target tracking, human behavior recognition and abnormal behavior detection; The three stages are processed in parallel in three processes; The monitoring data reading and human target detection detect the human target through human target detection on the collected data, and the detection result is pressed into a multi-process queue 1; The data in the multi-process queue 1 is processed by the human multi-target tracking to extract the target detection frame data; The target detection frame data is subjected to human behavior recognition and abnormal behavior detection by a static behavior recognition model in the human behavior recognition and abnormal behavior detection; The human behavior recognition and abnormal behavior detection specifically include real-time monitoring of falls and getting-up intentions based on infrared vision AI, including the following steps: 1) Set a static behavior, collect data of different categories of static behaviors in a target scene to form a fusion data set; the static behavior includes five situations: lying on the ground, standing, sitting, bending and squatting; 2) Create a static behavior recognition model for automatic data training; 3) Build a multi-process pipeline system for fall and getting-up intention monitoring to monitor falls and / or getting-up intentions in real time; 4) If the human body has a getting-up intention, send the old person's getting-up intention information to the caregiver; if the human body has fallen, alarm; The multi-process pipeline system for fall and getting-up intention monitoring includes three stages of infrared thermal instrument interface reading and human target detection, human multi-target tracking, and judging whether the human target is on the bed and getting-up intention detection or fall detection; the three stages are processed in parallel in three processes; The method for judging whether the human target is on the bed is: First, perform foreground separation to obtain a foreground mask; Second, take all id time-updated detection frames from the id library, and take the intersection of each id detection frame and the foreground mask as the human segmentation result of the id; Third, judge whether the human target is on the bed according to the obtained human segmentation result of each id; The specific method for getting-up intention detection or fall detection is: Use the trained static behavior recognition model to perform static behavior recognition on the corresponding content of the latest detection frame in the double-ended queue of each id detection frame on the original infrared image, and press the static behavior recognition result of the id into the static state queue belonging to the id; If the first two states of the static state queue of an id are any of squatting, bending, sitting, and standing, and at least two of the three states are lying states, it is considered that the human body represented by the id has fallen.

2. A real-time monitoring method of human behavior recognition and abnormal behavior detection as claimed in claim 1, wherein, The specific method for monitoring data reading and human target detection is: (1) After reading a frame of data through the monitoring device, use the target detection model to detect the human target; (2) Under the premise that the target detection model successfully detects the target, initialize the human frame visual data to form a multi-target tracker, and use the multi-target tracker to detect the human target in the subsequent sequence; (3) If the target detection model fails to detect the target in a future frame, the human bounding box name of the multi-target tracker is used to supplement it. (4) After obtaining the human body bounding box of each frame, put it into the multi-process queue 1.

3. A real-time monitoring method of human behavior recognition and abnormal behavior detection as claimed in claim 1, wherein, The specific method for multi-target human body tracking is as follows: 1) In order to identify different human bodies in the same frame and match them with the previous human body bounding boxes, id1, id2...idn are used to give different people in the same frame image aliases, and an ID library is used to store the target detection bounding box data of the current id. 2) In the ID database, each ID has a fixed-length double-ended queue used to store the detection box data belonging to that ID; 3) Retrieve all the detection box data from the multi-process queue 1, and select the detection box that is the most recent in time; 4) Extract the current detection bounding boxes for all IDs in the current ID database, perform IoU calculation with the latest detection bounding box, and obtain the cost matrix; The IoU formula is calculated as follows: where ID box is a detection box, ∩ is an intersection operator, ∪ is a union operator, and lastest detection box is a latest detection box. 5) After obtaining the cost matrix, optimize the allocation, delete allocations with IoUmin less than 0, update the latest detection box belonging to the current ID in the ID database, and reclaim IDs that have not appeared for a long time.

4. A real-time monitoring method of human behavior recognition and abnormal behavior detection as claimed in claim 1, wherein, The specific methods for human behavior recognition and abnormal behavior detection are as follows: Using the trained static behavior recognition model, perform static behavior recognition on the corresponding content of the latest detection box in the double-ended queue of each ID in the original image, and push the static behavior recognition result of that ID into the static state queue belonging to that ID. Abnormal behavior is detected based on the static behavior recognition results according to a certain logical relationship.

5. A real-time monitoring method of human behavior recognition and abnormal behavior detection as claimed in claim 1, wherein, The specific method for step 1) is as follows: 1-1) Video Acquisition: Data acquisition is performed on the target scene, and videos are taken from different people, different angles, and different distances using an infrared thermal imager; 1-2) Filtering video frames: Select frames from different times in the captured video to ensure that there are human targets in the extracted frames; 1-3) Second round of screening: In the images selected in the previous step, when the human target is occluded by less than 30%, use screenshot software to crop the determined human target according to its outer contour and save it as an image; 1-4) Data augmentation: Traverse all images, flip them horizontally, and save them in the original image path to multiply the acquired images.

6. A real-time monitoring method of human behavior recognition and abnormal behavior detection as claimed in claim 1, wherein, The specific method for step 2) is as follows: 2-1) Dataset expansion: After processing the newly collected data according to step 1), fill the old dataset with the labels corresponding to the new and old datasets to complete the dataset expansion; 2-2) Import the static behavior recognition model, import the index of all images in the dataset and classify them according to the file name; The static behavior recognition model is ResNet18; 2-3) Import the Adam optimizer and set the initial learning rate to 0.0001, using cross-entropy as the loss function; after each training round on the training set, the static behavior recognition model will automatically save the model and test it on the test set, saving the confusion matrix image to help trainers judge the recall and precision. 2-4) Once the expected recall and precision are achieved, interrupt the training process directly.

7. A real-time monitoring method of human behavior recognition and abnormal behavior detection as claimed in claim 1, wherein, The specific method for reading the infrared thermal sensor interface and detecting human targets is as follows: (1) A complete thermal image is obtained by reading the infrared thermal sensor interface, and the pixel value of each pixel represents the relative temperature of that point; (2) After reading a frame of data from the infrared thermal sensor interface, use YOLOv5 to detect human targets; (3) If YOLOv5 successfully detects the target, use the detected human bounding box to initialize the OpenCV multi-target tracker, and use the multi-target tracker to detect human targets in subsequent sequences. (4) If YOLOv5 does not detect the target in a future frame, the human bounding box of the tracker is used to fill in the missing target. (5) After obtaining the human body bounding box of each frame, put it into the multi-process queue 1.

Images