Behavior recognition method, device, server and system for multi-person tracking

Abstract

The invention discloses a behavior recognition method, device, server and system for multi-person tracking, and the method comprises the steps: obtaining a video sequence, and carrying out the foreground and background segmentation of the video sequence; performing fuzzy processing on the background image, fusing the background image with the foreground image to obtain a target image, and executing the following tracking steps: extracting a current video frame to be processed from the target image, and identifying each moving target in the current video frame; calculating the intersection-to-union ratio between the detection track of each moving object in the current video frame and the prediction tracks of all moving objects in the current video frame, and constructing a cost matrix according to the intersection-to-union ratio; optimizing the cost matrix, finding out a matching scheme with the minimum total cost, optimizing the matching of each moving target based on the matching scheme, updating the trajectory of the moving target, and repeatedly executing the tracking step until the tracking result of each moving target in the whole video sequence is obtained. And identifying the abnormal behavior of the moving target according to the tracking result.

Description

Technical Field

[0001] The present invention relates to the field of privacy protection intelligent monitoring technology, and in particular to a behavior recognition method, device, server and system for multi-person tracking. Background Art

[0002] The average life expectancy of people is increasing due to better quality of life and improved standards of healthcare. Under this trend, the number of elderly people will increase rapidly. Due to the contradiction between the scarcity of caregivers and the growing number of elderly people, the care of the elderly may become a big problem in today's society. In daily life, the elderly are very prone to various accidents, such as falls. Early detection of falls is crucial to the health of the elderly. If there is no one to take care of the elderly when a fall occurs, and the elderly are not treated in time, it may cause serious damage to the elderly's body and even threaten their life safety. A slight delay in detecting falls and providing medical help may be fatal. If a person lies on the floor for more than 1 hour after a fall, then he has a 50% chance of dying within 6 months after the fall. Therefore, it is very important to use existing intelligent technologies in a home environment to detect, predict and alert the abnormal behavior of the elderly in a timely manner, which can effectively improve the service quality of the elderly care industry and provide a more solid guarantee for the safety of the elderly.

[0003] In recent years, with the rapid development of computer vision and artificial intelligence, a behavior recognition technology based on machine vision has gradually entered people's field of vision. Behavior recognition has gradually become an increasingly popular research direction in the field of computer vision. Behavior recognition is mainly used to understand and analyze human behavior in the target scene and accurately identify human behavior. Behavior recognition in the field of computer vision mainly collects visual data such as pictures or videos through cameras, and then extracts features from the visual data. After a series of post-processing of the extracted features, the function of detecting human behavior is realized. Modern artificial intelligence, especially deep learning, is very effective for such tasks. Deep learning is also popular because of its versatility. A trained model can be used to deal with a variety of different problems, such as human-computer interaction systems. A trained behavior recognition model can provide the necessary basic knowledge for such intelligent terminal systems. In the past, human-computer interaction generally required corresponding responses according to human instructions, which required people to directly contact some input devices such as keyboards and mice to transmit information to machines. Machines are often in a passive position and cannot fully exert their initiative. Behavior recognition technology can enable machines to understand human behavior and respond accordingly to human behavior.

[0004] However, with the popularization of artificial intelligence technology based on computer vision, various image data are collected and applied to deep learning technology. In this process, data containing family information and life scene information may be leaked. When performing non-invasive behavior recognition based on computer vision for the elderly, privacy issues are particularly prominent. As a special group in society, the privacy of the elderly needs to be protected. It is very necessary to conduct research on behavior recognition for the elderly based on computer vision under privacy protection. In addition, behavior recognition for multiple people is also a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a behavior recognition method, device, server and system for multi-person tracking, which can distinguish different behavior individuals and realize the recognition of the behavior status of multiple individuals under the premise of protecting the privacy of the environment in which the moving target is located. This is beneficial to ensuring the privacy of the identified persons while improving the accuracy of abnormal behavior recognition of the moving target.

[0006] In a first aspect, the present invention provides the following technical solution through an embodiment of the present invention:

[0007] A behavior recognition method for multi-person tracking, applied to a server, comprising:

[0008] A video sequence is obtained, and the foreground and background of the video sequence are segmented to obtain a foreground image and a background image, wherein the foreground image is an image containing a moving target; the background image is blurred and fused with the foreground image to obtain a target image, and the following tracking steps are performed: a current video frame to be processed is extracted from the target image, each moving target in the current video frame is identified, and a trajectory entry is created for each moving target; an intersection-and-union ratio is calculated between the detected trajectory of each moving target in the current video frame and the predicted trajectory of all moving targets in the current video frame, and a cost matrix is ​​constructed according to the intersection-and-union ratio, wherein the predicted trajectory is the upper trajectory of the current video frame. The method comprises the following steps: using a Kalman filter to predict the trajectory of each moving target in the next video frame in a video frame; optimizing the cost matrix by using the Hungarian matching algorithm to find a matching scheme with the minimum total cost between the detected trajectories and the predicted trajectories of all moving targets; optimizing the matching of each moving target based on the matching scheme; and updating the trajectory of the moving target based on the matching result by using the Kalman filter, storing the updated trajectory in the corresponding trajectory entry, and repeating the tracking steps until the tracking result of each moving target in the entire video sequence is obtained; and identifying the abnormal behavior of each moving target according to the tracking result of each moving target in the entire video sequence.

[0009] Preferably, after creating a trajectory entry for each moving target respectively, it also includes: marking the trajectories of all moving targets in the current video frame in the initial state as unconfirmed trajectories; updating the moving target trajectory based on the matching result using the Kalman filter includes: if the detected trajectory of the moving target successfully matches the predicted trajectory, then using the Kalman filter to update the trajectory of the moving target; if the detected trajectory of the moving target fails to match the predicted trajectory, and the trajectory of the moving target is an unconfirmed trajectory, then deleting the trajectory of the moving target.

[0010] Preferably, if there is a detected trajectory of a moving target that successfully matches the predicted trajectory, then after using the Kalman filter to update the trajectory of the moving target, it also includes: if there is a detected trajectory of a moving target that successfully matches the predicted trajectory in three consecutive video frames, then all detected trajectories after the matching of the moving target are marked as confirmed trajectories; the updating of the moving target trajectory using the Kalman filter based on the matching result also includes: if there is a detected trajectory of a moving target that fails to match the predicted trajectory, and the trajectory of the moving target is a confirmed trajectory, then the trajectory of the moving target is deleted after a preset number of consecutive failures to match.

[0011] Preferably, after updating the moving target trajectory using the Kalman filter based on the matching result, it also includes: extracting the appearance features of each moving target; for each moving target, cascade matching the appearance features of the moving target, the detected trajectory of the moving target, and the predicted trajectory of the moving target; if the cascade matching is successful, updating the moving target trajectory using the Kalman filter based on the cascade matching result.

[0012] Preferably, the method further includes: if the cascade matching fails, performing intersection-and-union matching on the unconfirmed trajectory or the trajectory that failed to match with the unmatched moving target to construct a new cost matrix; inputting the new cost matrix into the Hungarian algorithm to obtain a linear matching result; if the linear matching is successful, using a Kalman filter to update the moving target trajectory based on the linear matching result.

[0013] Preferably, the foreground and background segmentation of the video sequence includes: inputting the video sequence into a YOLOv8-seg network model to obtain a mask of a human image; and extracting a foreground image and a background image based on the mask of the human image and the video sequence.

[0014] Preferably, the foreground image and the background image are extracted based on the mask of the human body image and the video sequence, including: performing an AND operation on the mask of the human body image and the video sequence to obtain a foreground image, wherein the pixel value of the human body part in the mask of the human body image is 1, and the pixel value of other parts except the human body is 0; performing an inverting operation on the mask of the human body image to obtain a mask of the background image, and performing an AND operation on the mask of the background image and the video sequence to obtain a background image, wherein the pixel value of the human body part in the mask of the background image is 0, and the pixel value of other parts except the human body is 1.

[0015] In a second aspect, the present invention provides the following technical solution through an embodiment of the present invention:

[0016] A behavior recognition device for multi-person tracking, applied to a server, comprising:

[0017] An image segmentation module is used to obtain a video sequence, perform foreground and background segmentation on the video sequence, and obtain a foreground image and a background image, wherein the foreground image is an image containing a moving target;

[0018] A fusion module is used to perform blur processing on the background image and fuse it with the foreground image to obtain a target image, and perform the following tracking steps: extracting a current video frame to be processed from the target image, identifying each moving target in the current video frame, and creating a trajectory entry for each moving target;

[0019] A matching module, used to calculate the intersection-and-union ratio between the detected trajectory of each moving target in the current video frame and the predicted trajectory of all moving targets in the current video frame, and to construct a cost matrix according to the intersection-and-union ratio, wherein the predicted trajectory is obtained by using a Kalman filter to predict the trajectory of each moving target in the next video frame when the previous video frame is the current video frame;

[0020] A trajectory update module is used to optimize the cost matrix using the Hungarian matching algorithm, find a matching scheme with the minimum total cost between the detection trajectory and the predicted trajectory of all moving targets, optimize the matching of each moving target based on the matching scheme, and use the Kalman filter to update the moving target trajectory based on the matching result, store the updated trajectory in the corresponding trajectory entry, and repeat the tracking step until the tracking result of each moving target in the entire video sequence is obtained;

[0021] The behavior recognition module is used to recognize the abnormal behavior of each moving target according to the tracking result of each moving target in the entire video sequence.

[0022] In a third aspect, the present invention provides the following technical solution through an embodiment of the present invention:

[0023] A server comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods described in the first aspect are implemented.

[0024] In a fourth aspect, the present invention provides the following technical solution through an embodiment of the present invention:

[0025] A state recognition system for multi-person tracking comprises: a visual sensor and a server as described in the third aspect above, wherein the visual sensor is communicatively connected to the server and is used to send a video sequence to the server.

[0026] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0027] The behavior recognition method for multi-person tracking provided by the embodiment of the present invention first performs foreground and background segmentation on the video sequence in the video frame, blurs the background image, and then merges it with the foreground image to obtain the target original image containing only the foreground. Since multiple people may appear in the picture, a trajectory entry is created for each identified moving target; then, the intersection over union (IOU) between the detection trajectory of each moving target and the predicted trajectory of all moving targets is calculated, and their IOU values ​​are listed as a cost matrix, and then the cost matrix is ​​optimized. The matching scheme includes the matching relationship between the detection trajectory and the predicted trajectory of each moving target, and the moving target trajectory is updated based on the matching result. Finally, the tracking result of each moving target in the entire video sequence is obtained, and the abnormal behavior of each moving target is identified. The method first performs background blurring on the video sequence, and then tracks the moving target trajectory after the background blurring process. It can achieve more accurate identification of the abnormal behavior of each moving target while preventing background interference, ensuring the safety of the elderly. The behavior recognition method based on privacy-preserving computer vision technology can meet the privacy protection and home safety requirements of the elderly's home environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 is a flow chart of a behavior recognition method for multi-person tracking in an embodiment of the present invention;

[0030] Figure 2is a flow chart of a privacy protection method according to an embodiment of the present invention;

[0031] Figure 3 is a flow chart of a target tracking method in an embodiment of the present invention;

[0032] Figure 4 is a flow chart of a behavior recognition process in an embodiment of the present invention;

[0033] Figure 5 Schematic diagram of the structure of a behavior recognition device for multi-person tracking in an embodiment of the present invention;

[0034] Figure 6 A schematic diagram of the structure of a server in an embodiment of the present invention;

[0035] Figure 7 Schematic diagram of the structure of a behavior recognition system for multi-person tracking in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The embodiments of the present application provide a behavior recognition method, device, server and system for multi-person tracking, which can distinguish different behavior individuals and realize the recognition of the behavior status of multiple individuals under the premise of protecting the privacy of the environment in which the moving target is located. This is beneficial to ensuring the privacy of the identified persons while improving the accuracy of abnormal behavior recognition of the moving target.

[0037] The technical solution of the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0038] A behavior recognition method for multi-person tracking, applied to a server, comprising:

[0039] Get the video sequence and perform the following tracking steps: get the video sequence of the current video frame in the video sequence in turn, segment the video sequence into foreground and background, and get the foreground image and background image, where the foreground image is the image containing the moving target; blur the background image and fuse it with the foreground image to get the target image, identify each moving target in the target image, and create a trajectory entry for each moving target; calculate the intersection-and-union ratio between the detected trajectory of each moving target in the current video frame and the predicted trajectory of all moving targets in the current video frame, and construct a cost matrix based on the intersection-and-union ratio, where the predicted trajectory is the previous video of the current video frame. The Kalman filter is used to predict the trajectory of each moving target in the next video frame; the Hungarian matching algorithm is used to optimize the cost matrix to find the matching scheme with the minimum total cost between the detected trajectory and the predicted trajectory of all moving targets, and the matching of each moving target is optimized based on the matching scheme. The Kalman filter is used to update the moving target trajectory based on the matching result, and the updated trajectory is stored in the corresponding trajectory entry. The tracking steps are repeated until the tracking result of each moving target in the entire video sequence is obtained; according to the tracking result of each moving target in the entire video sequence, the abnormal behavior of each moving target is identified.

[0040] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0041] It should be noted that the behavior recognition method provided in this application can be applied in a home environment to identify abnormal behavior states of elderly people at home and ensure their safety. It can also be applied in nursing homes and other environments. For ease of understanding, the following detailed description of this application is given using the monitoring of the elderly in a home environment as an example.

[0042] In a first aspect, an embodiment of the present invention provides a behavior recognition method for multi-person tracking, which is applied to a server. Specifically, Figure 1 As shown, the method includes the following steps S101 to S105:

[0043] Step S101, obtaining a video sequence, performing foreground and background segmentation on the video sequence to obtain a foreground image and a background image, wherein the foreground image is an image containing a moving target;

[0044] Step S102, blurring the background image and fusing it with the foreground image to obtain a target image, and performing the following tracking steps: extracting a current video frame to be processed from the target image, identifying each moving target in the current video frame, and creating a trajectory entry for each moving target.

[0045] Among them, a visual sensor (such as a camera) can be installed in the home environment to monitor environmental information in real time and obtain a video sequence. The execution subject of this application can be a server, and the visual sensor is connected to the server in communication. The video sequence is the original image.

[0046] In this embodiment, performing foreground and background segmentation on a video sequence may include: inputting the video sequence into a YOLOv8-seg network model to obtain a mask of a human image; and extracting a foreground image and a background image based on the mask of the human image and the video sequence.

[0047] Extracting a foreground image and a background image based on a mask of a human body image and a video sequence may include: performing an AND operation on the mask of the human body image and the video sequence to obtain a foreground image, wherein the pixel value of the human body part in the mask of the human body image is 1, and the pixel value of other parts except the human body is 0; performing an inversion operation on the mask of the human body image to obtain a mask of the background image, and performing an AND operation on the mask of the background image and the video sequence to obtain a background image, wherein the pixel value of the human body part in the mask of the background image is 0, and the pixel value of other parts except the human body is 1.

[0048] In the specific implementation process, a de-identification method based on deep learning is used to implement a privacy protection solution for the elderly's home environment. The main idea is to use a segmentation algorithm to separate the background of the elderly's home environment, and then Gaussian blur the background and merge it with the foreground to achieve privacy protection. Privacy protection processing is the basis for subsequent target tracking and behavior recognition. Since the discovery of abnormal behavior of the elderly needs to be highly real-time, privacy protection processing must also take into account the impact of real-time factors. Therefore, a lightweight YOLOv8-seg segmentation algorithm based on the YOLACT (You Only Look At Coefficients) algorithm principle is used for privacy protection processing. YOLACT is a fast one-stage segmentation model. For the one-stage detection model, it adds a branch for generating a prototype mask in parallel with the detection branch.

[0049] like Figure 2As shown in the figure, the real-time segmentation algorithm YOLOv8-seg is used to separate the foreground and background. The original image collected by the visual sensor is input into the YOLOv8-seg network model to obtain the mask of the human image, where the pixel value of the human part is 1 and the pixel value of the rest is 0. The human image mask is ANDed with the original image to obtain the foreground image containing only the human part. The human image mask is inverted to obtain the background image mask, where the pixel value of the human part is 0 and the pixel value of the rest of the background is 1. The background image mask is ANDed with the original image to obtain the image containing only the background. After the background part is Gaussian blurred separately, it is superimposed and fused with the foreground image to obtain the privacy-processed image.

[0050] In the privacy protection process, YOLOv8-seg is used to separate the foreground and background and perform Gaussian blur processing on the background. By inputting the original image captured by the camera into the pre-trained YOLOv8-seg model, the mask of the human image is obtained, and then the foreground and background images are extracted by operating with the original image. Subsequently, the background image is Gaussian blurred and fused with the foreground image to achieve privacy protection. This method not only ensures effective privacy protection for the elderly's home environment, but also ensures the real-time processing, meeting the high real-time requirements of abnormal behavior detection.

[0051] In the analysis of the first frame of the video sequence, the output of the target detector included in the segmentation model YOLOV8-seg is used to create a trajectory entry for each identified moving target to initialize the moving target trajectory.

[0052] When monitoring and guarding the elderly, there may be multiple elderly people appearing in the video at the same time, so it is necessary to distinguish different elderly people, and then associate the results of behavior recognition with different human targets. The technology for distinguishing different elderly people is target tracking. After distinguishing different elderly individuals, the results of behavior recognition are associated according to different individual IDs, which facilitates the analysis of continuous behavioral actions of the same individual.

[0053] Step S103, calculating the intersection-and-union ratio between the detected trajectory of each moving target in the current video frame and the predicted trajectory of all moving targets in the current video frame, and constructing a cost matrix according to the intersection-and-union ratio, wherein the predicted trajectory is obtained by using a Kalman filter to predict the trajectory of each moving target in the next video frame when the predicted trajectory is the previous video frame of the current video frame;

[0054] Step S104, using the Hungarian matching algorithm to optimize the cost matrix, find the matching scheme with the minimum total cost between the detection trajectory and the predicted trajectory of all moving targets, optimize the matching of each moving target based on the matching scheme, and use the Kalman filter to update the moving target trajectory based on the matching result, store the updated trajectory in the corresponding trajectory entry, and repeat the tracking step until the tracking result of each moving target in the entire video sequence is obtained.

[0055] It should be noted that the detection trajectory of the moving target is directly detected from the monitoring image of the current video frame.

[0056] like Figure 3 As shown, after initializing the moving target trajectory, the trajectory of each moving target is predicted. Calculating the detection trajectory of each moving target in the current video frame may include: using a Kalman filter to predict the position of the trajectory in the next video frame, so as to estimate the dynamic behavior of the moving target without observation data.

[0057] Next, the intersection-and-union ratio between the detected trajectory of each moving target in the current video frame and the predicted trajectory of all moving targets in the current video frame is calculated, and a cost matrix is ​​constructed based on the intersection-and-union ratio, which reflects the quality of potential matches.

[0058] The cost matrix contains the intersection over union (IOU) between the detected trajectory of each moving target in the current video frame and the predicted trajectory of all moving targets in the next video frame. For example, if the number of moving targets is N, the cost matrix is ​​N×N.

[0059] Among them, the Kalman filter uses the dynamic model of the system to predict the position and velocity of the target at a certain moment in the future based on the previous state estimate. When new detection data is available, the predicted trajectory is compared with the detection trajectory (actual measurement value), and the predicted state is adjusted to obtain a more accurate estimate. Over time, through continuous prediction and updating, the accuracy of the system state estimate is gradually improved. These steps of the Kalman filter form an iterative process that ensures that the estimate of the state of the dynamic system can be continuously optimized in a continuous time. This method is widely used in fields that require real-time state estimation because of its high efficiency in dealing with uncertainty and noise.

[0060] Next, the cost matrix is ​​optimized using the Hungarian matching algorithm to find the matching scheme that minimizes the total cost between the detection trajectories and the predicted trajectories of all moving targets. In the matching scheme, the detection trajectory of each moving target is matched with a predicted trajectory. The matching of each moving target is optimized based on the matching scheme, and the Kalman filter is used to update the moving target trajectory based on the matching result. The updated trajectory is stored in the corresponding trajectory entry, and the predicted trajectory is updated in the trajectory entry to obtain the tracking result of the moving target.

[0061] In one embodiment, after creating a trajectory entry for each moving target, the method may further include: marking the trajectories of all moving targets in the current video frame in the initial state as unconfirmed trajectories; updating the trajectories of the moving targets based on the matching results using a Kalman filter, which may include: if the detected trajectory of a moving target successfully matches the predicted trajectory, updating the trajectory of the moving target using a Kalman filter; if the detected trajectory of a moving target fails to match the predicted trajectory and the trajectory of the moving target is an unconfirmed trajectory, deleting the trajectory of the moving target.

[0062] The matching solution that minimizes the total cost may refer to a matching solution whose cost is less than a set cost threshold.

[0063] Furthermore, the matching scheme obtained above can satisfy the requirement of minimizing the total cost. In order to match the detected trajectory of each moving target with the predicted trajectory, after determining the matching scheme, it is also necessary to determine whether the detected trajectory of each moving target is successfully matched with the predicted trajectory. If the match is successful, the Kalman filter is used to update the trajectory of the moving target. If the match fails and the trajectory of the moving target is an unconfirmed trajectory, the trajectory of the moving target is deleted.

[0064] Among them, successful matching may mean that the similarity between the detected trajectory and the predicted trajectory meets the preset requirements, otherwise, the matching fails.

[0065] If the detected trajectory of a moving target successfully matches the predicted trajectory, after the trajectory of the moving target is updated using the Kalman filter, it may also include: if the detected trajectory of the moving target successfully matches the predicted trajectory in three consecutive video frames, all detected trajectories after the moving target is matched are marked as confirmed trajectories; based on the matching result, the moving target trajectory is updated using the Kalman filter, and it also includes: if the detected trajectory of the moving target fails to match the predicted trajectory, and the trajectory of the moving target is a confirmed trajectory, the trajectory of the moving target is deleted after a preset number of consecutive failures to match. When the moving target reappears on the screen, a trajectory entry is recreated for the moving target.

[0066] Therefore, for successfully matched tracks, the Kalman filter is used to update the state; for unmatched tracks, if the track is in an unconfirmed state, it is directly deleted; for unmatched tracks, if the track is in a confirmed state, it is deleted after a preset number of consecutive unmatched times (for example: 30 times). For unmatched new detection targets (Unmatched Detections), new track entries are created for these moving targets.

[0067] By creating a trajectory entry corresponding to a moving target and marking its initial state as "Unconfirmed", all trajectories are in an unconfirmed state because it is the starting frame of the video. A target must be stably detected in three consecutive frames and matched through the intersection over union (IOU) before its state can be upgraded to "Confirmed".

[0068] In this application, the Hungarian matching method is mainly used to solve the problem of data association, that is, to determine how to effectively match detected targets (detections) with existing tracks (tracks) between consecutive frames. By evaluating the similarity between the detection box and the tracking box, and using the deep learning features extracted by ReID (re-identification) technology, a cost matrix is ​​constructed. Then the Hungarian matching method is used to find a matching solution that minimizes the cost, thereby optimizing the target matching process.

[0069] Furthermore, in order to achieve more accurate trajectory tracking, after using the Kalman filter to update the trajectory of the moving target based on the matching result, it can also include: extracting the appearance features of each moving target; for each moving target, cascading matching the appearance features of the moving target, the detected trajectory of the moving target, and the predicted trajectory of the moving target; if the cascade matching is successful, using the Kalman filter to update the trajectory of the moving target based on the cascade matching result.

[0070] If the cascade matching fails, the unconfirmed trajectory or the trajectory that failed to match is matched with the unmatched moving target by intersection and union matching to construct a new cost matrix; the new cost matrix is ​​input into the Hungarian algorithm to obtain the linear matching result; if the linear matching is successful, the Kalman filter is used to update the moving target trajectory based on the linear matching result.

[0071] In a specific embodiment, for the confirmed trajectory, the appearance features extracted by the ReID technology and the position features predicted by the Kalman filter (predicted trajectory) are further used for more accurate matching. The matching results have the following three possibilities: if the cascade matching is successful, the trajectory is updated through the Kalman filter; if the moving target fails to match the trajectory, the previously unconfirmed trajectory or the trajectory that failed to match is IOU matched with the unmatched moving target, and a new cost matrix is ​​constructed based on the matching results.

[0072] The success of the cascade matching may refer to that the appearance features of the moving target are consistent with the appearance features of the previous video frame, otherwise, the cascade matching fails. The situation where the cascade matching fails may be that the moving target is blocked.

[0073] Here, we use the DeepSORT algorithm, which is improved based on the SORT algorithm, to track the target, and use the ReID (pedestrian re-identification) algorithm to extract appearance features. In the matching strategy, both the motion information and appearance information of the target are considered, so as to greatly reduce the ID Switch (incorrect exchange of the target's ID) caused by occlusion.

[0074] In this application, cascade matching is an iterative matching strategy that gradually builds a matching relationship through a cyclic process. The process starts with matching the trajectory with a missing age of 0 with the detection result. The trajectory with a missing age of 0 refers to a trajectory that has a matching record in each frame and has not been lost. Then, the matching process gradually extends to trajectories with a larger missing age until the set maximum loss time threshold is reached, which is generally 30 frames. In this strategy, trajectories with fewer missing trajectories will be given a higher priority for matching. Conversely, those trajectories with more missing times are arranged to be matched last. This design ensures that the matching process can give priority to those trajectories with higher continuity and stability, thereby improving the accuracy and reliability of matching.

[0075] Then, the new cost matrix formed in the cascade matching process is input into the Hungarian algorithm to obtain the linear matching result. If the trajectory linear matching fails and the trajectory is in the confirmed state, the trajectory is deleted after a preset number of consecutive failures; if the trajectory linear matching fails and the trajectory is in the unconfirmed state, the trajectory is directly deleted; for unmatched moving targets, a new trajectory is initialized for the moving target and set to the unconfirmed state. If the detection frame (detection trajectory) successfully matches the prediction frame (prediction trajectory), it means that the target is tracked successfully in the continuous frames, and the trajectory is updated through the Kalman filter.

[0076] After each moving target trajectory is updated, the updated trajectory is stored in the corresponding trajectory entry, and the aforementioned tracking steps are repeated until the tracking result of each moving target in the entire video sequence is obtained.

[0077] Specifically, the aforementioned trajectory initialization, trajectory prediction, trajectory matching and trajectory update steps are repeated until the end of the video frame sequence. During the entire video frame processing process, Tracks is maintained and updated, and finally the tracking results of each target are output, including the moving target ID, position and other information.

[0078] Step S105 , identifying abnormal behavior of each moving target according to the tracking result of each moving target in the entire video sequence.

[0079] According to the tracking results of each moving target in the entire video sequence, the abnormal behavior of each moving target is identified, which can include: inputting the tracking results of each moving target in the entire video sequence into a Slowfast behavior recognition model to extract the spatial semantic features and motion features of each moving target; based on the spatial semantic features and motion features of each moving target, the abnormal behavior of each moving target is identified.

[0080] In this embodiment, the Slowfast behavior recognition algorithm is used to perform behavior recognition for the elderly. The algorithm processes video frames with low sampling rates and high sampling rates respectively through the Slow and Fast paths to extract spatial semantic features and motion features. The Slow path samples the input video at a lower sampling rate, performs 2D+3D convolution, and extracts spatial semantic features that change slowly over time. At the same time, in order to make the spatial semantic features more robust, the Slow path obtains more spatial channels; the Fast path samples the input video at a higher sampling rate, performs 3D convolution to extract motion features that change quickly over time, and the Fast path has more pictures. In order to reduce the complexity of the channel, the Fast path obtains fewer spatial channels. In order to fuse these two features, a lateral connection method is also used to transfer the features extracted by the Fast path to the Slow path, thereby improving the accuracy and robustness of behavior recognition.

[0081] like Figure 4As shown, the overall process of the state recognition method shown in the present application is as follows: first, the camera captures an image, performs YOLOv8-seg algorithm operation on the image, obtains an image and a human mask, blurs the background image, and obtains an image list (i.e., a video sequence) of the target image after the background is blurred. If the number of images in the image list is greater than or equal to 64 pictures, the key frame is extracted, and the DeepSORT algorithm is performed on the key frame (i.e., the current video frame to be processed) to perform target detection and Slowfast behavior recognition to obtain a matching behavior recognition result; if the number of images is less than 64, the camera continues to capture images and performs YOLOv8-seg algorithm operation.

[0082] The moving target tracking process in this application includes: trajectory initialization, trajectory prediction, IOU matching and cost matrix construction, matching and updating, cascade matching, linear matching, loop iteration and output results. This process includes three trajectory matching updates, which makes the trajectory tracking more accurate and is conducive to more accurate behavior state recognition of the moving target.

[0083] In summary, a behavior recognition method for multi-person tracking is provided by an embodiment of the present invention. The method first performs background blurring on the surveillance image, and then tracks the trajectory of the moving target after the background blurring, so as to more accurately identify the abnormal behavior of each moving target and ensure the safety of the elderly. The behavior recognition method based on privacy-preserving computer vision technology can meet the needs of privacy protection and home safety in the elderly's home environment.

[0084] In the second aspect, based on the same inventive concept, this embodiment provides a behavior recognition device for multi-person tracking, which is applied to a server, such as Figure 5 As shown, the device comprises:

[0085] The image segmentation module 401 is used to obtain a video sequence, perform foreground and background segmentation on the video sequence, and obtain a foreground image and a background image, wherein the foreground image is an image containing a moving target;

[0086] The fusion module 402 is used to perform blur processing on the background image and fuse it with the foreground image to obtain a target image, and perform the following tracking steps: extracting a current video frame to be processed from the target image, identifying each moving target in the current video frame, and creating a trajectory entry for each moving target;

[0087] A matching module 403 is used to calculate the intersection-and-union ratio between the detected trajectory of each moving target in the current video frame and the predicted trajectory of all moving targets in the current video frame, and to construct a cost matrix according to the intersection-and-union ratio, wherein the predicted trajectory is obtained by using a Kalman filter to predict the trajectory of each moving target in the next video frame when the predicted trajectory is the previous video frame of the current video frame;

[0088] The trajectory updating module 404 is used to optimize the cost matrix by using the Hungarian matching algorithm, find a matching scheme with the minimum total cost between the detection trajectory and the predicted trajectory of all moving targets, optimize the matching of each moving target based on the matching scheme, and use the Kalman filter to update the moving target trajectory based on the matching result, store the updated trajectory in the corresponding trajectory entry, and repeat the tracking step until the tracking result of each moving target in the entire video sequence is obtained;

[0089] The behavior recognition module 405 is used to recognize the abnormal behavior of each moving target according to the tracking result of each moving target in the entire video sequence.

[0090] As an optional embodiment, the trajectory updating module 404 is specifically used to: mark the trajectories of all moving targets in the current video frame in the initial state as unconfirmed trajectories; and update the moving target trajectory based on the matching result using the Kalman filter, including: if the detected trajectory of the moving target successfully matches the predicted trajectory, then use the Kalman filter to update the trajectory of the moving target; if the detected trajectory of the moving target fails to match the predicted trajectory, and the trajectory of the moving target is an unconfirmed trajectory, then delete the trajectory of the moving target.

[0091] As an optional embodiment, the trajectory update module 404 is also specifically used for: if the detected trajectory of the moving target and the predicted trajectory are successfully matched in three consecutive video frames, all the detected trajectories after the matching of the moving target are marked as confirmed trajectories; the updating of the moving target trajectory based on the matching result using the Kalman filter also includes: if the detected trajectory of the moving target fails to match the predicted trajectory, and the trajectory of the moving target is a confirmed trajectory, the trajectory of the moving target is deleted after a preset number of consecutive failures to match.

[0092] As an optional embodiment, the trajectory update module 404 is also specifically used to: extract the appearance features of each moving target; for each moving target, cascade match the appearance features of the moving target, the detected trajectory of the moving target, and the predicted trajectory of the moving target; if the cascade match is successful, use a Kalman filter to update the trajectory of the moving target based on the cascade matching result.

[0093] As an optional embodiment, the trajectory update module 404 is further specifically used for: if the cascade matching fails, performing intersection-and-union matching on the unconfirmed trajectory or the trajectory that failed to match with the unmatched moving target to construct a new cost matrix; inputting the new cost matrix into the Hungarian algorithm to obtain a linear matching result; if the linear matching is successful, using the Kalman filter to update the moving target trajectory based on the linear matching result.

[0094] As an optional embodiment, the image segmentation module 401 includes:

[0095] The mask acquisition submodule is used to input the video sequence into the YOLOv8-seg network model to obtain the mask of the human body image;

[0096] The extraction submodule is used to extract the foreground image and the background image based on the mask of the human image and the video sequence.

[0097] As an optional embodiment, the extraction submodule is specifically used to: perform an AND operation on the mask of the human body image and the video sequence to obtain a foreground image, wherein the pixel value of the human body part in the mask of the human body image is 1, and the pixel value of other parts except the human body is 0; perform an inversion operation on the mask of the human body image to obtain a mask of the background image, and perform an AND operation on the mask of the background image and the video sequence to obtain a background image, wherein the pixel value of the human body part in the mask of the background image is 0, and the pixel value of other parts except the human body is 1.

[0098] The above modules can be implemented by software codes, in which case, the above modules can be stored in the memory of the control device. The above modules can also be implemented by hardware such as integrated circuit chips.

[0099] The embodiment of the present invention provides a behavior recognition device for multi-person tracking, and its implementation principle and technical effects are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment.

[0100] In a third aspect, based on the same inventive concept, this embodiment provides a server 500, such as Figure 6 The invention comprises: a memory 501, a processor 502 and a computer program 503 stored in the memory and executable on the processor, wherein when the processor 502 executes the program, the steps of the behavior recognition method for multi-person tracking described in the first aspect are implemented.

[0101] Since the server introduced in this embodiment is a server used to implement the control method of the squeeze roller in the embodiment of the present application, based on the control method of the squeeze roller introduced in the embodiment of the present application, the technical personnel in this field can understand the specific implementation of the server in this embodiment and its various variations, so how the server implements the method in the embodiment of the present application is not described in detail here. As long as the technical personnel in this field implement the server used in the control method of the squeeze roller in the embodiment of the present application, it belongs to the scope of protection of this application.

[0102] In a fourth aspect, based on the same inventive concept, this embodiment provides a state recognition system 50 for multi-person tracking, such as Figure 7 As shown, it includes: a visual sensor 400 and a server 500 as described in the third aspect above, and the visual sensor is communicatively connected to the server to send a video sequence to the server.

[0103] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A module that specifies functions in one or more boxes.

[0105] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction module, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0107] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0108] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A behavior recognition method for multi-person tracking, characterized in that: Applied to a server, the method comprises: Acquire a video sequence, perform foreground and background segmentation on the video sequence, and obtain a foreground image and a background image, wherein the foreground image is an image containing a moving target; The background image is blurred and fused with the foreground image to obtain a target image, and the following tracking steps are performed: a current video frame to be processed is extracted from the target image, each moving target in the current video frame is identified, and a trajectory entry is created for each moving target; Calculate the intersection-and-union ratio between the detected trajectory of each moving target in the current video frame and the predicted trajectory of all moving targets in the current video frame, and construct a cost matrix according to the intersection-and-union ratio, wherein the predicted trajectory is obtained by using a Kalman filter to predict the trajectory of each moving target in the next video frame when the previous video frame is the current video frame; The cost matrix is ​​optimized by using the Hungarian matching algorithm to find a matching scheme with the minimum total cost between the detection trajectory and the predicted trajectory of all moving targets, the matching of each moving target is optimized based on the matching scheme, and the moving target trajectory is updated by using a Kalman filter based on the matching result, and the updated trajectory is stored in the corresponding trajectory entry, and the tracking step is repeatedly performed until the tracking result of each moving target in the entire video sequence is obtained; According to the tracking result of each moving target in the entire video sequence, the abnormal behavior of each moving target is identified.

2. The method according to claim 1, characterized in that After creating a trajectory entry for each moving target, the method further includes: Mark all the trajectories of the moving targets in the current video frame in the initial state as unconfirmed trajectories; The updating of the moving target trajectory using a Kalman filter based on the matching result includes: If the detected trajectory of a moving target successfully matches the predicted trajectory, the Kalman filter is used to update the trajectory of the moving target; If the detected trajectory of a moving target fails to match the predicted trajectory, and the trajectory of the moving target is an unconfirmed trajectory, the trajectory of the moving target is deleted.

3. The method according to claim 2, characterized in that If the detected trajectory of a moving target successfully matches the predicted trajectory, the trajectory of the moving target is updated using the Kalman filter, and the following steps are also included: If the detected trajectory of a moving target successfully matches the predicted trajectory in three consecutive video frames, the trajectory under the trajectory entry corresponding to the moving target is marked as a confirmed trajectory; The updating of the moving target trajectory using a Kalman filter based on the matching result also includes: If there is a detected trajectory of a moving target that fails to match the predicted trajectory, and the trajectory of the moving target is a confirmed trajectory, the trajectory of the moving target is deleted after a preset number of consecutive failures to match.

4. The method according to claim 2, characterized in that After the Kalman filter is used to update the moving target trajectory based on the matching result, the method further includes: Extract the appearance features of each moving target; For each moving target, the appearance features of the moving target, the detection trajectory of the moving target and the predicted trajectory of the moving target are cascade matched; If the cascade matching is successful, the Kalman filter is used to update the moving target trajectory based on the cascade matching result.

5. The method according to claim 4, characterized in that Also includes: If the cascade matching fails, the unconfirmed trajectory or the trajectory that failed to match is matched with the unmatched moving target by intersection and union matching to construct a new cost matrix; Input the new cost matrix into the Hungarian algorithm to obtain the linear matching result; If the linear matching is successful, the Kalman filter is used to update the moving target trajectory based on the linear matching result.

6. The method according to claim 1, characterized in that The step of segmenting the foreground and background of the video sequence comprises: Input the video sequence into the YOLOv8-seg network model to obtain the mask of the human image; Based on the mask of the human image and the video sequence, the foreground image and the background image are extracted.

7. The method according to claim 6, characterized in that The method of extracting a foreground image and a background image based on a mask of a human body image and a video sequence includes: Performing an AND operation on the mask of the human body image and the video sequence to obtain a foreground image, wherein the pixel value of the human body part in the mask of the human body image is 1, and the pixel value of other parts except the human body is 0; The mask of the human body image is inverted to obtain the mask of the background image, and the mask of the background image is ANDed with the video sequence to obtain the background image, wherein the pixel value of the human body part in the mask of the background image is 0, and the pixel value of other parts except the human body is 1.

8. A state recognition device for multi-person tracking, characterized in that: Applied to a server, the device comprises: An image segmentation module is used to obtain a video sequence, perform foreground and background segmentation on the video sequence, and obtain a foreground image and a background image, wherein the foreground image is an image containing a moving target; A fusion module is used to perform blur processing on the background image and fuse it with the foreground image to obtain a target image, and perform the following tracking steps: extracting a current video frame to be processed from the target image, identifying each moving target in the current video frame, and creating a trajectory entry for each moving target; A matching module, used to calculate the intersection-and-union ratio between the detected trajectory of each moving target in the current video frame and the predicted trajectory of all moving targets in the current video frame, and to construct a cost matrix according to the intersection-and-union ratio, wherein the predicted trajectory is obtained by using a Kalman filter to predict the trajectory of each moving target in the next video frame when the previous video frame is the current video frame; A trajectory update module is used to optimize the cost matrix using the Hungarian matching algorithm, find a matching scheme with the minimum total cost between the detection trajectory and the predicted trajectory of all moving targets, optimize the matching of each moving target based on the matching scheme, and use the Kalman filter to update the moving target trajectory based on the matching result, store the updated trajectory in the corresponding trajectory entry, and repeat the tracking step until the tracking result of each moving target in the entire video sequence is obtained; The behavior recognition module is used to recognize the abnormal behavior of each moving target according to the tracking result of each moving target in the entire video sequence.

9. A server, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 8 when executing the program.

10. A state recognition system for multi-person tracking, characterized in that: include: A visual sensor and a server as claimed in claim 9, wherein the visual sensor is communicatively connected to the server and is used to send a video sequence to the server.

Images