Multi-person spatiotemporal abnormal behavior positioning method and system supporting real-time monitoring scenarios

Abstract

The present invention discloses a method and system for locating abnormal behaviors of multiple people in spatiotemporal domains that supports real-time monitoring scenarios. The method includes the following steps: using an acquisition device to capture video frames of real-time monitoring; performing multi-target detection and tracking and behavior recognition based on 2D motion modeling on the cached video frames to obtain character target frames and behavior confidence maps; using a multi-subject behavior matching algorithm to calculate the corresponding behavior confidence for each character target frame, and selecting the behavior with the highest confidence as the behavior of the person; using an abnormal risk assessment algorithm to score the risk factors contained in the behavior itself. The system includes modules: a data receiving module, a data persistence module, an abnormal behavior detection module, and a business processing module. The method and system for locating abnormal behaviors of multiple people in spatiotemporal domains of the present invention can complete the multi-target detection and tracking and behavior recognition processes in parallel, complete the abnormal behavior positioning of multiple people with high accuracy and low latency, and greatly improve the efficiency of abnormal detection.

Description

Technical Field

[0001] The present invention relates to the field of image and video processing technology, and in particular to a method and system for locating abnormal behaviors of multiple people in spatiotemporal domains supporting real-time monitoring scenarios. Background Art

[0002] In order to ensure the stability of social public security order, my country has proposed concepts such as "smart city" and "safe city", deployed a large number of video surveillance cameras, and built an intelligent urban monitoring management system. However, with the continuous expansion of urban population and the continuous expansion of urban scale, urban monitoring systems produce a large amount of monitoring videos every day, including indoor and outdoor monitoring. Outdoor scenes are mainly aimed at people coming and going, and can be used for anti-theft or external environment monitoring, such as outside the house and on the road. There is also a large part of monitoring shooting indoor scenes, including buses, offices, shops, houses, etc., mainly for indoor personnel monitoring, real-time care of the elderly and children in the home field, and constant monitoring of supervised persons in the security field.

[0003] However, the current monitoring method for these real-time monitoring is still mainly based on human inspection, which is very dependent on the experience of the monitor and consumes a lot of manpower and financial resources. The monitor often needs to monitor multiple monitors for a long time, which is prone to false detection and missed detection, and the inspection efficiency is very low.

[0004] In recent years, with the development and progress of artificial intelligence, an intelligent monitoring system that integrates computer vision technology has become popular. It uses deep learning algorithms to identify behaviors in surveillance videos and then matches them with pre-set abnormal behavior categories. Once abnormal behavior is identified, it immediately issues an early warning to supervisors. This algorithm-assisted approach can alleviate the pressure on supervisors to a certain extent and prevent missed detections. However, existing systems based on abnormal behavior recognition can often only identify single-person and group abnormalities, and cannot determine the spatial location of the abnormality, making them difficult to apply in multi-person scenarios. Therefore, how to achieve intelligent person-level abnormal behavior detection in multi-person monitoring scenarios has become a major challenge in the field of security monitoring.

[0005] Spatiotemporal action localization technology can effectively complete the task of multi-person behavior localization. It organically integrates the algorithm models of the two fields of behavior recognition and multi-target detection and tracking, and can determine the location of each person in the video and the corresponding behavior category.

[0006] Existing spatiotemporal action localization algorithms are mainly divided into two types: two-stage algorithms and end-to-end algorithms. The two-stage algorithm first predicts the candidate box of the target, and then performs behavior recognition on the target in the box. Since detection and recognition are performed in series, this method has poor real-time performance. The end-to-end algorithm uses multiple branches to perform detection and recognition tasks simultaneously. Although it reduces the amount of calculation, the algorithm's modeling of motion information is too rough, resulting in lower accuracy than the two-stage algorithm. In summary, the accuracy and real-time performance of the spatiotemporal action localization method have not reached a usable level, and it is difficult to show good results in the detection of abnormal behavior of people in multi-person scenes. Summary of the invention

[0007] The present invention aims at the problems existing in the field of security monitoring that it is impossible to effectively locate the behavior of multiple people, the accuracy of anomaly detection is low, and the real-time performance of the system is insufficient. A method and system for locating abnormal behavior of multiple people in spatiotemporal domain that supports real-time monitoring scenarios are proposed. The method and system are innovative methods and systems that are applied in the field of security and combine computer vision technology. The method and system utilize behavior recognition technology and multi-target detection and tracking technology to provide real-time personnel positioning and tracking and abnormal behavior recognition services in multi-person scenarios.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] On the one hand, the present invention provides a method for locating abnormal behaviors of multiple people in spatiotemporal domains supporting real-time monitoring scenarios, comprising the following steps:

[0010] S1: Use acquisition equipment to capture real-time monitoring video frames;

[0011] S2: Perform multi-target detection and tracking on the cached video frames and behavior recognition based on 2D motion modeling to obtain the character target frame and behavior confidence map;

[0012] S3: Use the multi-agent behavior matching algorithm to calculate the corresponding behavior confidence for each character target frame, and select the behavior with the highest confidence as the behavior of the person;

[0013] S4: Use an abnormal risk assessment algorithm to score the risk factors contained in the behavior itself.

[0014] Furthermore, in step S2, the multi-target detection and tracking algorithms are selected as YOLOv5 and DeepSort respectively. YOLOv5 is used to detect the target frames of all people appearing in the latest video frame, including the horizontal and vertical coordinates and the width and height of the upper left corner. DeepSort is used to associate the detected target frames with the person's trajectories and assign them to trajectory IDs.

[0015] Furthermore, the network structure of the behavior recognition algorithm based on 2D motion modeling includes a skeleton network, a motion modeling module and a prediction module. The behavior recognition algorithm process is as follows:

[0016] S21: Input: K video clips, each frame has an image size of 3×H×W, which represents the number of channels, height, and width respectively;

[0017] S22: Skeleton network: DLA-34 network is used to extract basic features of K frame images respectively. The feature map of , where C1 is the number of feature channels and R is the image downsampling ratio;

[0018] S23: Motion modeling module: The extracted K-frame feature map is sent to the motion modeling module, which uses 1×1 1D convolution to first reduce the number of channels to reduce the amount of motion modeling calculations, and then restores the number of channels to keep consistency, and supplements the motion modeling information in the form of residuals. Motion modeling is performed three times in a row to ensure that sufficient motion information is obtained; in each motion modeling unit, a 3×3 2D convolution is first used to shift and correct the features of the next frame, and then the features of the current frame are subtracted from the features of the next frame after the shift, and finally the results are spliced ​​together, and a feature map of all 0s is supplemented to maintain consistency in the time dimension;

[0019] S24: Prediction module: After obtaining the spatial features and motion features, the algorithm fuses the spatiotemporal information through the prediction module and predicts the behavior confidence. The prediction module first swaps the time and channel dimensions in the feature map, and then performs 3×3 2D convolutions channel by channel. Each convolution uses the feature values ​​of all frames within the 3x3 range to achieve the effect of fusing spatiotemporal information. Finally, a 1×1 convolution is performed to obtain the confidence map of each behavior category.

[0020] S25: Output: Behavior confidence map, where C is the number of behavior categories, H and W are the height and width of the video frame respectively.

[0021] Further, K=9, H=244, W=244, C1=1024, R=4, C=30.

[0022] Furthermore, in step S3, for each character target box box=x left ,y top ,w,h,The process of the multi-agent behavior matching algorithm is as follows:

[0023] S31: Behavior recognition algorithm output The behavior confidence map is converted into a behavior confidence map of size C×H×W through bilinear interpolation. For point x, y on the behavior confidence map, the probability that the behavior category is c is P c,xy ;

[0024] S32: Calculate the behavior confidence Fbox,x,y of the two-dimensional Gaussian distribution of the target box on the entire behavior confidence map according to formula (2), where: Then, according to formula (3), the behavior confidence corresponding to all points in the target box is normalized to obtain the confidence weight of all points. For point x, y in the target box, Wbox,x,y is the confidence weight of the point.

[0025]

[0026] S33: According to formula (4), the confidence weights generated in step S32 are used to perform weighted calculation on the confidence prediction values ​​of all points in the target box to obtain the comprehensive confidence of the target box for each behavior category:

[0027]

[0028] S34: According to formula (5), select the behavior with the highest confidence as the predicted behavior of the target box:

[0029]

[0030] Furthermore, the bilinear interpolation method in step S31 is: expand the 3×3 blue pixels to 5×5 green pixels according to the neighboring pixel values, that is, given the blue pixel values ​​Q of the four neighboring pixels 11 , Q 12 , Q 21 , Q 22 , use the bilinear interpolation algorithm to find the middle green pixel value P.

[0031] Furthermore, the process of solving the middle green pixel value P is as follows: 11 and Q 21 The value of R1 is calculated using a linear interpolation algorithm. 12 and Q 22 The pixel value of R2 is calculated using a linear interpolation algorithm based on the value of R1 and R2, and finally the pixel value of P is calculated using a linear interpolation algorithm based on the values ​​of R1 and R2.

[0032] Furthermore, the abnormal risk assessment algorithm process in step S4 is as follows:

[0033] S41: pre-define a behavior dictionary and a risk behavior template. The behavior dictionary divides common behaviors in indoor monitoring scenarios according to interaction methods, and the risk behavior template divides different behaviors into different risk levels.

[0034] S42: performing risk behavior template matching on each person's behavior recognition history results to obtain each person's risk assessment results;

[0035] S43: Determine whether an abnormality occurs based on the risk assessment result.

[0036] Furthermore, in order to further improve the real-time performance of abnormal behavior positioning, the present invention proposes to perform spatiotemporal abnormal behavior positioning only on key frames and motion frames, and adopts multiple processes to realize asynchronous real-time abnormal behavior positioning. The specific steps are as follows:

[0037] S1: Start the video stream reading process, the result video stream push process, the multi-target detection and tracking process, the motion trajectory prediction process, and the behavior recognition process;

[0038] S2: The multi-target detection and tracking process, motion trajectory prediction process, and behavior recognition process enter a loop waiting process; the video stream reading process continuously reads the real-time video stream and processes it; the result video stream push process cyclically reads the video frames with behavior positioning results and pushes them;

[0039] S2: The video stream reading process reads the latest video frame and sends it to the key frame determination algorithm to determine whether it is a key frame or a motion frame;

[0040] S3: If it is a key frame, the multi-target detection and tracking process is triggered; if it is not a key frame, the motion trajectory prediction process is triggered; if it is a motion frame, the behavior recognition process is triggered;

[0041] S4: The multi-target detection and tracking process predicts the character target frame and trajectory ID of the latest video frame; the motion trajectory prediction process predicts the trajectory ID of the latest frame; the behavior recognition process predicts the behavior confidence map of the latest frame sequence;

[0042] S5: Perform multi-agent behavior matching and risk assessment based on the target box, trajectory ID, and behavior confidence map, and draw the behavior positioning results on the latest frame;

[0043] S6: The result video stream push process pushes the latest video frames with the behavior positioning results to the RTSP streaming server in a loop.

[0044] On the other hand, the present invention also provides a multi-person spatiotemporal abnormal behavior positioning system supporting real-time monitoring scenarios, comprising the following modules to implement the above method steps:

[0045] Data receiving module: This module is used to capture the RTSP video stream pushed by the network surveillance camera, obtain the latest frame image of the video stream in real time, store it in the data persistence module, cache the latest frame sequence, and send it to the abnormal behavior detection module;

[0046] Abnormal behavior detection module: As the core module of the system, this module is used to complete multi-person positioning and tracking, behavior recognition and risk assessment functions. It obtains the latest frame sequence from the data receiving module, locates abnormal behaviors in the spatiotemporal domain, obtains behavior positioning records and risk assessment results, stores them in the data persistence module, and draws the positioning results on the latest frame image;

[0047] Data persistence module: This module is used to store structured data including video stream data from surveillance cameras, system log records, behavior location records, and risk assessment results;

[0048] Business processing module: This module is used to push image frames with behavior positioning information as video streams, and encapsulate the processed data into interface functions after analysis for calling;

[0049] System visualization module: This module is used to display the interface of the multi-person spatiotemporal abnormal behavior positioning system, including real-time monitoring with behavior positioning results, statistical data and abnormal alarms.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. The present invention proposes an innovative spatiotemporal domain abnormal behavior localization algorithm for personnel behavior localization and risk assessment. Compared with the traditional spatiotemporal action localization technology, the algorithm can complete multi-target detection and tracking and behavior recognition processes in parallel, and complete the abnormal behavior localization of multiple people with high accuracy and low latency, greatly improving the efficiency of anomaly detection.

[0052] 2. The behavior recognition algorithm based on 2D motion modeling and the multi-subject behavior matching algorithm proposed in the present invention provide new ideas for multi-person behavior recognition scenarios. The behavior recognition algorithm will infer the behavior confidence map of the current frame, while the multi-subject behavior matching algorithm performs Gaussian weighted calculation on the area where the person is located to obtain a comprehensive confidence, and then recognize the final behavior. In addition, the 2D motion modeling module proposed in the present invention can efficiently model the motion information of the characters in the video clips, improving the overall performance of the behavior recognition algorithm.

[0053] 3. The present invention proposes an asynchronous abnormal behavior location method based on key frame judgment and multi-process. The method can process multiple monitoring video streams in parallel, make full use of computer resources, and greatly improve the operating efficiency of the spatiotemporal domain abnormal behavior location algorithm.

[0054] 4. The present invention proposes a multi-person spatiotemporal abnormal behavior positioning system that supports real-time monitoring scenarios. The system realizes the visualization of monitoring video streams with behavior positioning results, supports real-time risk assessment and result display, and can effectively assist supervisors to quickly locate anomalies occurring during monitoring.

[0055] In summary, the real-time multi-person abnormal behavior positioning method provided by the present invention decomposes the original task into multiple subtasks such as target detection, multi-target tracking, behavior recognition, multi-subject behavior matching and abnormal risk assessment for solution. Each algorithm module is relatively independent, which can better ensure the accuracy of each task. At the same time, the present application designs a behavior recognition algorithm based on 2D motion modeling to improve the behavior recognition effect. For the proposed spatiotemporal abnormal behavior positioning method, the present application designs an asynchronous implementation method based on key frame determination and multi-process, which simultaneously completes personnel positioning tracking and behavior recognition tasks, further improving real-time performance. In addition, the present application also constructs a corresponding multi-person abnormal behavior positioning system, which can collect, process and analyze multiple monitoring video streams, locate and track, identify and assess the behavior of each person appearing in the video, and can efficiently perform medium and high risk warnings, display the behavior positioning results in real time on the monitoring interface, and summarize information and visualize real-time risks for all current monitoring from multiple angles, which can provide accurate and fast abnormal warning information for supervisors, thereby reducing manpower and time costs and improving the efficiency of abnormal behavior detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0057] Figure 1 A flow chart of a method for locating abnormal behaviors of multiple people in spatiotemporal domains supporting real-time monitoring scenarios provided by an embodiment of the present invention;

[0058] Figure 2 A diagram of a behavior recognition network structure based on 2D motion modeling provided by an embodiment of the present invention;

[0059] Figure 3 An example diagram of image upsampling using bilinear interpolation provided in an embodiment of the present invention;

[0060] Figure 4 A functional module diagram of a multi-person spatiotemporal abnormal behavior positioning system provided by an embodiment of the present invention;

[0061] Figure 5 A design diagram for locating asynchronous abnormal behavior based on key frame determination and multi-process provided in an embodiment of the present invention;

[0062] Figure 6 A business processing module operation diagram provided by an embodiment of the present invention;

[0063] Figure 7A user-side rendering of a multi-person spatiotemporal abnormal behavior positioning system provided by an embodiment of the present invention;

[0064] Figure 8 This is an architecture diagram of a multi-person spatiotemporal abnormal behavior positioning system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0066] The present invention proposes a method for locating abnormal behaviors of multiple people in spatiotemporal domains that supports real-time monitoring scenarios, such as Figure 1 As shown, the specific steps are as follows:

[0067] S1: Use acquisition equipment to capture real-time monitoring video frames;

[0068] S2: Perform multi-target detection and tracking on the cached video frames and behavior recognition based on 2D motion modeling to obtain the character target frame and behavior confidence map;

[0069] S3: Use the multi-agent behavior matching algorithm to calculate the corresponding behavior confidence for each character target frame, and select the behavior with the highest confidence as the behavior of the person;

[0070] S4: Use an abnormal risk assessment algorithm to score the risk factors contained in the behavior itself.

[0071] The following is a detailed explanation of the multi-target detection and tracking, behavior recognition based on 2D motion modeling, multi-agent behavior matching algorithm, and abnormal risk assessment algorithm in the algorithm.

[0072] 1. Multi-target detection and tracking algorithm

[0073] By investigating and comparing the existing multi-target detection and tracking technologies, this embodiment selects YOLOv5 and DeepSort, which are currently widely used in the industry and have better performance, as target detection and multi-target tracking algorithms, respectively. YOLOv5 is used to detect the target frames of all people appearing in the latest video frame, including the horizontal and vertical coordinates of the upper left corner and the width and height. DeepSort is used to associate the detected target frames with the track of the people and assign a track ID.

[0074] 2. Behavior recognition algorithm based on 2D motion modeling

[0075] The network structure of the behavior recognition algorithm based on 2D motion modeling is as follows Figure 2 As shown in the figure, it mainly includes skeleton network, motion modeling module, prediction module and other parts. The following is a detailed description of the process of processing video clips by this algorithm:

[0076] (1) Input: K video frames, each frame has an image size of 3×H×W, which represents the number of channels, height, and width respectively.

[0077] (2) Skeleton network: The DLA-34 network is used to extract basic features from K frames of images. The feature map is as follows, where C1 is the number of feature channels and R is the image downsampling ratio.

[0078] (3) Motion modeling module: The extracted K-frame feature maps are sent to the motion modeling module. This module uses 1×1 1D convolution to first reduce the number of channels to reduce the amount of motion modeling calculations, then restores the number of channels to maintain consistency, and supplements the motion modeling information in the form of residuals. Motion modeling is performed three times in a row to ensure that sufficient motion information is obtained. In each motion modeling unit, considering that the next frame may have motion shift, a 3×3 2D convolution is first used to correct the shift of the features of the next frame, and then the features of the current frame are subtracted from the features of the next frame after the shift. Finally, the results are concatenated and a feature map of all 0 is supplemented to maintain consistency in the time dimension.

[0079] (4) Prediction module: After obtaining spatial features and motion features, the algorithm fuses spatiotemporal information through the prediction module and predicts the behavior confidence. The prediction module first swaps the time and channel dimensions in the feature map, and then performs 3×3 2D convolutions channel by channel. Each convolution uses the feature values ​​of all frames within the 3x3 range to achieve the effect of fusing spatiotemporal information. Finally, a 1×1 convolution is performed to obtain the confidence map of each behavior category.

[0080] (5) Output: Behavior confidence map, where C is the number of behavior categories, H and W are the height and width of the video frame respectively.

[0081] Among them, K=9, H=244, W=244, C1=1024, R=4, C=30.

[0082] 3. Multi-agent behavior matching algorithm

[0083] The multi-agent behavior matching algorithm aims to calculate the comprehensive confidence of each behavior category for each target box generated by the multi-target detection and tracking algorithm, and then obtain the final predicted behavior of each person. left ,y top ,w,h,The process of behavior matching is as follows:

[0084] (1) Behavior recognition algorithm output The behavior confidence map is converted into a behavior confidence map of size C×H×W through bilinear interpolation. For point x, y on the behavior confidence map, the probability that the behavior category is c is P c,xy .

[0085] An example of bilinear interpolation for image upsampling is as follows Figure 3 As shown, it is necessary to expand the 3×3 blue pixels into 5×5 green pixels according to the neighboring pixel values, that is, given the blue pixel values ​​Q of the four neighbors 11 , Q 12 , Q 21 , Q 22 , use the bilinear interpolation algorithm to find the middle green pixel value P. The specific solution process is: according to Q 11 and Q 21 The value of R1 is calculated using a linear interpolation algorithm. 12 and Q 22 The pixel value of R2 is calculated using a linear interpolation algorithm based on the value of R1 and R2, and finally the pixel value of P is calculated using a linear interpolation algorithm based on the values ​​of R1 and R2.

[0086] Therefore, the problem is transformed into a linear interpolation problem: according to the values ​​of two points, find the value of any point on the line segment between the two points. 11 (x1,y1) and Q 21 Taking (x2, y1) as an example to find R1(x, y1), the three vertical coordinates are the same, so we only need to calculate the horizontal coordinate according to the line segment ratio, as shown in formula (1).

[0087]

[0088] (2) Use formula (2) to calculate the behavior confidence of the two-dimensional Gaussian distribution of the target box on the entire behavior confidence map, where: Then, the behavior confidence corresponding to all points in the target box is normalized to obtain the confidence weight of all points. As shown in formula (8-3), for point x, y in the target box, Wbox, x, y is the confidence weight of the point.

[0089]

[0090] (3) As shown in formula (4), the confidence weight generated by formula (3) is used to perform weighted calculation on the confidence prediction values ​​of all points in the target box to obtain the comprehensive confidence of the target box for each behavior category.

[0091]

[0092] (4) Finally, as shown in formula (5), the behavior with the highest confidence is selected as the predicted behavior of the target box.

[0093]

[0094] 4. Abnormal risk assessment algorithm

[0095] The abnormal risk assessment algorithm based on behavior template matching is as follows:

[0096] (1) Predefine behavior dictionary and risk behavior template;

[0097] (2) Then, based on each person’s behavior recognition history results, risk behavior template matching is performed to obtain each person’s risk assessment results;

[0098] (3) Determine whether an abnormality has occurred based on the risk assessment results.

[0099] As shown in Table 1, the behavior dictionary divides common behaviors in indoor monitoring scenarios into 3 categories and 30 subcategories according to the interaction mode, namely 10 behaviors of human-to-human interaction, 8 behaviors of human-to-object interaction, and 12 behaviors of single person. Then, as shown in Table 2, the risk behavior template divides different behaviors into different risk levels, including high-risk behavior, medium-risk behavior, and low-risk behavior. The abnormal risk assessment algorithm works based on the risk behavior template, which can realize risk assessment and abnormal alarm in a simple and efficient manner. Although the identifiable anomalies are limited to the risk behavior template, this module is relatively independent and can continuously improve the template in the subsequent use process to cover the common medium and high-risk behaviors in indoor scenarios as much as possible.

[0100] Table 1 Behavior dictionary

[0101]

[0102] Table 2 Risk behavior template

[0103]

[0104] The present invention also provides a real-time multi-person abnormal behavior positioning system to implement the above method steps. The functional module relationship is as follows: Figure 4 As shown in Figure 1, the software part of the entire system is divided into the following five modules: data receiving module, data persistence module, abnormal behavior detection module, business processing module, and system visualization module. Among them, the asynchronous multi-process implementation of the system's most critical multi-person abnormal behavior location function is as follows: Figure 5 As shown in the figure, it contains six processes: video stream reading process, multi-target detection and tracking process, motion trajectory prediction process, behavior recognition process, result video stream push process, and video stream saving process. These processes generally belong to different functional modules. The specific implementation of these functional modules will be described in detail below:

[0105] 1. Data receiving module: This module is used to capture the RTSP video stream pushed by the network surveillance camera, obtain the latest frame image of the video stream in real time, store it in the data persistence module, cache the latest frame sequence, and send it to the abnormal behavior detection module;

[0106] This module is mainly used to read the real-time image frames transmitted by the surveillance camera video stream, including the video stream reading process. The specific steps are as follows:

[0107] (1) Use Python's OpenCV library to initialize the RTSP video stream of the surveillance camera;

[0108] (2) Use Python's OpenCV library to read the latest frame image in the video stream;

[0109] (3) Send the latest frame to the video stream saving process and to the key frame extraction algorithm of the monitored object. After obtaining the motion information, perform two judgments (4) and (5) respectively;

[0110] (4) Determine whether the motion information of the current frame exceeds the threshold. If it exceeds the threshold, it is a key frame and is sent to the multi-target detection and tracking process; otherwise, it is sent to the motion trajectory prediction process;

[0111] (5) Determine whether the accumulated motion information exceeds the threshold. If it exceeds the threshold, it is a motion frame and is sent to the behavior recognition process;

[0112] (6) Repeat (2) to (5).

[0113] 2. Abnormal behavior detection module: As the core module of the system, this module is used to complete multi-person positioning and tracking, behavior recognition and risk assessment functions. It obtains the latest frame sequence from the data receiving module, locates abnormal behaviors in the spatiotemporal domain, obtains behavior positioning records and risk assessment results, and stores them in the data persistence module, and draws the positioning results on the latest frame image.

[0114] The abnormal behavior detection module includes multi-target detection and tracking process, motion trajectory prediction process, and behavior recognition process. The specific implementation of this module is described below:

[0115] 1) The multi-target detection and tracking process obtains the latest video frame in an asynchronous manner and performs detection and tracking. The specific implementation steps are as follows:

[0116] (1) The multi-target detection and tracking process will wait for queue tasks, and the monitoring video stream reading process will trigger queue tasks;

[0117] (2) Get the latest frame image from the monitoring object;

[0118] (3) The current frame image is sent to the YOLOv5 algorithm for target detection to obtain the position information of all people in the current frame, namely the target frame, including the horizontal and vertical coordinates of the upper left corner, width and height;

[0119] (4) Send all personnel location information and the current frame image to the DeepSort multi-target tracker to obtain personnel location and trajectory information, and update the result to the monitored object;

[0120] (5) Re-positioning behavior and re-assessing risk based on the new location and trajectory information of the monitored person.

[0121] 2) The motion trajectory prediction process uses the Kalman filter module in the DeepSort algorithm to directly predict the motion trajectory. It is more applicable when the motion is not significant, that is, when it is not a key frame. The specific steps are as follows:

[0122] (1) The motion trajectory prediction process will wait for queue tasks, and the monitoring video stream reading process will trigger queue tasks;

[0123] (2) Use the DeepSort multi-target tracker to directly perform Kalman filter motion trajectory prediction, calculate the new personnel position and trajectory information, and update the results to the monitored object;

[0124] (3) Re-positioning behavior and re-assessing risk based on the new location and trajectory information of the monitored person.

[0125] 3) The behavior recognition process uses the behavior recognition algorithm based on 2D motion modeling proposed by the present invention to calculate the current behavior confidence map. The specific process is as follows:

[0126] (1) The behavior recognition process will wait for queue tasks, and the monitoring video stream reading process will trigger queue tasks;

[0127] (2) Extract the most recent video frame sequence from the monitoring object;

[0128] (3) The obtained fragments are fed into a common behavior recognition algorithm model to obtain confidence maps of various behaviors, and the results are updated to the monitored objects;

[0129] (4) Re-position the behavior and conduct risk assessment based on the new behavior confidence map information of the monitored object.

[0130] 4) Behavior positioning and risk assessment function: In the multi-target detection and tracking process, motion trajectory prediction process, and behavior recognition process, the latest positioning tracking and behavior recognition results need to be used to re-perform behavior positioning and risk assessment. The specific steps are as follows:

[0131] (1) Take out the current behavior confidence map and personnel location trajectory from the monitored object and perform behavior matching on them;

[0132] (2) Conduct risk assessment on each person’s behavior, save the behavior location and risk assessment results into the database, and update the monitored objects;

[0133] (3) Draw the behavior localization results onto the current frame, and update the video frame with the behavior localization results to the monitored object.

[0134] 3. Data persistence module: This module is used to store structured data such as video stream data of surveillance cameras, system log records, behavior location records, and risk assessment results; this module mainly consists of two parts: the file system and the MySQL database. Among them, the video stream saving process stores the historical surveillance video stream to the file system, and structured data such as behavior location results, risk assessment results, and system log records are saved to the MySQL database.

[0135] 4. Business processing module: This module is used to push image frames with behavior positioning information as video streams, and encapsulate the processed data into interface functions after analysis for calling; the business processing module is mainly used for system application function implementation and external interface encapsulation. Figure 6 ,This module obtains relevant data from the data persistence module and the abnormal behavior ,detection module, analyzes it, and finally encapsulates it in the form of an interface and ,provides it to the system visualization module.

[0136] This module mainly includes four functions: obtaining all monitoring information, obtaining the latest monitoring data, obtaining historical anomalies, and pushing video streams with behavior positioning results. The first three functions are encapsulated as API interfaces for calling after reading the database and performing simple processing, and the last function is implemented by the result video stream push process. The result video stream push subprocess first starts the RTSP server, initializes the RTSP video stream using the ffmpeg command, establishes an input pipeline, and then loops to read the latest frame with behavior positioning results in the monitoring object, writes it into the pipeline for push.

[0137] 5. System visualization module: This module is used to display the interface of the multi-person spatiotemporal abnormal behavior positioning system, including real-time monitoring with behavior positioning results, statistical data and abnormal alarms.

[0138] The main interface of the system visualization module is as follows Figure 7 As shown, it mainly includes the following functions:

[0139] (1) Real-time monitoring with behavior location information: On the basis of completing the basic functions of the monitoring system, the video stream with the person target frame, trajectory ID, and behavior category is displayed to the front end in real time;

[0140] (2) Real-time personnel behavior location information table: The behavior identification results and risk levels of all supervised persons appearing in the monitoring are displayed in a table, which facilitates supervisors to quickly obtain the behavior status of supervised persons;

[0141] (3) Historical abnormality information: The medium- and high-risk events that have occurred are displayed as abnormalities in a table, including the time and cause of the abnormality, which can assist regulators in abnormality analysis;

[0142] (4) Risk statistics for each surveillance camera: The real-time risk assessment results of each surveillance camera are counted, and personnel are counted according to low, medium and high risks. Finally, the statistics are displayed in the form of a bar chart, which can prompt supervisors to focus their attention on surveillance cameras with more medium and high risks.

[0143] (5) Real-time statistical information: Summarizes the real-time monitoring information of the entire system, including the current total number of personnel, the current total number of medium and high-risk personnel, etc., so that regulators can control the overall risk situation from a macro perspective.

[0144] (6) This module also has the function of abnormal alarm. When a new abnormality occurs, a pop-up window will pop up on the front end of the system to prompt and display detailed abnormal personnel information to the supervisor.

[0145] The present invention provides a real-time multi-person abnormal behavior positioning method and system, which realizes simultaneous positioning tracking, behavior identification and risk assessment of multiple persons appearing in monitoring, visualizes the behavior positioning results in real time, and issues alarm prompts for abnormal behaviors.

[0146] The present invention provides a real-time multi-person abnormal behavior positioning system architecture as follows: Figure 8 As shown, in order to meet the needs of multi-monitoring expansion and reduce the degree of system coupling, the present invention divides the real-time multi-person abnormal behavior positioning system provided into a three-layer architecture: device layer, service layer and display layer. Among them, the multiple servers in the service layer work together in an asynchronous manner, which greatly improves the overall real-time performance of the system. The following is a detailed description:

[0147] (1) The device layer collects indoor real-time monitoring video streams and pushes them to the service layer through the RTSP protocol.

[0148] (2) The service layer mainly includes data servers, streaming media servers, business processing servers and computing servers. The data server is mainly used to store the data required by the system, including MySQL database and file system; the streaming media server is mainly used to push video streams with behavior positioning results, which is implemented using Nginx technology; the Python version of the spatiotemporal abnormal behavior positioning algorithm is deployed on the computing server, which is mainly based on Pytorch and OpenCV frameworks and is used for personnel positioning tracking, behavior recognition and risk assessment of video streams; the business processing server is mainly implemented using SpringBoot, which realizes the application functions of the system and encapsulates it as an API interface for front-end calls.

[0149] For the computing servers at the service layer, the present invention provides a new algorithm for locating abnormal behaviors in the spatiotemporal domain. As mentioned above, the algorithm first performs multi-target detection and tracking and behavior recognition in parallel, obtains personnel location information and behavior confidence maps, and then performs behavior matching and risk assessment on the detected personnel. For abnormal behavior detection in multi-person scenarios, the algorithm is innovative in its calculation method and greatly improves the efficiency of abnormal detection.

[0150] For the behavior recognition part of the spatiotemporal abnormal behavior location algorithm, the present invention provides a behavior recognition algorithm based on 2D motion modeling. As mentioned above, the algorithm introduces a new network module to extract motion information, and recognizes each position of the feature map, and finally outputs a behavior confidence map. The algorithm effectively solves the problem of inefficient extraction of key motion information, is innovative in prediction methods, and can provide data support for multi-person behavior recognition scenarios.

[0151] (3) The display layer is the visual front end of the system, which is mainly implemented based on the Vue framework. It is used to show the real-time monitoring status of the indoor scene to the supervisor and assist in analyzing the behavioral status of the supervised persons.

[0152] In the above-mentioned real-time multi-person abnormal behavior positioning system, in order to further improve the real-time performance of abnormal behavior positioning, the present invention also proposes a method for positioning abnormal behaviors in the spatiotemporal domain only for key frames and motion frames. For each monitoring, multiple processes are used to realize asynchronous real-time abnormal behavior positioning, and different tasks are performed respectively, including reading video streams, performing multi-target detection and tracking, performing motion trajectory prediction, performing behavior recognition, pushing result video streams, etc. This method can process multiple monitoring video streams in parallel, fully utilize computer resources, and greatly improve the real-time performance of multi-person abnormal behavior positioning.

[0153] The specific steps are as follows:

[0154] S1: Start the video stream reading process, the result video stream push process, the multi-target detection and tracking process, the motion trajectory prediction process, and the behavior recognition process;

[0155] S2: The multi-target detection and tracking process, motion trajectory prediction process, and behavior recognition process enter a loop waiting process; the video stream reading process continuously reads the real-time video stream and processes it; the result video stream push process cyclically reads the video frames with behavior positioning results and pushes them;

[0156] S2: The video stream reading process reads the latest video frame and sends it to the key frame determination algorithm to determine whether it is a key frame or a motion frame;

[0157] S3: If it is a key frame, the multi-target detection and tracking process is triggered; if it is not a key frame, the motion trajectory prediction process is triggered; if it is a motion frame, the behavior recognition process is triggered;

[0158] S4: The multi-target detection and tracking process predicts the character target frame and trajectory ID of the latest video frame; the motion trajectory prediction process predicts the trajectory ID of the latest frame; the behavior recognition process predicts the behavior confidence map of the latest frame sequence;

[0159] S5: Perform multi-agent behavior matching and risk assessment based on the target box, trajectory ID, and behavior confidence map, and draw the behavior positioning results on the latest frame;

[0160] S6: The result video stream push process pushes the latest video frames with the behavior positioning results to the RTSP streaming server in a loop.

[0161] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0162] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, electronic device embodiment, computer-readable storage medium embodiment, and computer program product embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0163] The above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical specialties therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for locating abnormal behaviors of multiple people in spatiotemporal domains supporting real-time monitoring scenarios, characterized in that: The following steps are involved: S1: Use acquisition equipment to capture real-time monitoring video frames; S2: Perform multi-target detection and tracking on the cached video frames and behavior recognition based on 2D motion modeling to obtain the character target frame and behavior confidence map; S3: Use the multi-agent behavior matching algorithm to calculate the corresponding behavior confidence for each character target frame, and select the behavior with the highest confidence as the behavior of each person; S4: Use an abnormal risk assessment algorithm to score the risk factors contained in the behavior itself; The network structure of the behavior recognition algorithm based on 2D motion modeling includes a skeleton network, a motion modeling module, and a prediction module. The behavior recognition algorithm process is as follows: S21: Input: K video clips, each frame has an image size of 3×H×W, which represents the number of channels, height, and width respectively; S22: Skeleton network: DLA-34 network is used to extract basic features of K frame images respectively. The feature map of , where C1 is the number of feature channels and R is the image downsampling ratio; S23: Motion modeling module: The extracted K-frame feature map is sent to the motion modeling module, which uses 1×1 1D convolution to first reduce the number of channels to reduce the amount of motion modeling calculations, and then restores the number of channels to keep consistency, and supplements the motion modeling information in the form of residuals. Motion modeling is performed three times in a row to ensure that sufficient motion information is obtained; in each motion modeling unit, a 3×3 2D convolution is first used to shift and correct the features of the next frame, and then the features of the current frame are subtracted from the features of the next frame after the shift, and finally the results are spliced ​​together, and a feature map of all 0s is supplemented to maintain consistency in the time dimension; S24: Prediction module: After obtaining the spatial features and motion features, the algorithm fuses the spatiotemporal information through the prediction module and predicts the behavior confidence. The prediction module first swaps the time and channel dimensions in the feature map, and then performs 3×3 2D convolutions channel by channel. Each convolution uses the feature values ​​of all frames within the 3x3 range to achieve the effect of fusing spatiotemporal information. Finally, a 1×1 convolution is performed to obtain the confidence map of each behavior category. S25: Output: Behavior confidence map, where C is the number of behavior categories, H and W are the height and width of the video frame respectively.

2. The method for locating abnormal behaviors of multiple people in spatiotemporal domain supporting real-time monitoring scenarios according to claim 1 is characterized in that: In step S2, the multi-target detection and tracking algorithms are selected as YOLOv5 and DeepSort respectively. YOLOv5 is used to detect the target frames of all people appearing in the latest video frame, including the horizontal and vertical coordinates of the upper left corner and the width and height. DeepSort is used to associate the detected target frames with the person's trajectory and assign a trajectory ID.

3. The method for locating abnormal behaviors of multiple people in spatiotemporal domain supporting real-time monitoring scenarios according to claim 1 is characterized in that: K=9, H=244, W=244, C1=1024, R=4, C=30.

4. The method for locating abnormal behaviors of multiple people in spatiotemporal domain supporting real-time monitoring scenarios according to claim 1 is characterized in that: In step S3, for each character target box box=(x left ,y top ,w,h), the process of multi-agent behavior matching algorithm is as follows: S31: Behavior recognition algorithm output The behavior confidence map is converted into a behavior confidence map of size C×H×W through bilinear interpolation. For a point (x, y) on the behavior confidence map, the probability that the behavior category is c is P (c,xy) ; S32: Calculate the behavior confidence F(box,x,y) of the two-dimensional Gaussian distribution of the target box on the entire behavior confidence map according to formula (2), where: Then, according to formula (3), the behavior confidence corresponding to all points in the target box is normalized to obtain the confidence weight of all points. For a point (x, y) in the target box, W(box, x, y) is the confidence weight of the point. S33: According to formula (4), the confidence weights generated in step S32 are used to perform weighted calculation on the confidence prediction values ​​of all points in the target box to obtain the comprehensive confidence of the target box for each behavior category: S34: According to formula (5), select the behavior with the highest confidence as the predicted behavior of the target box:

5. The method for locating abnormal behaviors of multiple people in spatiotemporal domain supporting real-time monitoring scenarios according to claim 4 is characterized in that: The bilinear interpolation method in step S31 is: expand the 3×3 blue pixels to 5×5 green pixels according to the neighboring pixel values, that is, given the blue pixel values ​​Q of the four neighboring pixels 11 , Q 12 , Q 21 , Q 22 , use the bilinear interpolation algorithm to find the middle green pixel value P.

6. The method for locating abnormal behaviors of multiple people in spatiotemporal domain supporting real-time monitoring scenarios according to claim 5 is characterized in that: The process of solving the green pixel value P in the middle is as follows: 11 and Q 21 The value of R1 is calculated using a linear interpolation algorithm. 12 and Q 22 The pixel value of R2 is calculated using a linear interpolation algorithm based on the value of R1 and R2, and finally the pixel value of P is calculated using a linear interpolation algorithm based on the values ​​of R1 and R2.

7. The method for locating abnormal behaviors of multiple people in spatiotemporal domain supporting real-time monitoring scenarios according to claim 1 is characterized in that: The abnormal risk assessment algorithm process in step S4 is as follows: S41: pre-define a behavior dictionary and a risk behavior template. The behavior dictionary divides common behaviors in indoor monitoring scenarios according to interaction methods, and the risk behavior template divides different behaviors into different risk levels. S42: performing risk behavior template matching on each person's behavior recognition history results to obtain each person's risk assessment results; S43: Determine whether an abnormality occurs based on the risk assessment result.

8. The method for locating abnormal behaviors of multiple people in spatiotemporal domain supporting real-time monitoring scenarios according to claim 1 is characterized in that: Only key frames and motion frames are used to locate abnormal behaviors in the spatiotemporal domain, and multiple processes are used to realize asynchronous real-time abnormal behavior location, including the following steps: S1: Start the video stream reading process, the result video stream push process, the multi-target detection and tracking process, the motion trajectory prediction process, and the behavior recognition process; S2: The multi-target detection and tracking process, motion trajectory prediction process, and behavior recognition process enter a loop waiting process; the video stream reading process continuously reads the real-time video stream and processes it; the result video stream push process cyclically reads the video frames with behavior positioning results and pushes them; the video stream reading process reads the latest video frame and sends it to the key frame determination algorithm to determine whether it is a key frame or a motion frame; S3: If it is a key frame, the multi-target detection and tracking process is triggered; if it is not a key frame, the motion trajectory prediction process is triggered; if it is a motion frame, the behavior recognition process is triggered; S4: The multi-target detection and tracking process predicts the character target frame and trajectory ID of the latest video frame; the motion trajectory prediction process predicts the trajectory ID of the latest frame; the behavior recognition process predicts the behavior confidence map of the latest frame sequence; S5: Perform multi-agent behavior matching and risk assessment based on the target box, trajectory ID, and behavior confidence map, and draw the behavior positioning results on the latest frame; S6: The result video stream push process pushes the latest video frames with the behavior positioning results to the RTSP streaming server in a loop.

9. A multi-person spatiotemporal abnormal behavior positioning system supporting real-time monitoring scenarios, characterized in that: The method comprises the following modules to implement any one of claims 1 to 8: Data receiving module: This module is used to capture the RTSP video stream pushed by the network surveillance camera, obtain the latest frame image of the video stream in real time, store it in the data persistence module, cache the latest frame sequence, and send it to the abnormal behavior detection module; Abnormal behavior detection module: As the core module of the system, this module is used to complete multi-person positioning and tracking, behavior recognition and risk assessment functions. It obtains the latest frame sequence from the data receiving module, locates abnormal behaviors in the spatiotemporal domain, obtains behavior positioning records and risk assessment results, stores them in the data persistence module, and draws the positioning results on the latest frame image; Data persistence module: This module is used to store structured data including video stream data from surveillance cameras, system log records, behavior location records, and risk assessment results; Business processing module: This module is used to push image frames with behavior positioning information as video streams, and encapsulate the processed data into interface functions after analysis for calling; System visualization module: This module is used to display the interface of the multi-person spatiotemporal abnormal behavior positioning system, including real-time monitoring with behavior positioning results, statistical data and abnormal alarms.

Images