Systems and methods for security video behavior recognition, query, and anomaly detection

Abstract

This invention provides a system and method for security video behavior recognition, querying, and anomaly detection, relating to the field of security video anomaly analysis technology. It utilizes digital network cameras to collect real-time video data, which is then preprocessed via multi-threading and input into a neural network. Actions are classified to obtain the associations of all identifiable actions. Action values ​​are defined and quantified. The correlation between actions is used to detect fluctuations in human movements within the video, extracting suspected abnormal segments. Within these suspected abnormal segments, a sliding window technique is used to group all actions and obtain representative actions for each group. The sliding window is then used to detect the mean of actions in the current group and its neighboring areas, yielding the action data for the current frame. A two-dimensional statistical histogram of the action mean and action value is calculated. Normal actions are generally distributed along the diagonal of a planar coordinate system; cluster analysis is then performed to identify abnormal behavior. This invention overcomes the scene dependence of existing anomaly detection methods, has a wide range of applications, and possesses online real-time detection capabilities.

Description

Technical Field

[0001] This invention relates to the field of security video anomaly analysis technology, and in particular to a system and method for security video behavior recognition, querying, and anomaly detection. Background Technology

[0002] my country's video surveillance system has grown increasingly large-scale, but a significant portion of the industry has failed to achieve its intended application effects. One reason for this is the lack of professional operation and maintenance (O&M) teams. Current O&M methods rely heavily on manual labor, with auxiliary management systems limited to basic inspections and equipment rounds, resulting in overall low O&M efficiency. Historically, video surveillance technology relied on personnel observing data to detect pedestrian movements and trajectories, analyzing and identifying abnormal behaviors. However, with the widespread installation of surveillance equipment and the dramatic increase in data volume, manual review of video content has become inadequate and prone to omissions. Pedestrian movement recognition and anomaly detection in video surveillance have gradually become a crucial technological prerequisite for strategic deployments in security monitoring and information security, with immeasurable academic and market value.

[0003] Video anomalies are generally considered to be unusual appearances, motion characteristics, or motion characteristics occurring at unusual locations or times. Current methods for identifying video anomalies are often scene-dependent, hindering the widespread deployment of anomaly detection technology in surveillance systems. The rapid development of artificial intelligence has driven advancements in video target recognition technology. Modern video human behavior recognition often uses models such as dual-stream, C3D, CNN-LSTM, and SlowFast dual-branch deep learning networks to provide pedestrian action recognition results. This invention utilizes relatively successful human target detection and tracking methods to acquire spatiotemporal data of pedestrian behavior, quantifies human behavior values ​​through statistical analysis, employs a comparative analysis strategy, and performs joint analysis of frame image spatial data and multi-frame temporal data. Anomalies are determined based on the numerical statistical characteristics of the sliding window neighborhood and clustering methods. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a system and method for security video behavior recognition, querying, and anomaly detection, providing an automated query approach and anomaly identification means for real-time human behavior recognition and anomaly detection in surveillance videos under multiple different application scenarios.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] On the one hand, the present invention provides a system for security video behavior recognition, query, and anomaly detection, including a digital network camera, a power supply, and a computer processor;

[0007] The digital network camera is used to capture real-time video data and input it into the SlowFast deep learning network in the computer processor; the power supply is responsible for powering the digital network camera.

[0008] The computer processor is used to receive RTSP video streams and convert them into video streams that can be processed by the SlowFast network. The SlowFast deep learning network is deployed in the computer processor to extract and analyze features from the identified action data, and to complete the functions of action statistics, abnormal behavior detection, search and display of the action of interest, as well as to control the database and display and use the web-based operating system.

[0009] The software units in the computer processor specifically include a video processing unit, a motion recognition and analysis unit, a data analysis unit, and an application system module;

[0010] The video processing unit includes a multi-threaded processing module and a preprocessing module. The multi-threaded processing module is used to reduce the system's computational pressure by separating video recognition from data processing, each forming a separate thread. The preprocessing module is used to convert the RTSP video stream into a video stream that can be recognized by the Python programming language and to preprocess the video, including processing the length, width, and resolution of image frames.

[0011] The action recognition and analysis unit includes a SlowFast network training module, a SlowFast model deployment and verification module, and an improved SlowFast input source module;

[0012] The SlowFast network training module is used to train the SlowFast deep learning network to obtain the SlowFast model.

[0013] The SlowFast model deployment and verification module is used to deploy SlowFast models to computer systems and verify that the model functions are compatible with the computer system.

[0014] The improved SlowFast input source module is used to improve the input source to a real-time camera based on the SlowFast computer system, and to continuously process the real-time video stream using the SlowFast model.

[0015] The data analysis unit includes a data extraction and calculation module and a statistical analysis module; the data extraction and calculation module includes a data extraction module and an action value and neighborhood mean calculation module; the statistical analysis module includes a nested sliding window module and a principal component analysis and cluster analysis module.

[0016] The data extraction module is used to continuously identify and extract human behavior in videos using the SlowFast model, based on the SlowFast computer system.

[0017] The motion value and neighborhood mean calculation module is used to analyze abnormal segments in the video and compare motion values ​​and motion mean using a sliding window.

[0018] The nested sliding window module is used to group data and compare it with the overall video frames using frames of a certain window size;

[0019] The principal component analysis and cluster analysis module is used to perform cluster analysis on the data within the sliding window, and to determine the number of abnormal actions by using the action value and the action mean.

[0020] The application system module includes a database storage module and a browser / server (B / S) system architecture module;

[0021] The database storage module is used to statistically analyze the behavior recognition data, facilitating its analysis and application in the system modules.

[0022] The B / S system architecture module is used to apply the entire model to the webpage. The database used is SQL Server, which serves as the data exchange method between the model and the webpage.

[0023] Furthermore, the SlowFast deep learning network includes two convolutional branches, namely the Slow branch and the Fast branch. Both the Slow and Fast channels use a 3D convolutional RestNet model. At the end of each branch, SlowFast performs global average pooling, and then a concat operation is performed to combine the features of the two channels for class prediction.

[0024] Furthermore, the multithreaded process in the multithreaded processing module includes:

[0025] Thread 1 is used to receive RTSP video streams and is responsible for converting videos in the buffer that have not been converted in time;

[0026] Thread 2 is responsible for preprocessing the converted video stream before feeding it into the SlowFast deep learning network;

[0027] Thread 3 is the real-time video analysis module of the SlowFast deep learning network, which is responsible for processing the input video source and outputting various types of analysis data for later use.

[0028] On the other hand, the present invention also provides a method for security video behavior recognition, query, and anomaly detection. Through the aforementioned system implementation, a SlowFast neural network is used to recognize human actions and perform secondary analysis on the recognized data. The action data is digitized, and a sliding window is used for neighborhood mean comparison. The input device is a digital network camera, and the action detection model is SlowFast. The extracted and analyzed data includes action name, action similarity, human anchor box data matrix, number of targets, time, and frame number. Specifically, the method includes the following steps:

[0029] Step 1: The SlowFast deep learning dual-branch model framework is adopted, with the input source being data collected by digital network cameras or local video data, to identify pedestrian actions and behaviors in remote monitoring videos in real time.

[0030] Step 2: Use the Fast R-CNN algorithm to perform simple target recognition, and feed the recognized target anchor box data into the SlowFast network for subsequent analysis;

[0031] Step 3: Use multithreading. One thread preprocesses the image frames, one thread performs the task of converting video into image frames, and one thread is responsible for inputting the image frames into the SlowFast network model.

[0032] Step 4: Collect and analyze the number of targets within the image frame, the target ID (UID), the action name of each target, the similarity of each action, the action anchor box data matrix of each target, and the dynamic time data separately.

[0033] Step 5: Analyze static and dynamic actions separately. Static anomaly detection is intra-frame action anomaly detection, and dynamic anomaly detection is inter-frame abnormal action detection.

[0034] Step 6: Inter-frame anomaly detection incorporates classification concepts and designs a method from the outside in and then from the inside out. By observing data fluctuations, it accurately locates segments that are abnormal compared to other frames. It then compares the current abnormal segment with the data of consecutive frames. If the difference is too large, it is considered that abnormal behavior has occurred.

[0035] Step 7: For the intra-frame data analysis method, calculate the average action value under different UIDs. UIDs that differ from the average by more than 0.3 are considered abnormal actions. Calculate the actions of all people within a frame and then compare them using Step 6.

[0036] Step 8: Implement video frame capture, motion statistics, anomaly detection, and special attention motion display functions through the web interface.

[0037] Furthermore, the specific method for step 6 is as follows:

[0038] Step 6.1: The behaviors identified by the SlowFast network are denoted as {ac1, ac2, ac3, ..., ac...} N The current number of actions is Num_action. All data will be unified into related actions for a certain action and the data will be digitized.

[0039] Step 6.2: Statistical analysis of action correlation vectors (ac) i ac j B), ac i ,ac j Let be any two actions ac that can be represented by a ratio to each other, and B be a certain baseline probability threshold. If action ac can always be found within the baseline probability threshold B, then... j Then ac j These are elements within the relevant action vector, expressed by the formula ac. j =ac i +(1-percent) indicates that the percent value is the frequency of the grouped actions (ac) i ac j The proportion of the probability of an action group to the total frequency of all actions is expressed by the following formula:

[0040]

[0041] Step 6.3: Calculate the mean action value AVG_action using the following formula:

[0042]

[0043] Clustering is performed using the mean action value and the distribution characteristics of action values ​​within consecutive frames to identify abnormal actions;

[0044] Step 6.4: Use a sliding window to complete the analysis from the outside in. The window size is K frames. Set a threshold TH to determine whether the current segment is an abnormal segment. Statistically count the motion values ​​within K frames and calculate the average motion value. Compare it with the overall average motion value to obtain abnormal segments.

[0045] Step 6.5: Group the action data within the abnormal segment, with each group consisting of m actions, denoted as set M. n The baseline action for this group is defined using the mode and similarity. A sliding window method is used for neighborhood mean comparison, where the neighborhood includes inter-frame neighborhood data Dm_frames. Data points; calculate the action mean of Dm_frames and form a pair with the current action value, i.e., perform coordinate analysis, perform cluster analysis, with the X-axis representing the action value and the Y-axis representing the action mean, and the actions on non-principal components... iThis refers to abnormal actions. The frequency of dynamic abnormal actions under different windows is obtained. The Z-axis is added to statistically analyze the frequency of abnormal actions and identify abnormal actions under one or even multiple consecutive windows.

[0046] The beneficial effects of adopting the above technical solution are as follows: The system and method for security video behavior recognition, query, and anomaly detection provided by this invention utilizes relatively successful methods such as human target detection and tracking to obtain spatiotemporal data of pedestrian behavior. It quantifies human behavior action values ​​through statistical analysis, employs a comparative analysis strategy, and performs joint analysis of frame image spatial data and multi-frame temporal data. Abnormal behavior is determined based on the statistical characteristics of sliding window neighborhood values ​​and clustering methods. Using the SlowFast deep learning dual-branch model framework, the input source is data collected from remote internet cameras or local video data, enabling real-time recognition of pedestrian actions in remote monitoring videos. By analyzing the internal relationships between various actions in the video scene, incorporating classification ideas, and introducing an outside-to-inside and inside-to-outside method, the method more accurately locates segments of abnormal behavior, improving detection accuracy. The method of this invention overcomes the scene dependence of existing anomaly detection methods, has a wide range of applications, possesses online real-time detection capabilities, and exhibits excellent results. Attached Figure Description

[0047] Figure 1 This is a system overall structure diagram of security video behavior recognition, query, and anomaly detection provided in an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of the system functional modules provided in an embodiment of the present invention;

[0049] Figure 3 A lateral link diagram of SlowFast provided for embodiments of the present invention;

[0050] Figure 4 This is a schematic diagram illustrating the reduction of false recognition rate for action groups provided in an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of the distribution of action values ​​and action mean values ​​provided in an embodiment of the present invention;

[0052] Figure 6 This is a flowchart of the process for extracting abnormal segments using a sliding window from the outside in, provided in an embodiment of the present invention.

[0053] Figure 7 A flowchart for analyzing abnormal actions of a sliding window from the inside out, provided for embodiments of the present invention;

[0054] Figure 8 This is a schematic diagram of a two-dimensional statistical histogram (action value, action mean) provided in an embodiment of the present invention. Detailed Implementation

[0055] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0056] Example 1

[0057] like Figure 1 As shown, this embodiment provides a system for security video behavior recognition, querying, and anomaly detection, including a network camera, a power supply, and a computer processor. Under the Browser / Server software development architecture, a remote monitoring video operation and maintenance management intelligent system is developed.

[0058] SlowFast networks are lightweight neural networks that use a human-like analytical approach. Fast channels use multiple frames with fewer channels, while slow channels use fewer frames with more channels. Lateral connections are then used, such as... Figure 3 As shown, learning motion information from video end-to-end is computationally efficient. Employing the existing SlowFast deep learning dual-branch model framework, with input sources including data from remote internet cameras or local video data, it can identify pedestrian actions in remote surveillance videos in real time.

[0059] By analyzing the intermolecular relationships between various actions in a video scene, incorporating classification concepts, and introducing an outside-in and inside-out approach, abnormal behavior segments can be located more accurately, improving detection accuracy. For special actions, a proximity detection method is used.

[0060] The specific module structure of the system is as follows: Figure 2 As shown, the system includes hardware modules and software units. The hardware modules include a digital network camera and a power supply, while the software unit is stored in the computer processor. The digital network camera is a Hikvision network camera used to capture real-time video data and input it to the SlowFast network in the computer processor. The power supply is responsible for providing power to the digital network camera.

[0061] The computer processor receives RTSP video streams and converts them into video streams that can be processed by the SlowFast network. The SlowFast deep learning network is deployed in the computer processor to extract and analyze features from the identified action data, and complete the functions of action statistics, abnormal behavior detection, search and display of the action of interest, as well as the manipulation of the database and the display and use of the web-based operating system.

[0062] The software units in a computer processor specifically include a video processing unit, a motion recognition and analysis unit, a data analysis unit, and an application system module.

[0063] The video processing unit includes a multi-threaded processing module and a preprocessing module. The multi-threaded processing module reduces the system's computational load by separating video recognition from data processing, each forming its own thread. The preprocessing module converts the RTSP video stream into a video stream recognizable by the Python programming language, performing preprocessing on the video, including adjusting the width, height, and resolution of image frames.

[0064] The action recognition and analysis unit comprises a SlowFast network training module, a SlowFast model deployment and validation module, and an improved SlowFast input source module. The action recognition and analysis unit is a SlowFast dual-branch network, using a model trained on an initial dataset to achieve real-time human behavior recognition. It can handle remotely acquired real-time video input sources as well as local video data. The SlowFast model, using data from remote monitoring cameras as input, converts RTSP video streams into video streams recognizable by the Python programming language. It preprocesses the video, including adjusting the length, width, and resolution of image frames to save unnecessary computational resources.

[0065] The SlowFast network training module is used to train the SlowFast deep learning network to obtain the SlowFast model. The SlowFast deep learning network consists of two convolutional branches: the Slow branch and the Fast branch. The Slow branch learns spatial semantic information using fewer frames and a larger number of channels. The Fast branch learns motion information using a larger number of frames and a smaller number of channels. The computational cost is proportional to the square of the number of channels. The Fast branch, due to its smaller number of channels, is more lightweight, accounting for only 20% of the overall computation. The Slow branch uses a larger stride τ to capture video frames, typically set to 16. For a 30fps video, this means approximately 2 frames can be captured per second, i.e., T = 2. The Slow branch has D channels, while the Fast branch uses a smaller stride τ / α to capture video frames, where α is typically set to 8. Therefore, for a 30fps video, 15 frames (αT) can be captured per second. The Fast branch maintains its lightweight nature by using a smaller number of channels (βD), with β typically set to 1 / 8 of the channel count. Both the Slow and Fast channels use a 3D convolutional RESTNet model. At the end of each branch, SlowFast performs global average pooling, followed by a concat operation to combine features from both channels for class prediction. Features extracted from the Slow and Fast branches need to be fused. SlowFast uses lateral connections to feed features from the Fast branch into the Slow branch for mixing. However, the feature dimensions of the two branches are inconsistent (Fast branch: {αT, S2, βC}, while Slow branch: {T, S2, αβC}), therefore SlowFast requires data transformation of the Fast branch results. Time-strided convolution: A 5×1×1 3D convolution is used, with 2βC output channels and a stride of α. The SlowFast network is used to identify pedestrian actions and provide action information. Simultaneously, the Fast R-CNN algorithm for target recognition is introduced into SlowFast for target localization and tracking, and the resulting target bounding boxes are fed into the SlowFast network for subsequent time-series data analysis.

[0066] The SlowFast model deployment and verification module is used to deploy SlowFast models to computer systems and verify that the model functions are compatible with the computer system.

[0067] The improved SlowFast input source module is used to improve the input source to a real-time camera based on the SlowFast computer system, and to continuously process the real-time video stream using the SlowFast model.

[0068] The data analysis unit includes a data extraction and calculation module and a statistical analysis module. The data extraction and calculation module includes a data extraction module and a module for calculating action values ​​and neighborhood means. The statistical analysis module includes a nested sliding window module and a principal component analysis and cluster analysis module.

[0069] The data extraction module is used to continuously identify and extract human behavior in videos using the SlowFast model, which is deployed on the computer system SlowFast.

[0070] The motion value and neighborhood mean calculation module is used to analyze abnormal segments in the video and compare motion values ​​and motion mean using a sliding window.

[0071] The nested sliding window module is used to group data and compare it with the overall video frames using frames of a certain window size.

[0072] The principal component analysis and cluster analysis module is used to perform cluster analysis on the data within the sliding window, and to determine the number of abnormal actions by using the action value and the action mean.

[0073] The application system module includes a database storage module and a browser / server (B / S) system architecture module.

[0074] The database storage module is used to statistically analyze the behavior recognition data, facilitating analysis and use by B / S systems.

[0075] The Browser / Server system architecture module is used to apply the entire model to web pages. The database used is SQL Server, which serves as the data exchange method between the model and the web pages.

[0076] Example 2

[0077] This embodiment provides a method for security video behavior recognition, query, and anomaly detection. It is implemented through the system of Embodiment 1. It uses a SlowFast neural network to recognize human actions and performs secondary analysis on the recognized data. The action data is digitized and a sliding window is used to compare the mean of neighboring areas. The input device is a digital network camera, and the action detection model is SlowFast. The extracted and analyzed data includes action name, action similarity, human anchor box data matrix, number of targets, time, and number of frames.

[0078] Based on a large amount of motion data, it is possible to remotely view pedestrian movements and comprehensively analyze abnormal movements in videos. According to the data distribution characteristics, it is divided into static anomaly detection, which is based on intra-frame data motion anomaly detection, and dynamic anomaly detection, which is based on inter-frame temporal data anomaly motion detection in video streams.

[0079] In multi-frame temporal data analysis, the results of the SlowFast neural network are used to compare and analyze the actions within different frames of the video, including the number of targets within the frame, the target ID (UID), the action name of each target, the similarity of each action, the action anchor box data matrix of each target within the frame, and the dynamic real-time time. Based on the occurrence and duration of detected abnormal actions, the time period of the abnormal behavior can be located, and an analysis method from the outside in and then from the inside out can be performed.

[0080] The SlowFast network can identify 79 behaviors, such as {sit, stand, drink, sing, ...}, which can be denoted as {ac1, ac2, ac3, ..., ac 79 Since the SlowFast model provides discrete text data, it is not conducive to direct analysis and detection of abnormal behavior. Extensive video analysis has revealed correlations between human action variables; that is, certain actions often occur sequentially, at intervals, or even simultaneously with other actions. For example, seeing someone standing still on a diving board suggests they will soon be diving. Therefore, this embodiment quantifies pedestrian actions by defining action values ​​and related action groups, assigning similar action values ​​to actions within related action groups. For instance, standing often occurs alongside walking, so they are considered related actions. The action value is determined jointly by the probability of related action groups occurring (percent) and a baseline action value. Actions that appear frequently in the video are defined as baseline actions. For example, based on baseline action ac1, a related action group Group_action_crl(ac1,ac2,B) is calculated, where B is defined as the baseline probability threshold. If action ac2 is always found within the baseline probability threshold B, then ac2 is an element in the related action vector, and its action value is set to be similar to ac1; otherwise, it is set to be much smaller. Due to the randomness of the model's action recognition results, some actions may be misidentified or incorrectly grouped actions may be identified. Using the Group_action_crl approach can reduce the misidentification rate to some extent. percent(ac1,ac2) represents the probability of related actions. The more frequently actions ac2 and ac1 appear together, the closer the value of ac2 will be to the value of ac1. This can be expressed by the formula ac2 = ac1 + (1 - percent). The percent value represents the proportion of the frequency of the current grouped action frequency(ac1,ac2) to the total frequency of all action combinations Num_frequency, which can be expressed by the following formula:

[0081]

[0082] Since this calculation requires two actions to be grouped together to form a related data set, when three or more actions appear in a group, they should be broken down into one-to-one corresponding action groups. If n actions occur, it will generate... Action groups. Typically, certain actions rarely occur in groups, such as standing and falling down, drinking and singing; these action groups can be considered unrelated. When it's necessary to reduce data instability caused by misidentification, an effective method is to statistically analyze all action groups and use the Group_action_crl data cleaning method. This can help improve data consistency and quality, ensuring data reliability and reducing the impact of potential misidentification, such as... Figure 4 As shown.

[0083] The background of videos captured by surveillance cameras in fixed locations is often static. For multi-frame time-series data, frames that are normal and appear frequently are usually considered normal; otherwise, abnormal behavior may be detected. An analysis strategy of going from the outside in and then from the inside out is adopted for anomaly detection in multi-frame time-series data. From the outside in, the fluctuations in video frame motion data are observed to identify segments with large fluctuations, precisely locating segments that are abnormal compared to other frames. These abnormal segments are then analyzed using an inside-out approach, comparing the data of the current frame with the data of the longer video stream. If the difference is too large, abnormal behavior is considered to have occurred.

[0084] The anomaly detection for different actions in different videos should be inconsistent. Anomalies should be specific to the current application scenario. This involves utilizing information within consecutive frames in the video and introducing a sliding window approach, comparing consecutive frame segments that slide forward over time with the overall frame. To reduce computational load, this embodiment designs a nested sliding window method for comparing the mean of neighboring frames. Neighborhood data for normal actions often exhibit similarity. By using a nested sliding window to statistically analyze action values ​​in time-series frame data and pedestrian action values ​​with different IDs in the frame space, the mean action value of Num_action actions within the neighborhood of the sliding window is calculated.

[0085]

[0086] A two-dimensional distribution histogram of statistical (action value, action mean) values ​​is shown in the diagram below. Figure 5 As shown, clustering methods can then be used to identify abnormal behavior.

[0087] A sliding window is used to analyze video segments from the outside in. The window size is set to K frames, typically K = 30 * frame rate. The value of K can be adjusted appropriately according to the video length and the required accuracy. A threshold TH = 1.2σ is set, where σ is the standard deviation of the motion values ​​in the video to be analyzed. Figure 6As shown below, the specific method is as follows:

[0088] Step ①: Statistically analyze the action values within the K-frame video segment, calculate the action mean and deviation difference. If the difference exceeds the set threshold TH, the segment is considered a suspected abnormal segment and is marked. Otherwise, continue to slide the window by D frames. Generally, 2 < D < K. Depending on the complexity of the problem and the requirements for solution efficiency, D can be a fixed value or adaptively adjusted for the D-frame length.

[0089] Step ②: Repeat Step ① until all video frames are processed.

[0090] Perform a second window slide on the suspected abnormal video segment. The window size is set to L frames. Generally, L = 30. As Figure 7 shown below, the process is as follows:

[0091] Step S1: Group the action data within the suspected abnormal segment to reduce the amount of data for analysis. Every m actions form a group, denoted as set M n , generally m = 3. If there is exactly one group with the highest frequency of occurrence within the set, that is, the mode of this group of data (the mode is the data that appears most frequently in a set of data), denoted as mode, select the mode of M n as the reference action for the current group; if there is no mode in set M n , select an action with a value close to the action as the action reference for this group of actions; if there are two or more modes in set M n , select the mode closest to the action mean of set M n as the reference action for this group;

[0092] Step S2: Use a sliding window to statistically analyze the neighborhood mean of actions. There are L data under this window, and the neighborhood includes multi-frame neighborhood data Dm_frames, with data. Calculate the binary tuple formed by the neighborhood action mean data of Dm_frames and the action value of the current window, (action value, action mean), and statistically analyze its two-dimensional distribution histogram. Perform clustering analysis on (X, Y) in the plane coordinate system. As Figure 5 shown, the distribution data on the diagonal is generally normal behavior. Calculate the principal component PCA value and then perform clustering analysis. The action ac i off the diagonal is the abnormal action. After the analysis is completed, slide forward by one group of action groups;

[0093] Step S3: Repeat step S2 until the video ends. When the video ends, a certain number of cluster analysis graphs will be generated, where the number of graphs is proportional to the video length. The purpose of these analysis graphs is to visualize the data to better understand the occurrence of anomalous actions. The sliding window size affects the amount of data and the probability of anomalous actions occurring. In these cluster analysis graphs, the location and number of anomalous data points appearing within them are recorded and counted. To more clearly represent the distribution of anomalous actions, three-dimensional images are used to visualize these statistics, such as... Figure 8 As shown, this method allows for the identification of patterns and distributions of unusual movements in video data, providing a better understanding of their occurrence and location. This facilitates further analysis and the implementation of necessary measures to address potential problems.

[0094] For intra-frame data analysis, the average value of the action value under different UIDs is calculated and compared. Actions of UIDs that differ significantly from the average value are considered abnormal actions.

[0095] SQL Server databases are responsible for data storage and usage functions. They work in conjunction with C# and Python languages ​​to complete data interaction and web page display.

[0096] The web interface should include modules for video display, frame capture, motion statistics, anomaly detection, and special attention action retrieval. The video module enables real-time monitoring of the data stream, the frame capture module captures abnormal image frames, the statistics module includes motion statistics, basic motion analysis such as mode and action confidence, as well as the number of detected frames and time, etc., and the anomaly detection module is responsible for playing back all the analyzed data and highlighting abnormal behaviors in red.

[0097] The method in this embodiment specifically includes the following steps:

[0098] Step 1: The SlowFast deep learning dual-branch model framework is adopted, with the input source being data collected by digital network cameras or local video data, to identify pedestrian actions and behaviors in remote monitoring videos in real time.

[0099] Step 2: Use the Fast R-CNN algorithm to perform simple target recognition, and feed the recognized target anchor box data into the SlowFast network for subsequent analysis;

[0100] Step 3: Use multithreading. One thread preprocesses the image frames, one thread performs the task of converting video into image frames, and one thread is responsible for inputting the image frames into the SlowFast network model.

[0101] Step 4: Collect and analyze the number of targets within the image frame, the target ID (UID), the action name of each target, the similarity of each action, the action anchor box data matrix of each target, and the dynamic time data separately.

[0102] Step 5: Analyze static and dynamic actions separately. Static anomaly detection is intra-frame action anomaly detection, and dynamic anomaly detection is inter-frame abnormal action detection.

[0103] Step 6: Inter-frame anomaly detection incorporates classification concepts and designs an outside-in and inside-out approach. By observing data fluctuations, it accurately locates segments that are abnormal compared to other frames. It then compares the current abnormal segment with the data from consecutive frames; if the difference is too large, it is considered an abnormal behavior. The specific method is as follows:

[0104] Step 6.1: The SlowFast network identifies 79 behaviors, such as {sit, stand, drink, sing, ...}, denoted as {ac1, ac2, ac3, ..., ac N The current number of actions is Num_action. All data will be unified into related actions for a certain action and the data will be digitized.

[0105] Step 6.2: Statistical analysis of action correlation vectors (ac) i ac j B), ac i ,ac j Let be any two actions ac that can be represented by a ratio to each other, and B be a certain baseline probability threshold. If action ac can always be found within the baseline probability threshold B, then... j Then ac j These are elements within the relevant action vector, expressed by the formula ac. j =ac i +(1-percent) indicates that the percent value is the frequency of the grouped actions (ac) i ac j The proportion of the probability of an action group to the total frequency of all actions is expressed by the following formula:

[0106]

[0107] Step 6.3: Calculate the mean action value AVG_action using the following formula:

[0108]

[0109] Clustering is performed using the mean action value and the distribution characteristics of action values ​​within consecutive frames to identify abnormal actions;

[0110] Step 6.4: Use a sliding window to complete the analysis from the outside in. The window size is K frames. Set a threshold TH to determine whether the current segment is an abnormal segment. Statistically count the motion values ​​within K frames and calculate the average motion value. Compare it with the overall average motion value to obtain abnormal segments.

[0111] Step 6.5: Group the action data within the abnormal segment, with each group consisting of m actions, denoted as set M. n The baseline action for this group is defined using the mode and similarity. A sliding window method is used for neighborhood mean comparison, where the neighborhood includes inter-frame neighborhood data Dm_frames. Data points; calculate the action mean of Dm_frames and form a pair with the current action value, i.e., perform coordinate analysis, perform cluster analysis, with the X-axis representing the action value and the Y-axis representing the action mean, and the actions on non-principal components... i This refers to abnormal actions. The frequency of dynamic abnormal actions under different windows is obtained. The Z-axis is added to statistically analyze the frequency of abnormal actions and identify abnormal actions under one or even multiple consecutive windows.

[0112] Step 7: For the intra-frame data analysis method, calculate the average value of the action value under different UIDs. UIDs that differ from the average value by more than 0.3 are considered abnormal actions. Calculate the actions of all people within a frame and then compare them using the numerical relationship from Step 6.

[0113] Step 8: Implement video frame capture, motion statistics, anomaly detection, and special attention motion display functions through the web interface.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A system for security video behavior recognition, querying, and anomaly detection, characterized in that: Includes digital network cameras, power supplies, and computer processors; The digital network camera is used to capture real-time video data and input it into the SlowFast deep learning network in the computer processor; the power supply is responsible for powering the digital network camera. The computer processor is used to receive RTSP video streams and convert them into video streams that can be processed by the SlowFast network. The SlowFast network is deployed in the computer processor to extract and analyze the features of the identified action data, and to complete the functions of action statistics, abnormal behavior detection, search and display of the action of interest, as well as to control the database and display and use the web-based operating system. The software units in the computer processor specifically include a video processing unit, a motion recognition and analysis unit, a data analysis unit, and an application system module; The video processing unit includes a multi-threaded processing module and a preprocessing module; the multi-threaded processing module is used to reduce the system's computational pressure by separating video recognition and data processing, each forming a separate thread. The preprocessing module is used to convert RTSP video streams into video streams that can be recognized by the Python programming language, and to preprocess the video, including processing the length, width, and resolution of image frames; The action recognition and analysis unit includes a SlowFast network training module, a SlowFast model deployment and verification module, and an improved SlowFast input source module; The SlowFast network training module is used to train the SlowFast deep learning network to obtain the SlowFast model. The SlowFast model deployment and verification module is used to deploy SlowFast models to computer systems and verify that the model functions are compatible with the computer system. The improved SlowFast input source module is used to improve the input source to a real-time camera based on the SlowFast computer system, and to continuously process the real-time video stream using the SlowFast model. The data analysis unit includes a data extraction and calculation module and a statistical analysis module; the data extraction and calculation module includes a data extraction module and an action value and neighborhood mean calculation module; the statistical analysis module includes a nested sliding window module and a principal component analysis and cluster analysis module. The data extraction module is used to continuously identify and extract human behavior in videos using the SlowFast model, based on the SlowFast computer system. The motion value and neighborhood mean calculation module is used to analyze abnormal segments in the video and compare motion values ​​and motion mean using a sliding window. The nested sliding window module is used to group data and compare it with the overall video frames using frames of a certain window size; The principal component analysis and cluster analysis module is used to perform cluster analysis on the data within the sliding window, and to determine the number of abnormal actions by using the action value and the action mean. The application system module includes a database storage module and a Browser / Server system architecture module; The database storage module is used to statistically analyze behavior recognition data, facilitating its use by Browser / Server systems. The Browser / Server system architecture module is used to apply the entire model to web pages. The database used is SQL Server, which serves as the data exchange method between the model and the web pages.

2. The system for security video behavior recognition, querying, and anomaly detection according to claim 1, characterized in that: The SlowFast deep learning network includes two convolutional branches: the Slow branch and the Fast branch. Both the Slow and Fast channels use a 3D convolutional RestNet model. At the end of each branch, SlowFast performs global average pooling, and then a concat operation combines the features of both channels to predict the class.

3. The system for security video behavior recognition, querying, and anomaly detection according to claim 1, characterized in that: The multi-threaded process in the multi-threaded processing module includes: Thread 1 is used to receive RTSP video streams and is responsible for converting videos in the buffer that have not been converted in time; Thread 2 is responsible for preprocessing the converted video stream before feeding it into the SlowFast deep learning network; Thread 3 is the real-time video analysis module of the SlowFast deep learning network, which is responsible for processing the input video source and outputting various types of analysis data for later use.

4. A method for security video behavior recognition, querying, and anomaly detection, characterized in that: The security video behavior recognition, query, and anomaly detection system described in claim 1 utilizes a SlowFast neural network to recognize human actions and performs secondary analysis on the recognized data. The action data is digitized, and a sliding window is used for neighborhood mean comparison. The input device is a digital network camera, and the action detection model is SlowFast. The extracted and analyzed data includes action name, action similarity, human anchor frame data matrix, number of targets, time, and frame count. Specifically, the system includes the following steps: Step 1: The SlowFast deep learning dual-branch model framework is adopted, with the input source being data collected by digital network cameras or local video data, to identify pedestrian actions and behaviors in remote monitoring videos in real time. Step 2: Use the Fast R-CNN algorithm to perform simple target recognition, and feed the recognized target anchor box data into the SlowFast network for subsequent analysis; Step 3: Use multithreading. One thread preprocesses the image frames, one thread performs the task of converting video into image frames, and one thread is responsible for inputting the image frames into the SlowFast network model. Step 4: Collect and analyze the data of the number of targets within the image frame, the target ID (UID), the action name of each target, the similarity of each action, the action anchor box data matrix of each target, and the dynamic time data separately. Step 5: Analyze static and dynamic actions separately. Static anomaly detection is intra-frame action anomaly detection, and dynamic anomaly detection is inter-frame abnormal action detection. Step 6: Inter-frame anomaly detection incorporates classification concepts and designs a method from the outside in and then from the inside out. By observing data fluctuations, it accurately locates segments that are abnormal compared to other frames. It then compares the current abnormal segment with the data of consecutive frames. If the difference is too large, it is considered that abnormal behavior has occurred. Step 7: For the intra-frame data analysis method, calculate the average action value under different UIDs. UIDs that differ from the average by more than 0.3 are considered abnormal actions. Calculate the actions of all people within a frame and then compare them using Step 6. Step 8: Implement video frame capture, motion statistics, anomaly detection, and special attention motion display functions through the web interface.

5. The method for security video behavior recognition, querying, and anomaly detection according to claim 4, characterized in that: The specific method for step 6 is as follows: Step 6.1: The behaviors identified by the SlowFast network are denoted as {ac1, ac2, ac3, ..., ac...} N The current number of actions is Num_action. All data will be unified into related actions for a certain action and the data will be digitized. Step 6.2: Statistical analysis of action correlation vectors (ac) i ac j B), ac i ,ac j Let be any two actions ac that can be represented by a ratio to each other, and B be a certain baseline probability threshold. If action ac can always be found within the baseline probability threshold B, then... j Then ac j These are elements within the relevant action vector, expressed by the formula ac. j =ac i +(1-percent) indicates that the percent value is the frequency of the grouped actions (ac) i ac j The proportion of the probability of an action group to the total frequency of all actions is expressed by the following formula: Step 6.3: Calculate the mean action value AVG_action using the following formula: Clustering is performed using the mean action value and the distribution characteristics of action values ​​within consecutive frames to identify abnormal actions; Step 6.4: Use a sliding window to complete the analysis from the outside in. The window size is K frames. Set a threshold TH to determine whether the current segment is an abnormal segment. Statistically count the motion values ​​within K frames and calculate the average motion value. Compare it with the overall average motion value to obtain abnormal segments. Step 6.5: Group the action data within the abnormal segment, with each group consisting of m actions, denoted as set M. n The baseline action for this group is defined using the mode and similarity. A sliding window method is used for neighborhood mean comparison, where the neighborhood includes inter-frame neighborhood data Dm_frames. Data points; calculate the action mean of Dm_frames and form a pair with the current action value, i.e., perform coordinate analysis, perform cluster analysis, with the X-axis representing the action value and the Y-axis representing the action mean, and the actions on non-principal components... i This refers to abnormal actions. The frequency of dynamic abnormal actions under different windows is obtained. The Z-axis is added to statistically analyze the frequency of abnormal actions and identify abnormal actions under one or even multiple consecutive windows.

Images