Video-based abnormal behavior detection method, device and electronic equipment

Abstract

This application belongs to the field of video understanding and action detection technology, and specifically discloses a method, device and electronic device for abnormal behavior detection in video. The method includes: acquiring a target video; determining abnormal behavior segments of the target video based on a temporal action detection algorithm and a spatiotemporal action detection algorithm, according to the priority of group abnormal behavior detection being greater than that of individual abnormal behavior detection. This method can simultaneously perform group abnormal behavior detection and individual abnormal behavior detection, thereby improving the accuracy of abnormal behavior detection.

Description

Technical Field

[0001] This application belongs to the field of video understanding and action detection technology, and more specifically, relates to a method, apparatus and electronic device for detecting abnormal behavior in videos. Background Technology

[0002] Currently, video understanding tasks for detecting abnormal behaviors in public safety involve identifying fights, crowd congestion, abnormal crowd movement, evacuation and escape, as well as abnormal behaviors such as falling, chasing, tailing, and loitering in video footage. Furthermore, group behaviors often include individual behaviors, such as falling or chasing in a group fight.

[0003] Currently used abnormal behavior detection methods generally only target specific group behaviors or individual behaviors. In terms of abnormal behavior detection algorithms, the difference between individual and group abnormal behaviors is that individual behaviors often involve a specific person engaging in abnormal activities, requiring the location of the abnormal person in the video footage, identification of the behavior category, and determination of the occurrence and end times of the abnormal activity. Group behaviors, on the other hand, usually involve multiple people interacting for a certain period of time, but it is not necessary to locate specific individuals; it is only necessary to identify the group behavior category, the occurrence and end times of the abnormal activity. Therefore, existing technologies cannot simultaneously detect individual and multiple abnormal behaviors in a video. If the detection process cannot distinguish between the two types of behaviors, it is easy to cause confusion in behavior types and occurrence times, leading to false alarms or false negatives and reducing the accuracy of abnormal behavior detection. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this application is to provide a method, apparatus and electronic device for detecting abnormal behavior in videos, which aims to solve the problem of low accuracy of abnormal behavior detection caused by the inability of the prior art to simultaneously detect abnormal behavior of single and multiple people in videos.

[0005] To achieve the above objectives, in a first aspect, this application provides a method for detecting abnormal behavior in videos, comprising:

[0006] Acquire the target video;

[0007] Based on temporal action detection algorithms and spatiotemporal action detection algorithms, abnormal behavior segments of the target video are determined according to the priority of group abnormal behavior detection being greater than individual abnormal behavior detection.

[0008] This application utilizes both temporal action detection and spatiotemporal action detection algorithms to simultaneously detect abnormal group behavior and abnormal individual behavior. It can comprehensively apply methods such as spatiotemporal feature fusion and temporal context information association to determine the location, behavior category, and behavior boundary of abnormal individuals. This solves the problem of false positives and false negatives caused by the inclusion relationship between abnormal group behavior and abnormal individual behavior during the abnormal behavior detection process. The detection is robust and improves the ability to understand multiple action categories contained in video data, thereby improving the accuracy of abnormal behavior detection.

[0009] According to the present invention, an abnormal behavior detection method for video is provided, wherein the abnormal behavior segments of the target video are determined based on a temporal action detection algorithm and a spatiotemporal action detection algorithm, with a priority of group abnormal behavior detection being greater than that of individual abnormal behavior detection, including:

[0010] The time-series action detection algorithm is used to detect abnormal group behavior in the target video, and the abnormal behavior categories and start and end times of all abnormal group behavior in the target video are obtained.

[0011] Based on the start and end times of the abnormal group behavior, the abnormal group behavior segments in the target video are cropped, and the remaining video segments are spliced ​​together.

[0012] The spatiotemporal action detection algorithm is used to detect abnormal single-person behaviors in the stitched video, and the abnormal behavior categories and start and end times of all abnormal single-person behaviors in the video are obtained.

[0013] Based on the start and end times of an individual's abnormal behavior, clips of that behavior are trimmed from the video.

[0014] This application prioritizes group abnormal behavior detection over individual abnormal behavior detection. It first uses a temporal action detection algorithm to obtain group abnormal behavior segments in the target video, then crops and splices the remaining video segments. Finally, it uses a spatiotemporal action detection algorithm to obtain individual abnormal behavior segments in the target video. It can comprehensively apply spatiotemporal feature fusion, temporal context information association, and coarse-fine granular feature fusion prediction to determine the location of abnormal persons, behavior categories, and behavior boundaries. It has strong adaptability to multiple scenarios and high detection efficiency.

[0015] According to the present invention, an abnormal behavior detection method for videos includes a training process for a temporal action detection algorithm comprising:

[0016] Create a dataset of abnormal group behavior;

[0017] The loss function value of the temporal action detection algorithm network is calculated using the cross-entropy loss function on the training data in the dataset;

[0018] The backpropagation algorithm is used to update and optimize all parameters of the temporal action detection algorithm network. Iterative training is performed repeatedly until the loss function reaches its minimum value and the recognition accuracy reaches its optimal value, thus obtaining the trained temporal action detection algorithm.

[0019] According to the present invention, an abnormal behavior detection method for videos is provided, wherein the training process of the spatiotemporal action detection algorithm includes:

[0020] Create a dataset of abnormal behavior in a single individual;

[0021] The cross-entropy loss function is used to calculate the loss function value of the spatiotemporal action detection algorithm network on the training data in the dataset;

[0022] The backpropagation algorithm is used to update and optimize all parameters of the spatiotemporal action detection algorithm network. Iterative training is performed repeatedly until the loss function reaches its minimum value and the recognition accuracy reaches its optimal value, thus obtaining the trained spatiotemporal action detection algorithm.

[0023] This application collects abnormal behavior datasets of groups and individuals respectively, which are used to train temporal action detection algorithms and spatiotemporal action detection algorithms to improve the detection accuracy of the algorithms.

[0024] According to the present invention, an abnormal behavior detection method for video is provided, which further includes, after acquiring the target video:

[0025] The target video is preprocessed.

[0026] According to the present invention, an abnormal behavior detection method for video is provided, wherein the preprocessing of the target video includes:

[0027] The target video is subjected to still frame clipping and frame sampling.

[0028] Convert the video to npy format video data.

[0029] This application reduces redundant video information and compresses video data by cropping still frames and sampling frames in the target video, thereby improving the efficiency of behavior detection. Converting the video to .npy format can improve the efficiency of format conversion.

[0030] Secondly, this application provides an abnormal behavior detection device for videos, comprising:

[0031] The acquisition module is used to acquire the target video;

[0032] The behavior detection module is used to determine the abnormal behavior segments of the target video based on the temporal action detection algorithm and the spatiotemporal action detection algorithm, according to the priority of group abnormal behavior detection being greater than individual abnormal behavior detection.

[0033] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the abnormal behavior detection method for video described in the first aspect or any possible implementation thereof.

[0034] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed on a processor, causes the processor to perform the abnormal behavior detection method for video described in the first aspect or any possible implementation thereof.

[0035] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to execute the abnormal behavior detection method for video described in the first aspect or any possible implementation of the first aspect.

[0036] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0037] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0038] By using temporal action detection algorithms and spatiotemporal action detection algorithms, both group abnormal behavior detection and individual abnormal behavior detection can be completed simultaneously. This approach comprehensively applies methods such as spatiotemporal feature fusion and temporal context information association to determine the location, behavior category, and behavior boundaries of abnormal individuals. It addresses the issue of false positives and false negatives caused by the inclusion relationship between group and individual abnormal behaviors during the abnormal behavior detection process. The detection exhibits strong robustness and improves the ability to understand multiple action categories contained in video data, thereby enhancing the accuracy of abnormal behavior detection. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1This is a flowchart illustrating the abnormal behavior detection method for videos provided in the embodiments of this application;

[0041] Figure 2 This is a flowchart illustrating the intelligent analysis of public security surveillance videos provided in an embodiment of this application;

[0042] Figure 3 This is a schematic diagram of the structure of the abnormal behavior detection device for video provided in the embodiments of this application;

[0043] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0046] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0047] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0048] First, let's introduce the following content:

[0049] In recent years, video surveillance systems have been widely used in real-time monitoring, command and dispatch, and investigation and evidence collection. The scale of surveillance video data is huge, but the probability of abnormal public safety behavior is often low. It is very difficult and time-consuming to rely on manual real-time monitoring and video search. Therefore, it is necessary to rely on efficient and robust video intelligent analysis methods to detect abnormal public safety video behavior.

[0050] Currently, video understanding tasks for detecting abnormal behaviors in public safety involve identifying fights, crowd congestion, abnormal crowd movement, evacuation and escape, as well as abnormal behaviors such as falling, chasing, tailing, and loitering in video footage. Furthermore, group behaviors often include individual behaviors; for example, group fights may include falling or chasing. If the detection process cannot distinguish between these two types of behaviors, confusion may arise regarding the behavior type and the time of occurrence, leading to false alarms or missed alarms.

[0051] Current common processing methods generally only target specific group behaviors or individual behaviors, and there is a lack of a robust video understanding method that can simultaneously detect and identify abnormal group and individual behaviors.

[0052] Regarding algorithms for detecting abnormal behavior, the difference between individual and group abnormal behavior lies in the fact that individual behavior often involves a specific person engaging in abnormal activity, requiring the location of the abnormal person in the video footage, identification of the behavior category, and determination of the occurrence and end times of the abnormal activity; while group behavior usually involves multiple people interacting for a certain period of time, but it does not require locating specific individuals, only identifying the group behavior category, the occurrence and end times of the abnormal activity. Therefore, different detection methods are required for the two types of tasks.

[0053] Next, combined Figures 1-2 The abnormal behavior detection method for videos provided in the embodiments of this application is described.

[0054] Figure 1 This is a flowchart illustrating the abnormal behavior detection method for videos provided in this application embodiment, as shown below. Figure 1 As shown, the method includes the following steps:

[0055] Step 100: Obtain the target video;

[0056] Optionally, this application does not limit the method of acquiring the target video or the video format of the target video.

[0057] In one embodiment of this application, video files in a public security surveillance video resource library are loaded locally or remotely for abnormal behavior detection.

[0058] Step 110: Based on the temporal action detection algorithm and the spatiotemporal action detection algorithm, determine the abnormal behavior segments of the target video according to the priority of group abnormal behavior detection being greater than that of individual abnormal behavior detection.

[0059] This application uses temporal action detection algorithms and spatiotemporal action detection algorithms to perform intelligent video analysis on target videos, obtaining the types of abnormal behaviors occurring in the video, the positions of people in the frame, the start and end times of the behavior, and video segments containing abnormal behaviors.

[0060] Optionally, the temporal action detection algorithm aims to detect the occurrence of actions in a video and locate the start and end times of the actions. The spatiotemporal action detection algorithm is an algorithm that simultaneously detects the occurrence of actions in video content and locates the temporal and spatial positions of the actions. This application uses the trained temporal action detection algorithm and spatiotemporal action detection algorithm, and determines the abnormal behavior segments of the target video, including group abnormal behavior segments and individual abnormal behavior segments, according to the priority of group abnormal behavior detection being greater than individual abnormal behavior detection, that is, group abnormal behavior detection is performed first and then individual abnormal behavior detection is performed.

[0061] Optionally, abnormal group behavior can include: fighting, crowd congestion, abnormal crowd movement, evacuation and escape as seen in police surveillance videos, while abnormal individual behavior can include: falling, running, theft, intoxication, loitering and tailing, etc.; the priority of setting abnormal group behavior categories is higher than that of abnormal individual behavior.

[0062] Optionally, after obtaining the abnormal behavior segments, the video name and the intelligent analysis results of the video can be used to create a key video segment index according to the abnormal behavior category and the time of occurrence, and stored in the abnormal behavior database.

[0063] Optionally, the progress of intelligent video analysis, key segments of abnormal behavior, types of abnormal behavior, and personnel locations can be displayed on a display device.

[0064] This application provides a method for detecting abnormal behavior in videos. By using temporal action detection algorithms and spatiotemporal action detection algorithms, it can simultaneously detect abnormal behavior in groups and individuals. It can comprehensively apply methods such as spatiotemporal feature fusion and temporal context information association to determine the location, behavior category, and behavior boundary of abnormal individuals. It solves the problem of false positives and false negatives caused by the inclusion relationship between abnormal behavior in groups and individuals during the abnormal behavior detection process. It has strong detection robustness and improves the ability to understand multiple action categories contained in video data, thereby improving the accuracy of abnormal behavior detection.

[0065] In some embodiments, step 110 specifically includes:

[0066] Step 1101: Use a time-series action detection algorithm to detect abnormal group behavior in the target video and obtain the abnormal behavior category and start and end time of all abnormal group behavior in the target video.

[0067] Step 1102: Based on the start and end times of the abnormal group behavior, crop the abnormal group behavior segments from the target video and splice the remaining video segments together;

[0068] Step 1103: Use the spatiotemporal action detection algorithm to detect abnormal single-person behaviors in the spliced ​​video, and obtain the abnormal behavior category and start and end time of all abnormal single-person behaviors in the video.

[0069] Step 1104: Based on the start and end times of the individual's abnormal behavior, trim the individual's abnormal behavior segments from the video.

[0070] This application performs intelligent video analysis on videos after preprocessing. Based on the priority prior information of abnormal behavior categories, it integrates group abnormal behavior temporal action detection and single-person abnormal behavior spatiotemporal action detection algorithms to identify the start and end frames of abnormal behavior, the type of abnormal behavior, and the position of the person in the frame.

[0071] Specifically, firstly, a temporal action detection algorithm is used to detect higher-priority group abnormal behaviors, obtaining the category and start and end times of each group abnormal behavior; based on the start and end times of the group abnormal behaviors, the target video is cropped to obtain video segments of group abnormal behaviors, the remaining video segments are spliced ​​together, and then a spatiotemporal action detection algorithm is used to detect lower-priority individual abnormal behaviors, obtaining the category, start and end times of each individual abnormal behavior, and the location of the person; based on the start and end times of the individual abnormal behaviors, the video is cropped to obtain video segments of individual abnormal behaviors.

[0072] In some embodiments, the training process of the temporal action detection algorithm includes:

[0073] Step 200: Create a dataset of abnormal group behavior;

[0074] Step 210: Calculate the loss function value of the temporal action detection algorithm network using the cross-entropy loss function on the training data in the dataset;

[0075] Step 220: Use the backpropagation algorithm to update and optimize all parameters of the temporal action detection algorithm network, and repeatedly perform iterative training until the loss function reaches the minimum value and the recognition accuracy reaches the optimal value, thereby obtaining the trained temporal action detection algorithm.

[0076] The training process of the temporal action detection algorithm is as follows: create a dataset of abnormal group behavior, use the cross-entropy loss function to calculate the loss function value of the temporal action detection algorithm network on the training data in the dataset; use the backpropagation algorithm to update and optimize all parameters of the network, and repeatedly perform iterative training until the loss function reaches the minimum value and the recognition accuracy reaches the optimal value, thereby obtaining the trained optimal temporal action detection algorithm network model, and using the optimal model to realize abnormal group behavior detection.

[0077] The network architecture of the temporal action detection algorithm is as follows: a 3D convolutional network is used to extract 3D features from video data, and then a feature pyramid network is used for temporal convolution to merge spatial and temporal features. The merged features are then used for coarse-grained regression of the start and end boundary positions of abnormal actions and prediction of abnormal action categories. Finally, fine-grained adjustment of the start and end boundary positions of abnormal actions and prediction of abnormal action categories are performed.

[0078] Optionally, the data in the abnormal group behavior dataset can be video data containing abnormal group behavior, with the behavior category and start and end time labeled. This application does not limit the method of obtaining abnormal group behavior data.

[0079] In some embodiments, the training process of the spatiotemporal action detection algorithm includes:

[0080] Step 300: Create a dataset of abnormal behaviors in a single individual;

[0081] Step 310: Calculate the loss function value of the spatiotemporal action detection algorithm network using the cross-entropy loss function on the training data in the dataset;

[0082] Step 320: Use the backpropagation algorithm to update and optimize all parameters of the spatiotemporal action detection algorithm network, and repeatedly perform iterative training until the loss function reaches the minimum value and the recognition accuracy reaches the optimal value, thereby obtaining the trained spatiotemporal action detection algorithm.

[0083] The training process of the spatiotemporal action detection algorithm is as follows: create a dataset of abnormal behavior of a single person; calculate the loss function value of the spatiotemporal action detection algorithm network using the cross-entropy loss function on the training data in the dataset; update and optimize all parameters of the network using the backpropagation algorithm, and iterate repeatedly until the loss function reaches the minimum value and the recognition accuracy reaches the optimal value, thereby obtaining a trained spatiotemporal action detection algorithm network model, and using the optimal model to realize the detection of abnormal behavior of a single person.

[0084] The network architecture of the spatiotemporal action detection algorithm is as follows: 2D convolutional network and 3D convolutional network extract spatial features of a single frame and contextual temporal features between multiple frames, then perform multi-level feature fusion to form spatiotemporal features, and finally use RPN network on the spatiotemporal features to predict the target coordinates of abnormal persons and the category of abnormal actions respectively.

[0085] Optionally, the data in the single-person abnormal behavior dataset can be video data containing single-person abnormal behavior, and the behavior category and start and end time can be labeled. This application does not limit the method of obtaining single-person abnormal behavior data.

[0086] In some embodiments, after step 100, the method further includes:

[0087] Step 101: Preprocess the target video.

[0088] After acquiring the target video, preprocessing can be performed on the target video, which can reduce the time spent processing large amounts of video data during video understanding and improve the efficiency of video data loading, processing, and storage.

[0089] In some embodiments, step 101 specifically includes:

[0090] Step 1011: Perform frame still image removal and frame sampling on the target video;

[0091] Step 1012: Convert the video into npy format video data.

[0092] By removing still frames and sampling frames from the target video, redundant information and data volume can be reduced, thus improving the efficiency of behavior detection. Converting the video to .npy format can improve the efficiency of format conversion.

[0093] Figure 2 This is a flowchart illustrating the intelligent analysis of public security surveillance videos provided in this application embodiment, such as... Figure 2 As shown, in one embodiment of this application, taking public security surveillance video as an example, the steps for detecting abnormal behavior in public security surveillance video are as follows:

[0094] Step 1: Load video files in formats such as mp4, avi, and flv from the video resource library, either locally or remotely;

[0095] Step 2: Perform preprocessing on the public security surveillance video files, such as frame still image removal, frame sampling, and format pre-conversion, to improve the efficiency of subsequent intelligent video analysis and processing;

[0096] Step 2.1: Specifically, for step 2, the video is converted into an image sequence V0. One frame is extracted from every 10 frames of the image sequence to form a new frame sequence V'0. The frame sequence is subjected to inter-frame difference processing, and the static frames without motion information are marked and their corresponding original video image sequences V1 are deleted to reduce video redundancy.

[0097] Step 2.2: For the video image sequence V1 processed in Step 2.1, perform a sampling operation of extracting 1 frame every 3 frames to form the video image sequence V2, and compress the video data volume;

[0098] Step 2.3: For the video image sequence V2 processed in Step 2.2 above, perform pre-format conversion, converting the video in mp4, avi, flv and other formats into npy format video data V3 in advance, and storing it in the npy video resource library to improve the format conversion efficiency;

[0099] Step 3: Perform intelligent video detection and analysis on the preprocessed video data V3. Based on the priority prior information of abnormal behavior categories, combine the group abnormal behavior temporal action detection and single abnormal behavior spatiotemporal action detection algorithms to identify the start and end frames of abnormal behavior, the type of abnormal behavior, and the position of the person in the frame.

[0100] Step 3.1: Determine the priority of public abnormal behavior categories; specifically, the original public security surveillance videos include group abnormal behaviors such as fighting, crowd congestion, abnormal crowd movement, and evacuation, as well as individual abnormal behaviors such as falling, running, theft, intoxication, and loitering / tailgating. The priority of group abnormal behavior categories is set higher than that of individual abnormal behaviors. Temporal action detection algorithms are used to detect the higher-priority group abnormal behaviors, while spatiotemporal action detection algorithms are used to detect the lower-priority individual abnormal behaviors.

[0101] Step 3.2: For the temporal action detection algorithm in Step 3.1, its characteristics are as follows: The temporal action detection algorithm is used to train the dataset of abnormal group behavior. First, a 3D convolutional network is used to extract 3D features F from the video data in the training set; then, a feature pyramid network is used to perform temporal convolution on the features F, merging the spatial and temporal dimensions to generate feature F. 1D , for F 1D Perform coarse-grained regression prediction B1 of the start and end boundary positions of the action, and predict the category of the action C1; finally, perform fine-grained adjustment of the start and end boundary positions of the action, and perform regression prediction B'1 of the start and end boundary positions of the action, and predict the category of the action C'1.

[0102] Step 3.3: For the temporal action detection algorithm in Step 3.2, create a dataset to train the algorithm. For each type of abnormal group behavior, such as fighting, crowd congestion, abnormal crowd movement, evacuation and escape, collect surveillance video data, label the behavior category and start and end time to form a group abnormal behavior dataset, and divide it into training set, validation set and test set in a ratio of 6:2:2.

[0103] Step 3.4: Train the temporal action detection algorithm using the training set prepared in Step 3.3. Specifically: Calculate the loss function value of the temporal action detection algorithm network using the cross-entropy loss function; update and optimize all weight parameters of the temporal action detection algorithm network using the backpropagation algorithm to obtain the updated network model; iteratively train the updated temporal action detection algorithm network until the loss function reaches its minimum value, repeating the training process to obtain the iteratively trained temporal action detection algorithm network model; use the validation set to verify the recognition accuracy of the iteratively trained temporal action detection algorithm network model until the recognition accuracy reaches its optimal value, thus obtaining the trained temporal action detection algorithm network model.

[0104] Step 3.5: Using a temporal action detection algorithm, the npy format video data V3 obtained after processing in Step 2 is used to detect abnormal group safety behaviors, resulting in the category M = {m} for each type of abnormal group behavior. n |n=0,1,2,...,N} and the start and end times of the behavior;

[0105] Step 3.6: Based on the start and end times of the abnormal group behavior obtained in the above steps, trim the video image sequence V2 from step 2.3 to obtain the video clip of the abnormal group behavior. The remaining video clips are spliced ​​together to obtain video V4;

[0106] Step 3.7: The spatiotemporal action detection algorithm in Step 3.1 is characterized by: firstly, using 2D convolutional networks and 3D convolutional networks respectively to extract the spatial features F of a single frame. 2D And the inter-frame contextual temporal features F 3D Then, for feature F 2D and F 3D Multi-level feature fusion is performed to form a fused spatiotemporal feature F'. Finally, the RPN network is used on feature F' to predict the target coordinates x of the person and the action category C2 respectively.

[0107] Step 3.8: For the spatiotemporal action detection algorithm in Step 3.1, create a dataset to train the algorithm. For each type of abnormal behavior of a single person, such as falling, running, tailing, loitering, etc., collect surveillance videos, label the behavior category, the target location of the abnormal person, and the start and end time of the behavior to form a dataset of abnormal behavior of a single person. Divide the dataset into training set, validation set and test set in a ratio of 6:2:2.

[0108] Step 3.9: Train the spatiotemporal action detection algorithm using the training set prepared in Step 3.8. Specifically: Based on the training set obtained in Step 3.8, calculate the loss function value of the spatiotemporal action detection algorithm network using the cross-entropy loss function; update and optimize all weight parameters of the spatiotemporal action detection algorithm network using the backpropagation algorithm to obtain the updated network model; iteratively train the updated spatiotemporal action detection algorithm network until the loss function reaches its minimum value, repeatedly iterating the loss function value of the algorithm network to obtain the iteratively trained spatiotemporal action detection algorithm network model; use the validation set to verify the recognition accuracy of the iteratively trained spatiotemporal action detection algorithm network model until the recognition accuracy reaches its optimal value, thus obtaining the trained spatiotemporal action detection algorithm network model.

[0109] Step 3.10: Apply a spatiotemporal motion detection algorithm to the video data V4 obtained after cropping in Step 3.6 to detect abnormal single-person safety behaviors, obtaining the location of the person involved and the behavior category K = {k} for each type of abnormal single-person safety behavior. n |n=0,1,2,...,N} and the start and end dates of the behavior;

[0110] Step 3.11: Based on the start and end times of the individual's abnormal behavior obtained in the above steps, trim video V4 to obtain a video clip of the individual's abnormal behavior.

[0111] Step 4: Based on the types of abnormal group behavior and individual safety anomalies obtained in Step 3 above, the location of the individuals in the footage, the start and end times of the abnormal behavior, and the video clips of the abnormal group behavior... Video clips of abnormal behavior by a single person Information such as these are used to create an abnormal behavior number, generate start and end timestamps for the behavior, save the abnormal video clip index, and store it in a public abnormal behavior database.

[0112] Step 5: Display local or remote video file resource library, public security monitoring video playback window, video intelligent analysis progress, abnormal behavior video clip list, corresponding abnormal behavior type, time and personnel location, etc. on the front-end interface.

[0113] For the aforementioned public safety video understanding scenarios, the video understanding task of abnormal behavior detection is subdivided into two sub-tasks: group abnormal behavior detection and individual abnormal behavior detection. Based on the priority prior information of abnormal behavior categories, and by integrating the temporal action detection results of group abnormal behavior and the spatiotemporal action detection results of individual abnormal behavior, the types of abnormal behavior of groups and individuals, the start and end times of occurrence, and the information of abnormal personnel are determined, thereby realizing intelligent video analysis for public safety abnormal behavior detection.

[0114] Figure 3 This is a schematic diagram of the structure of the abnormal behavior detection device for video provided in the embodiments of this application, as shown below. Figure 3 As shown, the device includes an acquisition module 310 and a behavior detection module 320, wherein:

[0115] Module 310 is used to acquire the target video;

[0116] The behavior detection module 320 is used to determine abnormal behavior segments of the target video based on the temporal action detection algorithm and the spatiotemporal action detection algorithm, according to the priority of group abnormal behavior detection being greater than individual abnormal behavior detection.

[0117] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0118] Based on the methods in the above embodiments, Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown in the illustration, this application provides an electronic device that may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions stored in the memory 430 to execute the abnormal behavior detection method for video described in the above embodiment.

[0119] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the abnormal behavior detection method for video described in the various embodiments of this application.

[0120] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the abnormal behavior detection method for video in the above embodiments.

[0121] Based on the methods in the above embodiments, this application provides a computer program product that, when running on a processor, causes the processor to execute the abnormal behavior detection method for video in the above embodiments.

[0122] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0123] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0124] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0125] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0126] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for anomaly behavior detection for a video, the method comprising: include: Acquire the target video; Based on temporal action detection algorithms and spatiotemporal action detection algorithms, abnormal behavior segments of the target video are determined according to the priority of group abnormal behavior detection being greater than single-person abnormal behavior detection. The method for determining abnormal behavior segments in the target video based on temporal action detection algorithms and spatiotemporal action detection algorithms, prioritizing group abnormal behavior detection over individual abnormal behavior detection, includes: The time-series action detection algorithm is used to detect abnormal group behavior in the target video, and the abnormal behavior categories and start and end times of all abnormal group behavior in the target video are obtained. Based on the start and end times of the abnormal group behavior, the abnormal group behavior segments in the target video are cropped, and the remaining video segments are spliced ​​together. The spatiotemporal action detection algorithm is used to detect abnormal single-person behaviors in the stitched video, and the abnormal behavior categories and start and end times of all abnormal single-person behaviors in the video are obtained. Based on the start and end times of an individual's abnormal behavior, clips of that behavior in the video are trimmed. The temporal motion detection algorithm aims to detect the occurrence of motion in a video and locate the start and end times of the motion. The spatiotemporal motion detection algorithm is an algorithm that simultaneously detects the occurrence of motion in video content and locates the temporal and spatial positions of the motion.

2. The method for anomaly behavior detection for a video according to claim 1, wherein, The training process of the time-series action detection algorithm includes: Create a dataset of abnormal group behavior; The loss function value of the temporal action detection algorithm network is calculated using the cross-entropy loss function on the training data in the dataset; The backpropagation algorithm is used to update and optimize all parameters of the temporal action detection algorithm network. Iterative training is performed repeatedly until the loss function reaches its minimum value and the recognition accuracy reaches its optimal value, thus obtaining the trained temporal action detection algorithm.

3. The abnormal behavior detection method for videos according to claim 1, characterized in that, The training process of the spatiotemporal action detection algorithm includes: Create a dataset of abnormal behavior in a single individual; The cross-entropy loss function is used to calculate the loss function value of the spatiotemporal action detection algorithm network on the training data in the dataset; The backpropagation algorithm is used to update and optimize all parameters of the spatiotemporal action detection algorithm network. Iterative training is performed repeatedly until the loss function reaches its minimum value and the recognition accuracy reaches its optimal value, thus obtaining the trained spatiotemporal action detection algorithm.

4. The abnormal behavior detection method for video according to claim 1, characterized in that, After acquiring the target video, the method further includes: The target video is preprocessed.

5. The abnormal behavior detection method for video according to claim 4, characterized in that, The preprocessing of the target video includes: The target video is subjected to still frame clipping and frame sampling. Convert the video to npy format video data.

6. An abnormal behavior detection device for video, characterized in that, include: The acquisition module is used to acquire the target video; The behavior detection module is used to determine the abnormal behavior segments of the target video based on the temporal action detection algorithm and the spatiotemporal action detection algorithm, according to the priority of group abnormal behavior detection being greater than single-person abnormal behavior detection. The behavior detection module is specifically used for: The time-series action detection algorithm is used to detect abnormal group behavior in the target video, and the abnormal behavior categories and start and end times of all abnormal group behavior in the target video are obtained. Based on the start and end times of the abnormal group behavior, the abnormal group behavior segments in the target video are cropped, and the remaining video segments are spliced ​​together. The spatiotemporal action detection algorithm is used to detect abnormal single-person behaviors in the stitched video, and the abnormal behavior categories and start and end times of all abnormal single-person behaviors in the video are obtained. Based on the start and end times of an individual's abnormal behavior, clips of that behavior in the video are trimmed. The temporal motion detection algorithm aims to detect the occurrence of motion in a video and locate the start and end times of the motion. The spatiotemporal motion detection algorithm is an algorithm that simultaneously detects the occurrence of motion in video content and locates the temporal and spatial positions of the motion.

7. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the abnormal behavior detection method for video as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, the processor performs the abnormal behavior detection method for video as described in any one of claims 1-5.

9. A computer program product, characterized in that, When the computer program product is run on a processor, the processor performs the abnormal behavior detection method for video as described in any one of claims 1-5.

Images