Abnormal behavior detection method and device for video and electronic equipment

Abstract

The invention belongs to the technical field of video understanding action detection, and particularly discloses an abnormal behavior detection method and device for a video and electronic equipment, and the method comprises the steps: obtaining a target video; on the basis of a time sequence action detection algorithm and a space-time action detection algorithm, the abnormal behavior segment of the target video is determined according to the priority that group abnormal behavior detection is larger than single-person abnormal behavior detection, the method can complete group abnormal behavior detection and single-person abnormal behavior detection at the same time, and the accuracy of abnormal behavior detection is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of video understanding action detection, and more specifically, to a method, device and electronic device for detecting abnormal behavior in a video. Background Art

[0002] The current video understanding task for public safety abnormal behavior detection is to identify abnormal behaviors such as fights, crowds, abnormal crowd movements, evacuations, falls, chases, stalking, and lingering in the video. Group behaviors often include individual behaviors. For example, group fights may include falls and chases.

[0003] At present, the commonly used abnormal behavior detection methods are generally only aimed at specific group behaviors or single-person behaviors. In terms of abnormal behavior detection algorithms, the difference between single-person and group abnormal behaviors is that single-person behavior is often abnormal activities performed by specific people. It is necessary to locate the position of the abnormal person in the video screen, identify the behavior category, and determine the occurrence time and end time of the abnormal activity; while group behavior usually includes a multi-person interaction process and lasts for a certain period of time, but there is no need to locate specific people. It is only necessary to identify the group behavior category and the occurrence and end time of the abnormal activity. Therefore, the existing technology cannot realize single-person and multi-person abnormal behavior detection in the video at the same time. If the detection process cannot distinguish between the two types of behavior, it is easy to cause confusion between the behavior type and the occurrence time, resulting in false alarms or missed alarms, and reducing the accuracy of abnormal behavior detection. Summary of the invention

[0004] In view of the defects of the prior art, the purpose of this application is to provide a method, device and electronic device for abnormal behavior detection in video, aiming to solve the problem of low accuracy of abnormal behavior detection caused by the inability to simultaneously detect abnormal behavior of single person and multiple people in video in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present application provides a method for detecting abnormal behavior in a video, comprising:

[0006] Get the target video;

[0007] Based on the temporal action detection algorithm and the spatiotemporal action detection algorithm, the abnormal behavior segment of the target video is determined according to the priority of group abnormal behavior detection over single person abnormal behavior detection.

[0008] This application uses a temporal action detection algorithm and a spatiotemporal action detection algorithm to simultaneously complete group abnormal behavior detection and single-person abnormal behavior detection. It can comprehensively apply methods such as spatiotemporal feature fusion and time context information association to determine the location, behavior category and behavior boundary of abnormal personnel, and solve the problem of false positives and negatives caused by the inclusion relationship between group abnormal behavior and single-person abnormal behavior in the abnormal behavior detection process. The detection has strong robustness and improves the ability to understand multiple action categories contained in video data, thereby improving the accuracy of abnormal behavior detection.

[0009] According to a method for detecting abnormal behavior in a video provided by the present invention, the method is based on a temporal action detection algorithm and a spatiotemporal action detection algorithm, and determines the abnormal behavior segment of the target video according to the priority of group abnormal behavior detection over single person abnormal behavior detection, including:

[0010] Detecting abnormal group behaviors in the target video using the temporal action detection algorithm to obtain abnormal behavior categories and start and end times of all abnormal group behaviors in the target video;

[0011] Based on the start and end time of the group abnormal behavior, cropping the group abnormal behavior segments in the target video, and splicing the remaining video segments;

[0012] Use the spatiotemporal action detection algorithm to detect abnormal behaviors of a single person in the spliced ​​video, and obtain the abnormal behavior categories and start and end times of all abnormal behaviors of a single person in the video;

[0013] Based on the start and end time of the abnormal behavior of a single person, the abnormal behavior clips of a single person in the video are cropped.

[0014] This application is based on the priority criterion that group abnormal behavior detection is greater than single-person abnormal behavior detection. It first uses a temporal action detection algorithm to obtain group abnormal behavior clips in the target video, crop and extract them, and then splice the remaining video clips. Then, it uses a spatiotemporal action detection algorithm to obtain single-person abnormal behavior clips in the target video. It can comprehensively apply methods such as spatiotemporal feature fusion, temporal context information association, coarse-grained and fine-grained feature fusion prediction to determine the location of abnormal personnel, behavior categories and behavior boundaries, and has strong adaptability to multiple scenarios and high detection efficiency.

[0015] According to a method for detecting abnormal behavior in a video provided by the present invention, the training process of the temporal action detection algorithm includes:

[0016] Create a dataset of abnormal group behavior;

[0017] The cross entropy loss function is used to calculate the loss function value of the temporal action detection algorithm network for the training data in the dataset;

[0018] The back propagation algorithm is used to update and optimize all parameters of the temporal action detection algorithm network, and iterative training is performed repeatedly until the loss function reaches the minimum value and the recognition accuracy reaches the optimal value, thereby obtaining a trained temporal action detection algorithm.

[0019] According to a method for detecting abnormal behavior in a video provided by the present invention, the training process of the spatiotemporal action detection algorithm includes:

[0020] Create a single person abnormal behavior dataset;

[0021] The cross entropy loss function is used to calculate the loss function value of the spatiotemporal action detection algorithm network for the training data in the dataset;

[0022] The back-propagation algorithm is used to update and optimize all parameters of the spatiotemporal action detection algorithm network, and iterative training is performed repeatedly until the loss function reaches the minimum value and the recognition accuracy reaches the optimal value, thereby obtaining a trained spatiotemporal action detection algorithm.

[0023] This application collects group and single-person abnormal behavior datasets respectively, which are used to train the temporal action detection algorithm and the spatiotemporal action detection algorithm respectively to improve the detection accuracy of the algorithm.

[0024] According to a method for detecting abnormal behavior in a video provided by the present invention, after acquiring the target video, the method further includes:

[0025] The target video is preprocessed.

[0026] According to a method for detecting abnormal behavior in a video provided by the present invention, the preprocessing of the target video includes:

[0027] Performing frame still picture cutting and frame sampling on the target video;

[0028] Convert video to npy format video data.

[0029] This application can reduce video redundant information and compress the amount of video data by trimming and sampling still frames of the target video, thereby improving the efficiency of behavior detection. By converting the video into the npy format, the efficiency of format conversion can be improved.

[0030] In a second aspect, the present application provides an abnormal behavior detection device for a video, comprising:

[0031] An acquisition module, used to acquire a target video;

[0032] The behavior detection module is used to determine the abnormal behavior segment of the target video based on the temporal action detection algorithm and the spatiotemporal action detection algorithm according to the priority of group abnormal behavior detection over single person abnormal behavior detection.

[0033] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the abnormal behavior detection method for videos described in the first aspect or any possible implementation of the first aspect.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the abnormal behavior detection method for a video described in the first aspect or any possible implementation of the first aspect.

[0035] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the abnormal behavior detection method for a video described in the first aspect or any possible implementation of the first aspect.

[0036] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0037] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0038] By using temporal action detection algorithms and spatiotemporal action detection algorithms, we can simultaneously complete group abnormal behavior detection and single-person abnormal behavior detection. We can comprehensively apply methods such as spatiotemporal feature fusion and temporal context information association to determine the location, behavior category and behavior boundary of abnormal personnel, and solve the problem of false positives and negatives caused by the inclusion relationship between group abnormal behavior and single-person abnormal behavior in the abnormal behavior detection process. The detection is highly robust and improves the ability to understand the various action categories contained in the video data, thereby improving the accuracy of abnormal behavior detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1It is a flowchart of an abnormal behavior detection method for a video provided in an embodiment of the present application;

[0041] Figure 2 It is a schematic diagram of the process of intelligent analysis of public security surveillance video provided by the embodiment of the present application;

[0042] Figure 3 is a structural schematic diagram of an abnormal behavior detection device for a video provided in an embodiment of the present application;

[0043] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0046] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0047] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0048] First, the following contents are introduced:

[0049] In recent years, video surveillance systems have been widely used in application scenarios such as real-time monitoring, command and dispatch, investigation and evidence collection. The scale of surveillance video data is huge, but the probability of abnormal public safety behavior is often low. It is very difficult and time-consuming to rely on manual work to complete real-time monitoring, video search and other tasks. Therefore, it is necessary to rely on efficient and robust video intelligent analysis methods to realize public safety video abnormal behavior detection.

[0050] The current video understanding task for public safety abnormal behavior detection is to identify abnormal behaviors such as fights, crowds, abnormal crowd movements, evacuations, falls, chases, stalking, and lingering in the video. Group behaviors often include individual behaviors. For example, group fights may include falls and chases. If the detection process cannot distinguish between the two types of behaviors, it is easy to cause confusion between the behavior types and occurrence times, leading to false alarms or missed alarms.

[0051] Currently, commonly used processing methods generally only target specific group behaviors or individual behaviors, and there is a lack of a robust video understanding method that can simultaneously detect and identify abnormal group and individual behaviors.

[0052] In terms of abnormal behavior detection algorithms, the difference between single-person and group abnormal behaviors is that single-person behavior often refers to abnormal activities performed by specific people. It is necessary to locate the abnormal person in the video, identify the behavior category, and determine the occurrence and end time of the abnormal activity. Group behavior usually involves multi-person interaction and lasts for a certain period of time, but there is no need to locate specific people. It is only necessary to identify the group behavior category and the occurrence and end time of the abnormal activity. Therefore, different detection methods are required for the two types of tasks.

[0053] Next, combine Figure 1-Figure 2 The abnormal behavior detection method for videos provided in the embodiments of the present application is introduced.

[0054] Figure 1 FIG. 1 is a flow chart of a method for detecting abnormal behavior in a video according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0055] Step 100, obtaining a target video;

[0056] Optionally, the present application does not limit the method for acquiring the target video and the video format of the target video.

[0057] In one embodiment of the present application, a video file in a public security surveillance video resource library is loaded locally or remotely to perform abnormal behavior detection.

[0058] Step 110 , based on the temporal action detection algorithm and the spatiotemporal action detection algorithm, the abnormal behavior segment of the target video is determined according to the priority of group abnormal behavior detection over single person abnormal behavior detection.

[0059] This application uses a temporal motion detection algorithm and a spatiotemporal motion detection algorithm to perform intelligent video analysis on the target video to obtain the type of abnormal behavior in the video, the position of the person in the picture, the start and end time of the behavior, and the video clips containing the abnormal behavior.

[0060] Optionally, the temporal action detection algorithm aims to detect the occurrence of action from the video and locate the start and end time of the action. The spatiotemporal action detection algorithm is an algorithm that simultaneously detects the occurrence of action and locates the position of action in time and space in the video content. The present application uses the trained temporal action detection algorithm and the spatiotemporal action detection algorithm to determine the abnormal behavior segments of the target video, including group abnormal behavior segments and single-person abnormal behavior segments, in accordance with the priority of group abnormal behavior detection over single-person abnormal behavior detection, that is, group abnormal behavior detection is performed first, and then single-person abnormal behavior detection.

[0061] Optionally, abnormal group behavior may include: fighting, crowd congestion, abnormal crowd movement, evacuation, etc. in public security surveillance videos, and abnormal single-person behavior may include: falling, running, stealing, drunkenness, lingering, etc.; setting the priority of the group abnormal behavior category is higher than that of the single-person abnormal behavior.

[0062] Optionally, after obtaining the abnormal behavior clip, the video name and the intelligent analysis result of the video can be used to create a key video clip index according to the abnormal behavior category and occurrence time, and stored in the abnormal behavior database.

[0063] Optionally, data such as the progress of the video intelligent analysis, key segments of abnormal behavior, types of abnormal behavior, and locations of personnel may be displayed through a display device.

[0064] The present application provides an abnormal behavior detection method for videos. By using a temporal action detection algorithm and a spatiotemporal action detection algorithm, group abnormal behavior detection and single-person abnormal behavior detection can be completed simultaneously. The method can comprehensively apply spatiotemporal feature fusion, temporal context information association and other methods to determine the location of abnormal persons, behavior categories and behavior boundaries, and solve the problem of false positives and negatives caused by the inclusion relationship between group abnormal behavior and single-person abnormal behavior during abnormal behavior detection. The detection has strong robustness and improves the ability to understand multiple action categories contained in video data, thereby improving the accuracy of abnormal behavior detection.

[0065] In some embodiments, step 110 specifically includes:

[0066] Step 1101, using a temporal action detection algorithm to detect abnormal group behaviors in a target video, and obtaining abnormal behavior categories and start and end times of all abnormal group behaviors in the target video;

[0067] Step 1102, based on the start and end time of the group abnormal behavior, cropping the group abnormal behavior segments in the target video, and splicing the remaining video segments;

[0068] Step 1103, using a spatiotemporal action detection algorithm to detect abnormal behaviors of a single person in the spliced ​​video, and obtaining abnormal behavior categories and start and end times of all abnormal behaviors of a single person in the video;

[0069] Step 1104 , based on the start and end time of the abnormal behavior of the single person, crop the abnormal behavior segment of the single person in the video.

[0070] This application performs intelligent video analysis on the video that has passed the preprocessing unit. Based on the priority prior information of abnormal behavior categories, it integrates the temporal motion detection of group abnormal behavior and the spatiotemporal motion detection algorithm of single-person abnormal behavior to identify the start and end frames of abnormal behavior, the type of abnormal behavior, and the position of the person in the picture.

[0071] Specifically, firstly, a temporal motion detection algorithm is used to detect group abnormal behaviors with higher priority, and the category and start and end time of each type of group abnormal behavior are obtained; according to the start and end time of the group abnormal behavior, the target video is cropped to obtain group abnormal behavior video clips, and the remaining video clips are spliced, and on this basis, a spatiotemporal motion detection algorithm is used to detect single-person abnormal behaviors with lower priority, and the category, start and end behaviors, and personnel location of each type of single-person abnormal behavior are obtained; according to the start and end time of the single-person abnormal behavior, the video is cropped to obtain the single-person abnormal behavior video clip.

[0072] In some embodiments, the training process of the temporal motion detection algorithm includes:

[0073] Step 200, creating a group abnormal behavior data set;

[0074] Step 210, using a cross entropy loss function to calculate the loss function value of the temporal action detection algorithm network for the training data in the data set;

[0075] Step 220, using the back propagation algorithm to update and optimize all parameters of the temporal action detection algorithm network, repeatedly performing iterative training until the loss function reaches a minimum value and the recognition accuracy reaches an optimal value, thereby obtaining a trained temporal action detection algorithm.

[0076] The training process of the temporal action detection algorithm is as follows: create a group abnormal behavior data set, use the cross entropy loss function to calculate the loss function value of the temporal action detection algorithm network for the training data in the data set; use the back propagation algorithm to update and optimize all parameters of the network, and repeatedly perform iterative training until the loss function reaches the minimum value and the recognition accuracy reaches the optimal value, thereby obtaining a trained optimal temporal action detection algorithm network model, and use this optimal model to realize group abnormal behavior detection.

[0077] The network architecture of the temporal action detection algorithm network is as follows: a 3D convolutional network is used to extract 3D features from video data, and then a feature pyramid network is used for temporal convolution, which combines spatial and temporal dimension features. The merged features are used to perform coarse-grained abnormal action start and end boundary position regression and abnormal action category prediction; finally, fine-grained abnormal action start and end boundary position adjustment and abnormal action category prediction are performed.

[0078] Optionally, the data in the group abnormal behavior data set may be video data containing group abnormal behavior, and the behavior category and start and end time are marked. The present application does not limit the method for obtaining the group abnormal behavior data.

[0079] In some embodiments, the training process of the spatiotemporal motion detection algorithm includes:

[0080] Step 300, creating a single person abnormal behavior data set;

[0081] Step 310, using a cross entropy loss function to calculate the loss function value of the spatiotemporal action detection algorithm network for the training data in the data set;

[0082] Step 320, using the back propagation algorithm to update and optimize all parameters of the spatiotemporal motion detection algorithm network, repeatedly performing iterative training until the loss function reaches a minimum value and the recognition accuracy reaches an optimal value, thereby obtaining a trained spatiotemporal motion detection algorithm.

[0083] The training process of the spatiotemporal action detection algorithm is as follows: create a single-person abnormal behavior dataset; use the cross-entropy loss function to calculate the loss function value of the spatiotemporal action detection algorithm network for the training data in the dataset; use the back-propagation algorithm to update and optimize all parameters of the network, and repeatedly perform iterative training until the loss function reaches the minimum value and the recognition accuracy reaches the optimal value, thereby obtaining a trained spatiotemporal action detection algorithm network model, and using this optimal model to realize single-person abnormal behavior detection.

[0084] The network architecture of the spatiotemporal action detection algorithm network is as follows: 2D convolutional network and 3D convolutional network extract the spatial features of a single frame and the contextual temporal features between multiple frames, and then perform multi-level feature fusion to form spatiotemporal features. Finally, the RPN network is used on the spatiotemporal features to predict the target coordinates of abnormal personnel and the category of abnormal actions respectively.

[0085] Optionally, the data in the single-person abnormal behavior data set may be video data containing the single-person abnormal behavior, with the behavior category and start and end time annotated. This application does not limit the method for obtaining the single-person abnormal behavior data.

[0086] In some embodiments, after step 100, the method further includes:

[0087] Step 101, pre-processing the target video.

[0088] After acquiring the target video, the target video can be preprocessed to reduce the time for processing a large amount of video data during the video understanding process and improve the efficiency of video data loading, processing, and storage.

[0089] In some embodiments, step 101 specifically includes:

[0090] Step 1011, cutting out still frames and sampling frames of the target video;

[0091] Step 1012, converting the video into video data in npy format.

[0092] By trimming still frames and sampling the target video, redundant information can be reduced, the amount of video data can be compressed, and the efficiency of behavior detection can be improved. By converting the video to the npy format, the efficiency of format conversion can be improved.

[0093] Figure 2 : is a flow chart of intelligent analysis of public security surveillance video provided by the embodiment of the present application, such as Figure 2 As shown, in one embodiment of the present application, taking the public security surveillance video as an example, the steps of performing abnormal behavior detection on the public security surveillance video are as follows:

[0094] Step 1: Locally or remotely call and load video files in mp4, avi, flv and other formats in the video resource library;

[0095] Step 2: Perform pre-processing on the public security surveillance video files, such as frame still image cutting, frame sampling, and format pre-conversion, to improve the efficiency of subsequent video intelligent analysis and processing;

[0096] Step 2.1: For step 2, specifically, the video is converted into an image sequence V0, one frame is extracted from every 10 frames of the image sequence to form a new frame sequence V'0, the frame sequence is subjected to inter-frame difference processing, still frames without motion information are marked, and the corresponding original video image sequence V1 is deleted to reduce video redundant information;

[0097] Step 2.2: For the video image sequence V1 processed in step 2.1, a sampling operation of extracting one frame every three frames is performed to form a video image sequence V2, thereby compressing the video data volume;

[0098] Step 2.3: For the video image sequence V2 processed in the above step 2.2, perform format pre-conversion, convert the videos in mp4, avi, flv and other formats into video data V3 in npy format, and store them in the npy video resource library to improve the format conversion efficiency;

[0099] Step 3: Perform intelligent video detection analysis on the preprocessed video data V3. Based on the priority prior information of abnormal behavior categories, the group abnormal behavior temporal action detection and the single person abnormal behavior spatiotemporal action detection algorithm are integrated to identify the start and end frames of the abnormal behavior, the abnormal behavior type, and the position of the person in the picture.

[0100] Step 3.1: Determine the priority of public abnormal behavior categories; specifically, the original public security surveillance video contains group abnormal behaviors such as fighting, crowd congestion, abnormal crowd movement, evacuation and escape, as well as individual abnormal behaviors such as falling, running, stealing, drunkenness, lingering and stalking. Set the priority of group abnormal behavior categories higher than individual abnormal behaviors. Use the temporal motion detection algorithm to detect higher priority group abnormal behaviors, and use the spatiotemporal motion detection algorithm to detect lower priority individual abnormal behaviors;

[0101] Step 3.2: For the temporal action detection algorithm in step 3.1, its characteristics are as follows: the temporal action detection algorithm is used to train the group abnormal behavior dataset, firstly, the 3D feature F is extracted from the video data in the training set through the 3D convolution network; then the feature F is temporally convolved using the feature pyramid network, the spatial and temporal dimension features are combined and the feature F is generated 1D , for F 1D Perform a coarse-grained regression prediction B1 of the action start and end boundary positions, and a category prediction C1 of the action; finally, perform a fine-grained adjustment of the action start and end boundary positions, and regressively predict the action start and end boundaries B'1, and a category prediction C'1 of the action;

[0102] Step 3.3: For the temporal action detection algorithm in step 3.2, create a data set for training the algorithm. For each type of abnormal group behavior, such as fighting, crowd congestion, abnormal crowd movement, evacuation, etc., collect surveillance video data, annotate the behavior category and start and end time, and form a group abnormal behavior data set. Divide the training set, validation set, and test set in a ratio of 6:2:2;

[0103] Step 3.4: Use the training set of the sequential action detection algorithm produced in step 3.3 to train the sequential action detection algorithm. Specifically: use the cross entropy loss function to calculate the loss function value of the sequential action detection algorithm network; use the back propagation algorithm to update and optimize all weight parameters of the sequential action detection algorithm network to obtain an updated network model; iteratively train the updated sequential action detection algorithm network until the loss function reaches the minimum value, and repeat the training of the loss function value of the iterative algorithm network to obtain the sequential action detection algorithm network model after iterative training; use the validation set to verify the recognition accuracy of the sequential action detection algorithm network model after iterative training until the recognition accuracy reaches the optimal value, thereby obtaining a trained sequential action detection algorithm network model;

[0104] Step 3.5: For the npy format video data V3 obtained after processing in step 2, a time sequence action detection algorithm is used to detect abnormal group safety behaviors, and each type of abnormal group behavior category M = {m n |n=0, 1, 2, ..., N} and the start and end time of the behavior;

[0105] Step 3.6: According to the start and end time of the group abnormal behavior obtained in the above steps, crop the video image sequence V2 in step 2.3 to obtain the group abnormal behavior video clip Splice the remaining video clips to obtain video V4;

[0106] Step 3.7: For the spatiotemporal action detection algorithm in step 3.1, its characteristics are as follows: first, a 2D convolutional network and a 3D convolutional network are used to extract the spatial features F of a single frame respectively. 2D , and multi-frame contextual temporal features F 3D , and then for the feature F 2D and F 3D Perform multi-level feature fusion to form fused spatiotemporal feature F', and finally use the RPN network to predict the target coordinates x and action category C2 of the person respectively on the feature F';

[0107] Step 3.8: For the spatiotemporal action detection algorithm in step 3.1, create a data set for training the algorithm. For each type of abnormal behavior of a single person, such as falling, running, following, lingering, etc., collect surveillance videos, annotate the behavior category, target location of the abnormal person, and the start and end time of the behavior to form a single-person abnormal behavior data set, and divide it into training set, validation set, and test set in a ratio of 6:2:2;

[0108] Step 3.9: Use the spatiotemporal motion detection algorithm training set produced in step 3.8 to train the spatiotemporal motion detection algorithm. Specifically: According to the training set obtained in step 3.8, use the cross entropy loss function to calculate the loss function value of the spatiotemporal motion detection algorithm network; use the back propagation algorithm to update and optimize all weight parameters of the spatiotemporal motion detection algorithm network to obtain an updated network model; iteratively train the updated spatiotemporal motion detection algorithm network until the loss function reaches the minimum value, and repeat the iterative algorithm network loss function value to obtain the spatiotemporal motion detection algorithm network model after iterative training; use the validation set to verify the recognition accuracy of the iteratively trained spatiotemporal motion detection algorithm network model until the recognition accuracy reaches the optimal value, thereby obtaining a trained spatiotemporal motion detection algorithm network model;

[0109] Step 3.10: The video data V4 obtained by trimming in step 3.6 is subjected to a spatiotemporal motion detection algorithm to detect abnormal single-person safety behaviors, and the location of the person involved in each type of abnormal single-person safety behavior and the behavior category K = {k n |n=0,1,2,...,N} and the start and end behaviors of the behaviors;

[0110] Step 3.11: Based on the start and end time of the abnormal behavior of a single person obtained in the above steps, crop the video V4 to obtain a video clip of the abnormal behavior of a single person

[0111] Step 4: Based on the types of group abnormal behaviors and individual safety abnormal behaviors obtained in step 3 above, the positions of the individuals in the picture, the start and end time of the abnormal behaviors, and the video clips of group abnormal behaviors Video clip of a person's abnormal behavior etc., create an abnormal behavior number, generate the start and end timestamps of the behavior, save the abnormal video clip index, and store it in the public abnormal behavior database;

[0112] Step 5: The front-end interface displays the local or remote video file resource library, the public security surveillance video playback window, the video intelligent analysis progress, the abnormal behavior video clip list, the type of corresponding abnormal behavior, time and personnel location and other information.

[0113] For the above-mentioned public safety video understanding scenario, the video understanding task of abnormal behavior detection is subdivided into two subtasks: group abnormal behavior detection and single person abnormal behavior detection. Based on the priority prior information of abnormal behavior categories, the temporal motion detection of group abnormal behavior and the spatiotemporal motion detection result information of single person abnormal behavior are integrated to determine the type of group and single person abnormal behavior, the start and end time of occurrence, and the information of abnormal personnel, so as to realize intelligent video analysis for public safety abnormal behavior detection.

[0114] Figure 3 is a schematic diagram of the structure of an abnormal behavior detection device for a video provided in an embodiment of the present application, such as Figure 3 As shown, the device includes an acquisition module 310 and a behavior detection module 320, wherein:

[0115] An acquisition module 310 is used to acquire a target video;

[0116] The behavior detection module 320 is used to determine the abnormal behavior segment of the target video based on the temporal action detection algorithm and the spatiotemporal action detection algorithm according to the priority of group abnormal behavior detection over single person abnormal behavior detection.

[0117] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0118] Based on the method in the above embodiment, Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, an embodiment of the present application provides an electronic device, which may include: a processor (processor) 410, a communication interface (CommunicationsInterface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logic instructions in the memory 430 to execute the abnormal behavior detection method for the video in the above embodiment.

[0119] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the abnormal behavior detection method for a video described in each embodiment of the present application.

[0120] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the abnormal behavior detection method for a video in the above embodiment.

[0121] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the abnormal behavior detection method for a video in the above embodiment.

[0122] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0123] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0124] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0125] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0126] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for detecting abnormal behavior in a video, characterized in that: include: Get the target video; Based on the temporal action detection algorithm and the spatiotemporal action detection algorithm, the abnormal behavior segment of the target video is determined according to the priority of group abnormal behavior detection over single person abnormal behavior detection.

2. The abnormal behavior detection method for video according to claim 1, characterized in that: The method of determining the abnormal behavior segment of the target video based on the temporal action detection algorithm and the spatiotemporal action detection algorithm according to the priority of group abnormal behavior detection over single person abnormal behavior detection includes: Detecting abnormal group behaviors in the target video using the temporal action detection algorithm to obtain abnormal behavior categories and start and end times of all abnormal group behaviors in the target video; Based on the start and end time of the group abnormal behavior, cropping the group abnormal behavior segments in the target video, and splicing the remaining video segments; Use the spatiotemporal action detection algorithm to detect abnormal behaviors of a single person in the spliced ​​video, and obtain the abnormal behavior categories and start and end times of all abnormal behaviors of a single person in the video; Based on the start and end time of the abnormal behavior of a single person, the abnormal behavior clips of a single person in the video are cropped.

3. The abnormal behavior detection method for video according to claim 1, characterized in that: The training process of the temporal action detection algorithm includes: Create a dataset of abnormal group behavior; The cross entropy loss function is used to calculate the loss function value of the temporal action detection algorithm network for the training data in the dataset; The back propagation algorithm is used to update and optimize all parameters of the temporal action detection algorithm network, and iterative training is performed repeatedly until the loss function reaches the minimum value and the recognition accuracy reaches the optimal value, thereby obtaining a trained temporal action detection algorithm.

4. The abnormal behavior detection method for video according to claim 1, characterized in that: The training process of the spatiotemporal action detection algorithm includes: Create a single person abnormal behavior dataset; The cross entropy loss function is used to calculate the loss function value of the spatiotemporal action detection algorithm network for the training data in the dataset; The back-propagation algorithm is used to update and optimize all parameters of the spatiotemporal action detection algorithm network, and iterative training is performed repeatedly until the loss function reaches the minimum value and the recognition accuracy reaches the optimal value, thereby obtaining a trained spatiotemporal action detection algorithm.

5. The abnormal behavior detection method for video according to claim 1, characterized in that: After acquiring the target video, the method further includes: The target video is preprocessed.

6. The abnormal behavior detection method for video according to claim 5, characterized in that: The preprocessing of the target video includes: Performing frame still picture cutting and frame sampling on the target video; Convert video to npy format video data.

7. A device for detecting abnormal behavior in a video, characterized in that: include: An acquisition module, used to acquire a target video; The behavior detection module is used to determine the abnormal behavior segment of the target video based on the temporal action detection algorithm and the spatiotemporal action detection algorithm according to the priority of group abnormal behavior detection over single person abnormal behavior detection.

8. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the abnormal behavior detection method for video as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is enabled to execute the abnormal behavior detection method for a video according to any one of claims 1 to 6.

10. A computer program product, characterized in that When the computer program product runs on a processor, the processor is enabled to execute the abnormal behavior detection method for a video according to any one of claims 1 to 6.

Images