A method and apparatus for detecting global abnormal behavior of a group in surveillance video.

Abstract

This application discloses a method for detecting global abnormal behavior of groups in surveillance videos, relating to the fields of image processing and computer vision. The method selects a pre-trained VGG16 model as the base network for the global abnormal behavior detection network model; replaces the final Softmax classifier of the VGG16 model with an LSTM network framework and a prediction error-based classifier to obtain a prediction-based global abnormal behavior detection network model; fine-tunes the obtained global abnormal behavior detection network model; uses the trained global abnormal behavior detection network model to detect input video data and outputs labels corresponding to the detected frame images. This method can accurately detect global abnormal behavior of groups in surveillance videos. This application also includes apparatus for implementing the method. This application solves the problems of existing detection frameworks being complex in structure and relying solely on local features for detection.

Description

Technical Field

[0001] This application relates to the fields of image processing and computer vision technology, and in particular to a method and apparatus for detecting global abnormal behavior of a group in surveillance video. Background Technology

[0002] With the widespread adoption of surveillance equipment, the intelligent analysis and in-depth mining of massive amounts of video information to build intelligent surveillance systems is of great significance for improving urban visual management, government early warning and emergency response capabilities, and maintaining urban public safety. Among these systems, detecting abnormal group behavior in surveillance videos is a key issue and a research hotspot in the field of computer vision. In recent years, the performance of computer-related hardware has continuously improved, especially the significant advancements in GPU-related technologies, further promoting the progress of deep learning based on neural networks. Deep learning-based detection techniques have also been introduced into the detection of abnormal group behavior in surveillance videos.

[0003] Currently, research on deep learning-based detection frameworks for detecting global abnormal behavior in groups has limitations, mainly in two aspects: First, these frameworks are built for detecting local abnormal behavior within a group, resulting in complex structures and limited applicability to global abnormal behavior detection. Second, the training data for these frameworks comes from databases corresponding to local abnormal behavior, which limits their application to global abnormal behavior detection. Global abnormal behavior detection focuses on the overall features of a frame image, meaning one frame corresponds to one feature vector. Existing frameworks, built for detecting local abnormal behavior, focus on local, detailed features, leading to complex structures and requiring lengthy fine-tuning and training. While detection methods based on local features can also detect global abnormal behavior, they are resource-intensive and time-consuming, and some frameworks do not outperform traditional methods. Based on this analysis, a simpler and more effective deep learning-based detection framework that focuses on the overall features of frame images is needed. Summary of the Invention

[0004] This application provides a method and apparatus for detecting global abnormal behavior of a group in surveillance videos, which solves the problems of existing detection frameworks being complex in structure and only relying on local features for detection.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] On the one hand, this application proposes a method for detecting global abnormal behavior of a group in surveillance video, comprising the following steps:

[0007] A pre-trained VGG16 model was selected as the base network for the group global abnormal behavior detection network model.

[0008] By replacing the final Softmax classifier of the VGG16 model with an LSTM network framework and a classifier based on prediction error, a prediction-based global abnormal behavior detection network model for the population is obtained.

[0009] During the training phase, the obtained group global abnormal behavior detection network model is fine-tuned;

[0010] During the detection phase, a pre-trained global abnormal behavior detection network model is used to detect the input video data and output the labels corresponding to the detected frame images.

[0011] On the other hand, this application also proposes a device for detecting global abnormal behavior of a group in surveillance video, used to implement the method described in any embodiment of this application, including a model building module, a model training module, and a video detection module; the model building module is used to build a global abnormal behavior detection network model based on a prediction mechanism; the model training module is used to train the global abnormal behavior detection network model built by the model building module; the video detection module uses the trained global abnormal behavior detection network model to detect global abnormal behavior of a group in the surveillance video.

[0012] The above-mentioned at least one technical solution adopted in the embodiments of this application can achieve the following beneficial effects:

[0013] A method for detecting global abnormal group behavior in surveillance videos replaces the final Softmax classifier of a VGG16 model with an LSTM network framework and a prediction error-based classifier, thereby obtaining a prediction-based network model for detecting global abnormal group behavior. Fine-tuning further enhances the network model's detection capability for specific scenarios. This method can accurately detect global abnormal group behavior in surveillance videos and has promising application prospects. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0015] Figure 1 A flowchart of a method for detecting global abnormal behavior of a group in surveillance video;

[0016] Figure 2 This is a diagram of the pre-trained VGG16 architecture;

[0017] Figure 3A network framework diagram consisting of a VGG16 model and an LSTM network;

[0018] Figure 4 A framework diagram of a global abnormal behavior detection network model for a group;

[0019] Figure 5 The diagrams are for normal events and abnormal events in scenario 1; where (a) is the diagram for normal events and (b) is the diagram for abnormal events.

[0020] Figure 6 The image shows the detection results for Scene 1.

[0021] Figure 7 The diagrams are for normal events and abnormal events in scenario 2; where (a) is the diagram for normal events and (b) is the diagram for abnormal events.

[0022] Figure 8 The image shows the detection results for Scene 2.

[0023] Figure 9 The diagrams are for normal events and abnormal events in scenario 3; where (a) is the diagram for normal events and (b) is the diagram for abnormal events.

[0024] Figure 10 This is the detection result image for Scene 3;

[0025] Figure 11 This is a structural block diagram of a device for detecting abnormal group behavior in surveillance videos. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0028] A method for detecting global abnormal behavior of a group in surveillance video, such as Figure 1 As shown, it includes the following steps:

[0029] Step S1: Pre-train on the ImageNet dataset as follows Figure 2 The VGG16 model shown consists of six sub-blocks: five convolutional blocks and one Softmax classifier. Each convolutional block contains three 2D convolutional layers and one 2D max pooling layer.

[0030] In this embodiment, only the first 5 convolutional blocks in the VGG16 model are selected for detecting global abnormal behavior of a group in the surveillance video, and the 3 convolutional layers contained in the last convolutional block are fine-tuned.

[0031] Step S2: Replace the final Softmax classifier of the VGG16 model with an LSTM network framework and a classifier based on prediction error to obtain a group global abnormal behavior detection network model based on prediction mechanism.

[0032] Step S3: Fine-tune the global abnormal behavior detection network model obtained in step S2. The specific method is as follows:

[0033] First, freeze five convolutional blocks of the pre-trained VGG16 model and train an LSTM network framework and a classifier based on prediction error on the UMN dataset.

[0034] Second, unfreeze the fifth convolutional block of the already trained VGG16 model and train it on the UMN dataset together with the LSTM network framework and the prediction error-based classifier.

[0035] The UMN dataset contains three scenes. In this embodiment, the first 400 frames of images from each scene are used to train the LSTM network framework and a classifier based on prediction error.

[0036] The main rationale for using this method to fine-tune the global abnormal behavior detection network model is that in the VGG16 model, the bottom convolutional layers generate reusable general features, while the top convolutional layers generate specialized features specific to the database. Furthermore, fine-tuning can avoid the risk of overfitting that occurs when training on smaller datasets.

[0037] Step S4: Given a surveillance video clip, use the finely tuned group global abnormal behavior detection network model to detect the frame images in the video clip;

[0038] In this embodiment, the network framework composed of the VGG16 model and the LSTM network is as follows: Figure 3 As shown, the segments of the surveillance video are denoted as {X1, X2, ..., X...} T}, where T is the number of images in the video clip. The set of motion features obtained from this video clip using the VGG16 model is {F1, F2, ..., F}. T The LSTM network framework corresponds to {X1, X2, ..., X}. T The output of} is {y1,y2,...,y T}. When image X tAfter being input into the VGG16 model, the output result of the LSTM network framework is y t It corresponds to the next frame X of the video sequence. t+1 The characteristics of , where t = 1, 2, L, T. Because y t This is the motion feature predicted based on the LSTM network framework. To maintain consistency with the motion features obtained from the VGG16 model in terms of representation notation, it is denoted as...

[0039] In the process of detecting abnormal behavior in a group, the training set during the training phase consists only of frames representing normal scenes. When the training set representing normal group behavior is used... Figure 3 After fine-tuning and training, the network framework shown exhibits good predictive ability for the features of each frame in a normal scene. When images are input into VGG16, similar images will yield similar features, and vice versa. The content of frames in normal scenes differs significantly from that in abnormal scenes; therefore, the features obtained by VGG16 for these two scenarios will also differ considerably. When using an LSTM network for prediction, the predicted feature values ​​for frames in abnormal scenes differ significantly from the true features.

[0040] To distinguish between normal and abnormal frames in group global abnormal behavior detection, the group global abnormal behavior detection network model uses, for example... Figure 4 The end-to-end detection framework is shown below. Here, PE is the defined prediction error function, as shown in the following formula:

[0041]

[0042] Among them, F t+1 For image X t+1 Obtain true features using the VGG16 model. The predicted features of the LSTM network framework; if the prediction error PE satisfies PE < τ, then the image X t+1 For normal group activity, if the prediction error PE satisfies PE≥τ, then the image X t+1 This represents abnormal group activity. Here, τ represents the threshold controlling the sensitivity of the detection framework. After calculating the prediction error and comparing it with the threshold, the detection framework outputs a corresponding label: "+1" indicates that image X... t+1 For group activities in normal scenarios, "-1" represents image X. T+1 Group activities under abnormal circumstances.

[0043] In this embodiment, the number T of images in the monitored video clip is set to 4. Figure 5 , Figure 7 and Figure 9 The detection was performed on frame images from the test video sequences of scenes 1, 2, and 3 shown. The detection results are as follows: Figure 6 , Figure 8 and Figure 10 As shown.

[0044] In this embodiment, a device for detecting global abnormal behavior of a group in a surveillance video is provided to implement the present application. Figures 1-10 The method described in any embodiment, wherein the apparatus is as follows Figure 11 As shown, the system includes a model building module 11, a model training module 12, and a video detection module 13. The model building module is used to construct a global abnormal behavior detection network model based on a prediction mechanism, as described in steps S1-S2. The model training module is used to train the global abnormal behavior detection network model constructed by the model building module, as described in step S3. The video detection module uses the trained global abnormal behavior detection network model to detect global abnormal behavior in the surveillance video, as described in step S4.

[0045] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0046] Therefore, this application also proposes a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the methods described in any embodiment of this application.

[0047] Furthermore, this application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any embodiment of this application.

[0048] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0049] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0050] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0051] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0052] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0053] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0054] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for detecting global abnormal behavior of a group in surveillance video, characterized in that, include: A pre-trained VGG16 model was selected as the base network for the group global abnormal behavior detection network model. By replacing the final Softmax classifier of the VGG16 model with an LSTM network framework and a classifier based on prediction error, a prediction-based global abnormal behavior detection network model for the population is obtained. During the training phase, the global abnormal behavior detection network model of the group is trained and fine-tuned using a training set consisting only of frame images from normal scenes. The specific fine-tuning method is as follows: freeze the 5 convolutional blocks of the pre-trained VGG16 model; unfreeze the 5th convolutional block of the pre-trained VGG16 model; and train it together with the LSTM network framework and the classifier based on prediction error. During the detection phase, surveillance video clips will be used. The trained global abnormal behavior detection network model is input, and the VGG16 model extracts the true feature set. Simultaneously, the LSTM network framework outputs the predicted features corresponding to the next frame. ; Calculate the predicted features With the true features Prediction error between ; The prediction error With preset threshold If a comparison is made, Then determine the image For normal group activities, if Then determine the image This is an abnormal group activity.

2. The method for detecting global abnormal behavior of a group in surveillance video according to claim 1, characterized in that, The VGG16 model includes 5 convolutional blocks and 1 Softmax classifier; each convolutional block includes 3 two-dimensional convolutional layers and 1 two-dimensional max pooling layer.

3. The method for detecting global abnormal behavior of a group in surveillance video according to claim 1, characterized in that, The prediction error function of the classifier based on prediction error is: in, For prediction error, For image Obtain true features using the VGG16 model. For the prediction features of the LSTM network framework, , T This represents the number of images in the video clip.

4. The method for detecting global abnormal behavior of a group in surveillance video according to claim 1, characterized in that, During the training phase, the first 400 frames of images from each scene are used to train the LSTM network framework and the prediction error-based classifier; during the detection phase, the number of images T in the surveillance video clip is set to 4.

5. The method for detecting global abnormal behavior of a group in a surveillance video according to claim 1, characterized in that, The prediction error-based classifier outputs a label "+1" when the image represents group activity in a normal scene, and outputs a label "-1" when the image represents group activity in an abnormal scene.

6. A device for detecting global abnormal behavior of a group in a surveillance video, used to implement the method described in any one of claims 1-5, characterized in that, It includes a model building module, a model training module, and a video detection module; the model building module is used to build a group global abnormal behavior detection network model based on a prediction mechanism; the model training module is used to train the group global abnormal behavior detection network model built by the model building module; the video detection module uses the trained group global abnormal behavior detection network model to detect group global abnormal behavior in the surveillance video.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

8. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-5.

Images