Video abnormal event parallel detection method based on secondary reconstruction

Abstract

The invention discloses a video abnormal event parallel detection method based on secondary reconstruction. The method comprises the following steps: collecting available video data; constructing a parallel computing platform, and performing target detection on the split video frames to obtain a target object; the parallel computing platform comprises a first reconstruction model and a second reconstruction model; in the training stage, an input target object is used as input for training of a first reconstruction model, and the target object and a first reconstruction object are used as input for training of a second reconstruction model; inputting the target object into the trained first reconstruction model and / or the trained second reconstruction model, calculating a reconstruction error between the target object and the first reconstruction object and / or the second reconstruction object, and normalizing the reconstruction error to obtain a first reconstruction score and / or a second reconstruction score; and carrying out weighted calculation on the two reconstruction scores to obtain a total reconstruction score so as to detect whether an abnormal event exists in the video frame. The invention provides a parallel computing strategy, the strategy can effectively reduce the operation time of the detection method, meanwhile, a secondary reconstruction model is designed, the model can highlight the occurrence area of the abnormal event in the video, and a memory module is introduced into the reconstruction model, so that the detection accuracy is improved. The module can record and compare whether an event in an input video frame is normal or not, and the accuracy of video abnormal event detection is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to a video anomaly detection technology, and in particular to a video anomaly event parallel detection technology based on secondary reconstruction. Background Art

[0002] Traditional video surveillance technology has become obsolete with the popularization of the Internet and the development of camera technology, and intelligent video surveillance has become a development trend. Relying solely on manpower to manage widely distributed video surveillance facilities will face problems such as huge workload, low alarm accuracy and slow response speed. It is crucial to improve the intelligence of video abnormal behavior detection tasks. At present, the reconstruction-based abnormal behavior detection method hopes that the reconstruction model will obtain a smaller reconstruction error when reconstructing normal video frames and a larger reconstruction error when reconstructing abnormal video frames. However, since the reconstruction model inherits its powerful representation ability when using convolutional neural networks, this will cause abnormal frames to have a probability of obtaining a smaller reconstruction error when reconstructing, and thus be detected as normal frames, making the detection results inaccurate.

[0003] The video abnormal behavior detection method relies heavily on deep learning and requires a lot of resources to adjust the model parameters. There are still limitations in detection efficiency. When detecting surveillance videos, a lot of time is required to extract information from the data and train the detection model. It is impossible to reasonably allocate the computing power required for the detection model to reduce the overall running time. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a parallel detection technology for video abnormal events based on secondary reconstruction, and introduces a memory module therein, which can memorize normal patterns for comparison, so as to effectively improve the accuracy of video abnormal event detection.

[0005] A first aspect of an embodiment of the present invention provides a method for parallel detection of abnormal video events based on secondary reconstruction, which is characterized by comprising:

[0006] S1: Collect the required video set and preprocess it to obtain one or more training video frames and test video frames;

[0007] S2: Establish a parallel computing strategy and use it in video abnormal event detection;

[0008] S3: performing target detection on the training video frame and / or the test video frame to obtain a training target object and / or a test target object;

[0009] S4: inputting the training target object into the first reconstruction model to train the model, thereby obtaining the first reconstruction target object;

[0010] S5: inputting the training target object and the first reconstruction target object into a second reconstruction model to train the model, thereby constructing a secondary reconstruction model;

[0011] S6: inputting the test target object into the secondary reconstruction model to obtain a first reconstruction score and / or a second reconstruction score;

[0012] S7: Perform weighted calculation on the two reconstruction scores to obtain a total reconstruction score, so as to detect whether there is an abnormal event in the video frame.

[0013] Furthermore, S2 proposes a parallel computing strategy, which is used in video abnormal event detection, and is applied in target detection, model building, anomaly detection and other steps to reduce the time required for each step and improve the detection efficiency of the overall detection method.

[0014] Further, the S3: performing target detection on the training video frame and / or the test video frame to obtain a training target object and / or a test target object includes:

[0015] The required surveillance video sets are distributed to different computing nodes in the parallel computing platform. Each computing node runs the target detection algorithm independently, and multiple computing nodes jointly complete the target detection and extraction task.

[0016] Furthermore, the target detection and extraction task includes:

[0017] Store the required surveillance video sets in the same path so that the parallel computing platform can access these video data;

[0018] Create SparkConf and SparkContext. The SparkContext is the entry point of the Spark application, responsible for communicating with the cluster manager, creating parallel computing core components such as RDD, accumulator, and broadcast variables, and providing some commonly used operation functions. The SparkConf is the configuration class of the Spark application, which can set various parameters of the Spark application.

[0019] Parallelizing the list containing the video data path into the RDD, using the parallel computing platform to divide the surveillance video data, and dividing the large task into multiple subtasks;

[0020] Dynamically allocate computing nodes according to the number of the divided subtasks, so that each computing node independently runs the target detection and extraction task;

[0021] After each computing node processes its own subtask, it obtains object information and aggregates the results of the object detection and extraction tasks into a central storage location or data structure;

[0022] When all subtasks have completed object detection and the results have been merged and processed, the object detection extraction task of the surveillance video is completed and the results are collected back to the driver node.

[0023] Further, the step S4: inputting the training target object into the first reconstruction model to train the model to obtain the first reconstruction target object includes:

[0024] Extracting a target object of interest from the training video frame by using a YOLOv5 algorithm, where the target object may be a person, a vehicle, or any other object of interest in the video frame;

[0025] The first reconstruction model is a generative adversarial network structure, which includes a generator and a discriminator part, wherein the generator is composed of an encoder, a decoder, a jump connection part and a memory module;

[0026] The target object is input into the encoder part of the autoencoder network, and the encoder processes the target object through multiple convolution and pooling operations to reduce the size of the feature map and increase the number of channels to obtain a potential feature map; the feature map is input into the decoder part, and the decoder performs an upsampling operation to restore the size of the feature map to the original size of the input image;

[0027] In order to better retain the detail information of the input image, the autoencoder network uses jump connections to connect feature maps of different levels, that is, splicing the feature map of each layer with the reconstructed feature map of the corresponding layer, and finally generates a first reconstructed video frame with the same size as the input video frame, that is, the first reconstructed target object.

[0028] After the first reconstruction target object is generated, it is subjected to adversarial training with the target object to further improve the ability of the first reconstruction model. The goal of this adversarial training is to narrow the distance between the first reconstruction frame and the normal training video frame, so that the first reconstruction model can better learn and capture the characteristics and distribution of the normal frame.

[0029] Further, the step S5: inputting the training target object and the first reconstruction target object into a second reconstruction model to train the model, thereby constructing a secondary reconstruction model, includes:

[0030] The second reconstruction model is a generative adversarial network structure, which includes a generator and a discriminator part, wherein the generator is composed of an encoder, a decoder, a jump connection part and a memory module, and the training method of the generator is:

[0031] Extracting the target object of interest from the training video frame by the YOLOv5 algorithm and inputting the first reconstructed target object into the encoder, and obtaining the first feature map of different sizes and different numbers of channels in each layer after multiple convolution and pooling processes, and obtaining the potential features at the lowest layer;

[0032] Inputting the latent features into a memory module to learn to model normal behavior and store normal patterns;

[0033] The latent features are input into a decoder, and the decoder performs an upsampling operation to generate a second feature map. At the same time, the first feature map and the second feature map at the same layer as the second feature map are connected by a jump connection to obtain a new second feature map. Finally, the size of the second feature map is restored to the original size of the input image to obtain a second reconstructed video frame with the same size as the input frame, that is, the second reconstructed target object;

[0034] The second reconstructed target object is subjected to adversarial training with the training target object, wherein the second reconstructed target object generated by the first reconstructed target object is subjected to adversarial training with the training target object corresponding to the first reconstructed target object, the first reconstructed target object generated by the training target object is subjected to adversarial training with the corresponding training target object, and the first reconstructed target object is subjected to adversarial training with the second reconstructed target object. The adversarial training shortens the distance between the two objects, so that the second reconstructed model can better learn and capture the characteristics and distribution of normal frames.

[0035] Further, the step S5: inputting the training target object and the first reconstruction target object into a second reconstruction model to train the model, thereby constructing a secondary reconstruction model, further includes:

[0036] The memory module learns to model normal behavior and store normal patterns:

[0037] The memory module stores a matrix of size N×C, defined as M∈R N×C , where N indicates that the memory module contains N valid feature information, and C indicates the value of the feature information, which is the same as the potential feature z dimension obtained by the encoder, and its calculation formula is:

[0038]

[0039] Where m i Represents the storage items in the memory module, the vector ω∈R 1×N is the attention coefficient calculated for the potential feature z;

[0040] Using the non-exponential softmax function, the weight ω is calculated i, the calculation formula is as follows:

[0041]

[0042] Where d(·) is the cosine similarity, which is calculated as follows:

[0043]

[0044] Although attention addressing is used, some anomalies may still be The complex combination of is well reconstructed. To solve this problem, a hard shrinkage strategy is adopted, which is only effective when the attention weight is greater than the set threshold, otherwise it is 0:

[0045]

[0046] However, the hard shrinkage strategy cannot be directly implemented because back propagation is not possible, so it is re-implemented using the ReLU function:

[0047]

[0048] Where max(·,·) is the ReLU function, ε is a very small positive term, and the threshold λ is usually set between [1 / N, 3 / N]. The normalized weight is calculated as follows:

[0049]

[0050] The final transformation formula containing the memory module can be obtained, and its calculation formula is as follows:

[0051]

[0052] Further, the step S4: inputting the training target object into the first reconstruction model to train the model to obtain the first reconstruction target object, further includes:

[0053] In order to minimize the distance between the first reconstructed frame and the input frame in the first reconstruction, constraints are imposed at the appearance level and motion level. The appearance constraint L app1 By the gradient constraint L gc1 and strength constraint L sc1 The gradient constraint L gc1 and the strength constraint L sc1 The calculation is as follows:

[0054]

[0055] Wherein, x represents the training target object, represents the first reconstructed target object, a, b represent the horizontal and vertical coordinates of the pixels of the training target object;

[0056] By the gradient constraint L gc1 and the strength constraint L sc1 , get the appearance constraint L app1 , the specific calculation method is as follows:

[0057]

[0058] Where m:n=1:1;

[0059] The motion constraint L opt1 is the optical flow constraint, and the calculation process is as follows:

[0060]

[0061] Where t represents the tth frame;

[0062] The objective function of the first reconstruction model is:

[0063] L1=αL app1 +βL opt1 .

[0064] Further, the step S5: inputting the training target object and the first reconstruction target object into a second reconstruction model to train the model, thereby constructing a secondary reconstruction model, further includes:

[0065] In order to minimize the distance between the normal frame and the second reconstructed frame in the second reconstruction, constraints are imposed at the appearance level and motion level. The appearance constraint L app2 By the gradient constraint L gc2 and strength constraint L sc2 The motion constraint L opt2 is the optical flow constraint, and the calculation method of both is the same as the appearance constraint L in the first reconstruction model app1 and the motion constraint L opt1 The calculation method is the same as Change to Indicates the second reconstruction of the target object;

[0066] In the memory module, the sparsity of ω is further constrained and regularized. The formula is as follows:

[0067]

[0068] Where Q is the number of memory modules, N is the number of memory items in a memory module, ω i,kis the similarity coefficient between the query feature and the memory item;

[0069] The specific optimization process is:

[0070] When training the discriminator, the generator is fixed. The goal is to enable the discriminator to accurately distinguish the second reconstructed frame from the normal training frame to improve the discriminator's discrimination ability. The loss function of the discriminator is as follows:

[0071]

[0072] Where i, j are the indices of the frames, D(·)∈[0,1], and L(·,·) represents the absolute value of the difference between the two;

[0073] When training the generator, the discriminator is fixed. The goal is to make the reconstructed frames generated by the generator and the original training frames indistinguishable from the discriminator to improve the generation ability of the generator. The loss function of the generator is as follows:

[0074]

[0075] The training goal of the generator is to reconstruct only normal frames and cannot reconstruct abnormal frames. That is, the generator and discriminator are only optimized for normal frames. The generator function is as follows:

[0076]

[0077] where λ app2 , opt2 , adv and λ m are the weight factors of appearance loss, motion loss, generator loss, and memory module loss respectively;

[0078] The objective function of the discriminator is as follows:

[0079]

[0080] Further, the step S6: inputting the test target object into the secondary reconstruction model to obtain a first reconstruction score and / or a second reconstruction score includes:

[0081] A first reconstruction error and a second reconstruction error, the first reconstruction error and / or the second reconstruction error

[0082] The calculation formula is:

[0083]

[0084] Where x′ represents the test target object, represents the first reconstruction target object, Representing the second reconstruction target object;

[0085] The first reconstruction error and / or the second reconstruction error are normalized by a normalization method to obtain the first reconstruction score Score(x′)1 and the second reconstruction score Score(x′)2, which are calculated as follows:

[0086]

[0087] Among them, S1 is the total set of first reconstruction errors of each of the test target objects, and S2 is the total set of second reconstruction errors of each of the test target objects.

[0088] Further, the step S7: performing weighted calculation on the first reconstruction score and the second reconstruction score to obtain a total reconstruction score, thereby detecting whether there is an abnormal event in the video frame, includes:

[0089] The total reconstruction score is calculated as:

[0090] Score(x′)=λ1Score(x′)1+λ2Score(x′)2

[0091] Wherein, λ1 is the weight of the first reconstruction score, λ2 is the weight of the second reconstruction score,

[0092] When Score(x′) is less than the threshold, it means that only normal behaviors exist in the video frame, otherwise it means that the video frame contains abnormal behaviors.

[0093] The present invention proposes a secondary reconstruction model, which reconstructs the video frame twice continuously, increases the reconstruction error between the abnormal frame and the original frame after reconstruction, and highlights the abnormal area, so as to better detect and identify abnormal events in the video. The video frame is input into the model, and the model reconstructs the frame twice to obtain reconstructed frames of different scales. Each scale captures different levels of features and detail information. If there is an abnormal event in the video frame, the abnormal event will cause abnormal signals of different degrees in the two reconstructions, so as to better detect the video frame with abnormal events. At the same time, a memory module is introduced into the secondary reconstruction model, which can help the detection model record the distribution pattern of normal frame features, so that the model has the necessary reference pattern when it is difficult to distinguish the video frame, and further distinguish normal frames from abnormal frames. In order to reduce the overall running time and reasonably allocate the detection model operation, the present invention adopts a parallel computing strategy. This strategy divides a single task into a multi-task mode, and solves the task simultaneously through multiple working nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1It is a flow chart of a video abnormal event detection technology based on secondary reconstruction according to an embodiment of the present invention;

[0095] Figure 2 It is a flow chart of a parallel detection technology of video abnormal events based on secondary reconstruction according to an embodiment of the present invention;

[0096] Figure 3 A generator of an autoencoder for a method for parallel detection of abnormal video events based on secondary reconstruction according to an embodiment of the present invention; Figure 4 It is a memory module of the video abnormal event parallel detection method based on secondary reconstruction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0097] The present invention will be described with reference to certain drawings and with respect to specific embodiments, but the present invention is not limited thereto, but only by the limitations of the claims. Any reference numerals in the claims should not be interpreted as limiting the scope. The drawings are merely schematic and non-restrictive. In the drawings, the size of some elements may be enlarged and not drawn in proportion for illustrative purposes. When the term "comprising" is used in the specification and claims, it does not exclude other elements or steps. When an indefinite article or a definite article such as "one (a)" or "(an)", "the" is used to refer to a singular noun, unless otherwise expressly stated, this includes the plural number of the noun. In addition, the terms first, second, third, etc. are used to distinguish similar elements in the specification and claims, and are not necessarily used to describe a sequential order or a temporal order. It should be understood that the terms used in this way can be interchangeable where appropriate, and the embodiments of the present invention described herein can be operated in other orders except for the order described or illustrated herein.

[0098] Unless otherwise defined herein, the scientific and technical terms and phrases used in the present invention shall have the meanings commonly understood by those of ordinary skill in the art. Unless otherwise indicated, the methods are generally performed according to conventional methods well known in the art and described in various general and more specific references, which are cited and discussed in this specification.

[0099] In order to more thoroughly understand the features and technical contents of the present invention, embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. The attached drawings and specific embodiments are only for reference and explanation, and are not intended to limit the present invention.

[0100] like Figure 1 The embodiment of the present invention provides a parallel detection technology for abnormal video events based on secondary reconstruction, comprising the following steps:

[0101] S1: Build a parallel computing platform and split the video into frames to obtain one or more training video frames and test video frames, and divide them into training set and test set;

[0102] S2: On a parallel computing platform, using a target detection algorithm to extract target regions of interest in the training set and the test set to obtain training target objects and / or test target objects;

[0103] S3: On the parallel computing platform, enter the training phase. Input the target object of the training set into the first reconstruction model, train the first reconstruction model, and obtain the first reconstruction training target object;

[0104] S4: Input the first reconstruction training target object and the target object of the normal training set into the second reconstruction model to obtain a trained second reconstruction model. Thus, a trained second reconstruction model is obtained;

[0105] S5: On the parallel computing platform, enter the test phase. Input the target object of the test set into the trained reconstruction model to obtain the first reconstructed frame and the second reconstructed frame;

[0106] S6: Calculate the reconstruction error between the reconstructed frame and the real frame to obtain the first reconstruction error and the second reconstruction error. Perform normalization on the two and then perform weighted calculation to obtain the total reconstruction score for detecting the video frame.

[0107] S7: Perform weighted calculation on the two reconstruction scores to obtain a total reconstruction score, so as to detect whether there is an abnormal event in the video frame.

[0108] In some embodiments, the method for extracting target objects in the video frames in S3 is: using a target detection algorithm on a parallel computing platform to perform parallel target detection and extraction processing on the video frames; allocating the required monitoring video sets to different computing nodes in the parallel computing platform, each computing node independently runs the target detection algorithm, and multiple computing nodes jointly complete the target detection and extraction task; first, the required monitoring video sets are stored in the same path, and the parallel computing platform divides the tasks; after dividing the tasks and obtaining several subtasks, scheduling the computing nodes with the same number of subtasks to perform calculations and processing simultaneously; finally, completing the target detection and extraction task of the monitoring video.

[0109] In some embodiments, the first reconstruction model in S4 is composed of an encoder, a skip connection module and a decoder, and the training method for the first reconstruction model (performed in a parallel computing platform) is:

[0110] The target object of the training set is input into the first reconstruction model. First, it enters the encoder part, and performs multiple convolution and pooling operations on the target object to extract the target object features and reduce the feature map size, increase the number of channels, and obtain the potential features at the bottom layer;

[0111] After obtaining the latent features, they are input into the decoder part. The decoder performs upsampling operation to restore the size of the feature map to the original size of the input image;

[0112] The feature maps of different levels are connected by using skip connections. In this process, the first reconstructed frame with the same size as the input frame is obtained by concatenating the feature map of each layer with the corresponding decoder output;

[0113] After the first reconstruction target object is generated, it is trained adversarially with the target object of the training set to further improve the first reconstruction model's ability to reconstruct normal events.

[0114] In some embodiments, the first reconstruction model is optimized using the appearance constraint and the motion constraint in S4, and the appearance constraint L app1 Divided into gradient constraints L gc1 With strength constraint L sc1 , the formula is as follows:

[0115]

[0116] Among them, x represents the target object of the input training set, represents the first reconstruction of the target object, a and b represent the horizontal and vertical coordinates of the target object pixels;

[0117] By the gradient constraint L gc1 and strength constraint L sc1 , we get the appearance constraint L app1 The specific calculation method is as follows:

[0118]

[0119] Where m:n=1:1;

[0120] Motion Constraint L opt1 is the optical flow constraint, and the calculation process is as follows:

[0121]

[0122] Where t represents the tth frame;

[0123] The objective function of the first reconstruction model is:

[0124] L1=αL app1 +βL opt1

[0125] Furthermore, in step S5, the second reconstruction model is composed of a generator and a discriminator, and the training method (performed in a parallel computing platform) is: use the first reconstruction target object obtained by the first reconstruction model and the target object of the training set to train the second reconstruction model, so that the reconstructed frame and the real frame are in competition; wherein the second reconstructed target object generated by the first reconstructed frame competes with the training frame corresponding to the first reconstructed target object, the reconstructed target object generated by the target object of the training set competes with the corresponding target object of the training set, and the first reconstructed target object competes with the second reconstructed target object, so as to shorten the distance between each other.

[0126] Furthermore, if Figure 2 As shown, the reconstruction generator in step S5 is composed of an autoencoder and a memory module. The specific adaptive training process is as follows Figure 3 As shown in the figure: the extracted normal training target object and the first reconstructed target object are input into the generator together; firstly, the encoder part of the autoencoder is entered, and after multiple convolution and pooling processes, features of different sizes and different numbers of channels in each layer are obtained, and latent features are obtained at the lowest layer; the latent features are input into the memory module, so that the module learns the normal event pattern and stores it; the latent features enter the decoder part, and the upsampling operation is performed, and the decoder restores the feature map size to correspond to the feature map size of each layer in the encoder and the input map size; in this process, the second reconstructed target object with the same size as the input is obtained by splicing the feature map of each layer with the corresponding decoder output.

[0127] In some embodiments, the memory module in step S5 can memorize and store normal event patterns, and perform query comparison during testing to improve the abnormality detection rate. The operation method is:

[0128] The memory module stores a matrix of size N×C, defined as M∈R N×C , where N means that the memory module contains N valid feature information, and C represents the value of the feature information, which is the same as the potential feature z dimension obtained by the encoder in the generator. The calculation formula is:

[0129]

[0130] Where m i Represents the storage item in the memory module, the vector ω∈R 1×N is the attention coefficient calculated for the latent feature z;

[0131] Using the non-exponential softmax function, the weight ω is calculated i , the calculation formula is as follows:

[0132]

[0133] Where d(·) is the cosine similarity, which is calculated as follows:

[0134]

[0135] Although attention addressing is used, some anomalies may still be passed through m i The complex combination of is well reconstructed. To solve this problem, a hard shrinkage strategy is adopted, which is only effective when the attention weight is greater than the set threshold, otherwise it is 0:

[0136]

[0137] However, this strategy cannot be directly implemented because backpropagation is not possible, so it is reimplemented using the ReLU function:

[0138]

[0139] Where max(·,·) is the ReLU function, ε is a small positive term, and the threshold λ is usually set between [1 / N, 3 / N].

[0140] Normalized weight, its calculation formula is:

[0141]

[0142] The final transformation formula containing the memory module can be obtained, and its calculation formula is as follows:

[0143]

[0144] In some embodiments, the second reconstruction model is optimized using the appearance constraint and the motion constraint in step S5. app2 Divided into gradient constraints L gc2 With strength constraint L sc2 , the formula is as follows:

[0145]

[0146] Where x represents the input video frame, represents the second reconstructed video frame, a and b represent the horizontal and vertical coordinates of the video frame pixels;

[0147] By the gradient constraint L gc2 and strength constraint L sc2 , we get the appearance constraint L app2 The specific calculation method is as follows:

[0148]

[0149] Where m:n=1:1;

[0150] Motion Constraint L opt2 is the optical flow constraint, and the calculation process is as follows:

[0151]

[0152] Where t represents the tth frame;

[0153] In the memory module, the sparsity of ω is further constrained and regularized. The formula is as follows:

[0154]

[0155] Where Q is the number of memory modules, N is the number of memory items in a memory module, ω i,k is the similarity coefficient between the query feature and the memory item;

[0156] The specific optimization process is:

[0157] When training the discriminator, the generator is fixed. The goal is to enable the discriminator to accurately distinguish the second reconstructed frame from the normal training frame to improve the discriminator's discrimination ability. The loss function of the discriminator is as follows:

[0158]

[0159] Where i, j are the indices of the frames, D(·)∈[0,1], and L(·,·) represents the absolute value of the difference between the two;

[0160] When training the generator, the discriminator is fixed. The goal is to make the reconstructed frames generated by the generator and the original training frames indistinguishable from the discriminator to improve the generator's generation ability. The loss function of the generator is as follows:

[0161]

[0162] The training goal of the generator is to reconstruct only normal frames and cannot reconstruct abnormal frames, that is, the generator and discriminator are only optimized for normal frames. The generator function is as follows:

[0163]

[0164] where λ app2 , opt2 , adv and λ m are the weight factors of appearance loss, motion loss, generator loss, and memory module loss respectively;

[0165] The objective function of the discriminator is as follows:

[0166]

[0167] In some embodiments, the specific test process in S6 is: in the test phase, the target object in the test frame is input into the trained first reconstruction model to obtain the first reconstruction frame of the test frame; the first reconstruction frame is input into the trained second reconstruction model to obtain the second reconstruction frame of the test frame.

[0168] In some embodiments, the specific process in S6 is: calculating the difference between the first reconstructed frame and the input test frame to obtain the first reconstruction error, and normalizing it to obtain the first reconstruction score; calculating the difference between the second reconstructed frame and the input test frame to obtain the second reconstruction error, and normalizing it to obtain the second reconstruction score; integrating the information of the first and second reconstructions, taking a weighted sum of the first reconstruction score and the second reconstruction score to obtain a comprehensive reconstruction score, thereby performing abnormal event detection on the input test frame.

[0169] In some embodiments, S6 specifically includes:

[0170] Calculate the first reconstruction error and the second reconstruction error. The calculation formulas for the two are:

[0171]

[0172] Where x′ represents the test sample, represents the first reconstructed frame, represents the second reconstructed frame;

[0173] The reconstruction error is normalized by the normalization method to obtain the first reconstruction score Score(x′)1 and the second reconstruction score Score(x′)2, which are calculated as follows:

[0174]

[0175] Among them, S1 is the total set of the first reconstruction errors of each test frame, and S2 is the total set of the second reconstruction errors of each test frame.

[0176] In some embodiments, the calculation formula of the total reconstruction score in S7 is:

[0177] Score(x′)=λ1Score(x′)1+λ2Score(x′)2

[0178] Among them, λ1 is the weight of the first reconstruction score, and λ2 is the weight of the second reconstruction score.

[0179] When Score(x′) is less than the threshold, it means that only normal events exist in the video frame, otherwise it means that the video frame contains abnormal events.

[0180] A second aspect of an embodiment of the present invention provides a parallel detection device for abnormal video events based on secondary reconstruction, comprising: a parallel computing platform acquisition module, which performs frame splitting on a video to obtain one or more training video frames and test video frames, and divides the frames into a training set and a test set;

[0181] The parallel computing platform target detection module uses a target detection algorithm to extract target regions of interest in the training set and the test set to obtain training target objects and / or test target objects;

[0182] The first training module of the parallel computing platform inputs the target object of the training set into the first reconstruction model, trains the first reconstruction model, and obtains the first reconstruction training target object;

[0183] The second training module of the parallel computing platform inputs the target objects of the first reconstruction training and the target objects of the normal training set into the second reconstruction model to obtain a trained second reconstruction model. Thus, a trained second reconstruction model is obtained;

[0184] The parallel computing platform test module inputs the target object of the test set into the trained reconstruction model to obtain the first reconstructed frame and the second reconstructed frame; calculates the reconstruction error between the reconstructed frame and the real frame to obtain the first reconstruction error and the second reconstruction error. After normalization, the two are weighted and calculated to obtain the total reconstruction score for video frame detection. The two reconstruction scores are weighted and calculated to obtain the total reconstruction score to detect whether there are abnormal events in the video frame.

[0185] The parallel detection device for abnormal video events based on secondary reconstruction provided in the embodiment of the present invention belongs to the same concept as the embodiment of the parallel detection method for abnormal video events based on secondary reconstruction in the above-mentioned embodiment. The specific implementation process is detailed in the method embodiment in the above-mentioned embodiment and will not be repeated here.

[0186] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-mentioned embodiment of the method for parallel detection of abnormal video events based on secondary reconstruction.

[0187] The computer-readable storage medium of the embodiment of the present invention may include any entity or device or recording medium capable of carrying computer program code, for example, ROM / RAM, magnetic disk, optical disk, flash memory and other memories.

[0188] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for parallel detection of abnormal video events based on secondary reconstruction, characterized in that: include: S1: Collect the required video set and preprocess it to obtain one or more training video frames and test video frames; S2: Establish a parallel computing strategy and use it in video abnormal event detection; S3: performing target detection on the training video frame and / or the test video frame to obtain a training target object and / or a test target object; S4: inputting the training target object into the first reconstruction model to train the model, thereby obtaining the first reconstruction target object; S5: inputting the training target object and the first reconstruction target object into a second reconstruction model to train the model, thereby obtaining the second reconstruction target object and constructing a secondary reconstruction model; S6: inputting the test target object into the secondary reconstruction model to obtain a first reconstruction score and / or a second reconstruction score; S7: Perform weighted calculation on the two reconstruction scores to obtain a total reconstruction score, so as to detect whether there is an abnormal event in the video frame.

2. The method for parallel detection of abnormal video events based on secondary reconstruction according to claim 1, characterized in that: The step S3: performing target detection on the training video frame and / or the test video frame to obtain a training target object and / or a test target object, includes: The required surveillance video sets are distributed to different computing nodes in the parallel computing platform. Each computing node runs the target detection algorithm independently, and multiple computing nodes jointly complete the target detection and extraction task.

3. The method for parallel detection of abnormal video events based on secondary reconstruction according to claim 2 is characterized in that: The target detection and extraction task includes: Store the required surveillance video sets in the same path so that the parallel computing platform can access these video data; Create SparkConf and SparkContext. The SparkContext is the entry point of the Spark application, responsible for communicating with the cluster manager, creating parallel computing core components such as RDD, accumulator, and broadcast variables, and providing some commonly used operation functions. The SparkConf is the configuration class of the Spark application, which can set various parameters of the Spark application. Parallelizing the list containing the video data path into the RDD, using the parallel computing platform to divide the surveillance video data, and dividing the large task into multiple subtasks; Dynamically allocate computing nodes according to the number of the divided subtasks, so that each computing node independently runs the target detection and extraction task; After each computing node processes its own subtask, it obtains object information and aggregates the results of the object detection and extraction tasks into a central storage location or data structure; When all subtasks have completed object detection and the results have been merged and processed, the object detection extraction task of the surveillance video is completed and the results are collected back to the driver node.

4. The method for parallel detection of abnormal video events based on secondary reconstruction according to claim 1, characterized in that: The step S4: inputting the training target object into the first reconstruction model to train the model to obtain the first reconstruction target object, including: Extracting a target object of interest from the training video frame by using a YOLOv5 algorithm, where the target object may be a person, a vehicle, or any other object of interest in the video frame; The first reconstruction model is a generative adversarial network structure, which includes a generator and a discriminator part, wherein the generator is composed of an encoder, a decoder, a jump connection part and a memory module; The target object is input into the encoder part of the autoencoder network, and the encoder processes the target object through multiple convolution and pooling operations to reduce the size of the feature map and increase the number of channels to obtain a potential feature map; the feature map is input into the decoder part, and the decoder performs an upsampling operation to restore the size of the feature map to the original size of the input image; The autoencoder network uses jump connections to connect feature maps of different levels, that is, splicing the feature map of each layer with the reconstructed feature map of the corresponding layer, and finally generates a first reconstructed video frame with the same size as the input video frame, that is, the first reconstructed target object.

5. The method for parallel detection of abnormal video events based on secondary reconstruction according to claim 1, characterized in that: S5: inputting the training target object and the first reconstruction target object into a second reconstruction model to train the model, thereby constructing a secondary reconstruction model, including: The second reconstruction model is a generative adversarial network structure, which includes a generator and a discriminator part, wherein the generator is composed of an encoder, a decoder, a jump connection part and a memory module, and the training method of the generator is: Extracting the target object of interest from the training video frame by the YOLOv5 algorithm and inputting the first reconstructed target object into the encoder, and obtaining the first feature map of different sizes and different numbers of channels in each layer after multiple convolution and pooling processes, and obtaining the potential features at the lowest layer; Inputting the latent features into a memory module to learn to model normal behavior and store normal patterns; The latent features are input into a decoder, and the decoder performs an upsampling operation to generate a second feature map. At the same time, the first feature map and the second feature map at the same layer as the second feature map are connected by a jump connection to obtain a new second feature map. Finally, the size of the second feature map is restored to the original size of the input image to obtain a second reconstructed video frame with the same size as the input frame, that is, the second reconstructed target object; The second reconstructed target object is subjected to adversarial training with the training target object, wherein the second reconstructed target object generated by the first reconstructed target object is confronted with the training target object corresponding to the first reconstructed target object, the first reconstructed target object generated by the training target object is confronted with the corresponding training target object, and the first reconstructed target object is confronted with the second reconstructed target object.

6. The method for parallel detection of abnormal video events based on secondary reconstruction according to claim 5 is characterized in that: The step S5: inputting the training target object and the first reconstruction target object into a second reconstruction model to train the model, thereby constructing a secondary reconstruction model, further comprising: The memory module learns to model normal behavior and store normal patterns: The memory module stores a matrix of size N×C, defined as M∈R N×C , where N indicates that the memory module contains N valid feature information, and C indicates the value of the feature information, which is the same as the potential feature z dimension obtained by the encoder, and its calculation formula is: Where m i Represents the storage items in the memory module, the vector ω∈R 1×N is the attention coefficient calculated for the potential feature z; Using the non-exponential softmax function, the weight ω is calculated i , the calculation formula is as follows: Where d(·) is the cosine similarity, which is calculated as follows: The hard shrinkage strategy is adopted, which is only effective when the attention weight is greater than the set threshold, otherwise it is 0: Secondly, the hard shrinkage strategy is re-implemented using the ReLU function: Where max(·,·) is the ReLU function, ε is a very small positive term, and the threshold λ is usually set between [1 / N, 3 / N]; Normalized weight, its calculation formula is: The final transformation formula containing the memory module can be obtained, and its calculation formula is as follows:

7. The method for parallel detection of abnormal video events based on secondary reconstruction according to claim 1, characterized in that: The step S4: inputting the training target object into the first reconstruction model to train the model to obtain the first reconstruction target object, further comprising: Constraints are imposed at the appearance level and the motion level. The appearance constraint L app1 By the gradient constraint L gc1 and strength constraint L sc1 The gradient constraint L gc1 and the strength constraint L sc1 The calculation is as follows: Wherein, x represents the training target object, represents the first reconstructed target object, a, b represent the horizontal and vertical coordinates of the pixels of the training target object; By the gradient constraint L gc1 and the strength constraint L sc1 , get the appearance constraint L app1 , the specific calculation method is as follows: Where m:n=1:1; The motion constraint L opt1 is the optical flow constraint, and the calculation process is as follows: Where t represents the tth frame; The objective function of the first reconstruction model is: L1=αL app1 +βL opt1 。 8. The method for parallel detection of abnormal video events based on secondary reconstruction according to claim 7, characterized in that: The step S5: inputting the training target object and the first reconstruction target object into a second reconstruction model to train the model, thereby constructing a secondary reconstruction model, further comprising: Constraints are imposed at the appearance level and the motion level. The appearance constraint L app2 By the gradient constraint L gc2 and strength constraint L sc2 The motion constraint L opt2 is the optical flow constraint, and the calculation method of both is the same as the appearance constraint L in the first reconstruction model app1 and the motion constraint L opt1 The calculation method is the same as Change to Indicates the second reconstruction of the target object; In the memory module, the sparsity of ω is further constrained and regularized. The formula is as follows: Where Q is the number of memory modules, N is the number of memory items in a memory module, ω i,k is the similarity coefficient between the query feature and the memory item; The specific optimization process is: When training the discriminator, the generator is fixed. The goal is to enable the discriminator to accurately distinguish the second reconstructed frame from the normal training frame to improve the discriminator's discrimination ability. The loss function of the discriminator is as follows: Where i, j are the indices of the frames, D(·)∈[0,1], and L(·,·) represents the absolute value of the difference between the two; When training the generator, the discriminator is fixed. The goal is to make the reconstructed frames generated by the generator and the original training frames indistinguishable from the discriminator to improve the generation ability of the generator. The loss function of the generator is as follows: The training goal of the generator is to reconstruct only normal frames and cannot reconstruct abnormal frames. That is, the generator and discriminator are only optimized for normal frames. The generator function is as follows: where λ app2 , opt2 , adv and λ m are the weight factors of appearance loss, motion loss, generator loss, and memory module loss respectively; The objective function of the discriminator is as follows:

9. The method for parallel detection of abnormal video events based on secondary reconstruction according to claim 1, characterized in that: The step S6: inputting the test target object into the secondary reconstruction model to obtain a first reconstruction score and / or a second reconstruction score includes: The first reconstruction error and the second reconstruction error, the calculation formula of the first reconstruction error and / or the second reconstruction error is: Where x′ represents the test target object, represents the first reconstruction target object, Representing the second reconstruction target object; The first reconstruction error and / or the second reconstruction error are normalized by a normalization method to obtain the first reconstruction score Score(x′)1 and the second reconstruction score Score(x′)2, which are calculated as follows: Among them, S1 is the total set of first reconstruction errors of each of the test target objects, and S2 is the total set of second reconstruction errors of each of the test target objects.

10. The method for parallel detection of abnormal video events based on secondary reconstruction according to claim 1, characterized in that: The step S7: performing weighted calculation on the first reconstruction score and the second reconstruction score to obtain a total reconstruction score, so as to detect whether there is an abnormal event in the video frame, includes: The total reconstruction score is calculated as: Score(x′)=λ1Score(x′)1+λ2Score(x′)2 Wherein, λ1 is the weight of the first reconstruction score, λ2 is the weight of the second reconstruction score, When Score(x′) is less than the threshold, it means that only normal behaviors exist in the video frame, otherwise it means that the video frame contains abnormal behaviors.

Images