Abnormal event detection method, apparatus, device, and computer-readable storage medium

Abstract

The application discloses an abnormal event detection method, device and equipment and a computer readable storage medium, belongs to the field of machine vision, considers that an adversarial autoencoder can perform abnormal detection on an image from two dimensions of reconstruction error and conditional probability, and considers that an abnormal event can be presented in motion features and appearance features of the image, so that the application can train an adversarial autoencoder based on a motion feature map and an appearance feature map of the image respectively, give abnormal scores corresponding to the motion feature map and the appearance feature map of the image to be detected by the two adversarial autoencoders respectively in an inference stage, and finally obtain an abnormal total score for judging the abnormal event, which can not only perform abnormal event analysis on multidimensional features of the image to be detected, but also perform abnormal scoring from two dimensions of reconstruction error and conditional probability by using the adversarial autoencoder, and is beneficial to improving the detection accuracy of the abnormal event in the image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine vision, in particular to an abnormal event detection method, device, equipment and computer readable storage medium. BACKGROUND

[0002] In recent years, image acquisition devices are widely used in various public areas, such as large shopping malls, airports, railway stations, etc., to ensure the safety of people's life and property, thereby bringing a large amount of image monitoring data, and more and more researchers in the field of machine vision are committed to mining effective information from these original image data; among them, abnormal event automatic detection has gradually become one of the research hotspots, but there is a lack of a mature abnormal event detection method in the related technology, which makes it difficult to accurately detect the abnormal events existing in the image.

[0003] Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present. SUMMARY

[0004] The purpose of the present application is to provide an abnormal event detection method, device, equipment and computer readable storage medium, which can not only analyze the multi-dimensional features of the test image, but also use the adversarial autoencoder to score from two dimensions of reconstruction error and conditional probability, which is beneficial to improve the detection accuracy of the abnormal event in the image.

[0005] To solve the above technical problems, the present application provides an abnormal event detection method, comprising:

[0006] The motion feature map of each image in the training image set is taken as an input image, the first adversarial autoencoder is trained, and the first training target and the second training target are achieved, wherein the images in the training image set do not exist abnormal events, the first training target refers to: the data distribution of the hidden layer feature of the input image tends to the preset prior distribution, and the second training target refers to: minimizing the reconstruction error of the reconstructed image of the input image;

[0007] The appearance feature map of each image in the training image set is taken as an input image, the second adversarial autoencoder is trained, and the first training target and the second training target are achieved;

[0008] The sum of the first detection parameter and the second detection parameter of the trained first adversarial autoencoder for the motion feature map of the test image is taken as the motion abnormality score, wherein the first detection parameter refers to the probability that the hidden layer feature of the test image conforms to the preset prior distribution, and the second detection parameter refers to the reconstruction error of the reconstructed image of the test image;

[0009] The sum of the first detection parameter and the second detection parameter of the trained second adversarial autoencoder for the appearance feature map of the to-be-tested image is taken as an appearance anomaly score.

[0010] According to the anomaly total score of the to-be-tested image, a determination result of whether an abnormal event exists in the to-be-tested image is obtained, wherein the anomaly total score refers to the sum of the motion anomaly score and the appearance anomaly score.

[0011] On the other hand, the motion feature map of each image in the training image set is taken as an input image to train the first adversarial autoencoder, and the first training target and the second training target are achieved, including:

[0012] The motion feature map of each image in the training image set is determined.

[0013] The motion feature block of each region in the motion feature map is determined.

[0014] Each motion feature block of each image in the training image set is taken as an input image to train the first adversarial autoencoder, and the first training target and the second training target are achieved.

[0015] The appearance feature map of each image in the training image set is taken as an input image to train the second adversarial autoencoder, and the first training target and the second training target are achieved, including:

[0016] The appearance feature map of each image in the training image set is determined.

[0017] The appearance feature block of each region in the appearance feature map is determined.

[0018] Each appearance feature block of each image in the training image set is taken as an input image to train the second adversarial autoencoder, and the first training target and the second training target are achieved.

[0019] Before taking the sum of the first detection parameter and the second detection parameter of the trained first adversarial autoencoder for the motion feature map of the to-be-tested image as a motion anomaly score, the anomaly event detection method further includes:

[0020] Each region of the specified image is taken as a to-be-tested image.

[0021] On the other hand, the adversarial autoencoder includes an encoder, a decoder, and a discriminator;

[0022] The adversarial autoencoder includes the first adversarial autoencoder and the second adversarial autoencoder.

[0023] The encoder is used to map an input image to a hidden layer feature.

[0024] The decoder is configured to reconstruct the input image according to the implicit layer feature.

[0025] The discriminator is configured to determine a probability that the input image conforms to the preset prior distribution.

[0026] In another aspect, the adversarial autoencoder further includes:

[0027] a memory module configured to compensate the implicit layer feature according to a target similarity, wherein the target similarity refers to a similarity between the implicit layer feature and a feature of a pre-stored typical normal event image.

[0028] The decoder is specifically configured to reconstruct the input image according to the compensated implicit layer feature.

[0029] In another aspect, the memory module is specifically configured to:

[0030] For any unit data of the implicit layer feature, a similarity between the unit data and unit data corresponding in position in the feature of the pre-stored typical normal event image is determined as a unit similarity.

[0031] Each of the unit similarities is binarized.

[0032] For any unit similarity after binarization, a product of a specific unit data and the unit similarity is taken as to-be-fused unit data, wherein the specific unit data refers to unit data corresponding to the unit similarity in the feature of the typical normal event image.

[0033] A feature sequentially spliced by the to-be-fused unit data is taken as the compensated implicit layer feature.

[0034] In another aspect, the objective function of the first training target includes:

[0035] ;

[0036] wherein En represents an identifier of the encoder, De represents an identifier of the decoder, represents that a function value of the objective function is minimized by optimizing network parameters of En, represents that a function value of the objective function is minimized by optimizing network parameters of De, is an input image, is a data distribution, is a data distribution of represents a data distribution condition, represents conforms to , The compensated implicit layer feature, The reconstructed input image, Refers to the Euclidean distance formula.

[0037] In another aspect, the second training objective of the target function includes:

[0038] ;

[0039] Wherein, Di is the identifier of the discriminator, Indicates that the network parameters are optimized to minimize the function value of the objective function, Indicates that the network parameters are optimized to maximize the function value of the objective function, Is a feature vector randomly sampled from a preset prior distribution, Indicates a preset prior distribution, Indicates the data distribution condition, Indicates Comply , Indicates the probability determined by the discriminator Comply The probability determined by the discriminator, Indicates that the encoder maps the input image to the implicit layer feature, Indicates the probability determined by the discriminator Comply The probability determined by the discriminator.

[0040] To solve the above technical problems, the present application also provides an abnormal event detection device, comprising:

[0041] The first training module is used for training the first adversarial autoencoder by taking the motion feature map of each image in the training image set as the input image, and achieving the first training objective and the second training objective, wherein the images in the training image set do not have abnormal events, the first training objective refers to making the data distribution of the implicit layer feature of the input image tend to the preset prior distribution, and the second training objective refers to minimizing the reconstruction error of the reconstructed image of the input image.

[0042] The second training module is used for training the second adversarial autoencoder by taking the appearance feature map of each image in the training image set as the input image, and achieving the first training objective and the second training objective.

[0043] a first scoring module configured to take the sum of a first detection parameter and a second detection parameter of the trained first adversarial autoencoder for the motion feature map of the to-be-tested image as a motion anomaly score, wherein the first detection parameter refers to a probability that the hidden layer feature of the to-be-tested image conforms to the preset prior distribution, and the second detection parameter refers to a reconstruction error of the reconstructed image of the to-be-tested image;

[0044] a second scoring module configured to take the sum of a first detection parameter and a second detection parameter of the trained second adversarial autoencoder for the appearance feature map of the to-be-tested image as an appearance anomaly score;

[0045] a result deriving module configured to derive a determination result of whether an abnormal event exists in the to-be-tested image according to an anomaly total score of the to-be-tested image, wherein the anomaly total score refers to the sum of the motion anomaly score and the appearance anomaly score.

[0046] To solve the above technical problems, the present application further provides an abnormal event detection device, comprising:

[0047] a memory configured to store a computer program;

[0048] a processor configured to implement the steps of the above abnormal event detection method when the computer program is executed.

[0049] To solve the above technical problems, the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above abnormal event detection method.

[0050] Advantages: The present application provides an abnormal event detection method, which considers that an adversarial autoencoder can perform abnormal detection on an image from two dimensions of a reconstruction error and a conditional probability, and also considers that an abnormal event can be presented in a motion feature and an appearance feature of an image. Therefore, the present application can train one adversarial autoencoder based on a motion feature map and an appearance feature map of an image respectively, and give an abnormal score corresponding to the motion feature map and the appearance feature map of a to-be-tested image by two adversarial autoencoders respectively in an inference stage. Finally, an anomaly total score is obtained for the determination of an abnormal event. The present application can not only analyze an abnormal event of a multi-dimensional feature of a to-be-tested image, but also use an adversarial autoencoder to perform abnormal scoring from two dimensions of a reconstruction error and a conditional probability, which is beneficial to improving the detection accuracy of an abnormal event in an image.

[0051] The present application further provides an abnormal event detection apparatus, device and computer readable storage medium, which have the same advantages as the above abnormal event detection method. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required by the related art and the embodiments. Obviously, the drawings described below are only some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0053] Figure 1 A flowchart of an abnormal event detection method provided by the present application is shown in the figure.

[0054] Figure 2 A flowchart of another abnormal event detection method provided by the present application is shown in the figure.

[0055] Figure 3 A flowchart of an image segmentation method provided by the present application is shown in the figure.

[0056] Figure 4 A structure diagram of a generative adversarial network provided by the present application is shown in the figure.

[0057] Figure 5 A structure diagram of a memory module provided by the present application is shown in the figure.

[0058] Figure 6 A structure diagram of an abnormal event detection device provided by the present application is shown in the figure.

[0059] Figure 7 A structure diagram of an abnormal event detection device provided by the present application is shown in the figure.

[0060] Figure 8 A structure diagram of a computer readable storage medium provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0061] The core of the present application is to provide an abnormal event detection method, device, equipment and computer readable storage medium, which can not only analyze the multi-dimensional features of the image to be tested, but also use the generative adversarial network to score the abnormality from two dimensions of reconstruction error and conditional probability, thereby improving the detection accuracy of the abnormal event in the image.

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the following will describe the technical solutions in the embodiments of the present application clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0063] Please refer to Figure 1 ,Figure 1 A flowchart of an abnormal event detection method provided by the present application is shown in the figure. The abnormal event detection method comprises the following steps:

[0064] S101: The motion feature maps of the images in the training image set are taken as input images to train the first adversarial autoencoder and achieve the first training target and the second training target. The images in the training image set do not have abnormal events. The first training target means that the data distribution of the hidden layer features of the input images tends to the preset prior distribution. The second training target means that the reconstruction error of the reconstructed images of the input images is minimized.

[0065] Specifically, considering the technical problems in the background art, and in combination with the consideration of (1) the adversarial autoencoder can perform abnormal event detection on images from two dimensions of reconstruction error and conditional probability, and (2) abnormal events can be presented in the motion features and the appearance features of images, the embodiments of the present application train one adversarial autoencoder for each of the motion feature maps and the appearance feature maps of the images in the training image set (which do not have abnormal events). For the training process of each adversarial autoencoder, two training targets are included, i.e., “minimizing the reconstruction error of the reconstructed images of the input images” and “making the data distribution of the hidden layer features of the input images tend to the preset prior distribution”. Thus, in the inference stage, the abnormal event detection on the test images can also be performed from the two aspects of motion features and appearance features, which is beneficial to improving the detection accuracy.

[0066] Specifically, in this step, the motion feature maps of the images in the training image set are first taken as input images to train the first adversarial autoencoder and achieve the first training target and the second training target.

[0067] The training image set can include a series of images, which can belong to the same specified scene and do not have abnormal events (i.e., all have normal events), such as images of normal events such as “people walking freely and entering and leaving a store” in a shopping mall scene. The embodiments of the present application do not limit this.

[0068] S102: The appearance feature maps of the images in the training image set are taken as input images to train the second adversarial autoencoder and achieve the first training target and the second training target.

[0069] Specifically, in the same way as the first adversarial autoencoder, the second adversarial autoencoder is trained by using the appearance feature maps of the images in the training image set, and the first training target and the second training target are achieved. The difference between the first adversarial autoencoder and the second adversarial autoencoder is that the feature maps used for training are different. The feature maps used for training the second adversarial autoencoder are appearance feature maps, and the feature maps used for training the first adversarial autoencoder are motion feature maps. The appearance feature refers to the characteristics or features exhibited by an object or a living body in appearance, which can be directly observed by vision or other senses. The motion feature refers to the representation of the motion characteristics of the object in the image.

[0070] S103: The sum of the first detection parameter and the second detection parameter of the trained first adversarial autoencoder for the motion feature map of the to-be-detected image is taken as a motion anomaly score, wherein the first detection parameter refers to the probability that the hidden layer feature of the to-be-detected image conforms to the preset prior distribution, and the second detection parameter refers to the reconstruction error of the reconstructed image of the to-be-detected image.

[0071] Specifically, due to the first training target of the adversarial autoencoder during training, the adversarial autoencoder can make the data distribution of the hidden layer feature of the image with a normal event tend to the preset prior distribution, but cannot make the data distribution of the hidden layer feature of the image with an abnormal event tend to the preset prior distribution. Therefore, the first detection parameter can be obtained when the to-be-detected image is detected. In addition, due to the second training target of the adversarial autoencoder during training, the adversarial autoencoder can minimize the reconstruction error of the reconstructed image of the image with a normal event, that is, the adversarial autoencoder is difficult to well reconstruct the image with an abnormal event (the reconstruction error is large). Therefore, the second detection parameter can also be obtained when the to-be-detected image is detected. The sum of the first detection parameter and the second detection parameter is taken as a motion anomaly score, and the motion anomaly score is taken as the data basis for the subsequent step.

[0072] S104: The sum of the first detection parameter and the second detection parameter of the trained second adversarial autoencoder for the appearance feature map of the to-be-detected image is taken as an appearance anomaly score.

[0073] Specifically, as in the logic of S103, the first detection parameter and the second detection parameter of the appearance feature map of the to-be-detected image can also be obtained by using the trained second adversarial autoencoder. The sum of the first detection parameter and the second detection parameter is taken as an appearance anomaly score, which is taken as the data basis for the subsequent step.

[0074] S105: According to the total anomaly score of the to-be-detected image, a determination result of whether an abnormal event exists in the to-be-detected image is obtained, wherein the total anomaly score refers to the sum of the motion anomaly score and the appearance anomaly score.

[0075] Specifically, the motion abnormality score can be analyzed from the perspective of "reconstruction of the motion feature map" and "distribution of the motion feature map data approaching the preset prior distribution", and the appearance abnormality score can be analyzed from the perspective of "reconstruction of the appearance feature map" and "distribution of the appearance feature map data approaching the preset prior distribution", and the abnormality total score obtained by adding the motion abnormality score and the appearance abnormality score can reflect "whether the abnormal event exists in the to-be-tested image" from the above perspectives, so that the determination result of whether the abnormal event exists in the to-be-tested image can be obtained according to the abnormality total score of the to-be-tested image in the embodiment of the application.

[0076] The determination result of whether the abnormal event exists in the to-be-tested image according to the abnormality total score of the to-be-tested image can specifically include judging whether the abnormality total score is greater than a preset score value, and if yes, determining that the abnormal event exists in the to-be-tested image, and if not, determining that the abnormal event does not exist in the to-be-tested image.

[0077] The application provides an abnormal event detection method, which considers that the adversarial autoencoder can perform abnormal event detection on the image from two dimensions of reconstruction error and conditional probability, and considers that the abnormal event can be presented in the motion feature and the appearance feature of the image, so that the application can train one adversarial autoencoder based on the motion feature map and the appearance feature map of the image respectively, and give the abnormality scores corresponding to the motion feature map and the appearance feature map of the to-be-tested image in the inference stage through the two adversarial autoencoders, and finally obtain the abnormality total score for the determination of the abnormal event, which not only can perform abnormal event analysis on the multidimensional features of the to-be-tested image, but also can use the adversarial autoencoder to perform abnormal scoring from two dimensions of reconstruction error and conditional probability, and is beneficial to improving the detection accuracy of the abnormal event in the image.

[0078] On the basis of the above embodiment:

[0079] As an optional embodiment, the motion feature map of each image in the training image set is taken as an input image to train the first adversarial autoencoder, and the first training target and the second training target are achieved, which include:

[0080] The motion feature map of each image in the training image set is determined;

[0081] The motion feature block of each region in the motion feature map is determined;

[0082] Each motion feature block of each image in the training image set is taken as an input image to train the first adversarial autoencoder, and the first training target and the second training target are achieved;

[0083] The apparent feature maps of the images in the training image set are taken as input images to train the second adversarial autoencoder, and the first training target and the second training target are achieved, which includes:

[0084] The apparent feature maps of the images in the training image set are determined.

[0085] The apparent feature blocks of each region in the apparent feature map are determined.

[0086] The apparent feature blocks of each image in the training image set are taken as input images to train the second adversarial autoencoder, and the first training target and the second training target are achieved.

[0087] Before the sum of the first detection parameter and the second detection parameter of the trained first adversarial autoencoder for the motion feature map of the to-be-detected image is taken as the motion anomaly score, the anomaly event detection method further includes:

[0088] The image blocks of each region in the specified image are taken as the to-be-detected image.

[0089] In order to better illustrate the embodiments of the present application, please refer to Figure 2 and Figure 3 , Figure 2 The flowchart of another anomaly event detection method provided by the present application is shown in Figure 3 The flowchart of an image segmentation method provided by the present application is shown in

[0090] Specifically, in Figure 2 , the training phase and the test phase are shown. The flow of the training phase can be represented by a solid black line with an arrow, and the flow of the test phase can be represented by a line segment with an arrow and “attached dots and lines”. In the training phase, the optical flow map (a kind of motion feature map) and the spatiotemporal gradient map (a kind of apparent feature map) can be obtained based on the training image set, respectively. Then, the optical flow feature blocks are obtained by segmenting the optical flow map, and the gradient feature blocks are obtained by segmenting the spatiotemporal gradient map. The first adversarial autoencoder can be trained by the optical flow feature blocks (i.e., the related weights are optimized by the loss function), and the second adversarial autoencoder can be trained by the gradient feature blocks (i.e., the related weights are optimized by the loss function). In the test phase, the spatiotemporal gradient map and the optical flow map are obtained based on the to-be-detected image set (including a plurality of specified images), and the gradient feature blocks are obtained by segmenting the spatiotemporal gradient map, and the optical flow feature blocks are obtained by segmenting the optical flow map. The motion anomaly scores corresponding to the optical flow feature blocks are obtained by the first adversarial autoencoder, and the apparent anomaly scores corresponding to the gradient feature blocks are obtained by the second adversarial autoencoder. Finally, the anomaly detection result of the to-be-detected image can be obtained by fusion (summing the motion anomaly scores and the apparent anomaly scores and determining).

[0091] Specifically, considering that a single image contains too much information, if only the presence of an abnormal event in the single image is determined, the specific position of the abnormal event in the image also needs to be identified by the staff, the positioning accuracy of the abnormal event is poor, and the user experience is reduced, therefore, in the embodiment of the present application, the motion feature blocks of the motion feature maps of each image in the training image set are taken as input images to train the first adversarial autoencoder, and the appearance feature blocks of the motion feature maps of each image in the training image set are taken as input images to train the second adversarial autoencoder; and the image patches of each region in the specified image are taken as the to-be-tested images, so that the abnormal event detection of the image patches of each region in the single image can be realized, which is beneficial to improving the positioning accuracy of the abnormal event and improving the user experience.

[0092] The specified image refers to a single image to be detected, and the source of the specified image can be various, for example, can be a photo taken by a camera, can be a single frame image in a video stream of a video camera, or can be an image given by a user through a human-computer interaction device, etc., which is not limited in the embodiment of the present application.

[0093] Specifically, specifically, Figure 3 The image segmentation process is shown in FIG. 1. The image to be segmented can include images in the training image set in the training and the specified image used in the test stage. When the motion feature blocks of each region of the images in the training image set are determined, the motion feature map (for example, the optical flow map) of the image can be determined first, and then the motion feature blocks (for example, the optical flow feature blocks) of each region in the motion feature map can be determined through the preset size sliding window. When the appearance feature blocks of each region of the images in the training image set are determined, the appearance feature map (for example, the spatiotemporal gradient map or the gradient feature map) of the image can be determined first, and then the appearance feature blocks (for example, the gradient feature blocks) of each region in the appearance feature map can be determined through the preset size sliding window. When the image patches of each region of the specified image are determined, the image patches of each region in the specified image can be determined through the preset size sliding window.

[0094] Specifically, the spatiotemporal gradient map of each image can contain three channels. The first two channels record the gradient values of the image in the horizontal direction and the vertical direction, describe the pose and shape of the target, and the third channel includes the gradient value of the image in the time direction, which describes the change of the appearance feature of the target with time. The derivation process of the optical flow map can be as follows: the horizontal direction optical flow value of each pixel point in the image is calculated by using the Lucas-Kanade optical flow method The vertical direction optical flow value of each pixel point in the image is calculated by using the Lucas-Kanade optical flow method Then, the optical flow amplitude value of each pixel point is calculated to compose the optical flow map of the image.

[0095] In addition, it is worth mentioning that, considering that there may be "blank" areas without objects in the image itself, such as large areas of floor, etc., the motion features and apparent features of these areas are relatively scarce, in order to increase the positioning accuracy of abnormal events and reduce the calculation amount, in the embodiment of the application, after determining the motion feature blocks of each area in the motion feature map, the motion feature blocks of each image in the training image set are taken as input images, the first adversarial autoencoder is trained, and before achieving the first training target and the second training target, the abnormal event detection method can further include:

[0096] The motion feature blocks meeting the first preset condition are retained, and the motion feature blocks not meeting the first preset condition are removed, wherein the first preset condition means that the motion feature values corresponding to at least a preset proportion (for example, 10%) of pixel points in the motion feature block are not zero;

[0097] After determining the apparent feature blocks of each area in the apparent feature map, the apparent feature blocks of each image in the training image set are taken as input images, the second adversarial autoencoder is trained, and before achieving the first training target and the second training target, the abnormal event detection method further includes:

[0098] The apparent feature blocks meeting the second preset condition are retained, and the apparent feature blocks not meeting the second preset condition are removed, wherein the second preset condition means that the apparent feature values corresponding to at least a preset proportion (for example, 10%) of pixel points in the apparent feature block are not zero;

[0099] The sum of the first detection parameter and the second detection parameter of the trained first adversarial autoencoder for the motion feature map of the to-be-detected image is taken as the motion abnormality score.

[0100] The sum of the first detection parameter and the second detection parameter of the trained first adversarial autoencoder for the motion feature map of the to-be-detected image meeting the first preset condition is taken as the motion abnormality score.

[0101] The sum of the first detection parameter and the second detection parameter of the trained second adversarial autoencoder for the apparent feature map of the to-be-detected image is taken as the apparent abnormality score.

[0102] The sum of the first detection parameter and the second detection parameter of the trained second adversarial autoencoder for the apparent feature map of the to-be-detected image meeting the second preset condition is taken as the apparent abnormality score.

[0103] The preset size can be autonomously set, and the preset size can include the height (letter h) and width (letter w) of the sliding window, etc., which are not limited in the embodiment of the application.

[0104] As an optional embodiment, the adversarial autoencoder comprises an encoder, a decoder and a discriminator;

[0105] The adversarial autoencoder comprises a first adversarial autoencoder and a second adversarial autoencoder.

[0106] The encoder is configured to map the input image to the hidden layer feature.

[0107] The decoder is configured to reconstruct the input image according to the hidden layer feature.

[0108] The discriminator is configured to determine a probability that the input image conforms to the preset prior distribution.

[0109] Specifically, the adversarial autoencoder in the embodiment of the present application has a simple structure and can quickly and efficiently determine the first detection parameter and the second detection parameter of the input image.

[0110] Of course, in addition to the specific structure, the adversarial autoencoder can also have other specific structures, which are not limited in the embodiment of the present application.

[0111] As an optional embodiment, the adversarial autoencoder further comprises:

[0112] The memory module is configured to compensate the hidden layer feature according to a target similarity, wherein the target similarity refers to a similarity between the hidden layer feature and a feature of a pre-stored typical normal event image.

[0113] The decoder is specifically configured to reconstruct the input image according to the compensated hidden layer feature.

[0114] In order to better illustrate the embodiment of the present application, please refer to Figure 4 , Figure 4 The figure is a structural diagram of an adversarial autoencoder provided by the present application.

[0115] Specifically, the adversarial autoencoder in the embodiment of the present application can comprise three neural networks: an encoder En, a decoder De and a discriminator Di. In order to extract deep features from the original feature blocks, each convolutional layer (1st to 3rd convolutional layers) in the encoder En adopts a 2D convolutional layer, each convolutional layer (4th to 6th convolutional layers) in the decoder De corresponds to use a 2D deconvolutional layer, and a fully connected layer (1st to 3rd fully connected layers) can be used in the discriminator Di, as shown in Figure 4 .

[0116] Specifically, En includes 3 layers of 2D convolution layers (2D-Conv1), and the output of each layer is subjected to batch normalization (BN, Batch Normalization) processing, and on this basis, each layer needs to pass through a linear rectifier (LReLU, Leaky Rectified Linear Unit) before obtaining the final output result, except for the last layer. En can map the original feature block (i.e. the input image of the adversarial autoencoder) to the hidden layer feature z; then, z is input into the memory module, which includes a memory unit for recording the features of a typical normal event image and an attention-based addresser (i.e. a memory addressing module), which has the function of storing the features of a typical normal event image, as shown in Figure 5

[0117] Specifically, De includes 3 layers of 2D deconvolution layers, and the hidden layer feature z is compensated by the memory module according to the target similarity to obtain (compensated hidden layer feature), and then is reconstructed into an input image; Di includes 3 layers of fully connected layers, and the 3rd fully connected layer uses a sigmoid (normalized) activation function to make the output result value range between 0 and 1. It is worth noting that the network structure in the apparent flow and the motion flow is consistent.

[0118] wherein, Figure 4 the structure of the adversarial autoencoder is introduced in the motion feature block (such as the gradient feature block) as an example, Figure 4 z~p(z) in the above formula means that the hidden layer feature z conforms to the preset prior distribution p(z), and z * ~p(z) means that the feature z conforms to the preset prior distribution p(z) * .

[0119] Specifically, the adversarial autoencoder in the embodiment of the present application can be called MemMGFC-AAE (Memory-augmented Multivariate Gaussian Fully Convolution Adversarial Autoencoder) according to its function structure, wherein the memory augmentation means that the adversarial autoencoder has a memory module, the multivariate Gaussian means that the preset prior distribution in the adversarial autoencoder can be a multivariate Gaussian distribution, and the full convolution means that the neural network in the adversarial autoencoder can adopt a convolutional neural network, except for the discriminator.

[0120] Of course, in addition to the multivariate Gaussian distribution, the preset prior distribution can also be other types, which are not limited in the embodiment of the present application.

[0121] ​Specifically, considering that even if the images in the training image set used in the training stage do not have abnormal events, but in the inference stage, the strong generalization ability of the neural network can also be able to better reconstruct the 'images with abnormal events', which is not conducive to accurate detection of abnormal events, in order to solve this problem, the adversarial autoencoder in the embodiment of the application can also include a memory module, the memory module is used to compensate the hidden layer features according to the target similarity (which refers to the similarity between the hidden layer features and the features of the pre-stored typical normal event image), and then the decoder is specifically used for: reconstructing the input image according to the compensated hidden layer features; in this case, if the to-be-tested image has abnormal events, then the similarity between the hidden layer features of the to-be-tested image and the features of the typical normal event image is poor, if the to-be-tested image does not have abnormal events, then the similarity between the hidden layer features of the to-be-tested image and the features of the typical normal event image is high, that is, the compensation degree of the memory module for the hidden layer features of the 'images with abnormal events' and the 'images without abnormal events' is different, thereby helping to resist the strong generalization ability of the neural network in the adversarial autoencoder, and being conducive to improving the precision of abnormal event detection.

[0122] Among them, the typical normal event image refers to: an image with a typical normal event, the typical normal event corresponds to the scene to which the image belongs, for example, the typical normal event of 'personnel normally entering and exiting the store' in the shopping mall, etc., which is not limited in the embodiment of the application.

[0123] As an optional embodiment, the memory module is specifically used for:

[0124] For any unit data of the hidden layer features, the similarity between the unit data and the unit data corresponding to the position in the pre-stored features of the typical normal event image is determined as a unit similarity;

[0125] Each unit similarity is binarized;

[0126] For any unit similarity after binarization, the product of the specific unit data and the unit similarity is taken as the to-be-fused unit data, wherein the specific unit data refers to the unit data in the features of the typical normal event image corresponding to the unit similarity;

[0127] The features sequentially spliced by the to-be-fused unit data are taken as the compensated hidden layer features.

[0128] In order to better illustrate the embodiment of the application, please refer to Figure 5 , Figure 5 The structure diagram of a memory module provided by the application.

[0129] Specifically, in Figure 5In the figure, the memory addressing module can first determine, for any unit data of the hidden layer feature, the unit data corresponding to the unit data position in the pre-stored feature of the typical normal event image (that is, in the hidden layer feature and the feature of the typical normal event image, each unit data is one-to-one corresponding according to the same position rule), and then can determine, for any unit data of the hidden layer feature, the similarity between the unit data and the unit data corresponding to the position in the pre-stored feature of the typical normal event image as the unit similarity, and the W in the figure is a set of each unit similarity, and each small square in the W represents a unit similarity of a unit data, and then the W is binarized to obtain , each small square in the W is a unit similarity after binarization; then, for any unit similarity after binarization, the product of the specific unit data and the unit similarity is taken as the to-be-fused unit data, and finally the feature formed by sequentially splicing the to-be-fused unit data can be taken as the compensated hidden layer feature.

[0130] Specifically, in order to better compensate the hidden layer feature, in the embodiment of the application, the similarity between the unit data and the unit data corresponding to the position in the pre-stored feature of the typical normal event image can be determined for any unit data of the hidden layer feature as the unit similarity; then each unit similarity is binarized, and for any unit similarity after binarization, the product of the specific unit data and the unit similarity is taken as the to-be-fused unit data, and finally the feature formed by sequentially splicing the to-be-fused unit data can be taken as the compensated hidden layer feature, which can more fully compensate the hidden layer feature according to the target similarity.

[0131] The specific process of binarizing the unit similarity can include: judging whether the unit similarity is greater than a pre-set similarity threshold, if yes, setting the unit similarity to 1, and if not, setting the unit similarity to 0.

[0132] Of course, in addition to the above specific manner, the compensation of the hidden layer feature according to the target similarity can also be other specific manners, which are not limited in the embodiment of the application.

[0133] As an optional embodiment, the target function of the first training target includes:

[0134] ;

[0135] wherein, En is the identifier of the encoder, De is the identifier of the decoder, indicates that the function value of the target function is minimized by optimizing the network parameters of En, minimizing the function value of the objective function by optimizing the network parameters of the encoder, for the input image, for the data distribution, for the data distribution, denotes the data distribution condition, denotes complies with , for the compensated hidden layer feature, for the reconstructed input image, refers to the Euclidean distance formula.

[0136] Specifically, through the objective function of the first training target as above, the network parameters of the encoder can be adjusted efficiently and accurately, so that the data distribution of the hidden layer feature of the input image tends to the preset prior distribution.

[0137] Of course, in addition to the specific form as above, the objective function of the first training target can also be of other specific types, which are not limited by the embodiments of the present application.

[0138] As an optional embodiment, the objective function of the second training target comprises:

[0139] ;

[0140] wherein Di is the identifier of the discriminator, minimizing the function value of the objective function by optimizing the network parameters of the encoder, maximizing the function value of the objective function by optimizing the network parameters of the discriminator, for the feature vector randomly sampled from the preset prior distribution, denotes the preset prior distribution, denotes the data distribution condition, denotes complies with , denotes the probability determined by the discriminator that complies with , denotes the mapping of the input image to the hidden layer feature by the encoder, denotes the probability determined by the discriminator that complies with .

[0141] Specifically, through the objective function of the second training target as above, the network parameters of the decoder can be adjusted efficiently and accurately, so as to minimize the reconstruction error of the reconstructed image of the input image.

[0142] Of course, in addition to the specific form as above, the objective function of the second training target can also be other specific types, and the embodiments of the present application are not limited here.

[0143] Specifically, the training process of the adversarial autoencoder in the embodiments of the present application (taking the second adversarial autoencoder corresponding to the apparent flow as an example) can be:

[0144] The gradient feature block set with normal events for training is represented as , is the gradient feature block in the set, a is the identifier of the apparent feature, i = 1, 2, 3, …, K, K is the number of gradient feature blocks in the set. For the gradient feature block , its data distribution is represented as . The hidden layer feature mapped by the encoder is , n is the preset dimension of the hidden layer feature, is a natural number.

[0145] The decoder takes (hidden layer feature compensated via the memory module) as input to reconstruct the gradient feature block . is the reconstructed gradient feature block . The reconstructed gradient feature block should be consistent with the original input, so the reconstruction objective function is a problem of minimizing the difference between input and output, and the Euclidean distance is used for calculation:

[0146] ;

[0147] At the same time, the encoder and the discriminator are trained in an adversarial manner to achieve the purpose that z conforms to the preset prior distribution , is the probability that the vector z is sampled from (i.e., conforms to) the preset prior distribution . Assuming represents a feature vector randomly sampled from the preset prior distribution , represents the probability that the vector is derived from . The objective function is defined as:

[0148] ;

[0149] In particular, the detailed network training process is as follows: first, in order to ensure that the gradient feature block based on the "implicit layer feature compensated by the memory module" has the normal event lossless reconstruction, the network parameters of the encoder, the memory module and the decoder are updated by minimizing the reconstruction loss in the following step (2); second, in order to ensure that the encoder can confuse the discriminator, the network parameters of the encoder are updated by maximizing the probability that the implicit layer representation z conforms to the preset prior distribution p(z) in steps (4) and (6). Step (5) updates the network parameters of the discriminator by minimizing the binary cross entropy, which increases the probability that z conforms to p(z) and reduces the probability that z conforms to p(z) to distinguish the distribution of the real sample z * from the distribution of the encoder generated sample z. *

[0150] wherein the input is: : a set of gradient feature blocks for training, m1 to m R The set is divided into R small batches, L: the number of gradient feature blocks in each small batch (batch size). N: the number of training iterations. The output is the network parameters of the encoder, the memory module, the decoder and the discriminator 、 、 and .

[0151] Step (1): the pth small batch is represented as wherein q is the starting sequence number of the gradient feature block in the pth small batch;

[0152] Step (2): update the network parameters of the encoder, the memory module and the decoder 、 and : ;

[0153] Step (3): randomly sample L n-dimensional feature vectors from the preset prior distribution p(z), represented as ;

[0154] Step (4): update the network parameters of the encoder : ;

[0155] Step (5): update the network parameters of the discriminator : ;

[0156] Step (6): update the network parameters of the encoder again : .

[0157] ​Specifically, the flow of the test (inference) stage of the abnormal event detection method in the embodiment of the present application is as follows:

[0158] Given a test image block , first, its corresponding optical flow feature block and gradient feature block are extracted, and then the trained MemMGFC-AAE is used to obtain its hidden layer representation in the dual-flow structure as follows:

[0159] ;

[0160] ;

[0161] wherein and are the hidden layers in the motion flow and the appearance flow respectively, and are the encoders of the trained MemMGFC-AAE in the motion flow and the appearance flow respectively.

[0162] With the hidden layer representation as the input, the first detection parameter in the motion flow and the first detection parameter in the appearance flow of the image block are calculated using the probability density function of the prior multivariate Gaussian distribution as follows:

[0163] ; ;

[0164] wherein n is the dimension of and , is the n×1-dimensional mean vector of the preset prior distribution , and is the n×n-dimensional covariance matrix.

[0165] After the compensation of and by the memory module, the following are obtained: and , and the reconstructed feature blocks in the motion flow and the appearance flow are obtained:

[0166] ;

[0167] ;

[0168] wherein and are the reconstructed input feature blocks (motion feature blocks or appearance feature blocks) in the motion flow and the appearance flow respectively, with respectively represent the decoders of the trained MemMGFC-AAE in the motion stream and the appearance stream.

[0169] The reconstruction error is used as the second detection parameter, wherein is the second detection parameter in the motion stream, is the second detection parameter in the appearance stream, as follows:

[0170] ;

[0171] ;

[0172] The final motion anomaly score is obtained by combining the anomaly scores based on the conditional probability and the reconstruction error and the appearance anomaly score :

[0173] ;

[0174] ;

[0175] The weight and The appearance anomaly score and the motion anomaly score are fused to obtain the total anomaly score , as follows:

[0176] ;

[0177] Finally, when the following formula is established, the test image block is determined to be abnormal:

[0178] ;

[0179] wherein, is a preset threshold.

[0180] Please refer to Figure 6 , Figure 6 is a structure diagram of an abnormal event detection device provided by the application, and the abnormal event detection device comprises:

[0181] The first training module 61 is configured to train the first adversarial autoencoder by taking the motion feature map of each image in a training image set as an input image, and to achieve a first training target and a second training target, wherein the images in the training image set do not have abnormal events, the first training target means that the data distribution of the hidden layer features of the input image tends to approach a preset prior distribution, and the second training target means to minimize the reconstruction error of the reconstructed image of the input image.

[0182] The second training module 62 is configured to train the second adversarial autoencoder by taking the appearance feature map of each image in the training image set as an input image, and achieve the first training target and the second training target.

[0183] The first scoring module 63 is configured to take the sum of the first detection parameter and the second detection parameter of the motion feature map of the to-be-detected image by the trained first adversarial autoencoder as a motion anomaly score, wherein the first detection parameter refers to a probability that the hidden layer feature of the to-be-detected image conforms to a preset prior distribution, and the second detection parameter refers to a reconstruction error of the reconstructed image of the to-be-detected image.

[0184] The second scoring module 64 is configured to take the sum of the first detection parameter and the second detection parameter of the appearance feature map of the to-be-detected image by the trained second adversarial autoencoder as an appearance anomaly score.

[0185] The result deriving module 65 is configured to derive a determination result of whether an abnormal event exists in the to-be-detected image according to an anomaly total score of the to-be-detected image, wherein the anomaly total score refers to the sum of the motion anomaly score and the appearance anomaly score.

[0186] On the basis of the above embodiment:

[0187] As an optional embodiment, the first training module 61 comprises:

[0188] The first determining module is configured to determine the motion feature map of each image in the training image set.

[0189] The second determining module is configured to determine the motion feature block of each region in the motion feature map.

[0190] The first training submodule is configured to train the first adversarial autoencoder by taking each motion feature block of each image in the training image set as an input image, and achieve the first training target and the second training target.

[0191] The second training module 62 comprises:

[0192] The third determining module is configured to determine the appearance feature map of each image in the training image set.

[0193] The fourth determining module is configured to determine the appearance feature block of each region in the appearance feature map.

[0194] The second training submodule is configured to train the second adversarial autoencoder by taking each appearance feature block of each image in the training image set as an input image, and achieve the first training target and the second training target.

[0195] The abnormal event detection device further comprises:

[0196] The first action module is configured to divide the image of each region in the specified image into image blocks as the image to be detected.

[0197] The abnormal event detection device provided by the embodiments of the present application is described above with reference to the embodiments of the abnormal event detection method, and the embodiments of the present application will not be described here again.

[0198] Please refer to Figure 7 , Figure 7 A structural diagram of an abnormal event detection device provided by the present application is shown in the figure, which comprises:

[0199] The memory 71 is configured to store a computer program.

[0200] The processor 72 is configured to execute the computer program to realize the steps of the abnormal event detection method in the foregoing embodiments.

[0201] The abnormal event detection device provided by the embodiments of the present application is described above with reference to the embodiments of the abnormal event detection method, and the embodiments of the present application will not be described here again.

[0202] Please refer to Figure 8 , Figure 8 A structural diagram of a computer readable storage medium provided by the present application is shown in the figure, and the computer readable storage medium 81 stores a computer program, and the computer program 82 is executed by a processor to realize the steps of the abnormal event detection method in the foregoing embodiments.

[0203] The computer readable storage medium provided by the embodiments of the present application is described above with reference to the embodiments of the abnormal event detection method, and the embodiments of the present application will not be described here again.

[0204] The present application also provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to realize the steps of the abnormal event detection method in the foregoing embodiments.

[0205] The computer program product provided by the embodiments of the present application is described above with reference to the embodiments of the abnormal event detection method, and the embodiments of the present application will not be described here again.

[0206] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section. It should also be noted that in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0207] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An abnormal event detection method characterized by, The method comprises: training a first adversarial autoencoder by taking the motion feature map of each image in a training image set as an input image, and achieving a first training target and a second training target, wherein the images in the training image set do not have abnormal events, the first training target refers to making the data distribution of the hidden layer feature of the input image tend to a preset prior distribution, and the second training target refers to minimizing the reconstruction error of a reconstructed image of the input image; training a second adversarial autoencoder by taking the appearance feature map of each image in the training image set as an input image, and achieving the first training target and the second training target; taking the sum of a first detection parameter and a second detection parameter of the motion feature map of a to-be-detected image by the trained first adversarial autoencoder as a motion abnormality score, wherein the first detection parameter refers to the probability that the hidden layer feature of the to-be-detected image conforms to the preset prior distribution, and the second detection parameter refers to the reconstruction error of a reconstructed image of the to-be-detected image; taking the sum of a first detection parameter and a second detection parameter of the appearance feature map of a to-be-detected image by the trained second adversarial autoencoder as an appearance abnormality score; obtaining a determination result of whether an abnormal event exists in the to-be-detected image according to an overall abnormality score of the to-be-detected image, wherein the overall abnormality score refers to the sum of the motion abnormality score and the appearance abnormality score.

2. The abnormal event detection method according to claim 1, characterized in that, The method comprises: determining the motion feature map of each image in the training image set; determining the motion feature block of each region in the motion feature map; training the first adversarial autoencoder by taking each motion feature block of each image in the training image set as an input image, and achieving the first training target and the second training target; The method comprises: determining the appearance feature map of each image in the training image set; determining the appearance feature block of each region in the appearance feature map; training the second adversarial autoencoder by taking each appearance feature block of each image in the training image set as an input image, and achieving the first training target and the second training target; The method further comprises: taking the image block of each region in a specified image as a to-be-detected image.

3. The abnormal event detection method of claim 1, wherein, The adversarial autoencoder comprises an encoder, a decoder, and a discriminator; The adversarial autoencoder comprises the first adversarial autoencoder and the second adversarial autoencoder; The encoder is configured to map an input image into a hidden layer feature; The decoder is configured to reconstruct the input image according to the hidden layer feature; The discriminator is configured to determine the probability that the input image conforms to the preset prior distribution.

4. The abnormal event detection method according to claim 3, characterized in that, The adversarial autoencoder further comprises: a memory module configured to compensate the hidden layer feature according to a target similarity, wherein the target similarity refers to a similarity between the hidden layer feature and a feature of a pre-stored typical normal event image; the decoder is specifically configured to reconstruct the input image according to the compensated hidden layer feature.

5. The abnormal event detection method according to claim 4, characterized in that, the memory module is specifically configured to: for each unit data of the hidden layer feature, determine a similarity between the unit data and unit data corresponding in position in the feature of the pre-stored typical normal event image as a unit similarity; perform binary processing on each unit similarity; for each unit similarity after the binary processing, take a product of a specific unit data and the unit similarity as a to-be-fused unit data, wherein the specific unit data refers to unit data corresponding to the unit similarity in the feature of the typical normal event image; take a feature sequentially spliced by the to-be-fused unit data as the compensated hidden layer feature.

6. The abnormal event detection method according to claim 4, characterized in that, the objective function of the first training target includes: ； wherein En is an identity of the encoder, De is an identity of the decoder, denotes minimizing a function value of the objective function by optimizing network parameters of En, denotes minimizing a function value of the objective function by optimizing network parameters of De, is an input image, is a data distribution, is a data distribution of En, denotes a data distribution condition, denotes complies with , is a compensated hidden layer feature, is a reconstructed input image, refers to a Euclidean distance formula.

7. The abnormal event detection method of claim 4, wherein, the objective function of the second training target includes: ； wherein Di is an identifier of the discriminator, denotes minimizing the function value of the objective function by optimizing the network parameters of En, denotes maximizing the function value of the objective function by optimizing the network parameters of Di, is a feature vector arbitrarily sampled from a preset prior distribution, denotes a preset prior distribution, denotes a data distribution condition, denotes complies with , denotes the probability that the discriminator determines complies with , denotes that the encoder maps the input image to the hidden layer feature, denotes the probability that the discriminator determines complies with .

8. An abnormal event detection apparatus characterized by comprising: including: a first training module configured to take a motion feature map of each image in a training image set as an input image, train the first adversarial autoencoder, and achieve a first training target and a second training target, wherein the images in the training image set are free of abnormal events, the first training target refers to making a data distribution of a hidden layer feature of the input image tend to a pre-set prior distribution, and the second training target refers to minimizing a reconstruction error of a reconstructed image of the input image; a second training module configured to take an appearance feature map of each image in the training image set as an input image, train the second adversarial autoencoder, and achieve the first training target and the second training target; a first scoring module configured to take a sum of a first detection parameter and a second detection parameter of a trained first adversarial autoencoder for a motion feature map of a to-be-tested image as a motion abnormality score, wherein the first detection parameter refers to a probability that a hidden layer feature of the to-be-tested image conforms to the pre-set prior distribution, and the second detection parameter refers to a reconstruction error of a reconstructed image of the to-be-tested image; a second scoring module configured to take a sum of a first detection parameter and a second detection parameter of a trained second adversarial autoencoder for an appearance feature map of a to-be-tested image as an appearance abnormality score; a result deriving module configured to derive a determination result of whether an abnormal event exists in the to-be-tested image according to an abnormality total score of the to-be-tested image, wherein the abnormality total score refers to a sum of the motion abnormality score and the appearance abnormality score.

9. An abnormal event detection device characterized by comprising: including: a memory configured to store a computer program; a processor configured to implement steps of the abnormal event detection method in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, the computer program stored on the computer readable storage medium is executed by the processor to implement steps of the abnormal event detection method in any one of claims 1 to 7.

Images