Abnormal event detection method, device and equipment and computer readable storage medium
By detecting the motion and apparent features of the image from the two dimensions of reconstruction error and conditional probability by adversarial autoencoder, the total abnormality score is calculated, and the problem of low detection accuracy of abnormal events in the prior art is solved, and the abnormal events in the image is realized efficient detection.
Patent Information
- Application Number
- CN202411161468.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-22
Smart Images

Figure CN119068254A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine vision, and in particular to an abnormal event detection method, device, equipment and computer-readable storage medium. Background Art
[0002] In recent years, image acquisition equipment has been widely used in various public areas, such as large shopping malls, airports, railway stations, etc., to ensure the safety of people's lives and property, which has brought about a huge amount of image monitoring data. More and more researchers in the field of machine vision are committed to mining effective information from these raw image data; among them, automatic detection of abnormal events has gradually become one of the research hotspots. However, there is a lack of a mature abnormal event detection method in related technologies, which makes it difficult to accurately detect abnormal events in images.
[0003] Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present. Summary of the invention
[0004] The purpose of the present invention is to provide an abnormal event detection method, device, equipment and computer-readable storage medium, which can not only perform abnormal event analysis on the multi-dimensional features of the image to be tested, but also use the adversarial autoencoder to perform abnormal scoring from the two dimensions of reconstruction error and conditional probability, which is beneficial to improve the detection accuracy of abnormal events in the image.
[0005] In order to solve the above technical problems, the present invention provides an abnormal event detection method, comprising:
[0006] Using the motion feature map of each image in the training image set as an input image, training the first adversarial autoencoder, and achieving a first training goal and a second training goal, wherein the images in the training image set do not have abnormal events, the first training goal refers to: making the data distribution of the hidden layer features of the input image approach a preset prior distribution, and the second training goal refers to: minimizing the reconstruction error of the reconstructed image of the input image;
[0007] Using the apparent feature map of each image in the training image set as an input image, training the second adversarial autoencoder, and achieving the first training goal and the second training goal;
[0008] The sum of the first detection parameter and the second detection parameter of the motion feature map of the image to be tested by the trained first adversarial autoencoder is used as the motion anomaly score, wherein the first detection parameter refers to: the probability that the hidden layer feature of the image to be tested conforms to the preset prior distribution, and the second detection parameter refers to: the reconstruction error of the reconstructed image of the image to be tested;
[0009] The sum of the first detection parameter and the second detection parameter of the apparent feature map of the image to be tested by the trained second adversarial autoencoder is used as the apparent abnormality score;
[0010] A determination result of whether an abnormal event exists in the image to be tested is obtained according to the total abnormality score of the image to be tested, wherein the total abnormality score refers to the sum of the motion abnormality score and the apparent abnormality score.
[0011] On the other hand, the motion feature map of each image in the training image set is used as an input image to train the first adversarial autoencoder, and achieving the first training goal and the second training goal includes:
[0012] Determine a motion feature map of an image in a training image set;
[0013] Determine the motion feature blocks of each area in the motion feature map;
[0014] Using each of the motion feature blocks of each image in the training image set as an input image, training the first adversarial autoencoder, and achieving the first training goal and the second training goal;
[0015] Taking the apparent feature map of each image in the training image set as an input image, training the second adversarial autoencoder, and achieving the first training goal and the second training goal includes:
[0016] Determining an apparent feature map of an image in a training image set;
[0017] Determine the apparent feature blocks of each region in the apparent feature map;
[0018] Using each of the appearance feature blocks of each image in the training image set as an input image, training the second adversarial autoencoder, and achieving the first training goal and the second training goal;
[0019] Before using the sum of the first detection parameter and the second detection parameter of the motion feature map of the image to be detected by the trained first adversarial autoencoder as the motion anomaly score, the abnormal event detection method further includes:
[0020] The image blocks in each area of the specified image are used as the image to be tested.
[0021] On the other hand, the adversarial autoencoder includes an encoder, a decoder, and a discriminator;
[0022] Wherein, the adversarial autoencoder includes the first adversarial autoencoder and the second adversarial autoencoder;
[0023] The encoder is used to map the input image into hidden layer features;
[0024] The decoder is used to reconstruct the input image according to the hidden layer features;
[0025] The discriminator is used to determine the probability that the input image conforms to the preset prior distribution.
[0026] On the other hand, the adversarial autoencoder further comprises:
[0027] A memory module, used for compensating the hidden layer features according to target similarity, wherein the target similarity refers to: the similarity between the hidden layer features and the features of the pre-stored typical normal event image;
[0028] The decoder is specifically used to reconstruct the input image according to the compensated hidden layer features.
[0029] On the other hand, the memory module is specifically used for:
[0030] For any unit data of the hidden layer feature, determining the similarity between the unit data and the unit data corresponding to the position in the pre-stored feature of the typical normal event image as the unit similarity;
[0031] Binarization processing is performed on each of the unit similarities;
[0032] For any of the unit similarities after binarization, the product of specific unit data and the unit similarity is used as the unit data to be fused, wherein the specific unit data refers to: the unit data corresponding to the unit similarity in the features of the typical normal event image;
[0033] The features formed by sequentially concatenating the unit data to be fused are used as the compensated hidden layer features.
[0034] On the other hand, the objective function of the first training objective includes:
[0035] ;
[0036] Wherein, En is the identifier of the encoder, De is the identifier of the decoder, It means that the function value of the objective function is minimized by optimizing the network parameters of En. It means that the function value of the objective function is minimized by optimizing the network parameters of De. is the input image, is the data distribution, for The data distribution, represents the data distribution condition, express conform to , is the hidden layer feature after compensation, is the reconstructed input image, Refers to the Euclidean distance formula.
[0037] On the other hand, the objective function of the second training objective includes:
[0038] ;
[0039] Wherein, Di is the identifier of the discriminator, It means that the function value of the objective function is minimized by optimizing the network parameters of En. It means that the function value of the objective function is maximized by optimizing the network parameters of Di. is a feature vector randomly sampled from a preset prior distribution, represents the preset prior distribution, represents the data distribution condition, express conform to , Denotes the discriminator determined conform to The probability of Indicates that the encoder maps the input image to hidden layer features. Indicates that the judgement determines conform to probability.
[0040] In order to solve the above technical problems, the present invention also provides an abnormal event detection device, comprising:
[0041] A first training module is used to use the motion feature map of each image in the training image set as an input image, train the first adversarial autoencoder, and achieve a first training goal and a second training goal, wherein the images in the training image set do not have abnormal events, the first training goal is to make the data distribution of the hidden layer features of the input image approach a preset prior distribution, and the second training goal is to minimize the reconstruction error of the reconstructed image of the input image;
[0042] A second training module, configured to use the apparent feature map of each image in the training image set as an input image, train the second adversarial autoencoder, and achieve the first training objective and the second training objective;
[0043] A first scoring module is used to use the sum of a first detection parameter and a second detection parameter of the motion feature map of the image to be tested by the trained first adversarial autoencoder as a motion anomaly score, wherein the first detection parameter refers to: a probability that the hidden layer feature of the image to be tested conforms to the preset prior distribution, and the second detection parameter refers to: a reconstruction error of a reconstructed image of the image to be tested;
[0044] A second scoring module is used to use the sum of the first detection parameter and the second detection parameter of the apparent feature map of the image to be tested by the trained second adversarial autoencoder as an apparent abnormality score;
[0045] The result obtaining module is used to obtain a determination result of whether an abnormal event exists in the image to be tested according to the total abnormality score of the image to be tested, wherein the total abnormality score refers to the sum of the motion abnormality score and the apparent abnormality score.
[0046] In order to solve the above technical problems, the present invention also provides an abnormal event detection device, comprising:
[0047] Memory for storing computer programs;
[0048] A processor is used to implement the steps of the abnormal event detection method as described above when executing the computer program.
[0049] In order to solve the above technical problem, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above abnormal event detection method are implemented.
[0050] Beneficial effect: The present invention provides an abnormal event detection method. Considering that an adversarial autoencoder can detect abnormalities in images from two dimensions, namely, reconstruction error and conditional probability, and considering that abnormal events can be reflected in the motion features and appearance features of the image, the present invention can train an adversarial autoencoder based on the motion feature map and the appearance feature map of the image respectively, and in the inference stage, two adversarial autoencoders are used to respectively give the abnormal scores corresponding to the motion feature map and the appearance feature map of the image to be tested, and finally the total abnormal score is obtained for the determination of abnormal events. Not only can the multi-dimensional features of the image to be tested be analyzed for abnormal events, but the adversarial autoencoder can also be used to score abnormalities from two dimensions, namely, reconstruction error and conditional probability, which is beneficial to improving the detection accuracy of abnormal events in the image.
[0051] The present invention also provides an abnormal event detection device, equipment and computer-readable storage medium, which have the same beneficial effects as the above abnormal event detection method. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the relevant technologies and the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0053] Figure 1 A schematic diagram of a flow chart of an abnormal event detection method provided by the present invention;
[0054] Figure 2 A schematic diagram of a flow chart of another abnormal event detection method provided by the present invention;
[0055] Figure 3 A schematic diagram of a process of image segmentation provided by the present invention;
[0056] Figure 4 A schematic diagram of the structure of an adversarial autoencoder provided by the present invention;
[0057] Figure 5 A schematic diagram of the structure of a memory module provided by the present invention;
[0058] Figure 6 A schematic diagram of the structure of an abnormal event detection device provided by the present invention;
[0059] Figure 7 A schematic diagram of the structure of an abnormal event detection device provided by the present invention;
[0060] Figure 8 A schematic diagram of the structure of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION
[0061] The core of the present invention is to provide an abnormal event detection method, device, equipment and computer-readable storage medium, which can not only perform abnormal event analysis on the multi-dimensional features of the image to be tested, but also use the adversarial autoencoder to score abnormalities from two dimensions: reconstruction error and conditional probability, which is beneficial to improving the detection accuracy of abnormal events in the image.
[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0063] Please refer to Figure 1 , Figure 1 A flowchart of an abnormal event detection method provided by the present invention is provided, and the abnormal event detection method comprises:
[0064] S101: using the motion feature map of each image in the training image set as an input image, training the first adversarial autoencoder, and achieving a first training goal and a second training goal, wherein there are no abnormal events in the images in the training image set, the first training goal is to make the data distribution of the hidden layer features of the input image approach a preset prior distribution, and the second training goal is to minimize the reconstruction error of the reconstructed image of the input image;
[0065] Specifically, considering the technical problems in the above background technology, and considering that (1) the adversarial autoencoder can detect abnormalities in images from two dimensions: reconstruction error and conditional probability, and (2) abnormal events can be reflected in the motion features and appearance features of the image, the embodiment of the present invention intends to train an adversarial autoencoder through the motion feature map and appearance feature map of each image in the training image set (without abnormal events), and the training process of any adversarial autoencoder includes two training goals: "minimizing the reconstruction error of the reconstructed image of the input image" and "making the data distribution of the hidden layer features of the input image approach the preset prior distribution". Therefore, in the inference stage, abnormal event detection can also be performed on the image to be tested from two aspects: motion features and appearance features, which is beneficial to improving the detection accuracy.
[0066] Specifically, in this step, the motion feature map of each image in the training image set can be first used as an input image to train the first adversarial autoencoder, and the first training goal and the second training goal can be achieved.
[0067] Among them, the training image set may include a series of images, which may belong to the same specified scene and do not have abnormal events (that is, they only have normal events), such as images of normal events such as "people walking freely and entering and exiting stores" in a shopping mall scene, etc. The embodiments of the present invention are not limited here.
[0068] S102: using the apparent feature map of each image in the training image set as an input image, training the second adversarial autoencoder, and achieving the first training goal and the second training goal;
[0069] Specifically, compared with the training logic type of the first adversarial autoencoder mentioned above, in this step, the appearance feature map of each image in the training image set can be used to train the second adversarial autoencoder and achieve the first training goal and the second training goal. The difference from the first adversarial autoencoder is that the feature maps used for training are different. The feature map used for the second adversarial autoencoder training is an appearance feature map, and the feature map used for the first adversarial autoencoder training is a motion feature map, wherein the appearance feature refers to the characteristics or features of an object or organism in appearance, which can be directly observed through vision or other senses; and the motion feature refers to the representation of the motion characteristics of the object in the image.
[0070] S103: taking the sum of the first detection parameter and the second detection parameter of the motion feature map of the image to be tested by the trained first adversarial autoencoder as the motion anomaly score, wherein the first detection parameter refers to: the probability that the hidden layer feature of the image to be tested conforms to the preset prior distribution, and the second detection parameter refers to: the reconstruction error of the reconstructed image of the image to be tested;
[0071] Specifically, due to the first training goal of the adversarial autoencoder during training, the adversarial autoencoder can make the data distribution of the hidden layer features of images with normal events approach the preset prior distribution, but cannot make the data distribution of the hidden layer features of images with abnormal events approach the preset prior distribution. Therefore, the first detection parameter can be obtained when detecting the image to be tested. Due to the second training goal of the adversarial autoencoder during training, the adversarial autoencoder can minimize the reconstruction error of the reconstructed image of the image with normal events. In other words, it is difficult for the adversarial autoencoder to reconstruct the image with abnormal events well (the reconstruction error is large). Therefore, this step can also obtain the second detection parameter when detecting the image to be tested, and use the sum of the first detection parameter and the second detection parameter as the motion anomaly score, and use the motion anomaly score as the data basis for subsequent steps.
[0072] S104: taking the sum of the first detection parameter and the second detection parameter of the apparent feature map of the image to be tested by the trained second adversarial autoencoder as the apparent abnormality score;
[0073] Specifically, as in the logic of S103, the first detection parameter and the second detection parameter of the apparent feature map of the image to be tested can also be obtained through the trained second adversarial autoencoder, and the sum of the first detection parameter and the second detection parameter is used as the apparent abnormality score, which serves as the data basis for subsequent steps.
[0074] S105: Obtaining a determination result of whether an abnormal event exists in the image to be tested according to the total abnormality score of the image to be tested, wherein the total abnormality score refers to the sum of the motion abnormality score and the appearance abnormality score.
[0075] Specifically, the motion abnormality score can be used to analyze the motion feature map of the image to be tested from the perspectives of "reconstructing the motion feature map" and "making the data distribution of the motion feature map approaching the preset prior distribution", while the apparent abnormality score can be used to analyze the apparent feature map of the image to be tested from the perspectives of "reconstructing the apparent feature map" and "making the data distribution of the apparent feature map approaching the preset prior distribution". The total abnormality score obtained by adding the motion abnormality score and the apparent abnormality score can reflect "whether there is an abnormal event in the image to be tested" from the above perspectives. Therefore, in an embodiment of the present invention, a determination result of whether there is an abnormal event in the image to be tested can be obtained based on the total abnormality score of the image to be tested.
[0076] Among them, according to the total abnormality score of the image to be tested, the determination result of whether there is an abnormal event in the image to be tested can specifically include judging whether the total abnormality score is greater than a preset score, if so, judging that there is an abnormal event in the image to be tested, if not, judging that there is no abnormal event in the image to be tested.
[0077] The present invention provides an abnormal event detection method. Considering that an adversarial autoencoder can detect abnormalities in images from two dimensions, namely, reconstruction error and conditional probability, and considering that abnormal events can be reflected in the motion features and appearance features of the image, the present invention can train an adversarial autoencoder based on the motion feature map and the appearance feature map of the image respectively, and in the inference stage, the motion feature map and the appearance feature map of the image to be tested are respectively given by two adversarial autoencoders. The abnormal scores corresponding to the motion feature map and the appearance feature map of the image to be tested are finally obtained, and the total abnormal score is used for the determination of abnormal events. Not only can the multi-dimensional features of the image to be tested be analyzed for abnormal events, but the adversarial autoencoder can also be used to score abnormalities from two dimensions, namely, reconstruction error and conditional probability, which is beneficial to improving the detection accuracy of abnormal events in the image.
[0078] Based on the above embodiments:
[0079] As an optional embodiment, the motion feature map of each image in the training image set is used as an input image to train the first adversarial autoencoder, and achieving the first training goal and the second training goal includes:
[0080] Determine a motion feature map of an image in a training image set;
[0081] Determine the motion feature blocks of each region in the motion feature map;
[0082] Using each motion feature block of each image in the training image set as an input image, training the first adversarial autoencoder, and achieving the first training goal and the second training goal;
[0083] The apparent feature map of each image in the training image set is used as an input image to train the second adversarial autoencoder, and achieving the first training goal and the second training goal includes:
[0084] Determining an apparent feature map of an image in a training image set;
[0085] Determine the apparent feature blocks of each region in the apparent feature map;
[0086] Taking each apparent feature block of each image in the training image set as an input image, training the second adversarial autoencoder, and achieving the first training goal and the second training goal;
[0087] Before using the sum of the first detection parameter and the second detection parameter of the motion feature map of the image to be tested by the trained first adversarial autoencoder as the motion anomaly score, the abnormal event detection method further includes:
[0088] The image blocks in each area of the specified image are used as the image to be tested.
[0089] To better illustrate the embodiments of the present invention, please refer to Figure 2 and Figure 3 , Figure 2 A flowchart of another abnormal event detection method provided by the present invention is shown below. Figure 3 A schematic diagram of an image segmentation process provided by the present invention.
[0090] Specifically, in Figure 2 The training phase and the testing phase are shown in FIG. 1 . The process of the training phase can be represented by a solid black line with an arrow, and the process of the testing phase can be represented by a line segment with an arrow and "attached with points and lines". In the training phase, the optical flow map (a type of motion feature map) and the spatiotemporal gradient map (a type of appearance feature map) can be obtained based on the training image set, and then the optical flow map is segmented to obtain various optical flow feature blocks, and the spatiotemporal gradient map is segmented to obtain various gradient feature blocks; the first adversarial autoencoder can be trained through the optical flow feature block (that is, the relevant weights are optimized through the loss function), and the gradient feature block is used to train the first adversarial autoencoder. The second adversarial autoencoder is trained (that is, the relevant weights are optimized through the loss function); in the testing phase, the spatiotemporal gradient map and the optical flow map are respectively obtained based on the image set to be tested (including several specified images), and the spatiotemporal gradient map is segmented to obtain gradient feature blocks, and the optical flow map is segmented to obtain optical flow feature blocks. The motion anomaly score corresponding to each optical flow feature block is obtained by the first adversarial autoencoder, and the apparent anomaly score corresponding to each gradient feature block is obtained by the second adversarial autoencoder. Finally, the anomaly detection result of the image to be tested can be obtained by fusion (summing and judging the motion anomaly score and the apparent anomaly score).
[0091] Specifically, considering that a single image contains too much information, if it is determined that there is an abnormal event in a single image, the staff is still required to identify the specific location of the abnormal event in the image. The positioning accuracy of the abnormal event is poor and the user experience is reduced. Therefore, in an embodiment of the present invention, the motion feature block of the motion feature map of each image in the training image set can be used as the input image to train the first adversarial autoencoder, and the apparent feature block of the motion feature map of each image in the training image set can be used as the input image to train the second adversarial autoencoder; and the image blocks of each area in the specified image are used as the image to be tested, so that abnormal event detection can be achieved for the image blocks of each area in a single image, which is beneficial to improving the positioning accuracy of abnormal events and improving the user experience.
[0092] Among them, the specified image refers to a single image to be detected. The source of the specified image can be various, for example, it can be a photo taken by a camera, a single frame image in the video stream of a camera, or an image given by the user through a human-computer interaction device, etc. The embodiment of the present invention is not limited here.
[0093] Specific, specific, Figure 3 , the image segmentation process is shown in , the images to be segmented may include images in the training image set during training and designated images used in the test phase, and when "determining the motion feature blocks of each region of the image in the training image set", the motion feature map of the image (for example, an optical flow map) can be determined first, and then the motion feature blocks of each region in the motion feature map (which can be optical flow feature blocks) can be determined through a sliding window of a preset size; when "determining the apparent feature blocks of each region of the image in the training image set", the apparent feature map of the image (for example, a spatiotemporal gradient map or a gradient feature map) can be determined first, and then the apparent feature blocks of each region in the apparent feature map (which can be gradient feature blocks) can be determined through a sliding window of a preset size; and when "determining the image blocks of each region of the designated image", the image blocks of each region in the designated image can be determined through a sliding window of a preset size.
[0094] Specifically, the spatiotemporal gradient map of each image can contain three channels. The first two channels record the gradient values of the image in the horizontal and vertical directions, describing the posture and shape of the target. The third channel includes the gradient value of the image in the time direction, which describes the changes of the target's apparent characteristics over time. The process of obtaining the optical flow map can be: the Lucas-Kanade optical flow method is used to calculate the horizontal optical flow value of each pixel in the image. The vertical optical flow value . Then calculate the optical flow amplitude of each pixel Optical flow map of the composed image.
[0095] In addition, it is worth mentioning that, considering that there may be "blank areas without objects" in the image itself, such as large areas of floor, etc., the motion features and appearance features of these areas are relatively sparse. In order to increase the accuracy of abnormal event positioning and reduce the amount of calculation, in the embodiment of the present invention, after determining the motion feature blocks of each area in the motion feature map, each motion feature block of each image in the training image set is used as an input image to train the first adversarial autoencoder, and before achieving the first training goal and the second training goal, the abnormal event detection method may also include:
[0096] The motion feature blocks meeting the first preset condition are retained, and the motion feature blocks not meeting the first preset condition are eliminated, wherein the first preset condition refers to: the motion feature values corresponding to at least a preset proportion (e.g., 10%) of the pixels in the motion feature block are not zero;
[0097] After determining the apparent feature blocks of each region in the apparent feature map, taking each apparent feature block of each image in the training image set as an input image, training the second adversarial autoencoder, and before achieving the first training goal and the second training goal, the abnormal event detection method further includes:
[0098] The apparent feature blocks meeting the second preset condition are retained, and the apparent feature blocks not meeting the second preset condition are eliminated, wherein the second preset condition refers to: the apparent feature values corresponding to at least a preset proportion (e.g., 10%) of the pixels in the apparent feature block are not zero;
[0099] The sum of the first detection parameter and the second detection parameter of the motion feature map of the image to be tested by the trained first adversarial autoencoder is used as the motion anomaly score including:
[0100] The sum of the first detection parameter and the second detection parameter of the motion feature map of the image to be tested "meeting the first preset condition" of the trained first adversarial autoencoder is used as a motion anomaly score;
[0101] The sum of the first detection parameter and the second detection parameter of the apparent feature map of the image to be tested by the trained second adversarial autoencoder is taken as the apparent abnormality score:
[0102] The sum of the first detection parameter and the second detection parameter of the apparent feature map of the image to be tested "meeting the second preset condition" trained by the second adversarial autoencoder is taken as the apparent abnormality score.
[0103] The preset size may be set independently, and may include the height (marked with letter h) and width (marked with letter w) of the sliding window, etc., which is not limited in the embodiment of the present invention.
[0104] As an optional embodiment, the adversarial autoencoder includes an encoder, a decoder, and a discriminator;
[0105] Wherein, the adversarial autoencoder includes a first adversarial autoencoder and a second adversarial autoencoder;
[0106] The encoder is used to map the input image to hidden layer features;
[0107] The decoder is used to reconstruct the input image based on the hidden layer features;
[0108] The discriminator is used to determine the probability that the input image conforms to the preset prior distribution.
[0109] Specifically, the adversarial autoencoder in the embodiment of the present invention has a simple structure and can quickly and efficiently determine the first detection parameter and the second detection parameter of the input image.
[0110] Of course, in addition to this specific structure, the adversarial autoencoder may also be other specific structures, which are not limited in the embodiments of the present invention.
[0111] As an optional embodiment, the adversarial autoencoder further includes:
[0112] A memory module is used to compensate the hidden layer features according to the target similarity, wherein the target similarity refers to the similarity between the hidden layer features and the features of the pre-stored typical normal event image;
[0113] The decoder is specifically used to reconstruct the input image according to the compensated hidden layer features.
[0114] To better illustrate the embodiments of the present invention, please refer to Figure 4 , Figure 4 A schematic diagram of the structure of an adversarial autoencoder provided by the present invention.
[0115] Specifically, the adversarial autoencoder in the embodiment of the present invention may include three neural networks: an encoder En, a decoder De, and a discriminator Di. In order to extract deep features from the original feature blocks, each convolution layer (the 1st to the 3rd convolution layer) in the encoder En uses a 2D convolution layer, each convolution layer (the 4th to the 6th convolution layer) in the decoder De uses a 2D deconvolution layer, and the discriminator Di may use a fully connected layer (the 1st to the 3rd fully connected layer), such as Figure 4 shown.
[0116] Specifically, En contains 3 layers of 2D convolutional layers (2D-Conv1), and the output of each layer is batch normalized (BN). On this basis, except for the last layer, each layer needs to pass through linear rectification (LReLU, Leaky Rectified Linear Unit) before obtaining the final output result. En can map the original feature block (that is, the input image of the adversarial autoencoder) to the hidden layer feature z; then, z is input into the memory module, which contains a memory unit for recording the features of typical normal event images and an attention-based addresser (that is, a memory addressing module), which has the function of storing the features of typical normal event images, such as Figure 5 shown.
[0117] Specifically, De contains three 2D deconvolution layers, and the memory module compensates the hidden layer feature z according to the target similarity. (hidden layer features after compensation) Reconstructed as the input image; Di contains 3 fully connected layers, and the third fully connected layer uses a sigmoid (normalized) activation function to make the output value range between 0 and 1. It is worth noting that the network structure in the apparent flow is consistent with that in the motion flow.
[0118] in, Figure 4 The structure of the adversarial autoencoder is introduced by taking the "motion feature block (such as gradient feature block)" as an example. Figure 4 The z~p(z) in the above sentence means that the hidden layer feature z conforms to the preset prior distribution p(z), z * ~p(z) refers to the feature z that conforms to the preset prior distribution p(z) * .
[0119] Specifically, the adversarial autoencoder in the embodiment of the present invention can be called MemMGFC-AAE (Memory-augmented Multivariate Gaussian Fully Convolution AdversarialAutoencoder) according to its functional structure, wherein memory enhancement means that the adversarial autoencoder has a memory module, multivariate Gaussian means that the preset prior distribution in the adversarial autoencoder can be a multivariate Gaussian distribution, and full convolution means that in addition to the discriminator, the neural network in the adversarial autoencoder can adopt a convolutional neural network.
[0120] Of course, in addition to the multivariate Gaussian distribution, the preset prior distribution may also be of other types, which is not limited in the embodiment of the present invention.
[0121] Specifically, considering that even if the images in the training image set used in the training stage do not have abnormal events, in the inference stage, the strong generalization ability of the neural network may also reconstruct the "images with abnormal events" better, which is not conducive to the accurate detection of abnormal events. In order to deal with this problem, the adversarial autoencoder in the embodiment of the present invention may also include a memory module, which is used to compensate the hidden layer features according to the target similarity (referring to the similarity between the hidden layer features and the features of the pre-stored typical normal event images). Then the decoder is specifically used to: reconstruct the input image according to the compensated hidden layer features; in this case, if there is an abnormal event in the image to be tested, then the similarity between the hidden layer features of the image to be tested and the features of the typical normal event image is poor, if there is no abnormal event in the image to be tested, then the similarity between the hidden layer features of the image to be tested and the features of the typical normal event image is high, that is, the memory module has different compensation degrees for the hidden layer features of the images to be tested that "have abnormal events" and "do not have abnormal events", which helps to enhance the strong generalization ability of the neural network in the adversarial autoencoder and improve the accuracy of abnormal event detection.
[0122] Among them, the typical normal event image refers to: an image with a typical normal event, and the typical normal event corresponds to the scene to which the image belongs, such as the typical normal event of "people entering and exiting a store normally" in a shopping mall, etc. The embodiment of the present invention is not limited here.
[0123] As an optional embodiment, the memory module is specifically used for:
[0124] For any unit data of the hidden layer feature, determine the similarity between the unit data and the unit data corresponding to the position in the pre-stored typical normal event image feature as the unit similarity;
[0125] Binarize the similarity of each unit;
[0126] For any unit similarity after binarization, the product of the specific unit data and the unit similarity is used as the unit data to be fused, wherein the specific unit data refers to: the unit data corresponding to the unit similarity in the features of the typical normal event image;
[0127] The features formed by sequentially concatenating the unit data to be fused are used as the compensated hidden layer features.
[0128] To better illustrate the embodiments of the present invention, please refer to Figure 5 , Figure 5 A schematic structural diagram of a memory module provided by the present invention.
[0129] Specifically, in Figure 5In the example, the memory addressing module can first determine the unit data corresponding to the unit data position of the hidden layer feature from the pre-stored typical normal event image features for any unit data of the hidden layer feature (that is, in the hidden layer features and the features of the typical normal event image, each unit data corresponds one to one according to the same position rule), and then for any unit data of the hidden layer feature, determine the similarity between the unit data and the unit data corresponding to the position in the pre-stored typical normal event image features as the unit similarity. In the figure, W is the set of each unit similarity, and each small square of W in the figure represents the unit similarity of a unit data. Then, W is binarized to obtain , Each small square in the image is labeled with a unit similarity that has been binarized; then, for any unit similarity after binarization, the product of the specific unit data and the unit similarity is used as the unit data to be fused, and finally, the features formed by sequentially splicing the unit data to be fused can be used as the compensated hidden layer features.
[0130] Specifically, in order to better compensate for the hidden layer features, in an embodiment of the present invention, for any unit data of the hidden layer features, the similarity between the unit data and the unit data corresponding to the position in the features of the pre-stored typical normal event image can be determined as the unit similarity; then each unit similarity is binarized, and for any unit similarity after the binarization, the product of the specific unit data and the unit similarity is used as the unit data to be fused, and finally the feature formed by sequentially splicing the unit data to be fused is used as the compensated hidden layer feature, which can more fully compensate the hidden layer features according to the target similarity.
[0131] The specific process of binarizing the unit similarity may include: determining whether the unit similarity is greater than a preset similarity threshold; if so, setting the unit similarity to 1; if not, setting the unit similarity to 0.
[0132] Of course, in addition to the above specific manners, the hidden layer features may be compensated according to the target similarity in other specific manners, which are not limited in the embodiment of the present invention.
[0133] As an optional embodiment, the objective function of the first training objective includes:
[0134] ;
[0135] Among them, En is the identifier of the encoder, De is the identifier of the decoder, It means that the function value of the objective function is minimized by optimizing the network parameters of En. It means that the function value of the objective function is minimized by optimizing the network parameters of De. is the input image, is the data distribution, for The data distribution, represents the data distribution condition, express conform to , is the hidden layer feature after compensation, is the reconstructed input image, Refers to the Euclidean distance formula.
[0136] Specifically, through the objective function of the first training objective as described above, the network parameters of the encoder can be adjusted efficiently and accurately, so that the data distribution of the hidden layer features of the input image approaches the preset prior distribution.
[0137] Of course, in addition to the above specific forms, the objective function of the first training objective may also be of other specific types, which is not limited in the embodiment of the present invention.
[0138] As an optional embodiment, the objective function of the second training objective includes:
[0139] ;
[0140] Among them, Di is the identifier of the discriminator, It means that the function value of the objective function is minimized by optimizing the network parameters of En. It means that the function value of the objective function is maximized by optimizing the network parameters of Di. is a feature vector randomly sampled from a preset prior distribution, represents the preset prior distribution, represents the data distribution condition, express conform to , Denotes the discriminator determined conform to The probability of Indicates that the encoder maps the input image to hidden layer features. Indicates that the judgement determines conform to probability.
[0141] Specifically, through the objective function of the second training objective as above, the network parameters of the decoder can be adjusted efficiently and accurately, thereby minimizing the reconstruction error of the reconstructed image of the input image.
[0142] Of course, in addition to the above specific forms, the objective function of the second training objective may also be of other specific types, which is not limited in the embodiment of the present invention.
[0143] Specifically, the training process of the adversarial autoencoder in the embodiment of the present invention (taking the second adversarial autoencoder corresponding to the apparent flow as an example) can be:
[0144] The set of gradient feature blocks with normal events used for training is expressed as , is the gradient feature block in the set, a is the identifier of the apparent feature, i=1,2,3,…,K, K is the number of gradient feature blocks in the set. , its data distribution is expressed as . Mapped to the hidden layer features through the encoder , n is the preset dimension of the hidden layer features, is a natural number.
[0145] The decoder (Hidden layer features compensated by memory modules) as input, reconstruct the gradient feature block . is the reconstructed gradient feature block , the reconstructed gradient feature block should be consistent with the original input, so the reconstruction objective function is a problem of minimizing the difference between input and output, and the Euclidean distance is used for calculation:
[0146] ;
[0147] At the same time, the encoder and the discriminator are trained adversarially to achieve z in line with the preset prior distribution Purpose, The vector z is sampled from (i.e. conforms to) the preset prior distribution The probability of . Assume Represents the default prior distribution Any feature vector sampled from Representation vector From The objective function is defined as:
[0148] ;
[0149] Specifically, the detailed network training process is as follows: First, in order to ensure that the gradient feature block with normal events is losslessly reconstructed based on the "hidden layer features compensated by the memory module", the following step (2) updates the network parameters of the encoder, memory module and decoder by minimizing the reconstruction loss; secondly, in order to ensure that the encoder can confuse the discriminator, steps (4) and (6) update the network parameters of the encoder by maximizing the probability that the hidden layer representation z conforms to the preset prior distribution p(z). Step (5) updates the network parameters of the discriminator by minimizing the binary cross entropy, increasing z * The probability of meeting p(z) is reduced while the probability of z meeting p(z) is reduced to distinguish the real sample z * The distribution of and the encoder generates the distribution of samples z.
[0150] The input is: : A set of gradient feature blocks used for training, m1 to m R For each gradient feature in the set, it is divided into R small batches. L: The number of gradient feature blocks in each small batch (batch size). N: The number of training iterations. The output is: the network parameters of the encoder, memory module, decoder and discriminator , , and .
[0151] Step (1): The pth mini-batch is represented as , where q is the starting number of the gradient feature block in the pth mini-batch;
[0152] Step (2): Update the network parameters of the encoder, memory module and decoder , and : ;
[0153] Step (3): Randomly sample L n-dimensional feature vectors from the preset prior distribution p(z), expressed as ;
[0154] Step (4): Update the encoder network parameters : ;
[0155] Step (5): Update the network parameters of the discriminator : ;
[0156] Step (6): Update the encoder network parameters again : .
[0157] Specifically, the process of the test (inference) phase of the abnormal event detection method in the embodiment of the present invention is as follows:
[0158] Given an image block to be tested , first extract the corresponding optical flow feature block With gradient feature block On this basis, the trained MemMGFC-AAE is used to obtain its hidden layer representation in the two-stream structure, as shown below:
[0159] ;
[0160] ;
[0161] in and are the hidden layers in motion flow and appearance flow respectively, and Encoders of the fully trained MemMGFC-AAE in motion flow and appearance flow, respectively.
[0162] Taking the hidden layer representation as input, the probability density function of the prior multivariate Gaussian distribution is used to calculate the image block The first detection parameter in the motion flow The first detection parameter with apparent flow , as shown below:
[0163] ; ;
[0164] Where n is and The dimension of The default prior distribution The n×1 dimensional mean vector of is the n×n dimensional covariance matrix.
[0165] The memory module and After compensation and , get Reconstructed feature blocks in motion flow and apparent flow:
[0166] ;
[0167] ;
[0168] in and are the input feature blocks (motion feature blocks or appearance feature blocks) reconstructed in the motion flow and the appearance flow respectively, and Denote the decoders of the fully trained MemMGFC-AAE in motion stream and appearance stream, respectively.
[0169] The reconstruction error is used as the second detection parameter, where is the second detection parameter in the motion flow, is the second detection parameter in the apparent flow, as shown below:
[0170] ;
[0171] ;
[0172] The final motion anomaly score is obtained by combining the anomaly score based on conditional probability and reconstruction error. Apparent abnormality score :
[0173] ;
[0174] ;
[0175] Use weights and Apparent abnormality score Motor abnormality score Fusion to get the total abnormal score , as shown below:
[0176] ;
[0177] Finally, when the following equation holds true, the test image block is determined For exception:
[0178] ;
[0179] in, is a pre-set threshold.
[0180] Please refer to Figure 6 , Figure 6 A schematic diagram of the structure of an abnormal event detection device provided by the present invention, the abnormal event detection device comprises:
[0181] A first training module 61 is used to use the motion feature map of each image in the training image set as an input image, train the first adversarial autoencoder, and achieve a first training goal and a second training goal, wherein there are no abnormal events in the images in the training image set, the first training goal is to make the data distribution of the hidden layer features of the input image approach a preset prior distribution, and the second training goal is to minimize the reconstruction error of the reconstructed image of the input image;
[0182] A second training module 62 is used to use the appearance feature map of each image in the training image set as an input image to train the second adversarial autoencoder and achieve the first training goal and the second training goal;
[0183] A first scoring module 63 is used to use the sum of a first detection parameter and a second detection parameter of the motion feature map of the image to be tested by the trained first adversarial autoencoder as a motion anomaly score, wherein the first detection parameter refers to: a probability that the hidden layer feature of the image to be tested conforms to a preset prior distribution, and the second detection parameter refers to: a reconstruction error of a reconstructed image of the image to be tested;
[0184] A second scoring module 64 is used to use the sum of the first detection parameter and the second detection parameter of the apparent feature map of the image to be tested by the trained second adversarial autoencoder as the apparent abnormality score;
[0185] The result obtaining module 65 is used to obtain a determination result of whether an abnormal event exists in the image to be tested according to the total abnormality score of the image to be tested, wherein the total abnormality score refers to the sum of the motion abnormality score and the apparent abnormality score.
[0186] Based on the above embodiments:
[0187] As an optional embodiment, the first training module 61 includes:
[0188] A first determination module is used to determine a motion feature map of an image in a training image set;
[0189] The second determination module is used to determine the motion feature blocks of each area in the motion feature map;
[0190] A first training submodule is used to use each motion feature block of each image in the training image set as an input image, train the first adversarial autoencoder, and achieve the first training goal and the second training goal;
[0191] The second training module 62 includes:
[0192] A third determination module is used to determine the apparent feature map of the image in the training image set;
[0193] A fourth determination module is used to determine the apparent feature blocks of each region in the apparent feature map;
[0194] A second training submodule is used to use each apparent feature block of each image in the training image set as an input image, train the second adversarial autoencoder, and achieve the first training goal and the second training goal;
[0195] The abnormal event detection device also includes:
[0196] The first action module is used to divide the image blocks in each area of the designated image into images to be tested.
[0197] For an introduction to the abnormal event detection device provided in the embodiment of the present invention, please refer to the aforementioned embodiment of the abnormal event detection method, and the embodiment of the present invention will not be described in detail here.
[0198] Please refer to Figure 7 , Figure 7 A schematic diagram of the structure of an abnormal event detection device provided by the present invention, the abnormal event detection device comprises:
[0199] A memory 71, used for storing computer programs;
[0200] The processor 72 is used to implement the steps of the abnormal event detection method in the above-mentioned embodiment when executing the computer program.
[0201] For an introduction to the abnormal event detection device provided in the embodiment of the present invention, please refer to the aforementioned embodiment of the abnormal event detection method, and the embodiment of the present invention will not be described in detail here.
[0202] Please refer to Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer-readable storage medium provided by the present invention. A computer program is stored on the computer-readable storage medium 81. When the computer program 82 is executed by the processor, the steps of the abnormal event detection method in the aforementioned embodiment are implemented.
[0203] For an introduction to the computer-readable storage medium provided in an embodiment of the present invention, please refer to the aforementioned embodiment of the abnormal event detection method, and the embodiment of the present invention will not be described in detail here.
[0204] The present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the abnormal event detection method in the above-mentioned embodiment when executed by a processor.
[0205] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the aforementioned embodiment of the abnormal event detection method, and the embodiment of the present invention will not be described in detail here.
[0206] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should also be noted that in this specification, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the element.
[0207] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for detecting abnormal events, characterized in that: include: Using the motion feature map of each image in the training image set as an input image, training the first adversarial autoencoder, and achieving a first training goal and a second training goal, wherein the images in the training image set do not have abnormal events, the first training goal refers to: making the data distribution of the hidden layer features of the input image approach a preset prior distribution, and the second training goal refers to: minimizing the reconstruction error of the reconstructed image of the input image; Using the apparent feature map of each image in the training image set as an input image, training the second adversarial autoencoder, and achieving the first training goal and the second training goal; The sum of the first detection parameter and the second detection parameter of the motion feature map of the image to be tested by the trained first adversarial autoencoder is used as the motion anomaly score, wherein the first detection parameter refers to: the probability that the hidden layer feature of the image to be tested conforms to the preset prior distribution, and the second detection parameter refers to: the reconstruction error of the reconstructed image of the image to be tested; The sum of the first detection parameter and the second detection parameter of the apparent feature map of the image to be tested by the trained second adversarial autoencoder is used as the apparent abnormality score; A determination result of whether an abnormal event exists in the image to be tested is obtained according to the total abnormality score of the image to be tested, wherein the total abnormality score refers to the sum of the motion abnormality score and the apparent abnormality score.
2. The abnormal event detection method according to claim 1, characterized in that: The motion feature map of each image in the training image set is used as an input image to train the first adversarial autoencoder, and achieving the first training goal and the second training goal includes: Determine a motion feature map of an image in a training image set; Determine the motion feature blocks of each area in the motion feature map; Using each of the motion feature blocks of each image in the training image set as an input image, training the first adversarial autoencoder, and achieving the first training goal and the second training goal; Taking the apparent feature map of each image in the training image set as an input image, training the second adversarial autoencoder, and achieving the first training goal and the second training goal includes: Determining an apparent feature map of an image in a training image set; Determine the apparent feature blocks of each region in the apparent feature map; Using each of the appearance feature blocks of each image in the training image set as an input image, training the second adversarial autoencoder, and achieving the first training goal and the second training goal; Before using the sum of the first detection parameter and the second detection parameter of the motion feature map of the image to be detected by the trained first adversarial autoencoder as the motion anomaly score, the abnormal event detection method further includes: The image blocks in each area of the specified image are used as the image to be tested.
3. The abnormal event detection method according to claim 1, characterized in that: The adversarial autoencoder includes an encoder, a decoder, and a discriminator; Wherein, the adversarial autoencoder includes the first adversarial autoencoder and the second adversarial autoencoder; The encoder is used to map the input image into hidden layer features; The decoder is used to reconstruct the input image according to the hidden layer features; The discriminator is used to determine the probability that the input image conforms to the preset prior distribution.
4. The abnormal event detection method according to claim 3, characterized in that: The adversarial autoencoder further comprises: A memory module, used for compensating the hidden layer features according to target similarity, wherein the target similarity refers to: the similarity between the hidden layer features and the features of the pre-stored typical normal event image; The decoder is specifically used to reconstruct the input image according to the compensated hidden layer features.
5. The abnormal event detection method according to claim 4, characterized in that: The memory module is specifically used for: For any unit data of the hidden layer feature, determining the similarity between the unit data and the unit data corresponding to the position in the pre-stored feature of the typical normal event image as the unit similarity; Binarization processing is performed on each of the unit similarities; For any of the unit similarities after binarization, the product of specific unit data and the unit similarity is used as the unit data to be fused, wherein the specific unit data refers to: the unit data corresponding to the unit similarity in the features of the typical normal event image; The features formed by sequentially concatenating the unit data to be fused are used as the compensated hidden layer features.
6. The abnormal event detection method according to claim 4, characterized in that: The objective function of the first training objective includes: ; Wherein, En is the identifier of the encoder, De is the identifier of the decoder, It means that the function value of the objective function is minimized by optimizing the network parameters of En. It means that the function value of the objective function is minimized by optimizing the network parameters of De. is the input image, is the data distribution, for The data distribution, represents the data distribution condition, express conform to , is the hidden layer feature after compensation, is the reconstructed input image, Refers to the Euclidean distance formula.
7. The abnormal event detection method according to claim 4, characterized in that: The objective function of the second training objective includes: ; Wherein, Di is the identifier of the discriminator, It means that the function value of the objective function is minimized by optimizing the network parameters of En. It means that the function value of the objective function is maximized by optimizing the network parameters of Di. is a feature vector randomly sampled from a preset prior distribution, represents the preset prior distribution, represents the data distribution condition, express conform to , Denotes the discriminator determined conform to The probability of Indicates that the encoder maps the input image to hidden layer features. Indicates that the judgement determines conform to probability.
8. An abnormal event detection device, characterized in that: include: A first training module is used to use the motion feature map of each image in the training image set as an input image, train the first adversarial autoencoder, and achieve a first training goal and a second training goal, wherein the images in the training image set do not have abnormal events, the first training goal is to make the data distribution of the hidden layer features of the input image approach a preset prior distribution, and the second training goal is to minimize the reconstruction error of the reconstructed image of the input image; A second training module, configured to use the apparent feature map of each image in the training image set as an input image, train the second adversarial autoencoder, and achieve the first training objective and the second training objective; A first scoring module is used to use the sum of a first detection parameter and a second detection parameter of the motion feature map of the image to be tested by the trained first adversarial autoencoder as a motion anomaly score, wherein the first detection parameter refers to: a probability that the hidden layer feature of the image to be tested conforms to the preset prior distribution, and the second detection parameter refers to: a reconstruction error of a reconstructed image of the image to be tested; A second scoring module is used to use the sum of the first detection parameter and the second detection parameter of the apparent feature map of the image to be tested by the trained second adversarial autoencoder as an apparent abnormality score; The result obtaining module is used to obtain a determination result of whether an abnormal event exists in the image to be tested according to the total abnormality score of the image to be tested, wherein the total abnormality score refers to the sum of the motion abnormality score and the apparent abnormality score.
9. An abnormal event detection device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the abnormal event detection method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the abnormal event detection method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Method for detecting an anomalous image among a first dataset of images using an adversarial autoencoder
CN109741292A
Video image anomaly detection method and system based on space-time cascade auto-encoder
CN111402237A
Visual anomaly detection method based on time sequence space information enhancement
CN116543335A
Video anomaly detection method and system based on appearance semantics and motion amplitude
CN118470611A
Abnormality determination system and method for constructing the same
JP2020177282A
