Semi-supervised anomaly detection method for crowd scenes based on masked transformer

Through the semi-supervised detection method based on mask transformer, the problem of insufficient real-time and accuracy in traditional crowd scene detection is solved, efficient and accurate abnormal detection is achieved, and calculation costs are reduced and detection accuracy is improved.

CN115100599BActive Publication Date: 2024-11-22湖南工商大学
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202210765871.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-01
Publication Date
2024-11-22
Estimated Expiration
2042-07-01

AI Technical Summary

Technical Problem

Traditional crowd scene abnormality detection methods have problems such as poor real-time, detection efficiency and accuracy, especially because the image abnormality data information is highly redundant, resulting in huge consumption of missed detection, false detection and computing resources.

Method used

Using a semi-supervised detection method based on mask transformer, the transformer's foreground feature extractor and potential are optimized by extracting optical flow sequences, target detection networks, and target detection networks to locate the foreground area, and perform redundant information filtering, combining masking mechanisms and multiple mask prediction losses. Representation optimizer, design inter-class difference constraint modules, implement semi-supervised training and exception detection.

Benefits of technology

It improves the real-time and accuracy of detection, reduces calculation costs, increases abnormal reconstruction errors, meets real-time detection needs and improves detection accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115100599B_ABST
    Figure CN115100599B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides a semi-supervised crowd scene anomaly detection method based on mask transformer, which belongs to the field of data recognition technology, and specifically includes: generating a foreground image sequence with abnormal information; generating a potential feature representation of normal events of crowd images; constructing a potential representation optimizer, inputting a feature map with mask symbols, predicting the real image of the masked part, and introducing multiple mask prediction losses, and optimizing the relevant parameters of the improved transformer foreground feature extractor in combination with the loss value; using deconvolution to construct a reconstructor, and defining a difference constraint module between normal event classes; optimizing network parameters to obtain a detection model; using the detection model to discriminate the crowd image sequence, outputting anomaly scores, and performing threshold discrimination on the anomaly scores, screening out abnormal images and outputting related abnormal data. Through the scheme of the present invention, the detection accuracy is improved while meeting the needs of real-time detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data recognition technology, and in particular to a semi-supervised crowd scene anomaly detection method based on mask transformer. Background Art

[0002] At present, with the development of informatization, in the detection of crowd scenes, image devices can be used to shoot scene images for anomaly detection. However, in traditional methods, due to the high redundancy of image anomaly data information, missed detection and false detection occur, and existing deep learning methods have problems such as difficulty in optimizing the potential representation of normal events, difficulty in constraining inter-class differences, and huge consumption of computing resources.

[0003] It can be seen that there is an urgent need for a real-time, efficient, and accurate detection method for semi-supervised crowd scene anomaly detection based on mask transformer. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a semi-supervised crowd scene anomaly detection method based on mask transformer, which at least partially solves the problems of poor real-time performance, detection efficiency and accuracy in the prior art.

[0005] The embodiment of the present invention provides a semi-supervised crowd scene anomaly detection method based on mask transformer, comprising:

[0006] Step 1: extract the optical flow sequence from the input crowd scene image, use the target detection network to locate the area with motion foreground, and filter the redundant information to generate a foreground image sequence with abnormal information;

[0007] Step 2: The foreground image sequence is equally divided and masked at a preset ratio. The unmasked part is input into the improved transformer foreground feature extractor for feature extraction to generate the latent feature representation of normal events in the crowd image.

[0008] Step 3: Combined with the mask mechanism, the transformer block is used to construct a latent representation optimizer, the feature map with mask symbols is input, the real image of the masked part is predicted, and multiple mask prediction losses are introduced. The relevant parameters of the improved transformer foreground feature extractor are optimized in combination with the loss value.

[0009] Step 4, construct a reconstructor using deconvolution and define a difference constraint module between normal event classes;

[0010] Step 5: Use a semi-supervised training method to train the improved transformer foreground feature extractor, potential representation optimizer and reconstructor, and combine the mask strategy to optimize the network parameters to obtain the detection model;

[0011] Step 6: Use the detection model to identify the crowd image sequence, output the abnormal score, and perform threshold identification on the abnormal score to filter out abnormal images and output related abnormal data.

[0012] According to a specific implementation of the embodiment of the present invention, step 1 specifically includes:

[0013] Step 1.1, use Flownet2 deep network to extract optical flow sequence;

[0014] Step 1.2, use the Cascade RCNN deep network to locate the motion foreground area;

[0015] In step 1.3, the filtered foreground images are normalized to a size of 32×32 to form a foreground image sequence with abnormal information.

[0016] According to a specific implementation of an embodiment of the present invention, the improved transformer foreground feature extractor includes a multi-layer transformer block, wherein each transformer layer includes an Attention layer and a feed-forward network layer.

[0017] According to a specific implementation of the embodiment of the present invention, step 2 specifically includes:

[0018] Step 2.1, the generated foreground image is divided into blocks of equal size and non-overlapping, and masked with a preset ratio suitable for image data, and the unmasked part is input into the improved transformer foreground feature extractor;

[0019] Step 2.2: Use the improved transformer foreground feature extractor to extract the unmasked part X = (x1, ...x t ) is block coded, position information is introduced and normalized, and then linear mapping is performed on X in three directions to generate three matrices Q, K, and V, which are expressed as:

[0020]

[0021] Step 2.3, perform average pooling on the matrix Q, K and input it into the Attention layer to calculate the attention;

[0022] Step 2.4, input the calculation results of the Attention layer into the feedforward network layer to complete the calculation of one layer of transformer blocks;

[0023] In step 2.5, foreground features are extracted through calculations of multi-layer transformer blocks to form latent feature representations of normal events in crowd images.

[0024] According to a specific implementation manner of an embodiment of the present invention, the feedforward network layer includes two fully connected layers.

[0025] According to a specific implementation of the embodiment of the present invention, step 3 specifically includes:

[0026] Step 3.1, use a layer of tranformer blocks and multiple convolutions to form a potential representation optimizer;

[0027] In step 3.2, the feature maps obtained by each transformer block except the last layer in the improved transformer foreground feature extractor are combined with the mask symbols in positional order and input into the potential representation optimizer. The real image of the masked part is predicted by the potential representation optimizer, and multiple losses are introduced, which are expressed as follows:

[0028]

[0029] Where G represents the number of times the optimizer receives the feature map obtained by the foreground feature extractor, Φ represents the true value of the masked area in the original foreground image, Represents the predicted value of the masked area when the optimizer receives the feature map for the i-th time.

[0030] According to a specific implementation of the embodiment of the present invention, step 4 specifically includes:

[0031] Step 4.1, a reconstructor is composed of deconvolution and multiple normal event class difference constraint modules, and the reconstructor receives the feature map output by the last layer transformer block of the improved transformer foreground feature extractor and the foreground map predicted by the optimizer as input;

[0032] Step 4.2, update the normal mode memory matrix M corresponding to each normal event inter-class difference constraint module, and the update expression is as follows:

[0033] M = softmax(MP T )M

[0034] Among them, P is the feature map of the input memory matrix M;

[0035] Step 4.3, introduce constraint loss into the normal event inter-class difference constraint module. The expression of constraint loss is as follows:

[0036]

[0037] Where D represents the number of memory matrices M, w i,j represents the matching probability of the jth memory slot in the i-th memory matrix.

[0038] According to a specific implementation of the embodiment of the present invention, step 5 specifically includes:

[0039] Step 5.1, train the improved transformer foreground feature extractor, latent representation optimizer, and reconstructor using the stochastic gradient descent algorithm and a training set that does not contain abnormal data;

[0040] Step 5.2, assuming the input data is y, For the reconstruction result, the reconstruction loss is defined as follows:

[0041]

[0042] Step 5.3, combining multiple mask losses and constraint losses, defines the total loss as:

[0043] L=λ RECON L RECON +λ MASK L MASK +λ CON L CON

[0044] Among them, λ RECON ,λ MASK ,λ CON A balance factor representing the corresponding loss;

[0045] Step 5.4, optimize the network parameters according to the total loss to obtain the detection model.

[0046] According to a specific implementation of the embodiment of the present invention, step 6 specifically includes:

[0047] Step 6.1, input the crowd image sequence into the detection model, reconstruct the original image according to the normal class features, and obtain the reconstruction error as the anomaly score;

[0048] Step 6.2: When the abnormal score is greater than the preset threshold, the abnormal image and abnormal information corresponding to the abnormal score are output.

[0049] The semi-supervised crowd scene anomaly detection scheme based on mask transformer in the embodiment of the present invention comprises: step 1, extracting an optical flow sequence from an input crowd scene image, using a target detection network to locate the area with motion foreground, and filtering redundant information to generate a foreground image sequence with abnormal information; step 2, equally dividing the foreground image sequence, and masking it at a preset ratio, inputting the unmasked part into an improved transformer foreground feature extractor for feature extraction, and generating a potential feature representation of normal events in the crowd image; step 3, combining the mask mechanism, using transformer blocks to construct a potential representation optimizer, and inputting the masked foreground feature extractor into the foreground feature extractor to extract the feature of the crowd image. The feature map of the code symbol is used to predict the real image of the masked part, and multiple mask prediction losses are introduced, and the relevant parameters of the improved transformer foreground feature extractor are optimized in combination with the loss value; Step 4, deconvolution is used to construct a reconstructor, and a difference constraint module between normal event classes is defined; Step 5, a semi-supervised training method is used to train the improved transformer foreground feature extractor, potential representation optimizer and reconstructor, and the mask strategy is combined to optimize the network parameters to obtain the detection model; Step 6, the detection model is used to discriminate the crowd image sequence, output the abnormal score, and threshold the abnormal score to screen out abnormal images and output related abnormal data.

[0050] The beneficial effects of the embodiments of the present invention are: using an improved transformer structure as a feature extractor, combined with a mask mechanism, while introducing multiple mask prediction losses to optimize the network and reduce computational costs, an inter-class difference constraint module is designed in the reconstructor to help the network increase the abnormal reconstruction error, thereby improving the detection accuracy while meeting the needs of real-time detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A flowchart of a semi-supervised crowd scene anomaly detection method based on a mask transformer provided by an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of a data processing flow involved in a semi-supervised crowd scene anomaly detection method based on a mask transformer provided in an embodiment of the present invention;

[0054] Figure 3A schematic diagram of a model involved in a semi-supervised crowd scene anomaly detection method based on a mask transformer provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0056] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0057] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present invention, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0058] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. The drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0059] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0060] An embodiment of the present invention provides a semi-supervised crowd scene anomaly detection method based on a mask transformer, which can be applied to the personnel analysis process in shopping malls, scenic spots or urban management and control scenarios.

[0061] See also Figure 1 , is a flow chart of a semi-supervised crowd scene anomaly detection method based on mask transformer provided by an embodiment of the present invention. Figure 1 As shown, the method mainly comprises the following steps:

[0062] Step 1: extract the optical flow sequence from the input crowd scene image, use the target detection network to locate the area with motion foreground, and filter the redundant information to generate a foreground image sequence with abnormal information;

[0063] Furthermore, the step 1 specifically includes:

[0064] Step 1.1, use Flownet2 deep network to extract optical flow sequence;

[0065] Step 1.2, use the Cascade RCNN deep network to locate the motion foreground area;

[0066] Step 1.3, normalize the filtered foreground image to a size of 32*32 to form a foreground image sequence with abnormal information.

[0067] In the specific implementation, the input is a crowd scene image sequence of any size and without anomalies. First, the pre-trained Flownet2 deep network is used to extract the corresponding optical flow image sequence. Then the pre-trained Cascade RCNN deep network is used to locate the area with motion foreground in the optical flow image. Considering that the number of foreground areas contained in the same size of optical flow images is different, and the specific size of the foreground area contained in each optical flow image is also inconsistent, in order to facilitate the subsequent calculation of the model, the foreground area can be normalized to 32×32×2 size to form a crowd foreground image sequence, where 32×32 is the resolution size and 2 is the number of channels.

[0068] Step 2: The foreground image sequence is equally divided and masked at a preset ratio. The unmasked part is input into the improved transformer foreground feature extractor for feature extraction to generate the latent feature representation of normal events in the crowd image.

[0069] Optionally, the improved transformer foreground feature extractor includes a multi-layer transformer block, wherein each transformer layer includes an Attention layer and a feed-forward network layer.

[0070] Furthermore, the step 2 specifically includes:

[0071] Step 2.1, the generated foreground image is divided into blocks of equal size and non-overlapping, and masked with a preset ratio suitable for image data, and the unmasked part is input into the improved transformer foreground feature extractor;

[0072] Step 2.2: Use the improved transformer foreground feature extractor to extract the unmasked part X = (x1, ...x t ) is block coded, position information is introduced and normalized, and then linear mapping is performed on X in three directions to generate three matrices Q, K, and V, which are expressed as:

[0073]

[0074] Step 2.3, perform average pooling on the matrix Q, K and input it into the Attention layer to calculate the attention;

[0075] Step 2.4, input the calculation results of the Attention layer into the feedforward network layer to complete the calculation of one layer of transformer blocks;

[0076] In step 2.5, foreground features are extracted through calculations of multi-layer transformer blocks to form latent feature representations of normal events in crowd images.

[0077] Optionally, the feedforward network layer includes two fully connected layers.

[0078] When implementing it, Figure 2 As shown in the figure, each 32×32×2 foreground image can be divided into equal-sized and non-overlapping blocks in a ratio of 16 in both horizontal and vertical directions, for a total of 256 blocks, each of which is 2×2×2 in size. All blocks are masked at a ratio of 75% suitable for image data, that is, the number of masked blocks is 192 and the number of unmasked blocks is 64. The 64 unmasked blocks are input into the foreground feature extractor.

[0079] The foreground feature extractor consists of four stages of transformers, and each stage transformer is set to the same size, with a specific dimension of 768, a depth of 3, and a multi-head number of 12. Each transformer is mainly composed of an attention layer and a feedforward network layer.

[0080] Transformer first performs block encoding on the 64 unmasked blocks using a two-dimensional convolution with a kernel of 2 and a stride of 2, and introduces block position information. A 64×768 feature map is generated. The feature map is linearly mapped in three directions to generate three matrices Q, K, and V, which have the same size of 64×64. Then, a one-dimensional adaptive average pooling with an output size of 4 is applied to the K and V matrices to reduce the size of the K and V matrices to 4×64, thereby reducing the computational cost of the transformer.

[0081] The three processed matrices are sent to the attention layer of the transformer for attention calculation, that is, the dot product attention of the K and V matrices is calculated to obtain a 64×4 attention matrix. The result of the probability mapping of the attention matrix through softmax is multiplied by the matrix Q to complete the attention calculation. At this time, the size of the matrix is ​​64×64. Because the number of multi-heads is 12, this part needs to be calculated in parallel 12 times to obtain a feature map of size 64×(64×12)=64×768. The feature map is normalized and sent to the feedforward network layer composed of two fully connected layers. Finally, the potential representation of the normal feature is obtained, and its feature map size is also 64×768. So far, the calculation of one stage of the transformer is completed. The entire foreground feature extractor has a total of four stages of transformers. The calculation of the entire transformer process does not change the size of the input feature map, so the feature map output at each stage is 64×768. It is represented as 64 unmasked blocks, each of which is represented by a 768-dimensional vector.

[0082] Specifically, the transformer first transforms the input block X = (x1, ...x t ) is block coded, position information is introduced, and normalization is performed. Then, linear mapping is performed on X in three directions to generate three matrices Q, K, and V, which are expressed as:

[0083]

[0084] Transformer is mainly divided into the Attention layer and the feedforward network layer. The main improvement is to perform average pooling on the matrix Q and K before calculating the attention in the Attention layer, which is recorded as: The purpose is to save computational costs. Then the transformer attention calculation is performed, which is specifically expressed as follows:

[0085]

[0086] d k represents the dimension of K, and t represents the number of blocks input to the transformer. Dot product generates a 2D correlation matrix Among them, the i-th row represents For all The attention value is calculated and the probability distribution of attention is calculated using the softmax algorithm for each row. The calculation result of the Attention layer is input into the feedforward network layer composed of two fully connected layers to complete the calculation of one layer of transformer blocks. The foreground features are extracted through the calculation of multiple layers of transformer blocks, and the potential representation of normal events is obtained.

[0087] Step 3: Combined with the mask mechanism, the transformer block is used to construct a latent representation optimizer, the feature map with mask symbols is input, the real image of the masked part is predicted, and multiple mask prediction losses are introduced. The relevant parameters of the improved transformer foreground feature extractor are optimized in combination with the loss value.

[0088] Based on the above embodiment, step 3 specifically includes:

[0089] Step 3.1, use a layer of tranformer blocks and multiple convolutions to form a potential representation optimizer;

[0090] In step 3.2, the feature maps obtained by each transformer block except the last layer in the improved transformer foreground feature extractor are combined with the mask symbols in positional order and input into the potential representation optimizer. The real image of the masked part is predicted by the potential representation optimizer, and multiple losses are introduced, which are expressed as follows:

[0091]

[0092] Where G represents the number of times the optimizer receives the feature map obtained by the foreground feature extractor, Φ represents the true value of the masked area in the original foreground image, Represents the predicted value of the masked area when the optimizer receives the feature map for the i-th time.

[0093] In specific implementation, the latent representation optimizer consists of a transformer stage and two downsampling operations, in order to optimize the foreground feature extraction network and provide information supplement for the reconstructor. The transformer has a dimension of 512, a depth of 2, and a multi-head number of 8. Downsampling is done by convolution operations.

[0094] The following is the optimizer's optimization operation on the foreground feature extractor. The potential representation extracted by the foreground feature extractor in the first three stages only contains the features of the unmasked part. The role of the transformer in the optimizer is to predict the masked part of the image and introduce multiple mask losses to optimize the foreground feature extractor. In order to obtain the feature representation of the complete image, the foreground feature extractor needs to introduce mask symbols in position order each time before sending the potential representation to the optimizer. The mask symbols are a series of randomly generated values. There are 192 mask symbols in total, and each mask symbol is represented as a 768-dimensional vector. Figure 2 The gray part in the middle represents the mask symbol. The latent representation feature map with the mask symbol added is linearly mapped into 512 dimensions and input into the transformer of the optimizer. The transformer calculation process described in step 2 is followed to complete the first stage of transformer calculation and generate a 256×512 feature map. The feature map is transformed into the same size as the original foreground map (32×32×2) through linear mapping and deconvolution operations, and all the values ​​of the unmasked part are set to 0. At this time, the optimizer's prediction result for the unmasked part is obtained, and this result is used to calculate the multi-mask loss with the original foreground map. During multiple trainings, the optimizer will perform backpropagation based on the loss value to achieve the purpose of optimizing the foreground feature extractor. Because the optimizer receives the latent representations of the three stages of the foreground feature extractor, this process will perform three optimizations at different stages in one training.

[0095] The following is the optimization operation of the optimizer on the reconstructor. The 256×512 feature map obtained by the optimizer through the transformer is expanded to 16×16×512 and input into the reconstructor as the first information supplement. Then, the 16×16×512 feature map is downsampled for the first time by the convolution operation to generate a feature map of 8×8×1024 as the second supplementary information input into the reconstructor. Finally, the 8×8×1024 feature map is downsampled for the second time by the convolution operation to generate a feature map of 4×4×2048 as the third supplementary information input into the reconstructor. The three information supplements complete the optimization of the reconstructor.

[0096] Step 4, construct a reconstructor using deconvolution and define a difference constraint module between normal event classes;

[0097] Furthermore, the step 4 specifically includes:

[0098] Step 4.1, a reconstructor is composed of deconvolution and multiple normal event class difference constraint modules, and the reconstructor receives the feature map output by the last layer transformer block of the improved transformer foreground feature extractor and the foreground map predicted by the optimizer as input;

[0099] Step 4.2, update the normal mode memory matrix M corresponding to each normal event inter-class difference constraint module, and the update expression is as follows:

[0100] M = softmax(MP T )M

[0101] Among them, P is the feature map of the input memory matrix M;

[0102] Step 4.3, introduce constraint loss into the normal event inter-class difference constraint module. The expression of constraint loss is as follows:

[0103]

[0104] Where D represents the number of memory matrices M, w i,j represents the matching probability of the jth memory slot in the i-th memory matrix.

[0105] In specific implementation, the reconstructor consists of deconvolution and multiple normal event inter-class difference constraint modules, which receive the feature map output by the last transformer layer of the foreground feature extractor and the foreground map predicted by the optimizer as input. Due to the diversity of normal events, the normal features extracted by the network have inter-class differences. In order to prevent this unconstrained inter-class difference from affecting the detection performance, a normal event inter-class difference constraint module is designed and applied to the reconstructor.

[0106] Each inter-class difference constraint module has a learnable normal mode memory matrix M∈□ N×C ,Each memory matrix M has N memory slots, each slot represents a normal event pattern, denoted as m i , where i = 1, 2…, N. The constraint module aims to transform the normal class features into more prominent N normal feature patterns, and maintain and update these patterns in each training. During reconstruction, these N patterns are used instead of the original potential features to prevent some non-prominent normal features from affecting the reconstruction effect.

[0107] Each slot in the matrix M is updated as the weighted sum of similar memory slots. The update expression of M is as follows, where P is the feature map of the input memory matrix M:

[0108] M = softmax(MP T )M

[0109] The constraint loss introduced by the inter-class difference constraint module is defined as follows, where D represents the number of memory matrices M, and w i,j Represents the matching probability of the jth memory slot in the i-th memory matrix:

[0110]

[0111] For example, the reconstructor consists of multiple inter-class difference constraint modules and upsampling parts, where R represents the inter-class difference constraint module. The reconstructor first receives the potential representation obtained by the fourth stage of the foreground feature extractor and adds the mask symbol in position order (the size of the feature map after adding the mask symbol is 16×16×512) and obtains the feature map (size is 2×2×4096) after 8 times downsampling.

[0112] Specifically, this feature map is first received by the inter-class difference constraint module in the reconstructor. The inter-class difference constraint module maintains a learnable normal pattern memory matrix M, which has 2000 memory slots, each of which records a normal class feature pattern. The role of the constraint module is to integrate all features into 2000 representative normal class features to mitigate the impact that may be caused by inconspicuous normal class features. In specific operations, the matrix M is first randomly initialized, and then the feature map is multiplied by the matrix M and softmax is performed to obtain 2000 matching weights, and then the weights are applied to the matrix M to form a new enhanced feature map of size (2×2×4096). This process introduces constraint loss in the training stage to help the inter-class difference module gradually correct the feature form recorded by the matrix M. At this point, the first inter-class difference constraint module is completed. The obtained 2×2×4096 feature map is upsampled twice by deconvolution and connected with the information supplemented by the optimizer for the third time (size 4×4×2048) on the channel, and then the number of channels is reduced to obtain a feature map of size 4×4×2048. At this point, the first upsampling is completed. Then enter the second inter-class difference constraint module, perform a second double upsampling and connect it with the information supplemented by the optimizer for the second time on the channel, and reduce the number of channels to obtain a feature map of size 8×8×1024. Finally, enter the third inter-class difference constraint module, perform a second double upsampling and connect it with the information supplemented by the optimizer for the first time on the channel, and reduce the number of channels to obtain a feature map of size 16×16×512. Linearly map the feature map to the same size as the original foreground map (32×32×2), which is the final reconstructed map obtained by the entire network. Finally, calculate the reconstruction error between the reconstructed map and the original foreground map, and perform backpropagation based on the reconstruction loss to optimize the overall network parameters.

[0113] Step 5: Use a semi-supervised training method to train the improved transformer foreground feature extractor, potential representation optimizer and reconstructor, and combine the mask strategy to optimize the network parameters to obtain the detection model;

[0114] Based on the above embodiment, step 5 specifically includes:

[0115] Step 5.1, train the improved transformer foreground feature extractor, latent representation optimizer, and reconstructor using the stochastic gradient descent algorithm and a training set that does not contain abnormal data;

[0116] Step 5.2, assuming the input data is y, For the reconstruction result, the reconstruction loss is defined as follows:

[0117]

[0118] Step 5.3, combining multiple mask losses and constraint losses, defines the total loss as:

[0119] L=λ RECON L RECON +λ MASK L MASK +λ CON L CON

[0120] Among them, λ RECON ,λ MASK ,λ CON A balance factor representing the corresponding loss;

[0121] Step 5.4, optimize the network parameters according to the total loss to obtain the detection model.

[0122] In the specific implementation, the stochastic gradient descent algorithm is used to train the foreground feature extractor, potential representation optimizer, and reconstructor. The above process is trained for at least 50 rounds. The semi-supervised training method is used during training, that is, only normal class data is used for training, so that the network learns normal class features. In this process, it is assumed that the input data is y. For the reconstruction result, the reconstruction loss is defined as follows:

[0123]

[0124] Combining the above multiple mask losses and constraint losses, the total loss is defined as:

[0125] L=λ RECON L RECON +λ MASK L MASK +λ CON L CON

[0126] Among them, λ RECON ,λ MASK ,λ CON Represents the balance factor of the corresponding loss. Then, the network parameters are optimized according to the total loss to obtain the detection model as follows Figure 3 shown.

[0127] Step 6: Use the detection model to identify the crowd image sequence, output the abnormal score, and perform threshold identification on the abnormal score to filter out abnormal images and output related abnormal data.

[0128] Furthermore, the step 6 specifically includes:

[0129] Step 6.1, input the crowd image sequence into the detection model, reconstruct the original image according to the normal class features, and obtain the reconstruction error as the anomaly score;

[0130] Step 6.2: When the abnormal score is greater than the preset threshold, the abnormal image and abnormal information corresponding to the abnormal score are output.

[0131] In the specific implementation, the crowd image sequence is input into the detection model, and the original image is reconstructed according to the normal class features. The input crowd image sequence contains abnormal images. It is difficult for the network to reconstruct the abnormal images well according to the learned normal class features. The reconstruction result and the original image will have a large reconstruction error. Then the reconstruction error value is mapped to the range of [0,1] as the abnormal score. Then the abnormal score is thresholded. The threshold setting can be customized according to the size of the abnormal score and actual needs. The initial setting is 0.5. Images with abnormal scores exceeding 0.5 are judged as abnormal. If the actual needs require the screening of images with obvious abnormalities, the threshold can be appropriately increased; if the actual needs require that no missed detection occurs, the threshold can be appropriately lowered. After the abnormal images are screened out, the relevant abnormal data is output. Of course, abnormal voice broadcast prompts can also be performed, and the detection log is recorded on the storage device in the system to facilitate subsequent analysis of the detection process.

[0132] The semi-supervised crowd scene anomaly detection method based on mask transformer provided in this embodiment adopts an improved transformer structure as a feature extractor, combined with a mask mechanism, while introducing multiple mask prediction losses to optimize the network and reduce computational costs. At the same time, an inter-class difference constraint module is designed in the reconstructor to help the network increase the anomaly reconstruction error, which can effectively improve the system anomaly detection efficiency and better adapt to crowd scenes with complex environments and unknown anomalies.

[0133] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof.

[0134] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A semi-supervised crowd scene anomaly detection method based on mask transformer, characterized in that: include: Step 1: extract the optical flow sequence from the input crowd scene image, use the target detection network to locate the area with motion foreground, and filter the redundant information to generate a foreground image sequence with abnormal information; Step 2: The foreground image sequence is equally divided and masked at a preset ratio. The unmasked part is input into the improved transformer foreground feature extractor for feature extraction to generate the latent feature representation of normal events in the crowd image. The step 2 specifically includes: Step 2.1, the generated foreground image is divided into blocks of equal size and non-overlapping, and masked with a preset ratio suitable for image data, and the unmasked part is input into the improved transformer foreground feature extractor; Step 2.2: Use the improved transformer foreground feature extractor to extract the unmasked part X = (x1, ...x t ) is block coded, position information is introduced and normalized, and then linear mapping is performed on X in three directions to generate three matrices Q, K, and V, which are expressed as: Step 2.3, perform average pooling on the matrix Q, K and input it into the Attention layer to calculate the attention; Step 2.4, input the calculation results of the Attention layer into the feedforward network layer to complete the calculation of one layer of transformer blocks; Step 2.5, extracting foreground features through calculation of multi-layer transformer blocks to form latent feature representation of normal events in crowd images; Step 3: Combined with the mask mechanism, the transformer block is used to construct a latent representation optimizer, the feature map with mask symbols is input, the real image of the masked part is predicted, and multiple mask prediction losses are introduced. The relevant parameters of the improved transformer foreground feature extractor are optimized in combination with the loss value. The step 3 specifically includes: Step 3.1, use a layer of tranformer blocks and multiple convolutions to form a potential representation optimizer; In step 3.2, the feature maps obtained by each transformer block except the last layer in the improved transformer foreground feature extractor are combined with the mask symbols in positional order and input into the potential representation optimizer. The real image of the masked part is predicted by the potential representation optimizer, and multiple mask losses are introduced, which are expressed as follows: Where G represents the number of times the optimizer receives the feature map obtained by the foreground feature extractor, Φ represents the true value of the masked area in the original foreground image, It represents the predicted value of the masked area when the optimizer receives the feature map for the i-th time; Step 4, construct a reconstructor using deconvolution and define a difference constraint module between normal event classes; The step 4 specifically includes: Step 4.1, a reconstructor is composed of deconvolution and multiple normal event class difference constraint modules, and the reconstructor receives the feature map output by the last layer transformer block of the improved transformer foreground feature extractor and the foreground map predicted by the optimizer as input; Step 4.2, update the normal mode memory matrix M corresponding to each normal event inter-class difference constraint module, and the update expression is as follows: M=softmax(MP T )M Among them, P is the feature map of the input memory matrix M; Step 4.3, introduce constraint loss into the normal event inter-class difference constraint module. The expression of constraint loss is as follows: Where D represents the number of memory matrices M, w i,j represents the matching probability of the jth memory slot in the i-th memory matrix; Step 5: Use a semi-supervised training method to train the improved transformer foreground feature extractor, potential representation optimizer and reconstructor, and combine the mask strategy to optimize the network parameters to obtain the detection model; The step 5 specifically includes: Step 5.1, train the improved transformer foreground feature extractor, latent representation optimizer, and reconstructor using the stochastic gradient descent algorithm and a training set that does not contain abnormal data; Step 5.2, assuming the input data is y, For the reconstruction result, the reconstruction loss is defined as follows: Step 5.3, combining multiple mask losses and constraint losses, defines the total loss as: L=λ RECON L RECON +λ MASK L MASK +λ CON L CON Among them, λ RECON ,λ MASK ,λ CON A balance factor representing the corresponding loss; Step 5.4, optimizing the network parameters according to the total loss to obtain the detection model; Step 6: Use the detection model to identify the crowd image sequence, output the abnormal score, and perform threshold identification on the abnormal score to filter out abnormal images and output related abnormal data.

2. The method according to claim 1, characterized in that , the step 1 specifically includes: Step 1.1, use Flownet2 deep network to extract optical flow sequence; Step 1.2, use the Cascade RCNN deep network to locate the motion foreground area; In step 1.3, the filtered foreground images are normalized to a size of 32×32 to form a foreground image sequence with abnormal information.

3. The method according to claim 2, characterized in that ,The improved transformer foreground feature extractor includes a multi-layer transformer block, wherein each transformer layer includes an Attention layer and a feed-forward network layer.

4. The method according to claim 3, characterized in that ,The feed-forward network layer includes two fully connected layers.

5. The method according to claim 4, characterized in that , the step 6 specifically includes: Step 6.1, input the crowd image sequence into the detection model, reconstruct the original image according to the normal class features, and obtain the reconstruction error as the anomaly score; Step 6.2: When the abnormal score is greater than the preset threshold, the abnormal image and abnormal information corresponding to the abnormal score are output.

Citation Information

Patent Citations

  • Human head identification method under complex scene

    CN101763504A

  • Abnormal event detection method of time-space variational self-encoding network based on self-attention enhancement

    CN113449660A

  • Abnormal behavior detection method based on appearance and action feature dual prediction

    CN113762007A

  • Abnormality detection method, system and equipment for time series data and storage medium

    CN114386521A

  • Deep neural network optimization system for machine learning model scaling

    US20220036194A1