Video anomaly detection method, and pre-training method and system of feature extractor

Abstract

The invention discloses a video anomaly detection method and a pre-training method and system of a feature extractor. The pre-training method comprises the following steps: partitioning a training video and carrying out mask covering; learning feature representation by using an encoder; reconstructing the masked video block by using a decoder, and updating parameters based on thousand similarity; the method specifically comprises the steps of obtaining a frame difference and a depth map; forming time and space probability graphs, and fusing to form a mask probability graph; selecting to form a mask pattern to selectively mask the video blocks; the probability of being selectively masked is positively correlated with the motion degree and negatively correlated with the depth value. According to the pre-training method provided by the invention, a specific visual feature extractor can be provided for anomaly detection of videos with fixed view angles, the problem of domain difference between a general behavior recognition data set used in current network model training and the videos with the fixed view angles is solved, and the feature representation customized for video anomaly detection is extracted, so that the feature representation of the videos with the fixed view angles is extracted. Therefore, the anomaly detection precision and efficiency of the fixed-view-angle video are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a video anomaly detection method, a pre-training method and a system for a feature extractor. Background Art

[0002] Video Anomaly Detection (VAD) plays a vital role in large-scale surveillance systems, and excellent visual features are the key to ensuring that anomaly detection networks achieve high accuracy. Currently, most studies still rely on 3D convolution-based networks such as I3D and C3D to extract features on general behavior recognition datasets. The domain difference between the general behavior recognition datasets (such as Kinetics400) used for I3D and C3D network training and the datasets of fixed-view videos makes these networks perform well in general videos, but their performance in special videos such as fixed-view videos, such as surveillance videos, is not satisfactory.

[0003] Since the attention-based backbone network Transformer was proposed, research results on Masked Visual Modeling (MVM) have continued to emerge. Masked Visual Modeling uses a simple masking strategy and pixel reconstruction tasks to learn general visual representations in large-scale unlabeled datasets and has powerful visual feature extraction capabilities. However, similar to the above-mentioned 3D convolution-based network, although it performs better in general videos, it is difficult to show good adaptability in fixed-view videos. Summary of the invention

[0004] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a video anomaly detection method, a pre-training method and a system for a feature extractor to solve the problems existing in the prior art.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention includes:

[0006] In a first aspect, the present invention provides a pre-training method for a feature extractor for video anomaly detection, wherein the feature extractor is used for anomaly detection in a video with a fixed viewing angle, comprising:

[0007] Divide the training video into blocks and perform masking, cover part of the video blocks, and keep the other part of the video blocks as input video blocks;

[0008] learning, using an encoder, a feature representation of the input video block;

[0009] Reconstructing the masked video block using a decoder based on the feature representation, and updating the parameters of the encoder based on the similarity between the reconstructed video block and the masked video block, and finally serving as the feature extractor;

[0010] The masking process specifically includes:

[0011] Obtaining a frame difference image of the training video by using inter-frame difference, and obtaining a depth image of the training video by using depth estimation;

[0012] Using the frame difference map to form a time probability map, using the depth map to form a spatial probability map, and fusing the spatial probability map and the time probability map to form a mask probability map;

[0013] Selecting to form a mask map based on the mask probability map, and selectively masking the video block based on the mask map;

[0014] When forming the mask map, the probability of any of the video blocks being selected for masking is positively correlated with the degree of motion in the corresponding temporal probability map, and negatively correlated with the depth value in the spatial probability map.

[0015] In a second aspect, the present invention further provides a method for detecting anomalies in a video with a fixed viewing angle, comprising:

[0016] The feature extractor obtained by pre-training using the above-mentioned pre-training method extracts features of the video to be detected to obtain video features;

[0017] The video features are evaluated for abnormality using a scorer to obtain an abnormality detection result.

[0018] In a third aspect, the present invention further provides a pre-training system for a feature extractor for video anomaly detection, wherein the feature extractor is used for anomaly detection in a video with a fixed viewing angle, comprising:

[0019] A mask module is used to divide the training video into blocks and perform masking to cover a part of the video blocks and retain another part of the video blocks as input video blocks;

[0020] An encoder module, configured to learn a feature representation of the input video block using an encoder;

[0021] A decoder module, configured to reconstruct the masked video block based on the feature representation using a decoder, and update the parameters of the encoder based on the similarity between the reconstructed video block and the masked video block, and finally serve as the feature extractor;

[0022] The mask module specifically includes:

[0023] A differential depth unit, used to obtain a frame difference map of the training video by using inter-frame difference, and to obtain a depth map of the training video by using depth estimation;

[0024] A probability map unit, configured to form a temporal probability map using the frame difference map, form a spatial probability map using the depth map, and fuse the spatial probability map and the temporal probability map to form a mask probability map;

[0025] A masking selection unit, configured to select and form a mask map based on the mask probability map, and selectively mask the video block based on the mask map;

[0026] When forming the mask map, the probability of any of the video blocks being selected for masking is positively correlated with the degree of motion in the corresponding temporal probability map, and negatively correlated with the depth value in the spatial probability map.

[0027] In a fourth aspect, the present invention further provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed, the steps of the above-mentioned pre-training method or video anomaly detection method are executed.

[0028] Based on the above technical solution, compared with the prior art, the beneficial effects of the present invention at least include:

[0029] The pre-training method provided by the present invention can provide a specific visual feature extractor for this type of video anomaly detection with a fixed viewpoint, solve the domain difference problem between the general behavior recognition dataset used in the current network model training and the fixed viewpoint video, and improve the accuracy and efficiency of anomaly detection in fixed viewpoint videos by extracting feature representations customized for video anomaly detection.

[0030] The above description is only an overview of the technical solution of the present invention. In order to enable those skilled in the art to more clearly understand the technical means of the present application and implement them according to the contents of the specification, the following is a description of the preferred embodiments of the present invention in conjunction with detailed drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the process of video anomaly detection provided by a typical implementation case of the present invention. DETAILED DESCRIPTION

[0032] In view of the deficiencies in the prior art, the inventor of this case has proposed the technical solution of the present invention after long-term research and extensive practice. The technical solution, its implementation process and principle will be further explained as follows.

[0033] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0034] Moreover, relational terms such as “first” and “second” are merely used to distinguish one component or method step from another with the same name, but do not necessarily require or imply any such actual relationship or order between these components or method steps.

[0035] See also Figure 1 As shown, an embodiment of the present invention provides a pre-training method for a feature extractor for video anomaly detection, wherein the feature extractor is used for anomaly detection of videos with a fixed viewing angle, and has the same general process as the attention-based backbone network Transformer in the prior art, including the following main steps:

[0036] Divide the training video into blocks and perform masking, cover part of the video blocks, and keep the other part of the video blocks as input video blocks;

[0037] learning, using an encoder, a feature representation of the input video block;

[0038] The decoder is used to reconstruct the masked video block based on the feature representation, and the parameters of the encoder are updated based on the similarity between the reconstructed video block and the masked video block, and finally used as the feature extractor.

[0039] In the specific implementation case provided by the present invention, VideoMAE is used as the basic architecture of the visual feature extractor. VideoMAE adopts a self-supervised learning method to obtain effective visual feature representation. One of its core designs is the mask mechanism, which includes two modules, the encoder and the decoder. The structures of the two modules are composed of traditional Transformer blocks. The encoder is responsible for learning feature representation from the input video block, while the decoder is used to reconstruct the masked video block.

[0040] During the training process of the model, the random masking strategy is a commonly used masking method, which randomly masks the input video blocks, and each video block has the same masking probability. However, the inventors of the present invention have found that this strategy may be simple and effective in general video recognition datasets, but its applicability is poor in videos with fixed viewing angles, such as video surveillance datasets.

[0041] This is because compared to general behavior recognition datasets, moving targets in surveillance scenes are usually smaller, and the foreground and background are clearly distinguished, and the target's motion range is often within a specific area. Therefore, the random mask strategy may cover up key targets in the surveillance scene, resulting in the loss of important information, making it impossible for the model to learn relevant contextual information. In order to solve this problem, the present invention proposes a visual mask modeling based on spatiotemporal clues for video surveillance scenes.

[0042] Therefore, the present invention has a key difference from the above-mentioned existing network model in that the masking process specifically includes:

[0043] Obtaining a frame difference image of the training video by using inter-frame difference, and obtaining a depth image of the training video by using depth estimation;

[0044] Using the frame difference map to form a time probability map, using the depth map to form a spatial probability map, and fusing the spatial probability map and the time probability map to form a mask probability map;

[0045] Selecting to form a mask map based on the mask probability map, and selectively masking the video block based on the mask map;

[0046] When forming the mask map, the probability of any of the video blocks being selected for masking is positively correlated with the degree of motion in the corresponding temporal probability map, and negatively correlated with the depth value in the spatial probability map.

[0047] Based on the observation and analysis of fixed-view datasets, the inventors have found two important characteristics of fixed-view videos: (1) objects in fixed-view videos are larger when they are closer and smaller when they are farther away; (2) in fixed-view scenes, only some areas usually experience motion changes. Based on these characteristics, the temporal and spatial semantic information (frame difference and depth information) of the video data is extracted to guide the selection of mask areas.

[0048] The target task of the pre-training stage of the visual feature extractor is pixel reconstruction. In order to emphasize the difference in contribution of different areas in the video scene to the reconstruction task, the depth information is used to assign corresponding weights to the reconstruction losses of different areas, so as to enhance the expressive ability of the visual extractor and enable it to better capture the visual features of key areas.

[0049] In the final application, in the anomaly detection feature extraction stage, the pre-trained visual feature extractor is applied to the video anomaly detection dataset, and the extracted features are input into the video anomaly detection network for anomaly scoring and judgment, which can achieve better detection performance than the general random mask method.

[0050] Specifically, in some embodiments, the mask probability map may be formed as follows:

[0051] P m =αP s +βP d ;;

[0052] Among them, P m represents the mask probability map; P s represents the time probability graph; P d represents the spatial probability map; α and β represent weight coefficients, and their sum is 1.

[0053] In some embodiments, the generation process of the temporal probability map and the spatial probability map can be expressed as:

[0054]

[0055] in, represents the probability value of the corresponding video block i in the temporal probability map; represents the probability value of the corresponding video block in the spatial probability map; s (·) Indicates the frame difference change amplitude of the video block; d (·) Indicates the depth value of the video block; i indicates the sequence number of the video block; HW / N 2 represents the number of video blocks, H and W are the number of pixels of the video frame height and width respectively. In some typical examples, H=W=224, N=14, but it is not limited to this parameter setting.

[0056] As for some specific technical details, in some embodiments, the masking process may also include:

[0057] The moving area and the still area are distinguished based on the frame difference map. Usually, since the background area remains unchanged and the variation range of the frame difference is small, the moving area and the still area can be effectively distinguished by setting a threshold. Of course, other technical means with the same function are used to distinguish the moving area and the still area.

[0058] In the motion area, a preset proportion of video blocks are randomly selected, and the probability of the selected video blocks being masked is specified as 0 in the mask probability map, and the selected video blocks are forcibly retained.

[0059] As some typical examples, for the input raw video V∈R T×H×W×C The present invention adopts the inter-frame difference method and the monocular depth estimation network (MiDas) to obtain the frame difference map S∈R of the video T×H×W and the depth map D∈R T×H×W , which represent the temporal information and spatial information of the video respectively. Next, softmax operations are performed on the temporal and spatial information to obtain the temporal probability map and spatial probability map The probability range is [0, 1]. These probability maps will be used to guide the selection process of the video mask block, which represents the mask probability of each input video block. The higher the probability value, the more likely the video block is to be masked; conversely, the smaller the probability value, the more likely the video block is to be retained.

[0060] In the spatial probability map, the foreground area has a higher mask probability, and the same target shows different density information in the same size video blocks in the foreground and background areas; for video blocks of the same size, the foreground area usually contains higher redundant information, while the background area contains more complex and high-density information.

[0061] In the temporal probability map, the motion region has a higher mask probability. The decoder reconstructs the motion region so that the model can better learn the feature representation of the motion target. However, in order to avoid the motion region being completely masked and the model being unable to obtain the context information of the motion target, the preferred implementation case of the present invention randomly selects about 25% of the motion region and sets its masked probability value to 0, so as to reserve some motion regions for the model to learn and prevent a significant decrease in the model training efficiency.

[0062] Specifically in some implementation schemes, the preset ratio can generally be 10-40%. In the implementation case provided by the present invention, the value can be set to 25%, but it does not mean that adaptive adjustment cannot be performed. The optimal value range may be different for different data sets and different basic models.

[0063] The final mask probability map is obtained by weighted summation of the temporal probability map and the spatial probability map. Through comparative analysis of the mask visualization effect, the usual method setting selects α=0.7 and β=0.3 as the optimal setting, which is more in line with the expected effect of model training in this scenario.

[0064] Specifically in some implementation schemes, the value range of α is generally 0.2-0.4, for example, it is set to about 0.3 in a typical implementation case.

[0065] The above presents the details on the generation of spatiotemporal probability information, and regarding the process of forming a mask map based on probability information, in some embodiments, the mask map formation process uses a roulette wheel selection algorithm to perform multiple rounds of selection. For any of the video blocks, the probability of being selected in the current round is related to the probability value in the corresponding mask probability map, until the proportion of the number of selected video blocks reaches a preset mask rate.

[0066] Specifically in some implementation schemes, the probability of the video block being selected is calculated as follows:

[0067]

[0068] Among them, p(xi) represents the selection probability of video block i; q(x i ) represents the cumulative probability; P(x.) represents the probability value of the corresponding video block in the mask probability map; j represents the sequence number of the video block; the remaining symbols have been clarified in the above formula and will not be repeated here.

[0069] The process of forming the mask image specifically includes repeating multiple rounds of:

[0070] Generate a random number r between [0, 1];

[0071] If r <q(x i ), then the video block i is selected;

[0072] Otherwise, jump to the i+1 video block for the next round of selection until a video block k is selected such that: q[Xk-1]<r≤q[x k ] holds; wherein k-1 represents a video block whose cumulative probability value is less than video block k and adjacent to video block k after being sorted by cumulative probability, and k represents that after selecting the kth video block, its cumulative probability value is greater than or equal to the random number r.

[0073] As a typical example, in the above process, according to the mask probability map representing all video blocks, a roulette selection algorithm is used to select the mask block. The roulette algorithm is a selection strategy in the genetic algorithm, and its core idea is that the probability of an individual being selected is proportional to its fitness. In the selection process of the mask block, the present invention regards the probability value in the mask probability map as the fitness of each video block, and calculates the selection probability of each video block. After that, the cumulative probability is calculated until the number of selected video blocks reaches the number of mask blocks required for the preset mask rate (for example, 90%, of course, it can also be appropriately adjusted up and down according to the training requirements, such as 60-95%). After that, the encoder learns the feature representation from the retained video blocks, and the decoder reconstructs and predicts the masked video blocks. Compared with the traditional random mask strategy, the probability mask strategy based on spatiotemporal clues proposed in the present invention can make the model pay more attention to the motion area and distant view area rich in semantic information, and achieve significantly different technical effects.

[0074] As for the loss function, in some embodiments, the loss function of the training method includes perspective reconstruction loss, which uses the depth value in the depth map as the weight of the corresponding video block, weightedly calculates the difference between the reconstruction result of the video block and the true value and summarizes them, wherein the weight of the video block is positively correlated with the depth value.

[0075] More specifically in some embodiments, the perspective loss function is expressed as:

[0076]

[0077] Among them, L rec represents the perspective loss function; ψ(·) represents the mapping function of the depth value, and the value is positively correlated with the depth value. In a typical representative case, it corresponds to Figure 1 The “loss window” in I represents the reconstructed video block; k，t represents the masked video block; k represents the serial number of the video block; t represents the serial number of the video frame, and the other symbols have been clarified in the above formula and will not be repeated here.

[0078] The mapping function can be a continuous function with positive correlation, or a piecewise function with positive correlation and fixed values ​​in each piece. Figure 1 As shown in , different depth values ​​are divided into 4 steps according to their relative sizes, and each step has a value of 1.2-1.0-0.8-0.6 from large to small. Of course, the feasible range is not limited to the method of this example.

[0079] The above scheme proposes a perspective reconstruction loss based on the depth information of the video. This loss uses the mean square error to measure the reconstruction quality of the model for the masked area, and introduces depth information to emphasize the difference in contribution of different areas in the surveillance video scene to the reconstruction task, thereby promoting the model to learn better feature representation.

[0080] This weight gives a higher weight to the distant area, thereby increasing the difficulty of the model to learn distant area targets, and enhancing the model's ability to learn distant area targets that contain rich semantic information. Compared with the reconstruction loss that pays equal attention to each area, the perspective reconstruction loss can guide the model to pay more attention to specific areas or targets, thereby improving the reconstruction quality and feature representation capabilities.

[0081] As a further application of the above technical solution, a second aspect of an embodiment of the present invention further provides a method for detecting anomalies in a video with a fixed viewing angle, which comprises the following steps:

[0082] A feature extractor obtained by pre-training using the pre-training method provided in any of the above embodiments extracts features of the video to be detected to obtain video features;

[0083] The video features are evaluated for abnormality using a scorer to obtain an abnormality detection result.

[0084] In the feature extraction process of anomaly detection, a typical implementation case of the present invention uses the above-mentioned pre-trained model as a visual feature extractor to replace traditional feature extractors such as I3D and C3D to extract features from anomaly detection datasets. The extracted features can then be used in a variety of anomaly detection networks (such as TEVAD) for anomaly prediction and scoring.

[0085] Correspondingly, an embodiment of the present invention further provides a pre-training system for a feature extractor for video anomaly detection, wherein the feature extractor is used for anomaly detection in a video with a fixed viewing angle, and comprises:

[0086] A mask module is used to divide the training video into blocks and perform masking to cover a part of the video blocks and retain another part of the video blocks as input video blocks;

[0087] An encoder module, configured to learn a feature representation of the input video block using an encoder;

[0088] A decoder module, configured to reconstruct the masked video block based on the feature representation using a decoder, and update the parameters of the encoder based on the similarity between the reconstructed video block and the masked video block, and finally serve as the feature extractor;

[0089] Also, correspondingly, and different from the existing network structure training system, the mask module specifically includes:

[0090] A differential depth unit, used to obtain a frame difference map of the training video by using inter-frame difference, and to obtain a depth map of the training video by using depth estimation;

[0091] A probability map unit, configured to form a temporal probability map using the frame difference map, form a spatial probability map using the depth map, and fuse the spatial probability map and the temporal probability map to form a mask probability map;

[0092] A masking selection unit, configured to select and form a mask map based on the mask probability map, and selectively mask the video block based on the mask map;

[0093] When forming the mask map, the probability of any of the video blocks being selected for masking is positively correlated with the degree of motion in the corresponding temporal probability map, and negatively correlated with the depth value in the spatial probability map.

[0094] In addition, an embodiment of the present invention further provides a readable storage medium, which stores a computer program. When the computer program is executed, the steps of the above-mentioned pre-training method or video anomaly detection method are executed.

[0095] The technical solution of the present invention is further described in detail below through several embodiments and in conjunction with the accompanying drawings. However, the selected embodiments are only used to illustrate the present invention, and do not limit the scope of the present invention.

[0096] Example 1

[0097] This embodiment applies the process of the pre-training method exemplified above to the pre-training of the feature extractor and the anomaly detection of the surveillance video, as shown below.

[0098] Experimental setup: During the pre-training phase of the model, ViT-B was used as the default architecture for the encoder, and the decoder was the same architecture. 800 rounds of training were performed on the surveillance dataset with a batch size of 512 and 128 Ascend 910N PUs. AdamW optimizer and cosine learning rate scheduler were used, with an initial learning rate of 1.5e-4, weight decay of 0.05, β1 of 0.9, β2 of 0.999, and a warm-up period of 40 rounds. During the mask generation process, a mask ratio of 90% was applied. The pre-trained model was used as the video encoder in the anomaly detection stage.

[0099] The training data uses a self-built surveillance data set, and actual detection is performed on the anomaly detection data set UCF-Crime and TAD. The area under the curve (AUC) of the frame-level receiver operating characteristic (ROC) curve is used to evaluate the performance of the method used in this embodiment on the UCF-Crime and TAD benchmarks.

[0100] Table 1. Comparison of method performance on UCF-Crime and TAD datasets

[0101]

[0102] Table 1 shows the performance comparison of this method with other methods, where an AUC performance of 86.0% is achieved on the UCF-Crime dataset, which is 1.7% to 3.9% higher than other methods; and a performance of 90.3% is achieved on the TAD dataset.

[0103] Comparative Example 1

[0104] This comparative example is substantially the same as Example 1, except that:

[0105] No statistics of spatiotemporal information are performed, but a random mask strategy is used to cover the video blocks, and the loss function does not involve weight terms related to depth values ​​(1oss window).

[0106] In order to fully demonstrate the effectiveness of the modules proposed in this method, we conducted ablation experiments on the UCF-Crime and TAD datasets. The specific settings are to adopt a random mask strategy, use only temporal information, and a mask strategy guided by the fusion of temporal and spatial information.

[0107] Table 2 Ablation experiment results of mask strategy

[0108]

[0109] Table 2 shows the ablation experiment comparison on the mask strategy. When the random mask strategy is used, the AUC performance of UCF-Crime and TAD is 82.6% and 88.1% respectively. When the mask strategy guided by temporal information is used alone, the performance of both datasets is improved by 1.0%; when the mask strategy guided by temporal and spatial information is integrated, the performance of UCF-Crime is further improved, and the performance of TAD is slightly reduced compared with the random mask strategy.

[0110] Comparative Example 2

[0111] This comparative example is substantially the same as Example 1, except that:

[0112] The weight term related to the depth value is deleted from the loss function (loss window).

[0113] Table 3 Ablation experiment of loss function

[0114]

[0115] Table 3 shows the ablation experiment of the proposed perspective reconstruction loss. The results show that after adopting the improved perspective reconstruction loss, the performance of UCF-Crime and TAD has been significantly improved, which fully demonstrates that the control of loss weight is very effective for video encoders to learn better feature representations.

[0116] Based on the above embodiments and comparative examples, it can be clearly seen that the pre-training method provided in the embodiments of the present invention can provide a specific visual feature extractor for this type of video anomaly detection with a fixed viewpoint, thereby solving the domain difference problem between the general behavior recognition dataset used in the current network model training and the fixed viewpoint video, and improving the accuracy and efficiency of anomaly detection in fixed viewpoint videos by extracting feature representations customized for video anomaly detection.

[0117] It should be understood that the above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A pre-training method for a feature extractor for video anomaly detection, wherein the feature extractor is used for anomaly detection in a video with a fixed viewing angle, comprising: Divide the training video into blocks and perform masking, cover part of the video blocks, and keep the other part of the video blocks as input video blocks; learning, using an encoder, a feature representation of the input video block; Reconstructing the masked video block using a decoder based on the feature representation, and updating the parameters of the encoder based on the similarity between the reconstructed video block and the masked video block, and finally serving as the feature extractor; It is characterized in that the masking process specifically includes: Obtaining a frame difference image of the training video by using inter-frame difference, and obtaining a depth image of the training video by using depth estimation; Using the frame difference map to form a time probability map, using the depth map to form a spatial probability map, and fusing the spatial probability map and the time probability map to form a mask probability map; Selecting to form a mask map based on the mask probability map, and selectively masking the video block based on the mask map; When forming the mask map, the probability of any of the video blocks being selected for masking is positively correlated with the degree of motion in the corresponding temporal probability map, and negatively correlated with the depth value in the spatial probability map.

2. The pre-training method according to claim 1, characterized in that: The formation method of the mask probability map is expressed as: P m =αP s +βP d ; Among them, P m represents the mask probability map; P s represents the time probability graph; P d represents the spatial probability map; α and β represent weight coefficients, and their sum is 1.

3. The pre-training method according to claim 2, characterized in that: The generation process of the time probability map and the space probability map is expressed as: in, represents the probability value of the corresponding video block i in the temporal probability map; represents the probability value of the corresponding video block i in the spatial probability map; s (·) Indicates the frame difference change amplitude of the video block; d (·) Indicates the depth value of the video block; i indicates the sequence number of the video block; HW / N 2 Represents the number of video blocks, H and W are the number of pixels of the video frame height and width respectively.

4. The pre-training method according to any one of claims 1 to 3, characterized in that: The masking process also includes: distinguishing a moving area and a stationary area based on the frame difference map; In the motion area, a preset proportion of video blocks are randomly selected, and the probability of the selected video blocks being masked is specified as 0 in the mask probability map, and the selected video blocks are forcibly retained.

5. The pre-training method according to claim 1, characterized in that: The mask map forming process adopts a roulette wheel selection algorithm to perform multiple rounds of selection. For any of the video blocks, the probability of being selected in the current round is related to the probability value in the corresponding mask probability map until the proportion of the number of selected video blocks reaches a preset mask rate.

6. The pre-training method according to claim 5, characterized in that: The probability of the video block being selected is calculated as follows: Among them, p(xi) represents the selection probability of video block i; q(x i ) represents the cumulative probability; P(x.) represents the probability value of the corresponding video block in the mask probability map; j represents the sequence number of the video block; The process of forming the mask image specifically includes repeating multiple rounds of: Generate a random number r between [0, 1]; If r <q(x i ), then the video block i is selected; Otherwise, jump to the i+1 video block for the next round of selection until a video block k is selected such that: q[xk-1]<r≤q[x k ] holds; wherein k-1 represents a video block whose cumulative probability value is less than video block k and adjacent to video block k after being sorted by cumulative probability, and k represents that after selecting the kth video block, its cumulative probability value is greater than or equal to the random number r.

7. The pre-training method according to claim 1, characterized in that: The loss function of the training method includes perspective reconstruction loss, which uses the depth value in the depth map as the weight of the corresponding video block, weightedly calculates the difference between the reconstruction result of the video block and the true value and summarizes it, wherein the weight of the video block is positively correlated with the depth value.

8. The pre-training method according to claim 7, characterized in that: The perspective loss function is expressed as: Among them, L rec represents the perspective loss function; ψ(·) represents the mapping function of the depth value, and the value is positively correlated with the depth value; I represents the reconstructed video block; k，t represents the masked video block; k represents the sequence number of the video block; t represents the sequence number of the video frame.

9. A method for detecting anomalies in a video with a fixed viewing angle, characterized in that: include: A feature extractor obtained by pre-training using the training method described in any one of claims 1 to 8 extracts features from the video to be detected to obtain video features; The video features are evaluated for abnormality using a scorer to obtain an abnormality detection result.

10. A pre-training system for a feature extractor for video anomaly detection, wherein the feature extractor is used for anomaly detection in a video with a fixed viewing angle, comprising: A mask module is used to divide the training video into blocks and perform masking to cover a part of the video blocks and retain another part of the video blocks as input video blocks; An encoder module, configured to learn a feature representation of the input video block using an encoder; A decoder module, configured to reconstruct the masked video block based on the feature representation using a decoder, and update the parameters of the encoder based on the similarity between the reconstructed video block and the masked video block, and finally serve as the feature extractor; It is characterized in that the mask module specifically includes: A differential depth unit, used to obtain a frame difference map of the training video by using inter-frame difference, and to obtain a depth map of the training video by using depth estimation; A probability map unit, configured to form a temporal probability map using the frame difference map, form a spatial probability map using the depth map, and fuse the spatial probability map and the temporal probability map to form a mask probability map; A mask selection unit, configured to select and form a mask map based on the mask probability map, and selectively mask the video block based on the mask map; When forming the mask map, the probability of any of the video blocks being selected for masking is positively correlated with the degree of motion in the corresponding temporal probability map, and negatively correlated with the depth value in the spatial probability map.

Images