Video visual feature extraction method and system based on spatiotemporal weighted pooling and application thereof

Abstract

The application discloses a kind of spatiotemporal weighted pooling video visual feature extraction method, system and application.The method includes: forming time domain information by background difference to video segment, obtaining space domain information by depth estimation;It is divided into multiple video blocks, corresponding to be summarized as time domain block information, space domain block information;Carrying out feature extraction, obtain multiple initial features;Screening is obtained using time domain block information Filter feature;With space domain block feature as the index of weight is weighted fusion, weight and depth value are positively correlated.The feature extraction method and video anomaly detection method provided by the application introduce spatiotemporal clues, guide feature selection and weighted fusion in the final pooling stage of reasoning to enhance feature representation, so that the extracted features pay more attention to dynamic regions with rich semantics, and the features of small targets are enhanced according to the perspective characteristics of fixed-angle videos, especially surveillance cameras, thereby significantly improving the performance of video anomaly detection.

Description

Technical Field

[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a spatiotemporal weighted pooling video visual feature extraction method, system and application. Background Art

[0002] Video anomaly detection is a key component of large-scale surveillance systems, and visual feature extractors are crucial to the performance of video anomaly detection networks. In recent years, visual feature extractors based on the Transformer architecture have shown great potential, achieving excellent performance in a variety of applications. Improving the Transformer-based visual feature extractor for surveillance video anomaly detection scenarios has become a valuable research topic.

[0003] However, there is a large amount of spatiotemporal information redundancy of irrelevant abnormal events in surveillance videos, which will confuse the final output representation in the final pooling stage of visual feature extraction, resulting in a decline in the performance of the feature extractor, affecting detection efficiency and adversely affecting detection accuracy. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a video visual feature extraction method, system and application based on spatiotemporal weighted pooling to solve the problems existing in the visual feature extractor of the Transformer architecture in the prior art.

[0005] To achieve the aforementioned object of the invention, the technical solutions adopted by the present invention include:

[0006] In a first aspect, the present invention provides a method for extracting video visual features by spatiotemporal weighted pooling, wherein the video visual features are used for anomaly detection in a video with a fixed viewing angle, and the method comprises:

[0007] Acquire a video clip and perform background subtraction on the video clip to form temporal information, and perform depth estimation to obtain spatial information;

[0008] Dividing the video clip into a plurality of video blocks, and correspondingly aggregating the temporal information into temporal block information and aggregating the spatial information into spatial block information;

[0009] Using a feature extractor with a Transformer architecture to extract features from each of the video blocks to obtain a plurality of initial features;

[0010] Filtering the initial features using the time domain block information, removing some features with small image change amplitudes in the initial features, and obtaining filtered features;

[0011] The spatial block feature is used as a weight indicator to perform weighted fusion on the screening features to obtain a video visual feature corresponding to the video clip, wherein the weight is positively correlated with the depth value of the video block feature.

[0012] In a second aspect, the present invention further provides a method for detecting anomalies in a video with a fixed viewing angle, comprising:

[0013] For videos with a fixed viewing angle, the above-mentioned video visual feature extraction method is used to extract video visual features;

[0014] The video visual feature is input into a scorer, and an abnormality score is performed on the video visual feature. The result of the abnormality score is used to indicate whether there is an abnormality in the video picture corresponding to the video visual feature and the degree of the abnormality.

[0015] In a third aspect, the present invention further provides a video visual feature extraction system based on spatiotemporal weighted pooling, wherein the video visual feature is used for anomaly detection in a video with a fixed viewing angle, and comprises:

[0016] A segment estimation module is used to obtain video segments and perform background difference on the video segments to form temporal information, and perform depth estimation to obtain spatial information;

[0017] A block aggregation module, configured to divide the video clip into a plurality of video blocks, and correspondingly aggregate the temporal information into temporal block information and aggregate the spatial information into spatial block information;

[0018] A feature extraction module, configured to extract features from each of the video blocks using a feature extractor with a Transformer architecture to obtain a plurality of initial features;

[0019] A feature screening module, configured to screen the initial features using the time domain block information, remove some features with small image change amplitudes from the initial features, and obtain screened features;

[0020] A weighted fusion module is used to perform weighted fusion on the screening features using the spatial block features as weight indicators to obtain video visual features corresponding to the video clips, wherein the weights are positively correlated with the depth values ​​of the video block features.

[0021] In a fourth aspect, the present invention further provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is run, the steps of the above-mentioned video visual feature extraction method are executed.

[0022] Based on the above technical solution, compared with the prior art, the beneficial effects of the present invention include at least:

[0023] The feature extraction method and video anomaly detection method provided by the present invention introduce spatiotemporal cues to guide feature selection and weighted fusion in the final pooling stage of Transformer reasoning to enhance feature representation, so that the extracted features pay more attention to semantically rich dynamic areas. At the same time, the features of small targets are enhanced according to the perspective characteristics of fixed-view videos, especially surveillance cameras, thereby significantly improving the performance of video anomaly detection.

[0024] The above description is only an overview of the technical solution of the present invention. In order to enable those skilled in the art to more clearly understand the technical means of this application and implement them according to the contents of the specification, the following is an explanation of the preferred embodiments of the present invention with detailed drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The figure is a flowchart of a video anomaly detection method provided by a typical implementation case of the present invention. DETAILED DESCRIPTION

[0026] The video visual feature extraction and video anomaly detection process of the Transformer architecture mentioned in the background technology is generally as follows:

[0027] 1. Adjust the video resolution to meet the input requirements of the visual feature extractor and evenly split the long video V into 16-frame video segments v i ∈R 16,H,W,3 , V = [v1, v2, ...].

[0028] 2. For each video segment v∈R 16,H,W,3 Divide the video into N equal-sized blocks Cube∈R without overlapping 2 ,16,16,3 , a video v is composed of N video blocks, N = (16 / 2)*(H / 16)*(W / 16).

[0029] 3. The embedding features of these video blocks can be expressed as X = [X i ∈R 1,C |1≤i≤N], where X i =Emb(Cube i ).

[0030] 4. Send X to the trained Transformer encoder Encoder(·) to extract visual features f∈R N,C :

[0031] f out =Encoder(X).

[0032] 5. Through average pooling, f out ∈RN,C Fusion is performed to obtain f∈R 1,C As the visual representation of the input video segment v:

[0033] f=Mean(f out ).

[0034] 6. Use the above method to process each video clip v i Extract visual features and get f i Then, it is sent to the anomaly scorer to get the anomaly score s of the segment. i :

[0035] s i =Scorer(f i ).

[0036] The above extraction method is generally applicable to video anomaly detection in various scenarios. However, for anomaly detection in some fixed-view videos, such as surveillance videos, the general scheme does not consider the local dynamics and perspective structure unique to fixed-view videos. Many coding tags may not contain information about video anomalies. When performing average pooling in step 5, the effective features will not be significant, resulting in a decrease in detection efficiency and accuracy.

[0037] Therefore, in view of the shortcomings of the prior art, the inventors of this case, after long-term research and extensive practice, have proposed the technical solution of the present invention. The following will further explain this technical solution, its implementation process and principles.

[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0039] Moreover, relational terms such as “first” and “second” are merely used to distinguish one component or method step from another with the same name, but do not necessarily require or imply any actual relationship or order between these components or method steps.

[0040] The main technical idea of ​​the present invention is to introduce the temporal dynamic information and spatial structural information of the video in the spatial domain on the basis of the existing feature extraction method. In the final pooling stage, the features with strong dynamic information are first retained based on the temporal information, and the spatial domain information is introduced to adjust the weights of the features before fusion.

[0041] Based on the above technical concept, the embodiment of the present invention first provides a method for extracting video visual features by spatiotemporal weighted pooling. The video visual features are used for anomaly detection in videos with a fixed viewing angle, which includes the following specific steps:

[0042] Acquire a video clip and perform background subtraction on the video clip to form temporal information, and perform depth estimation to obtain spatial information;

[0043] Dividing the video clip into a plurality of video blocks, and correspondingly aggregating the temporal information into temporal block information and aggregating the spatial information into spatial block information;

[0044] Using a feature extractor with a Transformer architecture to extract features from each of the video blocks to obtain a plurality of initial features;

[0045] Filtering the initial features using the time domain block information, removing some features with small image change amplitudes in the initial features, and obtaining filtered features;

[0046] The spatial block feature is used as a weight indicator to perform weighted fusion on the screening features to obtain a video visual feature corresponding to the video clip, wherein the weight is positively correlated with the depth value of the video block feature.

[0047] The above technical solution extracts dynamic information in the temporal domain and spatial structural information in the spatial domain from the original fixed-view video to guide visual feature extraction. The visual features are then fused through feature selection and weighted fusion. This extracts more significant visual features based on the characteristics of the fixed-view video, improving the efficiency and performance of video anomaly detection. However, since distant objects are relatively small in the video image and occupy fewer spatiotemporal blocks, their corresponding features are diluted by the features corresponding to a large number of other objects and the background during the fusion phase, resulting in insignificant features. The insignificant features of abnormal events in the distance can lead to missed anomalies. Therefore, the present invention gives greater weight to distant areas based on actual video features.

[0048] It should be noted that the various technical means involved in the above scheme, such as how to obtain video clips, how to divide video blocks, how to perform background difference and depth estimation, etc. (for example, frame difference method and Gaussian mixture model can be used for background difference, MiDaS and other deep learning-based algorithms can be used to achieve depth estimation, but not limited to this), can all be directly adopted or implemented with reference to various corresponding technical means in the existing technology, as long as the corresponding functions can be achieved; and the parameters involved in these technical means, such as the number of frames in the video clip, the number of video blocks, etc., can all be set or adjusted for reference, and will not have an essential impact on the achievement of the final technical effect.

[0049] Specifically regarding how to obtain spatiotemporal features, in some embodiments, the acquisition method of the temporal block information and the spatial block information may specifically include:

[0050] For any of the video blocks, the temporal information corresponding to each frame in the video block is summed to obtain the temporal block information, and the spatial information corresponding to each frame is summed to obtain the spatial block information.

[0051] Regarding the feature screening process, in some embodiments, the method for obtaining the screening features may specifically include:

[0052] Sort by image change amplitude, and search for part of the time domain block information with a large image change amplitude among the plurality of time domain block information using a preset screening ratio;

[0053] Part of the initial features corresponding to the found part of the time domain block information is retained as the screening features, and the remaining initial features are deleted.

[0054] Specifically, in some embodiments, the process of obtaining the screening feature can be expressed as:

[0055]

[0056] Among them, ids represents the index of the partial time domain block information found; represents the time domain block information; i represents the sequence number; N represents the total number of the time domain blocks; L represents the preset screening ratio; f slk represents the screening feature; Represents the spatial block feature; f out represents the initial features.

[0057] Regarding the final pooling (fusion) process, in some embodiments, the weighted fusion process can be expressed as:

[0058]

[0059] in, Represents the weight; ∈ is a very small value greater than 0, and in order to prevent miscalculation by division by 0, the order of magnitude is 10 -6 ; E(·) represents mean calculation; Var(·) represents variance calculation; f represents the visual features of the video; Mean(·) represents weighted fusion calculation.

[0060] The extraction, screening, and fusion of the above-mentioned spatiotemporal features are significant improvements of the present invention compared to the conventional video visual feature extraction method of the Transformer architecture, while other steps are basically consistent or similar to the current common implementation methods.

[0061] For example, in some embodiments, the feature extraction process can be specifically expressed as follows:

[0062] X i =Emb(Cube i );

[0063] X=[X i ∈R 1,C |1≤i≤N];

[0064] f out =Encoder(X);

[0065] Among them, X i represents the embedded features of the video block; X represents the input features; Encoder(·) represents the feature extractor.

[0066] In other aspects, in some embodiments, the video clip is formed by continuously and evenly segmenting a continuous video; or in some embodiments, the video blocks are formed by continuously, non-overlappingly, and equally sized segmentation of the frames of the video clip.

[0067] As some typical application examples of the above technical solutions, the specific process of video visual feature extraction based on the optimized and modified Transformer architecture provided by the technical solution of the present invention is as follows:

[0068] 1. Adjust the video resolution to meet the input requirements of the visual feature extractor and evenly split the long video V into 16-frame video segments v∈R 16,H,W,3 , V = [v1, v2, ...].

[0069] 2. At the same time, the time domain information v of each video clip is obtained by background subtraction and depth estimation T ∈R 16,H,W With airspace information v S ∈R 16,H,W ;

[0070] 3. For each video segment v∈R 16,H,W,3 Divide the video into N equal-sized blocks Cube∈R without overlapping 2 ,16,16,3 , a video v is composed of N video blocks, N = (16 / 2)*(H / 16)*(W / 16).

[0071] 4. Correspondingly, the time domain information v T The same method is used to divide the blocks and sum the contents of each block. The N values ​​obtained represent the temporal block information of the intensity of the video motion in each video block (Cube) in the video.

[0072] For airspace information v S The spatial domain block information representing the video depth information in each video block (Cube) in the video is obtained by the same processing method

[0073] 5. The embedding features of these video blocks can be expressed as X = [X i ∈R 1,C |1≤i≤N], where X i =Emb(Cube i ).

[0074] 6. Send X to the pre-trained Transformer encoder architecture encoder Encoder(·) to extract visual features f∈R N,C :

[0075] f out =Encoder(X).

[0076] As a specific example, the encoder used at this time can be trained using VideoMAE, but is not limited to this.

[0077] 7. No longer directly average pooling f out ∈R N,C Instead of fusion, firstly according to the corresponding time domain block information ( By doing frame difference and downsampling on the original video, some redundant features are filtered out. The specific method is: find The smallest proportion R (in this case R = 50%, generally choose 30-70% for adaptive adjustment) of the elements, and then f out The corresponding element is deleted.

[0078]

[0079] In the above formula, ids∈{1, 2…N} is f out The element index of . The temporal information obtained by performing frame subtraction and downsampling on the original video; f slk The visual features with high dynamic effects are screened.

[0080] 8. For the remaining features f_slk∈R that have not been screened out (1-R)N,C , according to the corresponding spatial information (estimated depth value) The weighted fusion is performed to obtain the representation feature (i.e., the video visual feature) f of the final video segment v. The distant part with a high depth value in f_slk will be given a higher weight:

[0081]

[0082] In the above formula, is the calculated feature weight; f slk The estimated depth value of the corresponding element in the original video.

[0083] Based on the above exemplary process, video visual features after screening and weighted fusion processing can be obtained. The data distribution of the video visual features conforms to the temporal and spatial characteristics of fixed-viewpoint videos, and can therefore accurately reflect the key information of such videos. The discriminator training and discriminator scoring based on the video visual features will both bring higher execution efficiency and accuracy.

[0084] As some typical applications of the above technical solutions, a second aspect of an embodiment of the present invention further provides a method for detecting anomalies in a video with a fixed viewing angle, which includes the following steps:

[0085] For a video with a fixed viewing angle, extracting video visual features using the video visual feature extraction method provided in any of the above embodiments;

[0086] The video visual feature is input into a scorer, and an abnormality score is performed on the video visual feature. The result of the abnormality score is used to indicate whether there is an abnormality in the video picture corresponding to the video visual feature and the degree of the abnormality.

[0087] Of course, in some cases, the scorer can be trained in the same way as the above process, including supervised, semi-supervised or unsupervised training, etc. The specific process is to score the training samples using the above method, and then evaluate the model performance based on labels or adversarial training and then update the parameters.

[0088] As a typical example of the above application, a representative anomaly detection process includes:

[0089] By using the exemplary method provided above, each video segment v i Extract visual features and get f i Then, it is sent to the anomaly scorer composed of the MLP network to infer the anomaly and obtain the anomaly score s of the segment. i :

[0090] s i =Scorer(f i ).

[0091] Of course, the MLP network used in this embodiment is a very typical scorer network structure, but it does not mean that the implementation of the present invention is limited to this network structure. Other network structures with similar functions can still achieve equivalent functions. The focus of the present invention is still on how to obtain highly representative video visual features.

[0092] Correspondingly, an embodiment of the present invention further provides a video visual feature extraction system using spatiotemporal weighted pooling, wherein the video visual feature is used for anomaly detection in a video with a fixed viewing angle, and the system includes:

[0093] A segment estimation module is used to obtain video segments and perform background difference on the video segments to form temporal information, and perform depth estimation to obtain spatial information;

[0094] A block aggregation module, configured to divide the video clip into a plurality of video blocks, and correspondingly aggregate the temporal information into temporal block information and aggregate the spatial information into spatial block information;

[0095] A feature extraction module, configured to extract features from each of the video blocks using a feature extractor with a Transformer architecture to obtain a plurality of initial features;

[0096] A feature screening module, configured to screen the initial features using the time domain block information, remove some features with small image change amplitudes from the initial features, and obtain screened features;

[0097] A weighted fusion module is used to perform weighted fusion on the screening features using the spatial block features as weight indicators to obtain video visual features corresponding to the video clips, wherein the weights are positively correlated with the depth values ​​of the video block features.

[0098] Furthermore, an embodiment of the present invention also provides a readable storage medium, in which a computer program is stored. When the computer program is executed, the steps of the video visual feature extraction method provided in any of the above embodiments are executed.

[0099] The technical solution of the present invention is further described in detail below through several embodiments and in conjunction with the accompanying drawings. However, the selected embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0100] Example 1

[0101] This embodiment uses the representative video visual feature extraction method and representative video anomaly detection method described above to conduct simulation experiments. In addition to the detailed parameters indicated above, other specific details include:

[0102] The video samples come from the UCF-Crime dataset, which includes 1900 videos. The number of video blocks N is 1568 (16*16*8). The parameters of weighted fusion are set as α=0.05; ∈=10 -6 .

[0103] Finally, real-time inference was achieved, with an AUC (Area Under the Curve) performance of 87.4 on the UCF-Crime dataset.

[0104] Comparative Example 1

[0105] This comparative example is substantially the same as Example 1, except that the video visual feature extractor of the CNN architecture in Example 1 is replaced by the video visual feature extractor of the traditional Transformer architecture, and average pooling is used for the final feature fusion.

[0106] Final anomaly detection performance evaluation: Real-time inference is achieved, and the AUC (Area Under the Curve) performance on the UCF-Crime dataset is 84.9.

[0107] Based on the above embodiments and control examples, it can be clearly seen that the feature extraction method and video anomaly detection method provided by the embodiments of the present invention introduce spatiotemporal cues to guide feature selection and weighted fusion in the final pooling stage of Transformer reasoning to enhance feature representation, so that the extracted features pay more attention to semantically rich dynamic areas. At the same time, according to the fixed-angle video, especially the perspective characteristics of surveillance cameras, the features of small targets are enhanced, thereby significantly improving the performance of video anomaly detection.

[0108] It should be understood that the above embodiments are merely illustrative of the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent variations or modifications made in accordance with the spirit and substance of the present invention are intended to be encompassed within the scope of protection of the present invention.

Claims

1. A method for extracting video visual features by spatiotemporal weighted pooling, wherein the video visual features are used for anomaly detection in fixed-view videos, characterized in that: include: Acquire a video clip and perform background subtraction on the video clip to form temporal information, and perform depth estimation to obtain spatial information; Dividing the video clip into a plurality of video blocks, and correspondingly aggregating the temporal information into temporal block information and aggregating the spatial information into spatial block information; Using a feature extractor with a Transformer architecture to extract features from each of the video blocks to obtain a plurality of initial features; Filtering the initial features using the time domain block information, removing some features with small image change amplitudes in the initial features, and obtaining filtered features; The spatial block information is used as a weight indicator to perform weighted fusion on the screening features to obtain video visual features corresponding to the video clip, wherein the weight is positively correlated with the depth value of the spatial block information.

2. The video visual feature extraction method according to claim 1, characterized in that: The method for obtaining the temporal block information and the spatial block information specifically includes: For any of the video blocks, the temporal information corresponding to each frame in the video block is summed to obtain the temporal block information, and the spatial information corresponding to each frame is summed to obtain the spatial block information.

3. The video visual feature extraction method according to claim 1, characterized in that: The method for obtaining the screening feature specifically includes: Sort by image change amplitude, and search for part of the time domain block information with a large image change amplitude among the plurality of time domain block information using a preset screening ratio; Part of the initial features corresponding to the found part of the time domain block information is retained as the screening features, and the remaining initial features are deleted.

4. The video visual feature extraction method according to claim 3, characterized in that: The process of obtaining the screening features is expressed as follows: Among them, ids represents the index of the partial time domain block information found; represents the time domain block information; i represents the sequence number; N represents the total number of the time domain blocks; L represents the preset screening ratio; f slk represents the screening feature; Indicates the spatial block information; f out represents the initial features.

5. The video visual feature extraction method according to claim 3, characterized in that: The weighted fusion process is expressed as: in, represents the weight; α represents a hyperparameter that controls the degree of influence of spatial features; ∈ is a very small value greater than 0, with an order of magnitude of 10 -6 ; E(·) represents mean calculation; Var(·) represents variance calculation; f represents the visual features of the video; Mean(·) represents weighted fusion calculation.

6. The video visual feature extraction method according to claim 3, characterized in that: The feature extraction process is expressed as: X i =Emb(Cube i ); X=[X i ∈R 1,C |1≤i≤N]; f out =Encoder(X); Among them, X i represents the embedded features of the video block; X represents the input features; Encoder(·) represents the feature extractor.

7. The video visual feature extraction method according to claim 1, characterized in that: The video clip is formed by continuously averaging a continuous video segment; And / or, the video blocks are formed by dividing the frames of the video clip into continuous, non-overlapping, and equal-sized segments.

8. A method for detecting anomalies in a video with a fixed viewing angle, characterized in that: include: For a video with a fixed viewing angle, extracting video visual features using the video visual feature extraction method described in any one of claims 1 to 7; The video visual feature is input into a scorer, and an abnormality score is performed on the video visual feature. The result of the abnormality score is used to indicate whether there is an abnormality in the video picture corresponding to the video visual feature and the degree of the abnormality.

9. A spatiotemporal weighted pooling video visual feature extraction system, wherein the video visual features are used for anomaly detection in fixed-view videos, characterized in that: include: A segment estimation module is used to obtain video segments and perform background difference on the video segments to form temporal information, and perform depth estimation to obtain spatial information; A block aggregation module, configured to divide the video clip into a plurality of video blocks, and correspondingly aggregate the temporal information into temporal block information and aggregate the spatial information into spatial block information; A feature extraction module, configured to extract features from each of the video blocks using a feature extractor with a Transformer architecture to obtain a plurality of initial features; A feature screening module, configured to screen the initial features using the time domain block information, remove some features with small image change amplitudes from the initial features, and obtain screened features; A weighted fusion module is used to perform weighted fusion on the screening features using the spatial block information as a weight indicator to obtain video visual features corresponding to the video clip, wherein the weight is positively correlated with the depth value of the spatial block information.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed, the steps of the video visual feature extraction method according to any one of claims 1 to 7 are executed.

Images