Video visual feature extraction method and system based on space-time weighted pooling and application

Abstract

The invention discloses a time-space weighted pooling video visual feature extraction method, system and application. The method comprises the following steps: performing background differencing on a video clip to form time domain information, and performing depth estimation to obtain space domain information; dividing the video blocks into a plurality of video blocks, and correspondingly summarizing the video blocks into time domain block information and space domain block information; performing feature extraction to obtain a plurality of initial features; performing screening by using the time domain block information to obtain screening characteristics; the spatial domain block features are used as indexes of weights for weighted fusion, and the weights are in positive correlation with depth values. According to the feature extraction method and the video anomaly detection method, space-time clues are introduced, feature selection and weighted fusion are guided in the final pooling stage of reasoning so as to enhance feature representation, the extracted features pay more attention to a dynamic region with rich semantics, and meanwhile, according to a video with a fixed view angle, the feature extraction efficiency is improved. And particularly, the features of the small target are enhanced according to the perspective characteristics of monitoring camera shooting, so that the performance of video anomaly detection is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a spatiotemporal weighted pooling video visual feature extraction method, system and application. Background Art

[0002] Video anomaly detection is a key component in large-scale surveillance systems, and visual feature extractors are crucial to the performance of video anomaly detection networks. In recent years, visual feature extractors based on the Transformer architecture have shown great potential and have achieved excellent performance in applications in many fields. How to improve the visual feature extractor based on the Transformer architecture for surveillance video anomaly detection scenarios so that it can be effectively applied in the field of surveillance video anomaly detection has become a valuable research topic.

[0003] However, there is a large amount of redundant spatiotemporal information of irrelevant abnormal events in surveillance videos, which will confuse the final output representation in the final pooling stage of visual feature extraction, resulting in a decline in the performance of the feature extractor, affecting detection efficiency and having an adverse impact on detection accuracy. Summary of the invention

[0004] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a spatiotemporal weighted pooling video visual feature extraction method, system and application to solve the problems existing in the Transformer architecture visual feature extractor in the prior art.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention includes:

[0006] In a first aspect, the present invention provides a method for extracting video visual features by spatiotemporal weighted pooling, wherein the video visual features are used for abnormality detection of videos with a fixed viewing angle, and the method comprises:

[0007] Acquire a video clip and perform background difference on the video clip to form time domain information, and perform depth estimation to obtain spatial domain information;

[0008] Divide the video clip into a plurality of video blocks, and correspondingly aggregate the temporal information into temporal block information and aggregate the spatial information into spatial block information;

[0009] Using a feature extractor with a Transformer architecture to extract features from each of the video blocks to obtain a plurality of initial features;

[0010] The initial features are screened using the time domain block information, and some features with small image change amplitudes in the initial features are removed to obtain screened features;

[0011] The spatial domain block feature is used as an indicator of weight, and the screening features are weightedly fused to obtain the video visual feature corresponding to the video clip, wherein the weight is positively correlated with the depth value of the video block feature.

[0012] In a second aspect, the present invention further provides a method for detecting anomalies in a video with a fixed viewing angle, comprising:

[0013] For a video with a fixed viewing angle, the video visual features are extracted using the above-mentioned video visual feature extraction method;

[0014] The video visual feature is input into a scorer, and an abnormality score is performed on the video visual feature. The result of the abnormality score is used to indicate whether there is an abnormality in the video picture corresponding to the video visual feature and the degree of the abnormality.

[0015] In a third aspect, the present invention further provides a spatiotemporal weighted pooling video visual feature extraction system, wherein the video visual feature is used for abnormality detection of a video with a fixed viewing angle, and comprises:

[0016] A segment estimation module, used to obtain a video segment and perform background difference on the video segment to form temporal information, and perform depth estimation to obtain spatial information;

[0017] A block aggregation module, used to divide the video clip into a plurality of video blocks, and correspondingly aggregate the time domain information into time domain block information and aggregate the spatial domain information into spatial domain block information;

[0018] A feature extraction module, configured to extract features from each of the video blocks using a feature extractor with a Transformer architecture to obtain a plurality of initial features;

[0019] A feature screening module, used to screen the initial features using the time domain block information, remove some features with small image change amplitude in the initial features, and obtain screening features;

[0020] The weighted fusion module is used to perform weighted fusion on the screening features using the spatial block features as weight indicators to obtain video visual features corresponding to the video clips, wherein the weights are positively correlated with the depth values ​​of the video block features.

[0021] In a fourth aspect, the present invention further provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed, the steps of the above-mentioned video visual feature extraction method are executed.

[0022] Based on the above technical solution, compared with the prior art, the beneficial effects of the present invention at least include:

[0023] The feature extraction method and video anomaly detection method provided by the present invention guide feature selection and weighted fusion in the final pooling stage of Transformer reasoning to enhance feature representation by introducing spatiotemporal cues, so that the extracted features pay more attention to semantically rich dynamic areas, and at the same time enhance the features of small targets according to the perspective characteristics of fixed-angle videos, especially surveillance cameras, thereby significantly improving the performance of video anomaly detection.

[0024] The above description is only an overview of the technical solution of the present invention. In order to enable those skilled in the art to more clearly understand the technical means of the present application and implement them according to the contents of the specification, the following is a description of the preferred embodiments of the present invention in conjunction with detailed drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of a video anomaly detection method provided by a typical implementation case of the present invention. DETAILED DESCRIPTION

[0026] The video visual feature extraction and video anomaly detection process of the Transformer architecture mentioned in the background technology is usually as follows:

[0027] 1. Adjust the video resolution to meet the input requirements of the visual feature extractor, and evenly split the long video V into 16-frame video segments v i ∈R 16,H,W,3 , V = [v1, v2, ...].

[0028] 2. For each video segment v∈R 16,H,W,3 Divide the video into N equal-sized blocks Cube∈R without overlapping 2 ,16,16,3 , a video v is composed of N video blocks, N = (16 / 2)*(H / 16)*(W / 16).

[0029] 3. The embedding features of these video blocks can be expressed as X = [X i ∈R 1,C |1≤i≤N], where X i =Emb(Cube i ).

[0030] 4. Send X to the trained Transformer encoder Encoder(·) to extract visual features f∈R N,C :

[0031] f out =Encoder(X).

[0032] 5. Through average pooling, f out ∈RN,C Fusion, get f∈R 1,C As a visual representation of the input video segment v:

[0033] f=Mean(f out ).

[0034] 6. Use the above method to perform each video clip v i Extract visual features and get f i After that, it is sent to the anomaly scorer for inference to obtain the anomaly score s for the segment. i :

[0035] s i = Scorer(f i ).

[0036] The above extraction method is generally applicable to video anomaly detection in various scenarios. However, for anomaly detection of some videos with fixed perspectives, such as surveillance videos, the above general scheme does not take into account the local dynamics and perspective structure unique to fixed-perspective videos. Many coding tags may not contain information about video anomalies. When average pooling is performed in step 5, effective features will not be significant, resulting in a decrease in detection efficiency and accuracy.

[0037] Therefore, in view of the deficiencies in the prior art, the inventor of this case has proposed the technical solution of the present invention after long-term research and extensive practice. The technical solution, its implementation process and principle will be further explained as follows.

[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0039] Moreover, relational terms such as “first” and “second” are merely used to distinguish one component or method step from another with the same name, but do not necessarily require or imply any such actual relationship or order between these components or method steps.

[0040] The main technical idea of ​​the present invention is to introduce the temporal dynamic information and spatial structural information of the video in the spatial domain on the basis of the existing feature extraction method. In the final pooling stage, the features with strong dynamic information are first retained based on the temporal information, and the spatial domain information is introduced to adjust the weights of the features before fusion.

[0041] Based on the above technical concept, the embodiment of the present invention first provides a method for extracting video visual features by spatiotemporal weighted pooling, wherein the video visual features are used for abnormality detection of videos with a fixed viewing angle, and the method comprises the following specific steps:

[0042] Acquire a video clip and perform background difference on the video clip to form time domain information, and perform depth estimation to obtain spatial domain information;

[0043] Divide the video clip into a plurality of video blocks, and correspondingly aggregate the temporal information into temporal block information and aggregate the spatial information into spatial block information;

[0044] Using a feature extractor with a Transformer architecture to extract features from each of the video blocks to obtain a plurality of initial features;

[0045] The initial features are screened using the time domain block information, and some features with small image change amplitudes in the initial features are removed to obtain screened features;

[0046] The spatial domain block feature is used as an indicator of weight, and the screening features are weightedly fused to obtain the video visual feature corresponding to the video clip, wherein the weight is positively correlated with the depth value of the video block feature.

[0047] The above technical solution extracts dynamic information in the time domain and spatial structure information in the air domain from the original fixed-view video to guide visual feature extraction, fuses visual features through feature selection and weighted fusion, and extracts more significant visual features based on the characteristics of the fixed-view video to improve the efficiency and performance of video anomaly detection. Since the distant target is smaller in the video screen and occupies fewer time and space blocks, its corresponding features are diluted by the features corresponding to a large number of other objects and backgrounds during the fusion stage, resulting in insignificant features. The insignificant features of abnormal events in the distance will lead to missed detection of abnormalities. Therefore, based on the actual video features, the present invention gives a greater weight to the distant area.

[0048] It should be noted that the various technical means involved in the above scheme, such as how to obtain video clips, how to divide video blocks, how to perform background difference and depth estimation, etc. (for example, the frame difference method and Gaussian mixture model can be used for background difference, MiDaS and other deep learning-based algorithms can be used to achieve depth estimation, but not limited to this), can be directly adopted or implemented with reference to a variety of corresponding technical means in the prior art, as long as the corresponding functions can be achieved; and the parameters involved in these technical means, such as the number of frames in the video clip, the number of video blocks, etc., can all be set or adjusted for reference, and will not have a substantial impact on the achievement of the final technical effect.

[0049] Regarding how to obtain the spatiotemporal features, in some implementation schemes, the acquisition method of the temporal block information and the spatial block information may specifically include:

[0050] For any of the video blocks, the time domain information corresponding to each frame in the video block is summed to obtain the time domain block information, and the spatial domain information corresponding to each frame is summed to obtain the spatial domain block information.

[0051] Regarding the feature screening process, in some embodiments, the method for obtaining the screening features may specifically include:

[0052] Sort by image change amplitude, and search for part of the time domain block information with a large image change amplitude among the plurality of time domain block information at a preset screening ratio;

[0053] Part of the initial features corresponding to the found part of the time domain block information is retained as the screening features, and the remaining initial features are deleted.

[0054] Specifically, in some embodiments, the process of obtaining the screening feature can be expressed as:

[0055]

[0056] Wherein, ids represents the index of the partial time domain block information found; represents the time domain block information; i represents the sequence number; N represents the total number of the time domain blocks; L represents the preset screening ratio; f slk represents the screening feature; represents the spatial domain block feature; f out represents the initial feature.

[0057] Regarding the final pooling (fusion) process, in some embodiments, the weighted fusion process can be expressed as:

[0058]

[0059] in, Represents the weight; ∈ is a very small value greater than 0, and in order to prevent miscalculation by division by 0, the order of magnitude is 10 -6 ; E(·) represents mean calculation; Var(·) represents variance calculation; f represents the visual features of the video; Mean(·) represents weighted fusion calculation.

[0060] The extraction, screening and fusion of the above-mentioned spatiotemporal features are significant improvements of the present invention compared to the conventional video visual feature extraction method of the Transformer architecture, while other steps are basically consistent or similar to the current common implementation methods.

[0061] For example, in some embodiments, the feature extraction process can be specifically expressed as:

[0062] X i =Emb(Cube i );

[0063] X=[X i ∈R 1,C |1≤i≤N];

[0064] f out =Encoder(X);

[0065] Among them, X i represents the embedded features of the video block; X represents the input features; Encoder(·) represents the feature extractor.

[0066] In other aspects, in some implementation schemes, the video segment is formed by continuously and evenly segmenting a continuous video; or in some implementation schemes, the video blocks are formed by continuously, non-overlappingly, and equally sized segmentation of the frames of the video segment.

[0067] As some typical application examples of the above technical solutions, the specific process of video visual feature extraction of the Transformer architecture after optimization and modification based on the technical solution provided by the present invention is as follows:

[0068] 1. Adjust the video resolution to meet the input requirements of the visual feature extractor, and evenly split the long video V into 16-frame video segments v∈R 16,H,W,3 , V = [v1, v2, ...].

[0069] 2. The temporal information v of each video clip is obtained by background subtraction and depth estimation at the same time T ∈R 16,H,W With airspace information v S ∈R 16,H,W ;

[0070] 3. For each video segment v∈R 16,H,W,3 Divide the video into N equal-sized blocks Cube∈R without overlapping 2 ,16,16,3 , a video v is composed of N video blocks, N = (16 / 2)*(H / 16)*(W / 16).

[0071] 4. Correspondingly, the time domain information v T The blocks are divided in the same way, and the contents of each block are summed. The N values ​​obtained represent the temporal block information of the intensity of the video picture motion in each video block (Cube) in the video.

[0072] For airspace information v S The spatial block information representing the video picture depth information in each video block (Cube) in the video is obtained by the same processing method

[0073] 5. The embedding features of these video blocks can be expressed as X = [X i ∈R 1,C |1≤i≤N], where X i =Emb(Cube i ).

[0074] 6. Send X to the pre-trained Transformer encoder architecture encoder Encoder(·) to extract visual features f∈R N,C :

[0075] f out =Encoder(X).

[0076] As a specific example, the Encoder used at this time can be trained using VideoMAE, but is not limited to this.

[0077] 7. No longer directly average pooling f out ∈R N,C Instead of fusion, firstly according to the corresponding time domain block information ( By doing frame difference and downsampling on the original video, some redundant features are filtered out. The specific method is: find The smallest proportion R (in this case, R = 50%, generally 30-70% is selected for adaptive adjustment) of the elements, and then f out The corresponding elements are deleted.

[0078]

[0079] In the above formula, ids∈{1, 2…N} is f out The element index of . The time domain information obtained by performing frame difference and downsampling on the original video; f slk Visual features with high dynamic effects are screened.

[0080] 8. For the remaining features f_slk∈R that have not been screened out (1-R)N,C , according to the corresponding spatial information (estimated depth value) The weighted fusion is performed to obtain the characterization feature (i.e., the video visual feature) f of the final video segment v. The distant view part with a high depth value in f_slk will be given a higher weight:

[0081]

[0082] In the above formula, is the calculated feature weight; f slk The estimated depth value of the corresponding element in the original video.

[0083] Based on the above exemplary process, video visual features after screening and weighted fusion processing can be obtained. The data distribution of the video visual features conforms to the temporal and spatial characteristics of fixed-viewing angle videos, and can therefore accurately reflect the key information of such videos. The discriminator training and discriminator scoring based on the video visual features will both bring higher execution efficiency and accuracy.

[0084] As some typical applications of the above technical solutions, a second aspect of an embodiment of the present invention further provides a method for detecting anomalies in a video with a fixed viewing angle, which comprises the following steps:

[0085] For a video with a fixed viewing angle, extracting video visual features using the video visual feature extraction method provided in any of the above implementations;

[0086] The video visual feature is input into a scorer, and an abnormality score is performed on the video visual feature. The result of the abnormality score is used to indicate whether there is an abnormality in the video picture corresponding to the video visual feature and the degree of the abnormality.

[0087] Of course, in some cases, the scorer can also be trained in the same way as the above process, including supervised, semi-supervised or unsupervised training, etc. The specific process is to score the training samples using the above method, and then evaluate the model performance based on labels or adversarial training and then update the parameters.

[0088] As a typical example of the above application, a representative anomaly detection process includes:

[0089] By using the exemplary method provided above, each video segment v i Extract visual features and get f i After that, it is sent to the anomaly scorer composed of the MLP network to infer the anomaly and obtain the anomaly score s of the clip. i :

[0090] s i = Scorer(f i ).

[0091] Of course, the MLP network used in this embodiment is a very typical scorer network structure, but it does not mean that the implementation of the present invention is limited to this network structure. Other network structures with similar functions can still achieve equivalent functions. The focus of the present invention is still on how to obtain highly representative video visual features.

[0092] Correspondingly, an embodiment of the present invention further provides a spatiotemporal weighted pooling video visual feature extraction system, wherein the video visual feature is used for abnormality detection of a video with a fixed viewing angle, and comprises:

[0093] A segment estimation module, used to obtain a video segment and perform background difference on the video segment to form temporal information, and perform depth estimation to obtain spatial information;

[0094] A block aggregation module, used to divide the video clip into a plurality of video blocks, and correspondingly aggregate the time domain information into time domain block information and aggregate the spatial domain information into spatial domain block information;

[0095] A feature extraction module, configured to extract features from each of the video blocks using a feature extractor with a Transformer architecture to obtain a plurality of initial features;

[0096] A feature screening module, used to screen the initial features using the time domain block information, remove some features with small image change amplitude in the initial features, and obtain screening features;

[0097] The weighted fusion module is used to perform weighted fusion on the screening features using the spatial block features as weight indicators to obtain video visual features corresponding to the video clips, wherein the weights are positively correlated with the depth values ​​of the video block features.

[0098] Furthermore, an embodiment of the present invention also provides a readable storage medium, in which a computer program is stored. When the computer program is executed, the steps of the video visual feature extraction method provided in any of the above-mentioned embodiments are executed.

[0099] The technical solution of the present invention is further described in detail below through several embodiments and in conjunction with the accompanying drawings. However, the selected embodiments are only used to illustrate the present invention, and do not limit the scope of the present invention.

[0100] Example 1

[0101] This embodiment uses the representative video visual feature extraction method and the representative video anomaly detection method recorded above to conduct simulation experiments. In addition to the detailed parameters indicated above, some other specific details include:

[0102] The video samples come from the UCF-Crime dataset, which includes 1900 videos in total. The number of video blocks N is 1568 (16*16*8). The parameters of weighted fusion are set as α=0.05; ∈=10 -6 .

[0103] Finally, real-time reasoning was achieved, with an AUC (Area Under the Curve) performance of 87.4 on the UCF-Crime dataset.

[0104] Comparative Example 1

[0105] This comparative example is substantially the same as Example 1, except that the video visual feature extractor of the CNN architecture in Example 1 is replaced by a video visual feature extractor of the traditional Transformer architecture, and average pooling is used for the final feature fusion.

[0106] Final anomaly detection performance evaluation: Real-time reasoning is achieved, and the AUC (Area Under the Curve) performance on the UCF-Crime dataset is 84.9.

[0107] Based on the above embodiments and control examples, it can be clearly seen that the feature extraction method and video anomaly detection method provided by the embodiments of the present invention guide feature selection and weighted fusion in the final pooling stage of Transformer reasoning to enhance feature representation by introducing spatiotemporal cues, so that the extracted features pay more attention to semantically rich dynamic areas, and at the same time, according to the video with a fixed perspective, especially the perspective characteristics of surveillance cameras, the features of small targets are enhanced, thereby significantly improving the performance of video anomaly detection.

[0108] It should be understood that the above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for extracting video visual features by spatiotemporal weighted pooling, wherein the video visual features are used for anomaly detection in videos with a fixed viewing angle, characterized in that: include: Acquire a video clip and perform background difference on the video clip to form time domain information, and perform depth estimation to obtain spatial domain information; Divide the video clip into a plurality of video blocks, and correspondingly aggregate the temporal information into temporal block information and aggregate the spatial information into spatial block information; Using a feature extractor with a Transformer architecture to extract features from each of the video blocks to obtain a plurality of initial features; The initial features are screened using the time domain block information, and some features with small image change amplitudes in the initial features are removed to obtain screened features; The spatial domain block feature is used as an indicator of weight, and the screening features are weightedly fused to obtain the video visual feature corresponding to the video clip, wherein the weight is positively correlated with the depth value of the video block feature.

2. The video visual feature extraction method according to claim 1, characterized in that: The method for acquiring the time domain block information and the space domain block information specifically includes: For any of the video blocks, the time domain information corresponding to each frame in the video block is summed to obtain the time domain block information, and the spatial domain information corresponding to each frame is summed to obtain the spatial domain block information.

3. The video visual feature extraction method according to claim 1, characterized in that: The method for obtaining the screening feature specifically includes: Sort by image change amplitude, and search for part of the time domain block information with a large image change amplitude among the plurality of time domain block information at a preset screening ratio; Part of the initial features corresponding to the found part of the time domain block information is retained as the screening features, and the remaining initial features are deleted.

4. The video visual feature extraction method according to claim 3, characterized in that: The process of obtaining the screening feature is expressed as follows: Wherein, ids represents the index of the partial time domain block information found; represents the time domain block information; i represents the sequence number; N represents the total number of the time domain blocks; L represents the preset screening ratio; f slk represents the screening feature; represents the spatial domain block feature; f out represents the initial feature.

5. The video visual feature extraction method according to claim 3, characterized in that: The weighted fusion process is expressed as: in, represents the weight; α represents a hyperparameter that controls the influence of spatial features; ∈ is a very small value greater than 0, with an order of magnitude of 10-6; E(·) represents mean calculation; Var(·) represents variance calculation; f represents the video visual feature; Mean(·) represents weighted fusion calculation.

6. The video visual feature extraction method according to claim 3, characterized in that: The feature extraction process is expressed as: X i =Emb(Cube i ); X=[X i ∈R 1,C |1≤i≤N]; f out =Encoder(X); Among them, X i represents the embedded features of the video block; X represents the input features; Encoder(·) represents the feature extractor.

7. The video visual feature extraction method according to claim 1, characterized in that: The video clip is formed by continuous average segmentation of a continuous video; And / or, the video blocks are formed by dividing the frames of the video clip into continuous, non-overlapping, and equal-sized segments.

8. A method for detecting anomalies in a video with a fixed viewing angle, characterized in that: include: For a video with a fixed viewing angle, extracting video visual features using the video visual feature extraction method described in any one of claim 17; The video visual feature is input into a scorer, and an abnormality score is performed on the video visual feature. The result of the abnormality score is used to indicate whether there is an abnormality in the video picture corresponding to the video visual feature and the degree of the abnormality.

9. A spatiotemporal weighted pooling video visual feature extraction system, wherein the video visual feature is used for anomaly detection of a video with a fixed viewing angle, characterized in that: include: A segment estimation module, used to obtain a video segment and perform background difference on the video segment to form temporal information, and perform depth estimation to obtain spatial information; A block aggregation module, used to divide the video clip into a plurality of video blocks, and correspondingly aggregate the time domain information into time domain block information and aggregate the spatial domain information into spatial domain block information; A feature extraction module, configured to extract features from each of the video blocks using a feature extractor with a Transformer architecture to obtain a plurality of initial features; A feature screening module, used to screen the initial features using the time domain block information, remove some features with small image change amplitude in the initial features, and obtain screening features; The weighted fusion module is used to perform weighted fusion on the screening features using the spatial block features as weight indicators to obtain video visual features corresponding to the video clips, wherein the weights are positively correlated with the depth values ​​of the video block features.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed, the steps of the video visual feature extraction method according to any one of claims 1 to 7 are executed.

Images