Local semantic enhancement encoder, window semantic enhancement Transform block, and aerial video classification model and method

Abstract

The invention discloses a local semantic enhancement encoder, a window semantic enhancement Transform block, an aerial video classification model and an aerial video classification method. The encoder comprises a window positioning module and a window time multi-head self-attention module. And the window positioning module adopts a non-filling convolution kernel with the same size as the local window to calculate the feature response of the input video features, so as to determine a key window region with the maximum feature response in the video features, and further to strip out the local video features in the key window region. And the window time multi-head self-attention module calculates window time multi-head self-attention of the local video features, and adds the window time multi-head self-attention into the video features through the residual block. In this way, background information insensitive to motion information is excluded, too high calculation amount caused by self-attention calculation of a too long video sequence is avoided, and the efficiency of aerial video recognition is improved. In addition, local motion information of the aerial video is enhanced, and the recognition accuracy of the later-period aerial video is improved.

Description

Technical Field

[0001] The present invention relates to the field of video recognition technology, and in particular to a local semantic enhancement encoder, a window semantic enhancement Transformer block, an aerial video classification model and a method. Background Art

[0002] With the continuous development of aviation automation technology and remote sensing imaging technology, drones can capture massive remote sensing images from different perspectives due to their high mobility, low cost and easy operation. At the same time, drones equipped with intelligent image analysis systems can capture and analyze videos and images, which has extremely high practical value in many application fields, such as target reconnaissance, disaster detection, logistics distribution, pest and disease analysis, etc.

[0003] Processing drone videos by manual interpretation is costly and slow, and it is difficult to adapt to the massive data obtained by drones. Therefore, a more effective and efficient way is needed to automatically interpret the content of drone videos. Deep learning is an important research branch of machine learning. It learns complex features and representations through the targeted design of deep neural networks, and is widely used in fields such as computer vision and natural language processing due to its excellent robustness and generalization.

[0004] Compared with manual interpretation, deep learning-based methods can automatically interpret the content of drone videos in a more effective and efficient way. Among them, convolutional neural networks and visual Transformers are the mainstream deep learning methods in the field of computer vision.

[0005] Transformer is a deep neural network based on self-attention. Compared with convolutional neural networks, Transformer naturally has excellent global feature modeling and parallel computing capabilities. It can effectively utilize the rich information of drone videos and efficiently process the massive data of drone videos. Current research on Transformer in the field of video recognition mainly focuses on human behavior in regular videos. Such video events have clear goals and obvious target behaviors.

[0006] However, drone videos collected in complex scenes contain a large amount of background information that is insensitive to human behavior information. This not only leads to excessive self-attention calculation of drone videos, reducing the recognition efficiency of drone videos, but also affects the recognition results of drone videos and reduces the recognition accuracy of drone videos. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention proposes a local semantic enhancement encoder, a window semantic enhancement Transformer block, an aerial video classification model and method, which can improve the efficiency and accuracy of aerial video recognition. The specific technical solution is as follows:

[0008] In a first aspect, a local semantic enhancement encoder is provided. In a first implementable manner of the first aspect, the encoder includes:

[0009] A window positioning module is configured to use an unfilled convolution kernel with the same size as the window to locate a key window area of ​​the video feature and strip off the local video features within the key window area;

[0010] The window time multi-head self-attention module is configured to calculate the window time multi-head self-attention of the local video feature and perform a residual connection between the window time multi-head self-attention and the video feature.

[0011] In combination with the first implementable manner of the first aspect, in a second implementable manner of the first aspect, the window positioning module includes:

[0012] A video pooling unit, configured to perform global average pooling and global maximum pooling on the input video features respectively to obtain corresponding pooling features;

[0013] The channel splicing unit is configured to splice the two pooling features obtained by the video pooling unit.

[0014] Obtain corresponding splicing features;

[0015] The feature response unit is configured to calculate the feature response of the spliced ​​feature using an unfilled convolution kernel with a consistent window area size, and normalize each obtained feature response to determine the weight of each feature response;

[0016] A window position unit, configured to perform global average pooling on the weights of all feature responses based on the time dimension, and locate the key window position of the video feature according to the feature response with the largest average weight;

[0017] A window area unit is configured to determine the key window area according to the key window position and the size of the unfilled convolution kernel.

[0018] In combination with the first implementable manner of the first aspect, in a third implementable manner of the first aspect, further comprising:

[0019] A first-layer normalization module configured to perform normalization processing on the video features and input the processed video features into the window positioning module;

[0020] A second layer normalization module is configured to perform normalization processing on the video features added with the window time multi-head self-attention;

[0021] The first multi-layer perceptron is configured to extract features of the video output by the second-layer normalization module, and the extracted features are residually connected with the video features input to the second-layer normalization module.

[0022] Secondly, a window semantics enhanced Transformer block is provided, including:

[0023] A local semantic enhancement encoder as described in any one of the first to third possible implementations of the first aspect;

[0024] The standard encoder performs feature extraction on the video features output by the local semantic enhancement encoder to obtain global scene information and local scene information of the video features.

[0025] In combination with the first implementable manner of the second aspect, in the second implementable manner of the second aspect, the standard encoder includes a third-layer normalization module, a multi-head attention module, a fourth-layer normalization module and a second multi-layer perceptron connected in sequence.

[0026] In combination with the first implementable manner of the second aspect, in a third implementable manner of the second aspect, the output of the multi-head attention module is connected to the input of the input standard encoder through a residual block;

[0027] The output of the second multi-layer perceptron is connected to the input of the fourth layer normalization module through a residual block.

[0028] In a third aspect, an aerial video classification model is provided. In a first implementable manner of the third aspect, the model includes:

[0029] Multiple layers of the window semantic enhancement Transformer block as described in the third possible implementation of the second aspect, configured to perform feature extraction on the aerial video to obtain global scene information and local scene information of the aerial video;

[0030] A global-local self-fusion and Transformer block is configured to fuse the global scene information and the local scene information of the aerial video using a self-fusion attention mechanism to obtain a fused video feature representation;

[0031] The classifier is configured to determine the classification result of the aerial video according to the fused video feature representation.

[0032] In combination with the first implementable manner of the third aspect, in a second implementable manner of the third aspect, the global local self-fusion and Transformer block includes:

[0033] A Transformer encoder configured to preprocess global scene information and local scene information of the aerial video;

[0034] The self-fusion attention module is configured to fuse the pre-processed global scene information and the local scene information to obtain the video feature representation.

[0035] In combination with the second implementable manner of the third aspect, in a third implementable manner of the third aspect, the self-fusion attention module includes:

[0036] A query vector calculation unit is configured to determine a global key value vector and a local key value vector respectively through a single linear mapping according to the global scene information and the local scene information, and obtain a global-local mixed query vector through channel splicing and linear mapping according to the global key value vector and the local key value vector;

[0037] A probability distribution calculation unit, configured to calculate a probability distribution matrix according to the query vector, the global key value vector and the local key value vector;

[0038] The video feature calculation unit is configured to calculate a fused video feature representation according to the probability distribution matrix, the global key value vector and the local key value vector.

[0039] In a fourth aspect, a method for classifying aerial videos is provided, comprising:

[0040] Acquire aerial photography data, and pre-process the aerial photography data to obtain an aerial video;

[0041] The preprocessed aerial video is classified using a trained aerial video classification model as described in any one of the first to third possible implementations of the third aspect.

[0042] Beneficial effects: The local semantic enhancement encoder, window semantic enhancement Transformer block, aerial video classification model and method of the present invention are used to locate the key window area with the maximum feature response in the aerial video through the window positioning module, and then the local video features in the key window area are stripped out. The window time multi-head self-attention of the local video features is calculated by the window time multi-head self-attention module, thereby eliminating background information that is insensitive to motion information, avoiding excessive calculation caused by calculating self-attention for too long video sequences, and improving the efficiency of aerial video recognition. And the window time multi-head self-attention is added to the aerial video through the residual connection, which enhances the local motion information of the aerial video and improves the recognition accuracy of the aerial video. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation of the present invention, the following will briefly introduce the drawings required for use in the specific implementation. In all the drawings, each element or part is not necessarily drawn according to the actual scale.

[0044] Figure 1 A schematic diagram of the structure of a local semantic enhancement encoder provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the positioning principle of a window positioning module provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the structure of a window semantics enhanced Transformer block provided in one embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the structure of an aerial video classification model provided by an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of the structure of the global local self-fusion and Transformer block provided by an embodiment of the present invention;

[0049] Figure 6 A schematic diagram of the structure of a self-fusion attention module provided by an embodiment of the present invention;

[0050] Figure 7 The present invention provides a flowchart of an aerial video classification method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0052] It should be understood that in this embodiment, the purpose of aerial video recognition is to analyze and understand the spatiotemporal visual patterns in the video, and its events include multiple concepts such as objects, scenes, and behaviors, such as disaster events such as fires and floods, human activity events such as running and swimming, and traffic events such as car accidents and traffic jams. Event recognition and behavior recognition are not in opposition. When the video content to be recognized is the behavior of a certain target, the event recognition task can be regarded as a behavior recognition task.

[0053] like Figure 1 The schematic diagram of the structure of the local semantic enhancement encoder shown in FIG. 1 includes:

[0054] A window positioning module is configured to use an unfilled convolution kernel with the same size as the window to locate a key window area of ​​the video feature and strip off the local video features within the key window area;

[0055] The window time multi-head self-attention module is configured to calculate the window time multi-head self-attention of the local video feature and perform a residual connection between the window time multi-head self-attention and the video feature.

[0056] Specifically, the local semantic enhancement encoder includes a window positioning module and a window time multi-head self-attention module. Among them, the window positioning module can use an unfilled convolution kernel with the same size as the local window to calculate the feature response of the input video feature, and thereby determine the key window area with the largest feature response in the video feature, and then strip out the local video features in the key window area. The window time multi-head self-attention of the local video feature is then calculated by the window time multi-head self-attention module, and the window time multi-head self-attention is added to the video feature through the residual block, so as to learn the identity mapping, alleviate the gradient disappearance, promote the rapid convergence of the model, and improve the efficiency of aerial video recognition.

[0057] This not only excludes background information that is insensitive to motion information, avoids excessive computational complexity caused by calculating self-attention for overly long video sequences, but also includes spatially complete targets, achieves efficient modeling of temporal information, and improves the efficiency of aerial video recognition. It also enhances the local motion information of aerial videos and improves the accuracy of subsequent aerial video recognition.

[0058] In this embodiment, optionally, the encoder further includes:

[0059] A first-layer normalization module configured to perform normalization processing on the video features and input the processed video features into the window positioning module;

[0060] A second layer normalization module is configured to perform normalization processing on the video features added with the window time multi-head self-attention;

[0061] The first multi-layer perceptron is configured to extract features of the video output by the second-layer normalization module, and the extracted features are residually connected with the video features input to the second-layer normalization module.

[0062] Specifically, the local semantic enhancement encoder is a feedforward neural network structure, including a first-layer normalization module, a window positioning module, a window time multi-head self-attention module, a second-layer normalization module and a first multi-layer perceptron connected in sequence.

[0063] Among them, the first-layer normalization module can perform layer normalization on the input video features to prevent the encoder from internal covariate shift during the learning process, accelerate the learning convergence speed of the encoder, and improve the efficiency of aerial video recognition.

[0064] The window positioning module can use an unfilled convolution kernel with the same size as the local window to calculate the feature response of the video features processed by the first layer normalization module, and thereby determine the key window area with the largest feature response in the video features, and then extract the local video features in the key window area.

[0065] In this embodiment, optionally, the window positioning module includes:

[0066] A video pooling unit, configured to perform global average pooling and global maximum pooling on the input video features respectively to obtain corresponding pooling features;

[0067] The channel splicing unit is configured to splice the two pooling features obtained by the video pooling unit.

[0068] Obtain corresponding splicing features;

[0069] The feature response unit is configured to calculate the feature response of the spliced ​​feature using an unfilled convolution kernel with a consistent window area size, and normalize each obtained feature response to determine the weight of each feature response;

[0070] A window position unit, configured to perform global average pooling on the weights of all feature responses based on the time dimension, and locate the key window position of the video feature according to the feature response with the largest average weight;

[0071] A window area unit is configured to determine the key window area according to the key window position and the size of the unfilled convolution kernel.

[0072] Specifically, if Figure 2 As shown, the window positioning module includes a video pooling unit, a channel stitching unit, a feature corresponding unit, a window position unit and a window area unit. Among them, the video pooling unit can perform global average pooling and global maximum pooling on the input video features through a global average pooling unit and a global maximum pooling unit based on the channel dimension, respectively, so as to obtain the global average pooling features and global maximum pooling features corresponding to the video features.

[0073] The channel splicing unit can splice the global average pooling features and the global maximum pooling features calculated by the video pooling unit along the channel dimension. The specific calculation formula is as follows:

[0074] X p =Cat c (AvgPool(X),MaxPool(X));

[0075] Where X∈R T×N×C , X p ∈R T×N×2, T, C represent the number of sampling frames and channel dimension respectively; N = H × W / P 2 It represents the feature dimension, H and W represent the height and width of the input video frame, that is, the spatial size, and P represents the convolution kernel size in the block embedding operation; Cat c (·) indicates channel splicing, AvgPool C (·) and MaxPool C (·) represent channel-based global average pooling and global maximum pooling, respectively.

[0076] The feature response unit can combine the features X p The feature dimension is transformed from one dimension to two dimensions To perform subsequent convolution operations, In order to ensure that the feature response can be mapped to the corresponding window area, the feature response can be calculated by unfilled convolution with the convolution kernel consistent with the window size, thereby reducing the number of feature channels to 1. The feature response unit normalizes the feature response using the Softmax function to determine the feature response weights corresponding to each window obtained by convolution calculation. The specific calculation is as follows:

[0077]

[0078] In the formula, X map ∈R T×u×u Represents the feature response weight of each frame image; u=n-s+1 represents the size of the feature response weight; Conv s×s (·) represents an unfilled convolution kernel of size s. The size of the convolution kernel s is the size of the window. The window size is a hyperparameter for network training and can be determined according to the actual training task. If the drone is flying at low altitude, the window can be enlarged if the target is obvious. If the drone is flying at high altitude, the window can be reduced if the target is small. The window size parameter can be adjusted multiple times during training to ensure that the optimal window response area can be learned.

[0079] It should be understood that in aerial videos, due to the small size of moving objects, the spatial displacement of moving objects along the time axis is often more important than the movement itself. For example, in a cycling event, the movement information of the bicycle in space is more valuable than the movement information of the athlete's legs, and the latter is difficult to capture.

[0080] In order to obtain this information, one video corresponds to one window, and the window position unit can average the feature response weights X of all windows in the time dimension. map Then get the average weight The maximum position is used to obtain the window position with high response in the input video feature X as the key window position. The specific calculation is as follows:

[0081]

[0082]

[0083] Where pos∈R 2 , indicating the key window position; AvgPool T (·) represents global average pooling based on the time dimension; Argmax(·) represents the location where the highest value of the feature response weight is calculated.

[0084] After locating the key window position, the window area unit can infer the key window area based on the key window position and the size of the unfilled convolution kernel, thereby stripping out the video features in the key window area. This local video feature mainly focuses on the moving target and excludes scene information that is not sensitive to spatiotemporal information. The specific calculation is as follows:

[0085] X w =Loc(X,pos,s),

[0086] Among them, X w ∈R (T×s×s)×C Represents a local video feature; Loc represents the operation of locating the local video feature according to the window position.

[0087] The window time multi-head self-attention module can calculate the window time multi-head self-attention of the local video features stripped by the window positioning module, and add the window time multi-head self-attention to the video features through the residual block. The specific calculation formula is as follows:

[0088]

[0089] Among them, X l is the input of the l-th layer encoder, X w is a local video feature.

[0090] It should be understood that several existing Transformer-based spatiotemporal information modeling methods include:

[0091] Joint spatiotemporal multi-head self-attention performs self-attention on the spatiotemporal joint video features. This approach tends to focus on richer scene information, while too long feature sequences will lead to excessive computation.

[0092] Temporal multi-head self-attention based on pixel blocks performs self-attention on pixel blocks in the same space at different times. Compared with joint spatiotemporal multi-head self-attention, although the computational complexity is lower, it destroys the integrity of the target in the spatial dimension.

[0093] In this embodiment, the windowed temporal multi-head self-attention module uses windowed temporal multi-head self-attention to calculate self-attention only for locally complete subjects that are time-sensitive and spatially complete, enabling efficient modeling of temporal information. The computational complexity of windowed temporal multi-head self-attention and other methods on a video frame sequence of length T is as follows:

[0094] Ω(WT-MSA) = 4s 2 TC 2 + 2s 4 T 2 C,

[0095] Ω(JST-MSA) = 4hwTC 2 + 2(hw) 2 T 2 C,

[0096] Ω(PT-MSA) = 4hwTC 2 + 2hwT 2 C,

[0097] where JST-MSA is spatio-temporal joint multi-head self-attention; PT-MSA is pixel-block-based temporal multi-head self-attention; since s 2 << hw, compared with the other two self-attention methods, windowed temporal multi-head self-attention exhibits lower computational complexity, and together with the window positioning module, it enables accurate modeling of temporal information.

[0098] The second layer normalization module can perform layer normalization on the video features with windowed temporal multi-head self-attention added, preventing internal covariate shift during the learning process of the encoder, accelerating the learning convergence speed of the encoder, and improving the efficiency of aerial video recognition.

[0099] The first multi-layer perceptron can extract features from the video features processed by the second layer normalization module, obtain high-level features of the video features, and add the extracted high-level features to the video features input to the second layer normalization module through a residual block to obtain video features with local motion information, so as to improve the recognition accuracy of subsequent aerial videos. The specific calculation formula is as follows:

[0100]

[0101] where LN represents layer normalization and MLP represents multi-layer perceptron.

[0102] As Figure 3 shown in the structural schematic diagram of the window semantic enhancement Transformer block, this Transformer block includes:

[0103] The above-mentioned local semantic enhancement encoder;

[0104] The standard encoder performs feature extraction on the video features output by the local semantic enhancement encoder to obtain global scene information and local scene information of the video features.

[0105] Specifically, the window semantic enhancement Transformer block includes two serially connected encoders, one of which is a local semantic enhancement encoder and the other is a standard encoder. The local semantic enhancement encoder can enhance the local motion information of the aerial video, and the standard encoder can extract the video features output by the local semantic enhancement encoder, thereby obtaining the global scene information and local scene information of the video features for subsequent aerial video recognition.

[0106] In this embodiment, optionally, the standard encoder includes a third-layer normalization module, a multi-head attention module, a fourth-layer normalization module and a second multi-layer perceptron connected in sequence.

[0107] Specifically, the standard encoder is a feedforward neural network structure, including a third-layer normalization module, a multi-head attention module, a fourth-layer normalization module, and a second multi-layer perceptron connected in sequence. Among them, the third-layer normalization module can perform layer normalization processing on the video features output by the local semantic enhancement encoder. The multi-head attention module can calculate the multi-head attention of the video features processed by the third-layer normalization module, and add the multi-head attention to the input video features through the residual block. The specific calculation formula is as follows:

[0108]

[0109] Among them, MSA represents the calculation operation of performing multi-head attention.

[0110] The fourth layer normalization module can perform layer normalization on the video features with multi-head attention, and extract the global scene information of the layer-normalized video features through the second multi-layer perceptron. The specific calculation formula is as follows:

[0111]

[0112] like Figure 4 The schematic diagram of the structure of the aerial video classification model shown in FIG. 1 includes:

[0113] The multiple layers of the above-mentioned window semantic enhancement Transformer blocks are configured to perform feature extraction on the aerial video to obtain global scene information and local scene information of the aerial video;

[0114] A global-local self-fusion and Transformer block is configured to fuse the global scene information and the local scene information of the aerial video using a self-fusion attention mechanism to obtain a fused video feature representation;

[0115] The classifier is configured to determine the classification result of the aerial video according to the fused video feature representation.

[0116] Specifically, the classification model includes a multi-layer window semantic enhancement Transformer block, a global local self-fusion and Transformer block and a classifier. The window semantic enhancement Transformer block can enhance the local motion information of the aerial video, and quickly and accurately extract the global scene information and local scene information of the enhanced aerial video. The local scene information includes the local class embedding of the aerial video, and the global scene information includes the global class embedding sequence of the aerial video. By averaging the features of the global class embedding, the global class embedding of the aerial video can be obtained. The global local self-fusion and Transformer block can fuse the local class embedding and global class embedding extracted by the window semantic enhancement Transformer block using self-fusion and attention mechanisms to obtain a fused video feature representation. The classifier can accurately identify the category of the aerial video based on the fused video feature representation to obtain the classification result of the aerial video.

[0117] In this embodiment, optionally, the classification model further includes a pixel block embedding module, a position embedding module, and a global local class embedding module. Since the high resolution of the image is not conducive to the modeling of self-attention, the pixel block embedding module can be used to perform pixel block embedding processing on the aerial video to reduce the resolution of the image. The specific method is to perform a convolution operation on the aerial video through the pixel block embedding module, thereby converting P×P pixels in the aerial video into a pixel block.

[0118] Self-attention is different from convolution and does not naturally have an inductive bias. Therefore, a learnable position encoding can be introduced into the aerial video through the position embedding module to better process the aerial video sequence. For the convenience of classification, the global and local class embedding modules can also be used to embed global and local class embedding information that is different from the semantics of video features into the aerial video.

[0119] In this embodiment, optionally, the global local self-fusion and Transformer block includes:

[0120] A Transformer encoder configured to preprocess global scene information and local scene information of the aerial video;

[0121] The self-fusion attention module is configured to fuse the pre-processed global scene information and the local scene information to obtain the video feature representation.

[0122] Specifically, if Figure 5 As shown in Figure 1, the global local self-fusion and Transformer block includes a Transformer encoder and a self-fusion attention module. Among them, the Transformer encoder is a standard encoder, and its specific composition is the same as the above-mentioned standard encoder. Through the Transformer encoder, a preliminary interaction can be established between the global scene information and the local scene information. The specific calculation formula is as follows:

[0123] CLS gl =Cat(CLS global ,CLS local )

[0124] [CLS g ,CLS l ]=MSA(LN(CLS gl ))+CLS gl ;

[0125] Among them, CLS local For local class embedding, CLS global For global class embedding, self-fusion attention module.

[0126] The self-fusion attention module can deeply fuse the preprocessed local class embedding and global class embedding, thereby realizing adaptive feature fusion based on the self-attention method and obtaining the corresponding video feature representation.

[0127] It should be understood that the commonly used fusion strategies are feature averaging and score averaging, which averages the feature or score weights. This strategy is easy to implement but too simple and easily ignores detailed features. There are also some adaptive fusion strategies, such as channel adaptation and adaptive weighting, which are often based on convolution and may not be suitable for the deep features of the Transformer network.

[0128] In this embodiment, optionally, the self-fusion attention module includes:

[0129] A query vector calculation unit is configured to determine a global key value vector and a local key value vector respectively through a single linear mapping according to the global scene information and the local scene information, and obtain a global-local mixed query vector through channel splicing and linear mapping according to the global key value vector and the local key value vector;

[0130] A probability distribution calculation unit, configured to calculate a probability distribution matrix according to the query vector, the global key value vector and the local key value vector;

[0131] The video feature calculation unit is configured to calculate a fused video feature representation according to the probability distribution matrix, the global key value vector and the local key value vector.

[0132] Specifically, if Figure 6 As shown in Figure 1, the self-fusion attention module includes a query vector calculation unit, a probability distribution calculation unit, and a video feature calculation unit. The query vector calculation unit can obtain the corresponding global key value vector V by a single linear mapping method based on the local class embedding and the global class embedding. g , K g With the local key vector V l , K l After channel concatenation and linear mapping, the global-local mixed query vector Q is obtained. m , the specific calculation formula is as follows:

[0133]

[0134]

[0135]

[0136] Where V g , K g 、V l , K l , Q m ∈R 1×C ; represents channel dimension concatenation; W represents the weight of the linear mapping, and R is the real number field.

[0137] The probability distribution calculation unit can calculate the global key value vector V according to the query vector calculation unit. g , K g With the local key vector V l , K l , and the global-local hybrid query vector Q m , determine the channel probability distribution matrix of the mixed vector to the double input vector. The specific calculation formula is as follows:

[0138]

[0139] Where map∈R C×C Represents the channel probability distribution matrix of the mixed vector to the double input vector; Represents the dimension of vector K.

[0140] The video feature calculation unit can map the channel probability distribution matrix g 、map l Sum value vector V g 、Vl The two are spliced ​​together, and the video feature representation after the fusion of global class embedding and local class embedding is obtained by matrix multiplication. The specific calculation formula is as follows:

[0141]

[0142] in, CLS fusion ∈R 1×2C Represents the feature vector after multivariate fusion, that is, the video feature representation after the fusion of global class embedding and local class embedding.

[0143] Since the self-fusion attention module mixes the query vector of the input features through mixed linear mapping and self-fuses with the key and value vectors of each feature, compared with the existing feature averaging, score averaging, channel adaptation and adaptive weighting, the self-fusion attention module can obtain richer video feature representation, which helps to further improve the accuracy of aerial video recognition.

[0144] After obtaining the video feature representation after the fusion of the global class embedding and the local class embedding, the video feature representation can be input into the classifier, and the final classification result of the video can be obtained through the classifier. The specific calculation formula is as follows:

[0145] C=Max(FC(CLS fusion ));

[0146] Among them, FC(·) is a fully connected layer classifier with an input dimension of 768 and an output dimension of the total category, Max(·) represents the maximum score result for all, and C is the final classification category of the video.

[0147] like Figure 7 The flowchart of the aerial video classification method shown in FIG. 1 includes:

[0148] Acquire aerial photography data, and pre-process the aerial photography data to obtain an aerial video;

[0149] The pre-processed aerial video is classified using the trained aerial video classification model as described above.

[0150] Specifically, first, the aerial data collected by the drone can be obtained, and the aerial data can be preprocessed to obtain the corresponding aerial video. In the embodiment of the present invention, there is no restriction on the method of obtaining the aerial video data. The method of self-collecting video includes: controlling the drone to shoot the target scene or subject from multiple angles, for a long time, partially blocked and under different lighting conditions, and selecting multiple different target scenes or subjects for shooting. The method of obtaining sample videos through a computer includes: searching for video clips containing the target scene or subject as aerial video data on the Internet, or directly obtaining it from existing drone aerial data sets (such as ERA, MOD20, UAVhuman).

[0151] Specifically, the aerial data can be edited first, and blurred, unstable and invalid video frames in the aerial data can be deleted to obtain high-quality and equal-length video clips. Then, the video sampling frequency can be adjusted according to the existing computing resources, and the corresponding video frames can be extracted from the video clips according to the adjusted video sampling frequency to generate a video frame sequence of corresponding length as the aerial video, and the resolution of the video frame can be adjusted to a preset size. Then, the aerial video obtained after preprocessing is input into the trained aerial video classification model, and the aerial video can be classified by the trained aerial video classification model.

[0152] In this embodiment, the training process of the aerial video classification model includes:

[0153] First, collect multiple groups of aerial video data, and use the same method as the above preprocessing to preprocess each group of aerial video data, so as to obtain the corresponding aerial video to build a data set. When preprocessing the aerial video data, these aerial video data can be edited into valid and equal-length video clips according to the needs and computing conditions, and used as training sets and test sets respectively to ensure that the data collected at the same frame rate are consistent. In order to avoid the impact of data imbalance on model training, the number of videos of different categories should be as equal as possible. In addition, in order to avoid the influence of homologous videos on test results, multiple clips edited from the same video need to be put into the same training set or test set to ensure the training effect of the model.

[0154] It should be understood that in this embodiment, there is no restriction on the number of samples in the data set, and the number of sample videos can be one or more. Exemplarily, the number of sample videos is multiple to ensure the model training effect. Exemplarily, in the case where the number of sample videos is multiple, the number of video frames in different sample videos is the same to ensure the model training effect.

[0155] Afterwards, the above-mentioned aerial video classification model can be trained through the training set. During the training process, the parameters of the aerial video classification model can be continuously updated through the loss function until the loss function is reduced to the minimum, and then the trained aerial video classification model is verified through the test set. It can be understood that the training, verification and testing processes of the model have similar or identical processing procedures. Those skilled in the art should know that in the verification and testing stage, the pre-processed aerial video data can be processed according to the same process as the training to obtain the corresponding aerial video classification results.

[0156] In this embodiment, the parameters of the model can be updated by gradient backpropagation, the cross entropy loss function is used as the loss function, and the stochastic gradient descent optimizer is used for updating. The stochastic gradient descent optimizer can select the cosine descent strategy to keep the learning rate linearly updated. The cross entropy loss function is specifically:

[0157]

[0158] Where N represents the total number of categories of training data; q i represents the probability of the model being trained as the i-th category, and q i ∈{0,1};p i represents the probability that the model predicts the i-th category, and p i ∈{0,1},

[0159] By continuously calculating the loss function and updating the network parameters using back propagation, the model is continuously optimized and its recognition accuracy is improved. When the loss function reaches the lowest point, the model training is completed.

[0160] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A local semantic enhancement encoder, characterized in that: include: A window positioning module is configured to use an unfilled convolution kernel with the same size as the window to locate a key window area of ​​the video feature and strip off the local video features within the key window area; The window time multi-head self-attention module is configured to calculate the window time multi-head self-attention of the local video feature and perform a residual connection between the window time multi-head self-attention and the video feature.

2. The local semantic enhancement encoder according to claim 1, characterized in that The window positioning module comprises: A video pooling unit, configured to perform global average pooling and global maximum pooling on the input video features respectively to obtain corresponding pooling features; A channel splicing unit is configured to splice the two pooling features obtained by the video pooling unit to obtain corresponding splicing features; A feature response unit is configured to calculate feature responses of the spliced ​​features using an unfilled convolution kernel with a consistent window area size, and normalize each obtained feature response to determine a weight of each feature response; A window position unit, configured to perform global average pooling on the weights of all feature responses based on the time dimension, and locate the key window position of the video feature according to the feature response with the largest average weight; A window area unit is configured to determine the key window area according to the key window position and the size of the unfilled convolution kernel.

3. The local semantic enhancement encoder according to claim 1, characterized in that Also includes: A first-layer normalization module configured to perform normalization processing on the video features and input the processed video features into the window positioning module; A second layer normalization module is configured to perform normalization processing on the video features added with the window time multi-head self-attention; The first multi-layer perceptron is configured to extract features of the video output by the second-layer normalization module, and the extracted features are residually connected with the video features input to the second-layer normalization module.

4. A window semantics enhanced Transformer block, characterized in that include: The local semantic enhancement encoder as described in any one of claims 1 to 3; The standard encoder performs feature extraction on the video features output by the local semantic enhancement encoder to obtain global scene information and local scene information of the video features.

5. The window semantics enhanced Transformer block according to claim 4, characterized in that The standard encoder includes a third-layer normalization module, a multi-head attention module, a fourth-layer normalization module and a second multi-layer perceptron connected in sequence.

6. The window semantics enhanced Transformer block according to claim 5, characterized in that: The output of the multi-head attention module is connected to the input of the standard encoder through a residual block; The output of the second multi-layer perceptron is connected to the input of the fourth layer normalization module through a residual block.

7. An aerial video classification model, characterized in that: include: Multiple layers of the window semantic enhancement Transformer block as claimed in claim 6, configured to perform feature extraction on the aerial video to obtain global scene information and local scene information of the aerial video; A global-local self-fusion and Transformer block is configured to fuse the global scene information and the local scene information of the aerial video using a self-fusion attention mechanism to obtain a fused video feature representation; The classifier is configured to determine the classification result of the aerial video according to the fused video feature representation.

8. The aerial video classification model according to claim 7, characterized in that: The global local self-fusion and Transformer block includes: A Transformer encoder configured to preprocess global scene information and local scene information of the aerial video; The self-fusion attention module is configured to fuse the pre-processed global scene information and the local scene information to obtain the video feature representation.

9. The aerial video classification model according to claim 8, characterized in that: The self-fusion attention module includes: A query vector calculation unit is configured to determine a global key value vector and a local key value vector respectively through a single linear mapping according to the global scene information and the local scene information, and obtain a global-local mixed query vector through channel splicing and linear mapping according to the global key value vector and the local key value vector; A probability distribution calculation unit, configured to calculate a probability distribution matrix according to the query vector, the global key value vector and the local key value vector; The video feature calculation unit is configured to calculate a fused video feature representation according to the probability distribution matrix, the global key value vector and the local key value vector.

10. A method for classifying aerial videos, characterized in that: include: Acquire aerial photography data, and pre-process the aerial photography data to obtain an aerial video; The preprocessed aerial video is classified using the trained aerial video classification model as described in any one of claims 7 to 9.

Images