A short video event detection method and device based on multi-scale attention fusion

Abstract

This invention discloses a method and apparatus for short video event detection based on multi-scale attention fusion. The method includes: generating short video sub-sequences at three scales using sliding windows of different scales as input; uniformly dividing the input at each scale into local blocks; using an inner transformer and an outer transformer to divide the local blocks into smaller patch blocks; calculating the attention of each patch block; and feeding the attention of each patch block together with the overall attention of the local block into the outer transformer for calculation. The inner transformer processes the visual information within the patch blocks, and the outer transformer processes the visual information after aggregating the features of the local blocks and patches. This visual information is applied to short video event detection. The apparatus includes a processor and a memory. This invention can learn the relationships and connections between different visual elements in an image, thereby improving the accuracy of classification tasks.

Description

Technical Field

[0001] This invention relates to the field of short video event detection, and in particular to a short video event detection method and apparatus based on multi-scale attention fusion. Background Technology

[0002] With the rapid development of the internet and short video platforms, short videos have become increasingly popular among young people due to their short duration, rich content, simple production, and strong dissemination capabilities. These short video data contain a wealth of information and value, such as people's interests, social hot topics, and cultural trends. Therefore, the research and analysis of this data has become a very important topic. As an emerging multimedia form, short videos have become an indispensable part of people's lives. Researching and analyzing this massive amount of short video data can not only help people better understand and grasp current cultural and social trends, but also provide useful references for short video content production, user profiling, and personalized recommendations.

[0003] Currently, research on short video content mainly focuses on popularity prediction, scene classification, and short video recommendation. For example, low-rank multi-view learning frameworks can improve the popularity prediction of short videos; constructing Neural Multimodal Cooperative Learning (NMCL) neural networks can calculate attention scores through perceptual attention mechanisms to measure the correlation between different modalities, thereby achieving scene classification of short videos. Furthermore, given that short videos have diverse audiences, establishing a user-video shared attention network can facilitate short video recommendation tasks.

[0004] Compared to tasks such as popularity prediction, scene classification, and short video recommendation, short video event detection is more challenging and practical. Short video event detection aims to automatically discover and identify events from short videos, such as sporting events, concerts, exhibitions, and celebrations, thereby providing more accurate video content understanding and management. Short video event detection can provide users with more accurate, richer, and more timely information services, and provide short video platforms with more efficient management and recommendation functions.

[0005] Short video event detection technology refers to a technique for detecting the features of subjects and objects involved in specific scenarios. In videos, simple gestures such as clapping, running, and smiling are considered actions, while a collection of actions occurring in special occasions such as birthday parties and weddings is called an event. Currently, event detection technology is mainly divided into two categories: abnormal event detection in surveillance videos and sports event detection in traditional videos. Among them, mainstream technologies include: a hybrid autoencoder model combining long short-term memory networks and convolutional neural networks for abnormal event detection; a hybrid modulation method for abnormal event detection based on feature expectation subgraph calibration classification for video surveillance scenarios; and a semi-supervised learning scheme to address the different distributions of normal and abnormal events in surveillance videos.

[0006] To better utilize multimodal fusion, it is essential to establish a reliable model capable of extracting visual information at different scales and finer granularities. Furthermore, the lack of mainstream datasets also hinders the development of short video event detection technology. Summary of the Invention

[0007] This invention provides a method and apparatus for short video event detection based on multi-scale attention fusion. It offers a novel video feature extraction method that generates three video sub-sequences at different scales and encodes their features, leading to a better understanding of the complexity and richness of short videos. By applying transformer processing (known to those skilled in the art) with varying precision to local blocks and patch blocks in each frame of the sub-sequence, the relationships and connections between different visual elements in the image can be learned, thereby improving the accuracy of classification tasks. Details are described below:

[0008] A short video event detection method based on multi-scale attention fusion, the method comprising:

[0009] Three short video subsequences at different scales were generated as input using sliding windows of different scales;

[0010] The input at each scale is evenly divided into local blocks. The inner transformer and outer transformer are used to divide the local blocks into smaller patch blocks. The attention of each patch block is calculated. The attention of each patch block and the overall attention of the local block are fed into the outer transformer for calculation.

[0011] The inner transformer is used to process visual information within the patch block, while the outer transformer is used to process visual information after aggregating the features of the local area and the patch. This visual information is then applied to short video event detection.

[0012] The three scales of short video subsequences are represented as follows:

[0013]

[0014] in, , .

[0015] The step of uniformly dividing the input at each scale into local blocks is as follows:

[0016]

[0017] Each local block is further divided into m patch blocks, meaning a local block consists of m patch blocks:

[0018]

[0019] Furthermore, the step of feeding the attention of each patch block together with the overall attention of that local block into the outertransformer for calculation is as follows:

[0020] The sequence transformation of the patch block is embedded into the local block through linear projection:

[0021]

[0022] Among them, the sequence of patch blocks and fully connected layer Match the dimensions of the two for use in addition;

[0023] Embedding is implemented using the outer transformer module:

[0024]

[0025]

[0026] An outer transformer is used to model the relationships between local block codes.

[0027] A short video event detection device based on multi-scale attention fusion, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the steps of the method according to any one of the claims.

[0028] The beneficial effects of the technical solution provided by this invention are:

[0029] 1. Due to the richness and complexity of video content, extracting information from a single time scale is often insufficient. To better process this short video data, this invention uses three sliding windows of different scales to generate three different scales of video subsequences. Sliding windows of different scales can capture specific features and information within different time periods of the video. For example, smaller-scale sliding windows can better capture fast-moving actions and detailed information in the video, while larger-scale sliding windows can better capture the overall features and background information in the video. By feature encoding each subsequence, the complexity and richness of short videos can be better understood.

[0030] 2. By combining internal and external transformers, better visual information representation capabilities can be achieved, and short video data can be processed more effectively. The internal transformer focuses on processing visual information within patch blocks, which can better capture local details, thereby improving the model's accuracy and performance. Furthermore, the internal transformer is very effective in processing complex visual information, as it can learn the relationships and connections between different visual elements in the image. Feature aggregation of local and patch elements enhances representation capabilities. This method can better capture global features and background information of the image, thereby improving the model's robustness and generalization ability.

[0031] 3. Short video event detection is a very challenging task, requiring high accuracy and robustness from the algorithm. However, due to the lack of mainstream datasets in this area, this invention constructs a new short video event detection dataset. This dataset contains short videos from different fields, time scales, and perspectives, covering a wide variety of scenes and events. A large amount of short video data was collected from multiple data sources and carefully labeled and classified to ensure the accuracy and effectiveness of the dataset.

[0032] This invention fully utilizes the multimodal information of short videos to perform event detection, and also provides a novel keyframe extraction method and an innovative network architecture, thereby improving the accuracy of short video event detection tasks. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of keyframe extraction;

[0034] Figure 2 This is a flowchart of the network model framework. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0036] Example 1

[0037] This invention provides a short video event detection method based on multi-scale attention fusion. This invention fully utilizes the multi-scale and fine-grained visual information of short videos, aiming to solve the problem of short video event detection. (See also...) Figure 1 and Figure 2 The method includes the following steps:

[0038] 101: Use sliding windows of different scales to generate short video subsequences of three scales as input;

[0039] In order to better process complex and rich short video data and make full use of its visual information, three scales of short video subsequences are generated as input using sliding windows of different scales. This can better capture specific features and information in different time periods of the video.

[0040] 102: Divide the input at each scale into local blocks evenly. Use inner transformer (patch block transformer) and outer transformer (local block transformer) to explore a new architecture. Divide the local blocks into smaller patches, compute the attention of each patch, and feed the attention of each patch along with the overall attention of the local block into the outer transformer for computation.

[0041] In practical applications, the Transformer helps solve complex tasks by encoding input data and extracting powerful feature representations. In natural image processing, extracting object features at different scales and locations becomes particularly important for handling complex details and color information. For processing short video data, this paper first transforms it into a frame sequence suitable for the Transformer, and then proposes a novel Transformer network architecture for visual information processing. This architecture utilizes three sliding windows (with strides of 1, 2, and 4) to generate video subsequences at three different scales and calculates the correlation matrices at these three scales, thus obtaining three different scale video subsequences from the original video as input to the network model. This method can effectively handle multi-scale problems in natural images and helps to extract object features at different scales and locations.

[0042] First, this embodiment of the invention uniformly divides the input at each scale into local blocks. For the information within each local block, attention representation is considered essential and high-performance; therefore, inner transformers and outer transformers are used to explore a novel architecture. The local blocks are further divided into smaller patch blocks, for example, 4 or 9 blocks. The patch blocks are divided into 2, and the attention of each patch block is calculated together with other patches in a given local block. The computational cost is negligible. In specific implementations, the embodiments of the present invention do not limit the number or size of the blocks, but select them according to the needs of the actual application.

[0043] 103: The inner transformer is used to process visual information within the patch block, while the outer transformer is used to process visual information after aggregating the features of the local area and the patch. This visual information can be used for a series of downstream tasks.

[0044] For each scale of input, this invention uniformly divides it into a series of local blocks, and further subdivides each local block into smaller patch blocks. To better process this information, this invention proposes a novel architecture that includes an inner transformer and an outer transformer. The inner transformer processes the visual information within the patch block, while the outer transformer processes the visual information of the local block and adds the feature information of the patch block to the local block to enhance its representational power. When computing the attention representation, the attention of each patch block is computed together with other patches in a given local block, and the computational cost is negligible. This method can extract visual information at different granularities while overcoming the problem of feature information loss due to information loss between different granularities.

[0045] The inner transformer processes visual information within the patch block, while the outer transformer simultaneously processes visual information from local blocks and aggregates features from both local and patch areas to enhance representational capabilities. Ultimately, this model can be used for downstream tasks, such as event detection. Through this model, embodiments of the present invention can extract visual information at different scales and finer granularities, providing more detailed features.

[0046] Finally, this invention utilizes visual information of different scales and finer granularities obtained from the embodiments of the invention for event detection tasks. Experimental results on the Flickr short video event detection dataset demonstrate that the classification accuracy of this method outperforms existing mainstream methods. This approach can help to better understand the complexity of natural images and provide more detailed features.

[0047] Finally, this network was applied to a short video event detection task. This method has wide applications in short videos and other areas of image processing, and can be applied to many more downstream tasks.

[0048] The performance of the scheme is evaluated using three metrics: accuracy (ACC), average precision (AP), and average recall (AR) to ensure the objectivity and accuracy of the experimental results.

[0049] In summary, the embodiments of the present invention fully utilize the visual information of short videos at various scales and fine granularities through the above steps 101-103, thus solving the problem of short video event detection.

[0050] Example 2

[0051] The scheme in Example 1 will be further described below with reference to calculation formulas and examples. See the description below for details:

[0052] 201: Uniform sampling is a commonly used short video feature extraction method that captures key information from a video by selecting frames at equal intervals.

[0053] The implementation steps of this method include: video decomposition, selection of sampling interval, and frame extraction. During frame extraction, an appropriate sampling interval needs to be selected based on task requirements and dataset characteristics to determine which frames can be used as input features. Furthermore, the method's equal-interval frame extraction can improve feature diversity and uniformity while reducing redundancy, thereby improving model performance.

[0054] The three scales of short video subsequences extracted in this embodiment of the invention are represented as follows:

[0055] (1)

[0056] in, ,

[0057] 202: First, let's briefly describe the basic components of the transformer, including MSA (Multi-head Self-Attention), MLP (Multi-layer Perceptron), and LN (Low-Normalization). These will be explained in detail below:

[0058] (1) MSA: It can establish connections between different positions and aggregate information from each position by weighted calculation of attention weights. Multi-head self-attention introduces multiple heads on the basis of the self-attention mechanism. Each head will perform a different attention calculation on the input, thereby extracting richer and more diverse feature representations.

[0059] In MSA, Q, K, and V represent the query vector, key vector, and value vector, respectively, which are obtained from the input word vectors through linear transformation. During the calculation of attention weights, for a given query vector Q, the model calculates a similarity score between each input word vector and the query vector. A higher similarity score indicates a greater relevance between the word vector and the query vector. The model uses the key vector K and the value vector V to calculate the similarity score.

[0060] (2)

[0061] Finally, a linear layer is used to generate the output, and then the output values ​​of each head are concatenated and linearly projected to form the final output.

[0062] (2) MLP: MLP is applied to feature transformation and nonlinear transformation between self-attention layers:

[0063]

[0064] (3)

[0065] Among them, and These are the weights and biases of the fully connected layer, respectively. It is an activation function.

[0066] (3) LN: Layer normalization is a key component in transformers that enables stable training and fast convergence:

[0067] (4)

[0068] in, , These refer to the mean and standard deviation of the feature, respectively. , Both refer to parameters that can be changed.

[0069] 203: For any two-dimensional image in a short video subsequence of different scales, divide it uniformly into n local blocks:

[0070] (5)

[0071] The proposed network model architecture learns global and local information in images. Therefore, in this embodiment of the invention, each local block is further divided into m patch blocks, that is, a local block consists of m patch blocks:

[0072] (6)

[0073] in, It refers to the j-th patch block of the i-th local block.

[0074] Through linear projection, embodiments of the present invention convert it into an embedded sequence:

[0075]

[0076] (7)

[0077] in, This refers to the embedding sequence of the j-th patch block, while It refers to quantization operations.

[0078] In this embodiment of the invention, two main processing operations are performed: processing local blocks and processing patch blocks. A transformer is used to explore the relationships between them. The process can be represented as follows:

[0079]

[0080] (8)

[0081] in, = 1, 2, ..., L is the first The index of each block is given, and L is the total number of stacked blocks. This part can be viewed as the inner transformer module, which establishes relationships between patch blocks by calculating the interactions between any two patch blocks within a local block. For example, in a local block of a face, the word corresponding to the eyes is more relevant to other words related to the eyes, while it interacts less with the mouth.

[0082] For the processing of local blocks, the sequence transformation of the patch block is first embedded into the local block through linear projection:

[0083] (9)

[0084] Among them, the sequence of patch blocks and fully connected layer Match the dimensions of the two for use in addition.

[0085] Through the above operations, the embedded representation of the patch block enhances the image features. The embedding is performed using a standard transformer, specifically the outer transformer module.

[0086]

[0087] (10)

[0088] An outer transformer is used to model the relationships between local block codes.

[0089] Therefore, the inputs and outputs of this embodiment of the invention include the local block encoding and patch block encoding described above. Finally, it is applied to short video event detection.

[0090] 204: This study used three publicly available datasets: UCF-101, HMDB51, and a newly established Flickr short video event detection dataset. The newly established dataset was divided into training and testing sets in a 10:2 ratio. UCF-101 and HMDB51 are two publicly available action recognition datasets. UCF-101 contains 13,320 short videos from 101 different action categories, while HMDB51 contains 6,766 short videos covering 51 different categories. Throughout the training and testing process, the model's performance was evaluated using three metrics: Accuracy (ACC), Average Precision (AP), and Average Recall (AR). The specific meanings of these three metrics are as follows:

[0091] (1) Accuracy (ACC) is a metric used to evaluate the accuracy of a classification algorithm. It represents the proportion of samples correctly classified out of the total number of samples. The higher the accuracy, the better the performance of the classifier. The formula for calculating accuracy is:

[0092] (11)

[0093] Wherein, TP represents the number of true positives (the number of samples correctly identified as positive), TN represents the number of true negatives (the number of samples correctly identified as negative), FP represents the number of false positives (the number of samples incorrectly identified as positive), and FN represents the number of false negatives (the number of samples incorrectly identified as negative).

[0094] Accuracy is calculated very simply as the ratio of correctly classified samples to the total number of samples. In binary classification problems, higher accuracy indicates better classifier performance because the classifier correctly classifies more samples as positive or negative. However, accuracy does not always fully reflect a classifier's performance. Especially in cases of imbalanced sampling, if the classifier consistently classifies all samples as negative, the accuracy may still be high, but the classifier's performance will be poor. Therefore, in evaluating model performance, other metrics besides accuracy need to be considered.

[0095] (2) Average Precision (AP) is a metric used to evaluate information retrieval algorithms and can also be used to evaluate the performance of classification algorithms. It represents the average precision across all recall values, where recall represents the proportion of correctly classified positive samples out of all positive samples, and precision represents the proportion of correctly classified positive samples out of all positive samples. A higher average precision indicates better classifier performance. The formula for calculating average precision is:

[0096] (12)

[0097] in, This represents the accuracy at the i-th position. This represents the difference between the recall rate at position i and the recall rate at position (i-1). The calculation of average precision is relatively complex; it requires calculating precision across all recall rates and then multiplying precision by recall.

[0098] (3) Average Recall (AR) is a metric used to evaluate the performance of classification algorithms. It represents the average recall across all precision levels, where precision represents the proportion of correctly classified positive samples out of all correctly classified positive samples, and recall represents the proportion of correctly classified positive samples out of all positive samples. A higher average recall indicates better classifier performance. The formula for calculating average recall is:

[0099] (13)

[0100] in, This represents the recall rate at position i. This represents the difference between the precision at position i and the precision at position i-1. Similar to average precision, it is necessary to calculate the corresponding recall or precision at different precision levels and then calculate their product. Average recall is mainly used to evaluate the performance of the classifier at different precision levels, and can more comprehensively reflect the performance of the classifier. In the application of the classifier, embodiments of this invention typically use multiple thresholds to generate different classification results. Average recall can help comprehensively evaluate the performance of the classifier at different thresholds and select the optimal threshold.

[0101] In summary, to better process natural images with high complexity and rich details and color information, the embodiments of this invention cannot simply extract visual information using the granularity of local block partitioning using traditional transformers. Therefore, mining the features of objects at different scales and locations becomes particularly important. To better process short video data, the embodiments of this invention first convert the short video into a frame sequence format suitable for transformer input and propose a novel transformer network architecture for visual information processing. First, three video subsequences at different scales of the original video are obtained as input. To further extract features, the embodiments of this invention uniformly divide the input at each scale into local blocks, and then further divide the local blocks into smaller patch blocks. The embodiments of this invention explore a new architecture, namely inner transformer and outer transformer, for processing information within local blocks and feature aggregation between local blocks. By calculating the attention of each patch block, the visual information within local blocks and the visual information within patch blocks are aggregated to enhance representational power. Finally, the embodiments of this invention use this model for downstream tasks such as short video event detection.

[0102] Example 3

[0103] A short video event detection device based on multi-scale attention fusion includes a processor and a memory. The memory stores program instructions, and the processor calls the program instructions stored in the memory to cause the device to execute any one of the following method steps:

[0104] Three short video subsequences at different scales were generated as input using sliding windows of different scales;

[0105] The input at each scale is evenly divided into local blocks. The inner transformer and outer transformer are used to divide the local blocks into smaller patch blocks. The attention of each patch block is calculated. The attention of each patch block and the overall attention of the local block are fed into the outer transformer for calculation.

[0106] The inner transformer is used to process visual information within the patch block, while the outer transformer is used to process visual information after aggregating the features of the local area and the patch. This visual information is then applied to short video event detection.

[0107] The short video subsequences at the three scales are represented as follows:

[0108]

[0109] in, , .

[0110] The input at each scale is uniformly divided into local blocks as follows:

[0111]

[0112] Each local block is further divided into m patch blocks, meaning a local block consists of m patch blocks:

[0113]

[0114] The attention of each patch block is fed together with the overall attention of that local block into the outer transformer for calculation as follows:

[0115] The sequence transformation of the patch block is embedded into the local block through linear projection:

[0116]

[0117] Among them, the sequence of patch blocks and fully connected layer Match the dimensions of the two for use in addition;

[0118] Embedding is implemented using the outer transformer module:

[0119]

[0120]

[0121] An outer transformer is used to model the relationships between local block codes.

[0122] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0123] The execution entities of the aforementioned processor and memory can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementations, the embodiments of the present invention do not limit the execution entities and can select them according to the needs of actual applications.

[0124] Data signals are transmitted between the memory and the processor via a bus, which will not be elaborated upon in this embodiment of the invention.

[0125] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0126] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A short video event detection method based on multi-scale attention fusion, characterized in that, The method includes: Three short video subsequences at different scales were generated as input using sliding windows of different scales; The input at each scale is evenly divided into local blocks. The inner transformer and outer transformer are used to divide the local blocks into small patch blocks. The attention of each patch block is calculated. The attention of each patch block and the overall attention of the local block are fed into the outer transformer for calculation. The inner transformer is used to process visual information within the patch block, and the outer transformer is used to process visual information after aggregating the features of the local area and the patch. This visual information is applied to short video event detection. The method of using inner transformer and outer transformer to divide local blocks into smaller patch blocks is as follows: Each local block is further divided into m patch blocks, meaning a local block consists of m patch blocks: ； in, It refers to the j-th patch block of the i-th local block; Transform it into an embedded sequence using linear projection: ； ； in, This refers to the embedding sequence of the j-th patch block, while It refers to quantization operations; The process of using a transformer to explore the relationship between local blocks and patch blocks can be represented as follows: ； ； in, = 1, 2, ..., L is the first The index of each block, and L is the total number of stacked blocks; this part is the innertransformer module, which establishes the relationship between patch blocks by calculating the interaction between any two patch blocks within a local block; Multi-head self-attention; LN for normalization; MLP for multilayer perceptron; The process of feeding the attention of each patch block together with the overall attention of that local block into the outer transformer for calculation is as follows: The sequence transformation of the patch block is embedded into the local block through linear projection: ； Among them, the sequence of patch blocks and fully connected layer Match the dimensions of the two for use in addition; Embedding is implemented using the outer transformer module: ； ； An outer transformer is used to model the relationships between local block codes.

2. The short video event detection method based on multi-scale attention fusion according to claim 1, characterized in that, The three scales of short video subsequences are represented as follows: ； in, , .

3. The short video event detection method based on multi-scale attention fusion according to claim 1, characterized in that, The process of uniformly dividing the input at each scale into local blocks is as follows: ； Each local block is further divided into m patch blocks, that is, a local block consists of m patch blocks. composition:

4. A short video event detection device based on multi-scale attention fusion, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the steps of the method according to any one of claims 1-3.

Images