Video retrieval feature extraction method based on high-efficiency attention mechanism

Abstract

The invention discloses a video retrieval feature extraction method based on a high-efficiency attention mechanism, and belongs to the field of computer vision. Comprising the following steps: constructing a feature network with semantic perception based on a high-efficiency attention mechanism; acquiring an image training sample and a training sample of the text image pair; pre-training by using a self-supervised learning method based on image data random enhancement to obtain an image feature encoder; pre-training to an image semantic feature encoder by using a self-supervised learning method based on cross-modal semantics of image text pairs; and carrying out further joint training on the two encoders to obtain a final video feature extraction network for short video retrieval, carrying out frame extraction on a video to be queried, and inputting the video frames into the video feature extraction network, so that the features of the video image can be extracted. According to the method, on the premise that the video feature extraction efficiency of short video retrieval work is greatly improved, it can be guaranteed that the feature network gives consideration to shallow detail features and time semantic features of video pictures, and the final performance of retrieval is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a video retrieval feature extraction method based on a high-efficiency attention mechanism, which can be applied to short video retrieval scenarios. Background Art

[0002] As video data continues to grow, effectively organizing, finding, and analyzing video content becomes increasingly challenging. This is where video retrieval technology comes in. Video retrieval involves extracting, matching, and retrieving relevant information from massive amounts of video data. Its development has benefited from rapid advances in computer vision, machine learning, and artificial intelligence. By analyzing visual and semantic information in videos, video retrieval systems can automatically identify objects, scenes, actions, and other key elements, making it easier for users to search for and access desired content.

[0003] However, the complexity, large scale, and diversity of video data make accurate retrieval difficult. At the same time, cross-language, cross-cultural, and multimodal content also increase the complexity of the technology. Video retrieval primarily involves frame extraction, feature extraction, matching and sorting, and current research focuses on feature extraction. One approach is to extract features at the video level, while another is to extract features at the frame level. The feature vector form at the video level can be directly used for similarity calculation to complete retrieval and sorting, while the feature vector form at the frame level requires matching and aggregation operations before calculating the similarity relationship to complete sorting.

[0004] In terms of research, previous frame-level methods mainly focused on solving scenarios with almost completely repeated videos. With the release of open source datasets and the development of real-world scenario needs, video retrieval tasks have begun to solve tasks with the same semantic videos, retrieving and recalling two videos that are partially duplicated or only semantically related. In most existing work, when processing video images, video feature extraction networks find it difficult to simultaneously take into account the feature invariance of the images and the understanding of the semantic information of the images, or some networks have high design complexity, which requires huge computing resources for subsequent model training and inference, and does not meet the needs of achieving high-efficiency retrieval tasks.

[0005] Therefore, it is necessary to design a feature extraction network with both high efficiency and high performance for short video retrieval tasks based on the needs of real scenarios, improve the traditional high computational complexity feature network based on the self-attention mechanism, improve the feature processing efficiency of the feature network, design the feature processing flow, and add a semantic perception module to improve the performance of the model; in the model training stage, by setting a reasonable model training method, the trained feature network has the ability to extract features without deformation and understand event semantic information while using fewer computing resources, and has higher image feature extraction quality. Summary of the Invention

[0006] In order to overcome the above technical problems, the present invention discloses a video retrieval feature extraction method based on a high-efficiency attention mechanism.

[0007] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0008] A video semantic feature extraction method based on self-supervised learning includes the following steps:

[0009] (1) Construct a semantically aware feature network based on a highly efficient attention mechanism; obtain multiple image training samples and multiple text-image pair training samples;

[0010] (2) using a self-supervised learning method based on random enhancement of image data, using part of the image training samples to train the feature network, and the trained feature network is the image feature encoder;

[0011] (3) The image feature encoder is used as the image tower, and the pre-trained text encoder is used as the text tower to form a dual-tower model. The dual-tower model is trained using a cross-modal semantic self-supervised learning method with some text images as training samples. The image feature encoder obtained after training is the image semantic feature encoder.

[0012] (4) using the remaining image training samples and text image training samples as training data, the image feature encoder and the image semantic feature encoder obtained in step (2) and step (3) are fused and trained;

[0013] (5) Extract frames from the video to be queried, and input the obtained video frames into the jointly trained image feature encoder and image semantic feature encoder to obtain the semantic feature representation of the video frames. The semantic feature representation of each video frame is aggregated into representative features of the video to be queried through a clustering algorithm, and the representative features are used for short video retrieval.

[0014] The present invention also proposes a feature extraction system for short video retrieval based on a high-efficiency attention mechanism to implement the above method.

[0015] Beneficial effects of the present invention:

[0016] The present invention discloses a video retrieval feature extraction method based on a high-efficiency attention mechanism, which includes deriving and designing a feature extraction network with a high-efficiency attention mechanism with semantic perception, constructing a self-supervised learning method based on image data enhancement for repeated event scenarios and a self-supervised learning method based on cross-modal semantics of image-text pairs for complementary shots and event information scenarios according to different short video retrieval scenarios, and jointly training the feature encoders obtained after the two self-supervised learning. Finally, a feature extraction method suitable for high-performance and high-efficiency short video retrieval scenarios is obtained, which has the ability to extract short video image features without deformation and understand event semantic information while using fewer computing resources, and has high image feature extraction quality. At the same time, this method does not need to construct a local manually labeled data set, and sets a reasonable self-supervised learning signal, which can not only achieve the purpose of training the network, but also reduce the training cost, and facilitate the subsequent optimization of the model effect.

[0017] The present invention improves the feature network based on the self-attention mechanism and designs a high-efficiency attention mechanism to reduce its feature calculation complexity, while taking into account multi-scale learning and global information, thereby improving the performance of the high-efficiency attention mechanism in short video retrieval tasks. In the model training stage, self-supervised learning based on random enhancement of image data and self-supervised learning based on cross-modal semantics of image-text pairs are constructed respectively to obtain image feature encoders and image semantic feature encoders. The feature encoder after fusion training has the ability to extract features without deformation and understand event semantic information while using fewer computing resources. It has high image feature extraction quality and is expected to be applied in real scenarios to improve users' short video retrieval experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a high-efficiency feature network with a high-efficiency attention mechanism module with semantic perception in the present invention;

[0019] Figure 2 This is a flow chart of the short video retrieval feature extraction method based on the high-efficiency attention mechanism of the present invention. DETAILED DESCRIPTION

[0020] The method of the present invention will be further described below with reference to the accompanying drawings.

[0021] In the short video retrieval scenario, the traditional feature network based on the self-attention mechanism has the characteristics of high computational complexity. To improve the feature processing efficiency of the feature network, the present invention designs a high-performance and efficient short video feature extraction method to address this problem. The design ideas of the present invention include:

[0022] Key Point 1: By comparing the common feature extraction networks used in existing short video retrieval tasks, analyzing their performance and efficiency, and deriving and designing the feature network based on the high-efficiency attention mechanism in this invention, it can be used for subsequent inference training;

[0023] In this step, we conduct experiments on the commonly used feature extraction models in the current short video retrieval field and determine the image feature extraction network with the best efficiency and performance as the image feature encoder; at the same time, we uniformly extract frames from the video dataset in step (3) and save them in the corresponding files according to the file names. i The data is H×W×C. First, divide the image into blocks and then flatten it. Assume that the length and width of each block are (P, P), the number of blocks is N = H×W / (P×P), and each image block is flattened into a one-dimensional vector. The size of each vector is P×P×C. The total input is N×(P 2 ×C); perform a linear transformation (fully connected layer) on each vector, compressing the dimension to D, so that the input dimension is converted to N×D, and at the same time introduce a trainable variable position in each block embedding As the location information of each block; when inputting, it is necessary to add [CLS] at the front. This is an artificially added learnable variable. The specific reason can be explained in the ViT related paper;

[0024] The sequence features obtained above are normalized and then input into the multi-head attention layer. The multi-head attention layer is composed of multiple Self-Attention layers.

[0025] The multi-head self-attention mechanism is the main source of the model's computational complexity. The input matrix (N represents the number of tokens, and f represents the model embedding dimension.) After linear transformation of the matrix, we obtain three different spatial vectors Q, K, and V for query, key, and value. We calculate the attention score between the query vector Q and the key vector K, and multiply the attention weights between tokens by the value vector V to obtain the vectorized representation of the word in the entire sentence. By breaking down the complexity of the above operation process, we can analyze the computational complexity of the calculation model separately:

[0026]

[0027] In QK T In the operation of The time complexity of the naive algorithm for matrix multiplication is O(N·f·N)=O(N 2 f), due to but The time complexity is O(N 2 ).right Execute the Softmax function to find the probability value. The time complexity of the operation is O(N 2 ), then calculate the time complexity of SV operation, The time complexity of SV is O(N 2 f) Adding the time complexity of each of the above processes, we can get the total time complexity of the self-attention module, as shown below:

[0028] O(N 2 f+N 2 +N 2 +N 2 f)≈O(N 2 f)

[0029] Here, N represents the length of the sequence tokens, and f represents the embedding dimension of the model, or the number of channels. When the input token length is small, the computational complexity is acceptable. However, if the input image resolution is high, the sequence length N of the model will increase, causing the computational complexity of ViT to grow quadratically.

[0030] If you use the associative law of multiplication, first execute K T Matrix operations on V, Its operation time complexity is O(f·N·f)=O(Nf 2 ), then multiply Q by K on the left T V, get The time complexity is O(N·f·f)=O(Nf 2 ), add them together to get the total computational complexity, as shown in the following formula:

[0031] O(Nf 2 +Nf 2 )=O(Nf 2 )

[0032] According to the above formula derivation, if we use the multiplication associative law to change the calculation order of Q, K, and V and discard the Softmax function, we can reduce the computational complexity of the attention mechanism from O(N 2 f) becomes O(Nf 2 ), if the f dimension is set as a fixed value, the complexity is reduced from the quadratic level to the linear level. At the same time, in order to enhance the expression ability of the model, the present invention introduces nonlinear functions to improve the nonlinear fitting ability of the network. After empirical experiments, the original computational complexity O(N 2 ) is replaced by the Rectified Linear Unit (ReLU) function. Finally, the expression of the high-efficiency linear attention mechanism module of the present invention is as follows:

[0033]

[0034] Among them, O i To represent the i-th row of matrix O, we only need to calculate and Once each, and then reuse these results in each query. Through the above improvements, the computational complexity of the self-attention mechanism is reduced from quadratic to linear level, which greatly improves the computational efficiency of the attention mechanism module in the present invention;

[0035] Key point 2: To solve the problem of lack of semantic perception in the feature network based on the high-efficiency attention mechanism module in the short video retrieval task, relevant field knowledge such as semantic segmentation is introduced, and two key information, global information and multi-scale learning, are taken into account in the feature processing process; after optimization, a high-efficiency feature network with a high-efficiency attention mechanism module with semantic perception is obtained, such as Figure 1 shown.

[0036] The attention mechanism has strong global feature extraction capabilities. Applying the attention mechanism to input features can obtain fine-grained global feature information. Furthermore, feature networks based on the linear attention mechanism have high feature processing efficiency, meeting the high efficiency requirements of short video retrieval tasks. Multi-scale learning is achieved by performing multiple small-kernel convolutions on the original image and fusing them. High-resolution information is aggregated into low-resolution features with higher semantic meaning through convolution. The linear attention mechanism is then applied to this aggregated feature information to obtain holistic semantic information. Finally, information is fused between feature maps with different levels of aggregation to ensure that both high-level and low-level feature information of the input are captured. For this purpose, 3×3 and 5×5 convolution kernels are commonly used to achieve image resolution processing.

[0037] Experiments have found that performing multiple small-kernel convolutions of varying degrees and then fusing them together on the features of the same input image can easily result in a huge amount of computation in practice, increasing the model's computational complexity and hindering large-scale, efficient short video retrieval applications. This design evenly divides the input image's feature map H*W*C (the number of channels, C) into several branches, each performing a different computational process. The channel features from all branches are then fused and spliced ​​together to balance feature information at different aggregation scales and reduce feature loss at different granularities. This approach achieves the same results without increasing the computational complexity of the model.

[0038] A comparative experiment was set up to use image features passed through different feature networks in related video retrieval tasks. According to the evaluation indicator mAP, it was confirmed that the high-efficiency feature network with a high-efficiency attention mechanism module with semantic awareness has better feature image extraction quality and is very suitable for the application scenario of short video retrieval tasks.

[0039] Figure 2 This is a flowchart of a short video retrieval feature extraction method based on a high-efficiency attention mechanism of the present invention, which includes the following steps:

[0040] Step (1). Collect existing public industry dataset information. For datasets in the image field, datasets in the video field, and datasets of text-image pairs, obtain and download datasets;

[0041] The research collected information on existing public industry datasets in the fields of images, text-image pairs, and videos. For example, in the image field, ImageNet is a huge image library for image and visual training, with each image containing a corresponding label category; in the short video field, YouTube-8M contains the IDs of six million YouTube videos, each ranging from 120 seconds to 500 seconds in length; in the text-image pair field, the WIT dataset is the largest text-image dataset based on Wikipedia, containing 36.7 million text-image pairs from 108 pre-researched categories.

[0042] The image dataset in the present invention preferably uses a dataset in the field of short videos. The complete video sequence is extracted, and an image dataset is constructed for each video frame extracted to serve as an image training sample. The image dataset in the present invention can also use an open source image dataset such as the ImageNet dataset.

[0043] For the text-image pair dataset, it is randomly shuffled and divided into some text-image pair training samples for model training in step (5) and the remaining text-image pair training samples for joint training.

[0044] Step (2). Construct a self-supervised learning method based on image data enhancement for repeated event scenarios, and obtain an image feature encoder after training; in the self-supervised learning based on image enhancement, a high-efficiency feature network with a high-efficiency attention mechanism module with semantic perception determined in key points one and two is used as the image feature encoder; design a loss function, use self-supervised learning to learn the feature encoder, and optimize the parameters.

[0045] In the context of video / image retrieval, it is necessary to recall images that are highly similar to the query image. In addition to learning the distinguishing information of the main category, it is also necessary to learn as much image information as possible. In the video retrieval scenario, the first requirement that needs to be solved is to retrieve videos that are almost completely repeated in the query input or to perform some editing operations based on the original query input, such as video image flipping, brightness changes, random cropping, etc. Self-supervision based on data enhancement is very suitable for solving these secondary creation video images such as editing and picture editing.

[0046] Construct the following loss function for self-supervised learning: images that have undergone some editing techniques such as cropping and flipping should have a higher similarity with the feature vector of the original image in the feature space, while images that have undergone secondary editing techniques that are not identical to the original photo should have a lower similarity. The specific steps are as follows:

[0047] The obtained image dataset is used as the image training sample, and the images are randomly shuffled and divided into several image subsets X={X0,X1...X m-1}, X k Represents one of the subsets, containing N pictures batch The image subsets are subjected to two random data augmentations to obtain the edited image sets Z0 and Z1, and the feature vectors of the image sets are obtained and normalized by the feature extraction encoder determined in steps (1) and (2);

[0048] Z0=random_augmentation(X=X k ,SEED=seed0)

[0049] Z1=random_augmentation(X=X k ,SEED=seed1)

[0050] T0=Norm(VRHF(Z0))

[0051] T1=Norm(VRHF(Z1))

[0052] Among them, random_augmentation(.) is a random data augmentation operation (including random horizontal flipping, brightness and color adjustment, random cropping, etc.), X is the input image, SEED is the random data augmentation method, seed0 and seed1 are two different random seeds, T0 and T1 are normalized image feature vectors, and the dimensions of T0 and T1 are N batch *D, VRHF(.) is the network proposed in step (1) and step (2), D represents the output dimension of the network, and Norm(.) is the normalization operation;

[0053] Then, by calculating the mutual similarity matrix M=T0T1 of T0 and T1 T , The similarity matrix M is the cross-similarity value between the image vectors obtained by two different data augmentation methods for a group of images in the same batch. The ultimate training goal is to hope that the model can make the similarity of images from the same source much higher than the similarity of images from different sources, and construct the cross entropy loss function:

[0054]

[0055] Among them, L i (M) is the image x after the image enhancement operation seed0 i and N after image enhancement operation seed1 batch Calculate the cross entropy loss for each image, M i,i is the same source image x i The similarity value obtained after two random image enhancements, M i,j are images x from different sources i and x j The similarities calculated using different random image enhancements are: τ is a temperature coefficient that acts as a hyperparameter, controlling the model's ability to distinguish negative samples; exp(.) is an exponential function with the natural constant e as its base. The cross-entropy loss function is implemented on the i-th row of the mutual similarity matrix M using the above formula, increasing the similarity of the target and reducing the similarity with other non-homologous images. Similar calculations are performed on each row and column of the mutual similarity matrix to achieve self-supervision of the target image.

[0056] Since the same source image is obtained by two different random enhancement methods, that is, the rows and columns of the mutual similarity matrix M are obtained by different changes, consider implementing the cross entropy loss function on the j-th column of the mutual similarity matrix M at the same time, and calculate the loss function between the column and the images in different rows:

[0057]

[0058] Finally, the following loss function is obtained:

[0059]

[0060] Among them, L v (M) is the number of N in the batch batch The total loss of the image. Calculate the other subsets X in turn k The loss function is updated according to the gradient descent model.

[0061] Step (3). Construct a self-supervised learning method for cross-modal semantics based on image-text pairs for complementary shots and event information scenes, and obtain an image semantic feature encoder after training; the image semantic feature encoder in the self-supervised learning of cross-modal semantics based on image-text pairs adopts the image feature encoder obtained in step (4) to train the designed loss function, and uses self-supervised learning to learn the image semantic feature encoder and optimize the parameters;

[0062] For more complex scenarios, such as when the content of different images is the same but the overall expression is the same, the effect of self-supervised learning based on image enhancement in step (2) will be limited. It is necessary to construct a method that can understand more and deeper semantic content information of videos / images from another perspective. To cope with scenarios where the surface content of images is different but the deep semantic information is the same, a dual-tower model is constructed, namely the text tower and the image tower. The model is trained using open source text-image pairs. The image tower learns image information from the text-image pair, and the image feature extraction of the image tower is trained to focus on the deep semantic information of the picture.

[0063] Text feature extraction is a model method in the field of natural language processing. HuggingFace contains a large number of pre-trained language models, which are used as feature extraction tools for sentence text. At the same time, the text model has excellent semantic generalization performance and can bring different expressions of the same semantics closer in the feature space. The image tower of the multimodal self-supervised method is a visual extraction model that compares and supervises the output feature vectors of the image tower and text tower, so that the features of the image and the text are close in the semantic space or feature space.

[0064] The feature extraction model of the Image Tower can use the image feature encoder pre-trained in step (2) to obtain the feature vector representation of the image; the Text Tower uses the pre-trained model M3E-base in the Sentence-BERT library. M3E Models is an embedding model trained on a dataset of tens of millions (22 million+) Chinese sentences. It surpasses openai-ada-002 (chatGPT official model) in both text classification and retrieval tasks. The pre-trained weights of this model can be obtained through HuggingFace. After the feature vectors are extracted by the Image Tower and the Text Tower respectively, the training goal is to shorten the distance between the text image pairs input simultaneously in the feature space, while at the same time increasing the spatial distance between them and other negative samples as much as possible. The specific steps are as follows:

[0065] Using the text image pairs obtained in step (1), construct the training dataset Datasets; randomly shuffle the order and split Datasets according to the size of Batch = {(X0, T0), (X1, T1) ... (X k ,T k )...}, each image subset X k Contains N Batch images, each text subset T k Contains N Batch The text corresponding to the item;

[0066] For (X k ,Tk ) Each text-image pair in the set needs to undergo similar operations as in step (2):

[0067] 1) For image x i The image encoder proposed in step (1) and step (2) extracts the image feature v i , after further normalization, we get The entire image subset X k Processed as

[0068] 2) For text t i After the text pre-training model is processed, the text feature text is obtained i , after further normalization, we get The entire text subset T k Processed as

[0069] 3) Calculate the mutual similarity matrix

[0070] 4) Calculate the corresponding multimodal loss according to the loss function in step 3), update the image encoder according to the cross-modal loss combined with the learning rate, and obtain the image semantic encoder after training;

[0071] It can be expressed as:

[0072] v i = VRHF(x i )

[0073] text i =Sentence_BERT(t i )

[0074]

[0075] Among them, Sentence_BERT(.) represents the M3E-base deep neural network as an embedding model for text features, Norm(.) represents the normalization operation, and L v-t (M′) represents the loss function of the image-text subset, and its construction principle is similar to the method in step (2) above.

[0076] During the multimodal self-supervised training process, the parameters of the Text Tower are first frozen and do not participate in training. The Text Tower is a pre-trained model that, after sufficient training, has already achieved good generalization performance. The training process of the dual-tower model is to ensure that the image processed by the image encoder can approximate the corresponding text vector representation in the feature space.

[0077] There are more and more open-source pre-trained text feature extraction models, and the fields are becoming more refined. Different pre-trained models on HuggingFace can be loaded to perform multimodal self-supervised training, and image encoders under different text feature extraction models can be obtained, which can be used in subsequent recall effect evaluation.

[0078] Step (4). In order to obtain a feature encoder capable of capturing shallow detail information and deep temporal semantic information of short video images, the image feature encoder and the image semantic feature encoder obtained in steps (2) and (3) are fused and trained; the fused trained network is used as the feature encoder for the final application in the short video retrieval scenario;

[0079] Partial order learning is primarily used to process data with a partially ordered structure, focusing on the relative order between data. It has broad applications in information retrieval, recommendation systems, and ranking tasks. Pairwise Ranking Loss is a commonly used loss function in partial order learning. It ensures the correct order by comparing the scores of two samples, penalizing samples with higher scores to be ranked before samples with lower scores, thus preserving the relative order between samples.

[0080] The specific method of fusion training is as follows: the input types of the model are mainly images and image-text pairs. Batch data is constructed for each of these two types of data, and the two types of input samples are randomly shuffled and fused.

[0081] 1. If the current input is image x, directly use the same self-supervised method based on image data enhancement as step (2) for learning. Specifically, input image x into the image feature encoder and use the cross entropy loss function L i (M) Train the image feature encoder.

[0082] 2. If the current input is an image-text pair (x, t), a partial order learning fusion training method is constructed to learn the cross-modal image semantic feature encoder of the image-text pair. Specifically, the image x in the same batch of input data as the current (x, t) is iRandom data augmentation is performed to obtain an enhanced image ra = RandAugment(x), thereby constructing a triplet (x, ra, t), where x represents the original input image vector, ra represents the vector of the original input image after random data augmentation, and t represents the text vector of the original input image-text pair. The original image x and text t are used as the input of the twin-tower model, and the enhanced image is used as the input of the image encoder. The text tower parameters and image feature encoder parameters of the twin-tower model are fixed, and a ternary loss function is constructed. The image semantic feature encoder in the twin-tower model is optimized and learned to minimize the loss function. This approach brings the original image x and the corresponding text vector t closer in vector space, making the vector distance between the original image x and the enhanced image ra closer than the distance between the image-text pair. While ensuring that the enhanced image ra and the corresponding text vector t are close to the original image, the enhanced image ra is closer to the original image x than the input text t, thus adding fine-grained image feature encoding to the image semantic feature encoding. Finally, using the cosine similarity measure, the ternary loss function is expressed as follows:

[0083]

[0084] Margin is a hyperparameter greater than 0. The ultimate learning goal is to ensure that the enhanced image ra is closer to the original image x than the text t, and that the difference between the vector distance of text t and the enhanced image ra is at least margin. This ensures that a certain relationship is satisfied between the trained image feature encoder and the image semantic feature encoder. The feature extraction model after fusion training has richer expressive capabilities when vectorizing video frames, and the corresponding retrieval performance indicators are significantly improved. Experiments show that a margin of 0.3 achieves optimal results.

[0085] After joint training is complete, a converged image feature encoder and image semantic encoder are obtained. The query video frame is input into the two image encoders to obtain two image encoder outputs. The two outputs are weighted and summed to obtain the final semantic feature representation. In this embodiment, the output weight of the image feature encoder and the image semantic encoder is 1:0.5.

[0086] The process of short video retrieval using the semantic feature representation of the present invention can be carried out in the following manner:

[0087] A large amount of short video data is acquired, and each short video is subjected to frame extraction at a one-second interval. Several video frames are extracted, and semantic feature representations are extracted from each frame using the encoder network optimized and trained by the present invention. The semantic feature representations of each video frame are aggregated into K representative features using a clustering algorithm, and these are stored in a retrieval database representing the short video. During retrieval, semantic feature representations of several video frames of the query video are extracted using the same method, and representative features are clustered to obtain. The representative features of the query video are compared with the representative features in the retrieval database, and their cosine similarity is calculated. The results are sorted from largest to smallest based on cosine similarity to obtain retrieval results.

[0088] This embodiment also provides a feature extraction system for short video retrieval based on a high-efficiency attention mechanism, which is used to implement the above embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the system described in the following embodiments is preferably implemented in software, it is also possible to implement it in hardware, or a combination of software and hardware.

[0089] Network building module, used to build semantically aware feature networks based on a highly efficient attention mechanism;

[0090] A training data acquisition module is used to acquire image training samples and text-image pair training samples;

[0091] An encoder training module is used to train the feature network using image training samples and text image pair training samples to obtain an image feature encoder and an image semantic feature encoder;

[0092] A joint training module for jointly training the image feature encoder and the image semantic feature encoder;

[0093] The feature extraction module is used to obtain the semantic feature representation of the video frame to be queried through the jointly trained image feature encoder and image semantic feature encoder.

[0094] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Ordinary technicians in this field can understand and implement it without paying any creative work.

[0095] Embodiments of the system of the present invention can be applied to any device with data processing capabilities, such as a computer or other device. System embodiments can be implemented through software, hardware, or a combination of software and hardware. For example, a software implementation, as a logical device, is implemented by a processor of any device with data processing capabilities, reading corresponding computer program instructions from non-volatile memory into internal memory and executing them.

[0096] The above description is merely an illustration of the specific implementation of this application and the technical principles of its application. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or equivalent features without departing from the concept of this application. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A video retrieval feature extraction method based on a high-efficiency attention mechanism, characterized in that: The following steps are involved: (1) Construct a semantically aware feature network based on a highly efficient attention mechanism; obtain multiple image training samples and multiple text-image pair training samples; (2) using a self-supervised learning method based on random enhancement of image data, using part of the image training samples to train the feature network, and the trained feature network is the image feature encoder; (3) The image feature encoder is used as the image tower, and the pre-trained text encoder is used as the text tower to form a dual-tower model. The dual-tower model is trained using a cross-modal semantic self-supervised learning method with some text images as training samples. The image feature encoder obtained after training is the image semantic feature encoder. (4) using the remaining image training samples and text image training samples as training data, the image feature encoder and the image semantic feature encoder obtained in step (2) and step (3) are fused and trained; (5) Extract frames from the video to be queried, and input the obtained video frames into the jointly trained image feature encoder and image semantic feature encoder to obtain the semantic feature representation of the video frames. The semantic feature representation of each video frame is aggregated into representative features of the video to be queried through a clustering algorithm, and the representative features are used for short video retrieval.

2. The video retrieval feature extraction method based on a high-efficiency attention mechanism according to claim 1 is characterized in that: The semantically aware feature network based on the high-efficiency attention mechanism includes an image preprocessing module, a high-efficiency attention mechanism module, and a fusion output module; The image preprocessing module is used to divide the input image into n sub-images according to the channel dimension, and then process each sub-image into sub-images of different scales through convolution kernels of different sizes. After the sub-images of different scales pass through a linear layer, n image outputs are obtained; The n image outputs obtained by the image preprocessing module are respectively input into n high-efficiency attention mechanism modules. The expression of the high-efficiency attention mechanism module is as follows: Among them, O i represents the i-th row of the output matrix O of the high-efficiency attention mechanism module, ReLU(.) is the ReLU function, Q, K and V are the query, key and value of the attention mechanism respectively, and N is the number of tokens; The fusion output module is used to splice and fuse the outputs of each high-efficiency attention mechanism module, restore it to the size of the original input image and output it. The output is the feature vector of the input image.

3. The video retrieval feature extraction method based on a high-efficiency attention mechanism according to claim 1 is characterized in that: The adopting of partial image training samples to train the feature network specifically includes: Some image training samples are randomly shuffled and then divided into several image subsets. The image subsets are subjected to two random data augmentations to obtain image sets Z0 and Z1. The feature vectors of image sets Z0 and Z1 are extracted and normalized through the semantically aware feature network based on the high-efficiency attention mechanism: T0=Norm(VRHF(Z0)) T1=Norm(VRHF(Z1)) Among them, T0 and T1 are normalized image feature vector sets, and the dimensions of T0 and T1 are N batch *D, N batch is the number of images contained in the image subset, VRHF(.) is the feature network, D is the output dimension of the feature network, and Norm(.) is the normalization operation; Calculate the mutual similarity matrix M=T0T1 T , A cross entropy loss function is constructed to perform self-supervisory training on the feature network. The cross entropy loss function is: Among them, L i (M) is the cross entropy loss, M i,i 、M i,j and M j,i is the similarity value in the mutual similarity matrix M, τ is a temperature coefficient, and exp(.) is an exponential function with the natural constant e as the base.

4. The feature extraction method for short video retrieval based on a high-efficiency attention mechanism according to claim 1 is characterized in that: The pre-trained text encoder is the pre-trained model M3E-base in the Sentence-BERT library.

5. The video retrieval feature extraction method based on a high-efficiency attention mechanism according to claim 1 is characterized in that: The method of using partial text images to train the twin tower model for training samples specifically includes: Freezing parameters of the text tower; For some text image pairs, the images and texts in the training samples are extracted by the image tower and text tower in the dual tower model respectively, and further normalized to obtain the image feature set and text feature sets Calculate the mutual similarity matrix The total loss of the text-image pair is calculated and the image encoder is unsupervisedly trained based on the total loss. The total loss of the text-image pair is: Among them, L v-t (M′) represents the total loss of the text-image pair, M′ i,i , M′ i,j and M′ j,i is the mutual similarity value in the mutual similarity matrix M′, τ is the temperature coefficient, exp(.) is the exponential function with the natural constant e as the base, N batch Indicates the training batch size.

6. The video retrieval feature extraction method based on a high-efficiency attention mechanism according to claim 1 is characterized in that: The method utilizes the remaining image training samples and text image training samples as training data to perform fusion training on the image feature encoder and image semantic feature encoder obtained in step (2) and step (3), specifically including: For the image training samples used for fusion training, the image training samples are randomly enhanced, and then the image feature encoder is learned using a self-supervised learning method based on image data enhancement. For the text image pair training samples used for fusion training, the image x is randomly enhanced to obtain an enhanced image ra, and the corresponding unenhanced image x and the corresponding text t are combined to form a triple (x, ra, t), and a ternary loss function is constructed. The image x and the text t are used as the input of the image semantic feature encoder and the text encoder in the dual-tower model, and the enhanced image ra is used as the input of the image semantic feature encoder. The parameters of the text encoder and the image feature encoder are fixed, and the image semantic feature encoder in the dual-tower model is trained and optimized to minimize the ternary loss function. The expression of the ternary loss function Loss(x, ra, t) is: Among them, margin is a hyperparameter greater than 0; the triples are optimized according to the ternary loss function, and all text image pairs used for joint training are processed into training samples to obtain a triplet dataset.

7. The video retrieval feature extraction method based on a high-efficiency attention mechanism according to claim 1 is characterized in that: The video frame is input into the jointly trained image feature encoder and image semantic feature encoder to obtain the semantic feature representation of the video frame, specifically including: The video frame is input into the jointly trained image feature encoder and image semantic feature encoder. The image feature encoder and image semantic feature encoder respectively extract and output corresponding features. The two output features are weightedly summed to obtain the semantic feature representation of the corresponding video frame.

8. A video retrieval feature extraction system based on a high-efficiency attention mechanism, used to implement the method of claim 1, characterized in that: The system comprises: Network building module, used to build semantically aware feature networks based on a highly efficient attention mechanism; A training data acquisition module is used to acquire image training samples and text-image pair training samples; An encoder training module is used to train the feature network using image training samples and text image pair training samples to obtain an image feature encoder and an image semantic feature encoder; A joint training module for jointly training the image feature encoder and the image semantic feature encoder; The feature extraction module is used to obtain the semantic feature representation of the video frame to be queried through the jointly trained image feature encoder and image semantic feature encoder.

Images