Video data processing method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides a video data processing method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring an initial video feature vector of a video to be retrieved; inputting the initial video feature vector into a target video encoder, wherein the target video encoder comprises a convolutional neural network, a recurrent neural network and a selective sequence model; acquiring a global feature vector of the initial video feature vector through a convolutional neural network, and acquiring a time sequence feature vector of the initial video feature vector through a recurrent neural network; fusing the global feature vector and the time sequence feature vector to obtain a first video feature vector; different weights are configured for different first video feature vectors through the selective sequence model, so that the calculation efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing, and in particular relates to a video data processing method, device, electronic equipment and readable storage medium. Background Art

[0002] With the increase in online video uploads, video text retrieval has become an emerging demand for people to efficiently find relevant videos. Video text retrieval is a basic research task in multimodal vision and language understanding. Among them, video is not just a simple collection of static frames, but a dynamic and continuous information flow, a complex long sequence information. The core issue of long sequence modeling is how to achieve effectiveness and efficiency at the same time. Effectiveness requires that the model can selectively memorize historical information and solve long-distance dependency problems; while efficiency requires efficient computing.

[0003] The current video text retrieval is implemented through the CLIP4Clip model, which pre-trains the model through a pixel-level method, that is, using the pre-trained CLIP model to learn from the original video, and transfer the knowledge from the image text pre-training to the video text retrieval task. Different from the sparse sampling strategy adopted by the ClipBERT model, the CLIP4Clip model uses average sampling to extract relatively continuous image frames from the video clip, and then encodes them through the video encoder to obtain a series of features. In order to enhance the extraction of temporal features, the model uses 3D linear projection in the video encoder, replacing the kernel in the two-dimensional linear with three-dimensional convolution.

[0004] However, the video encoder of the CLIP4Clip model mainly adopts the Transformer structure, which has the advantages of processing input of arbitrary length and establishing global relationships. However, its attention mechanism needs to create a weight matrix to compare the correlation between each token when processing long sequences, resulting in high video memory consumption and low efficiency in calculating the sequence. Summary of the invention

[0005] The present invention provides a video data processing method, device, electronic device and readable storage medium to solve the problem that the current video text retrieval model video encoder adopts a Transformer structure. When processing long sequences, it is necessary to create a weight matrix to compare the correlation between each token, resulting in large video memory consumption and low efficiency in calculating the sequence.

[0006] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0007] In a first aspect, the present invention provides a method for processing video data, the method comprising:

[0008] Obtaining an initial video feature vector of the video to be retrieved;

[0009] Inputting the initial video feature vector into a target video encoder, wherein the target video encoder includes a convolutional neural network, a recurrent neural network and a selective sequence model;

[0010] Acquire a global feature vector of the initial video feature vector through the convolutional neural network, and acquire a temporal feature vector of the initial video feature vector through the recurrent neural network;

[0011] The global feature vector and the temporal feature vector are fused to obtain a first video feature vector;

[0012] Different weights are configured for different first video feature vectors through the selective sequence model.

[0013] Optionally, before obtaining the initial video feature vector of the video to be retrieved, the method further includes:

[0014] Get the text to be retrieved;

[0015] Performing word segmentation processing on the text to be searched to obtain target word segments;

[0016] According to a preset vocabulary, obtain the digital code corresponding to the target word segmentation;

[0017] The digital code is input into a text editor, and a text feature vector of the text to be retrieved is output.

[0018] Optionally, the obtaining of an initial video feature vector of the video to be retrieved includes:

[0019] Get the video frame of the video to be retrieved;

[0020] Extracting target video frames from the video frames at preset intervals;

[0021] Divide the target video frame into blocks to obtain image blocks;

[0022] The image block is converted into an initial video feature vector through linear projection.

[0023] Optionally, after configuring different weights for different first video feature vectors by using the selective sequence model, the method further includes:

[0024] Divide the text feature vector of the text to be retrieved and the first video feature vector carrying the weight in the video to be retrieved into blocks to obtain vector blocks;

[0025] Adding a category vector to the vector block, wherein the category vector includes a text category vector and a video category vector;

[0026] The vector block with the added category vector is input into a decoder, and the similarity between the text to be retrieved and the video to be retrieved is output.

[0027] Optionally, after inputting the vector block to which the category vector is added into a decoder and outputting the similarity between the text to be retrieved and the video to be retrieved, the method further includes:

[0028] Obtaining a plurality of matched video-text pairs and similarity scores of the video-text pairs;

[0029] generating a similarity matrix according to the similarity scores;

[0030] Through the similarity matrix, a first loss from video to text and a second loss from text to video are obtained.

[0031] Optionally, after obtaining the first loss from video to text and the second loss from text to video through the similarity matrix, the method further includes:

[0032] obtaining a loss sum of the first loss and the second loss;

[0033] The target parameter matrix in the selective sequence model is adjusted by the loss and.

[0034] Optionally, configuring different weights for different first video feature vectors through the selective sequence model includes:

[0035] Pre-setting a target parameter matrix for the selective sequence model, wherein the target parameter matrix is ​​adjusted according to the input parameters, and the target parameter matrix includes a first parameter matrix, a second parameter matrix and a third parameter matrix;

[0036] Obtaining a hidden state value of a previous time step in the selective sequence model;

[0037] Obtaining the hidden state value of the current time step in the selective sequence model through the hidden state value of the previous time step, the first video feature vector, the first parameter matrix and the second parameter matrix;

[0038] The first video feature vector configured with different weights is obtained through the hidden state value of the current time step and the third parameter matrix.

[0039] In a second aspect, the present invention provides a video data processing device, the device comprising:

[0040] A first acquisition module, used to acquire an initial video feature vector of a video to be retrieved;

[0041] An input module, used for inputting the initial video feature vector into a target video encoder, wherein the target video encoder includes a convolutional neural network, a recurrent neural network and a selective sequence model;

[0042] A second acquisition module is used to acquire a global feature vector of the initial video feature vector through the convolutional neural network, and to acquire a temporal feature vector of the initial video feature vector through the recurrent neural network;

[0043] A fusion module, used for fusing the global feature vector and the temporal feature vector to obtain a first video feature vector;

[0044] A weight configuration module is used to configure different weights for different first video feature vectors through the selective sequence model.

[0045] In a third aspect, the present invention provides an electronic device, comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor;

[0046] The processor is used to read the program in the memory to implement any of the above-mentioned video data processing methods.

[0047] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute any of the above-mentioned video data processing methods.

[0048] In an embodiment of the present invention, an initial video feature vector of a video to be retrieved is obtained; the initial video feature vector is input into a target video encoder, the target video encoder includes a convolutional neural network, a recurrent neural network and a selective sequence model; a global feature vector of the initial video feature vector is obtained through the convolutional neural network, and a time series feature vector of the initial video feature vector is obtained through the recurrent neural network; a first video feature vector is obtained by fusing the global feature vector and the time series feature vector; different weights are configured for different first video feature vectors through the selective sequence model, and by using the convolutional neural network and the recurrent neural network of the target encoder, the time relationship between features in large-scale video data can be effectively established without increasing the time dimension calculation, which greatly saves computing resources and can more efficiently and directly extract the spatiotemporal characteristics in the video content. The selective sequence model can further assign different weights to the input vector, so as to more accurately and efficiently capture the key frames or dynamics in the video content, reduce unnecessary calculations, and thus greatly improve the efficiency and accuracy of video encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 is a flowchart of a video data processing method provided by an embodiment of the present invention;

[0051] Figure 2 yes Figure 1 The flowchart of step 101 in a video data processing method provided by an embodiment of the present invention is shown;

[0052] Figure 3 yes Figure 1 The flowchart of step 105 in a video data processing method provided by an embodiment of the present invention is shown;

[0053] Figure 4 is a structural diagram of a video data processing device provided by an embodiment of the present invention;

[0054] Figure 5 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Video-Text Retrieval is a classic task in the field of vision and language, which requires retrieving videos based on texts, or retrieving texts based on videos. The current mainstream method is to learn a joint embedding space to encode videos and texts into feature vectors. Vectors with similar meanings in the space are also located similarly, so retrieval is achieved by calculating the similarity between vectors. Currently, video-text retrieval is mainly divided into three directions: 1. Fine-grained matching: A single feature vector is difficult to encode rich detailed information, and more fine-grained video-text matching is required. Taking the Hierarchical Graph Reasoning (HGR) model, the Hierarchical Alignment Network (HANet) and the Fine-Grained Cross-Modal Alignment Network (FCA-Net) as examples, it is mainly divided into three steps: multi-level text encoding, multi-level video encoding and fine-grained matching. The fine-grained matching model decomposes video-text matching into a global to local level, performs global matching on the event level, and local matching on the action and entity level. The similarity of the three levels is averaged as the final retrieval similarity. 2. Multimodal features: Videos have rich multimodal information. Using multiple multimodal features can enhance retrieval performance. Representative models include MMT (Multi-Modal Transformer) and MDMMT (Multidomain Multimodal Transformer). The advantage is that Transformer is used to jointly encode the multimodal information of the video (objects, actions, scenes, audio, etc.); the disadvantage is that multimodal features (embedding) with huge distribution differences are input into a black box, which not only cannot achieve effective fusion, but also increases the difficulty of model training. 3. Large-scale pre-training: In recent years, large-scale pre-training has been widely used, and the retrieval ability of pre-trained models has been significantly improved. It is mainly divided into three categories: 1) Fine-grained, multi-level (HiT): The input video passes through the video encoder, the input text passes through the text encoder, and then two retrieval methods (text retrieval video, video retrieval text) are used on two network levels (feature bottom layer, semantic high layer) to complete a total of four cross-modal comparisons and matching. 2) End-to-end (ClipBERT, CLIP4Clip): Taking advantage of end-to-end learning, by reducing preprocessing steps and learning directly from raw input video and text, the efficiency and accuracy of video text understanding and retrieval are effectively improved. However, due to the heterogeneity of structure and content between video and text, previous clip-based models are prone to overfitting during the training phase, resulting in relatively poor retrieval performance.3) Hubness problem: In video text retrieval, we face the "Hubness" problem: some target videos are closely related to many query texts, forming a "hub" phenomenon, which is similar to the long-tail distribution, resulting in some options being frequently retrieved while other options are generally less similar to the query. To solve this problem, a method combining a single-gate mixture of experts (CAMoE) and a double softmax loss function (DSL) is introduced. CAMoE extracts video features such as actions, entities, scenes, etc. from multiple perspectives through a hybrid expert model, and matches these features with the corresponding text parts; at the same time, borrowing strategies from natural language processing, a query library is used to re-normalize the similarity between the query text and the target video to solve the heterogeneity problem of different natures.

[0057] However, there are still some challenges in video-text retrieval. One of the core challenges is how to effectively understand and process the temporal dimension of videos. Video is not just a simple collection of static frames, but a dynamic and continuous information flow, in which the development and transformation of actions, events in time series are crucial to understanding the content of the video. Although models such as CLIP4Clip process videos by decomposing them into multiple frames, thus making use of CLIP's ability in image and text alignment to a certain extent, this approach still faces challenges in capturing dynamic changes and complex interactions in video time series: 1. Understanding of temporal relationships: How to effectively capture the temporal relationship between video frames and understand the temporal coherence of action development, event transformation, etc. in the video is a major challenge in video-text retrieval; 2. Processing of long videos: For long videos, how to effectively extract key information from them while maintaining temporal coherence and avoiding information loss is another technical difficulty; 3. Temporal sensitivity of the model: Understanding video content requires the model to be sensitive to temporal changes, which requires the model to not only recognize objects and scenes in static images, but also understand dynamic information such as the movement of objects and the transformation of scenes.

[0058] Among them, Clip is a visual classification model launched by OpenAI. Through self-supervised contrastive learning methods, it learns the relationship between images and texts from 400 million image-text pairs to achieve alignment between vision and language. The model consists of two parts: a text encoder and an image encoder, which are used to extract text and image features respectively. One of the main applications of CLIP is text search for images, that is, through feature matching of the text encoder and the image encoder, efficient retrieval is achieved using cosine similarity. In addition, CLIP can also be applied to text-video retrieval, extracting image features by analyzing video key frames. The encoder of the CLIP model mainly adopts the Transformer structure.

[0059] Clip4Clip uses pre-trained CLIP to extract features from a set of videos (or video clips) and a set of texts, and solves the problem of video text retrieval through three similarity calculation methods: parameter-free, sequential, and compact. Among them, the parameter-free method integrates video information through average pooling technology without adding additional parameters. The sequential method and the compact method introduce new weights. The sequential method uses weights of different sizes for learning, while the compact method uses the Transformer encoder to process the multimodal interaction between video and text, and predicts the similarity through a linear layer, which introduces new uninitialized weights in this process. It can be seen that the Clip4Clip model also uses the Transformer structure in the encoder. Transformer is a deep learning model based on the self-attention mechanism, which specializes in processing sequence data and performs tasks in fields such as natural language processing in an efficient and accurate manner. Its core advantage is that it can simultaneously utilize the information of all vectors (tokens) in the input sequence, but this consumes a lot of video memory when processing long sequences because a matrix needs to be created to compare the correlation between each token and all other tokens. Although attention calculation can be accelerated in parallel during training, attention needs to be recalculated for each new token during inference, which becomes a major performance bottleneck. Based on this, an embodiment of the present invention provides a video data processing method.

[0060] Reference Figure 1 , Figure 1 is a flowchart of a video data processing method provided by an embodiment of the present invention. Figure 1 As shown, the method may include:

[0061] Step 101: Obtain an initial video feature vector of a video to be retrieved.

[0062] The embodiment of the present invention is an improved method for text video retrieval. Text video retrieval needs to obtain the feature vectors of text and video respectively, and then obtain the similarity. When obtaining the text feature vector, the text to be retrieved is first extracted from the video. The text to be retrieved can be subtitles, titles, description information, etc. in the video. Use a word segmentation tool or algorithm to perform word segmentation processing on the text to be retrieved, and divide the text into independent words (word segments) (such as vocabulary, characters or subwords). Word segmentation is a basic step in text processing, which is used to convert continuous text into discrete word sequences. Use a preset vocabulary (dictionary) to convert the target word segmentation into corresponding digital codes. The vocabulary is a mapping table that maps each word to a unique digital code. The digital code is used to represent the words in the text, which is convenient for subsequent text feature extraction and processing. The digital code is input into a text editor to generate a text feature vector. The text feature vector is a numerical vector that represents the feature information of the text. The text feature vector can be used for tasks such as text classification, sentiment analysis, and information retrieval. The text editor is composed of multiple Transformer architectures to establish a global relationship between text vectors. The specific steps include:

[0063] Get the text to be retrieved;

[0064] Perform word segmentation on the text to be retrieved to obtain the target word segmentation;

[0065] According to the preset vocabulary, obtain the digital code corresponding to the target word segmentation;

[0066] Input the digital code into the text editor and output the text feature vector of the text to be retrieved.

[0067] For example, the text to be retrieved is the video subtitle "Natural language processing is an important branch of artificial intelligence". The word segmentation of "Natural language processing is an important branch of artificial intelligence" is performed to obtain the target word segmentation: ["natural language processing", "is", "artificial intelligence", "of", "one", "important", "branch"]. Assume that the vocabulary is as follows: "natural language processing"->1, "is"->2, "artificial intelligence"->3, "of"->4, "one"->5, "important"->6, "branch"->7, and the digital code corresponding to the target word segmentation is: [1,2,3,4,5,6,7]. Input the digital code [1,2,3,4,5,6,7] into the text editor and output the text feature vector. Assume that the text feature vector is represented as: [1,2,3,4,5,6,7].

[0068] Through the above steps, the text to be retrieved in the video can be segmented, and the digital code corresponding to the target segmentation can be obtained according to the preset vocabulary, and finally a text feature vector is generated. The text feature vector can be used for various text processing tasks, such as text classification, sentiment analysis, information retrieval, etc. In the video text retrieval task, the text feature vector can be used to match and retrieve related video content.

[0069] After obtaining the text feature vector, the video to be retrieved is processed and decomposed into a series of static image frames. By extracting a frame of image at fixed time intervals (for example, every 5 seconds), a representative image sequence, namely the target video frame, is extracted from the entire video. The above operation is because the video is composed of continuous image frames, and directly processing the entire video will be very time-consuming and resource-intensive. By reducing the number of frames, the complexity and amount of data processing can be effectively reduced. In addition, regular extraction of image frames helps to capture the key content and features of the video that change over time, while avoiding excessive redundant information that affects processing efficiency and effect. Then, the target video frame is divided into blocks, which can capture and utilize the local features inside the image in a more detailed manner, which is beneficial to understanding the image content and structure. In addition, the image blocks can be processed independently, which facilitates subsequent parallel computing and feature extraction, and can significantly improve processing speed and efficiency. Finally, through linear projection, the image block can be converted into a numerical vector with a fixed dimension, namely the initial video feature vector, which is convenient for subsequent calculation and analysis. The vectorized data can be directly processed by the video encoder, where linear projection helps to extract more abstract and useful information from the original pixel data, laying the foundation for further analysis and understanding of the video content.

[0070] Further, step 101, such as Figure 2 As shown:

[0071] Step 1011, obtaining the video frame of the video to be retrieved.

[0072] Step 1012: extract target video frames from video frames at preset intervals.

[0073] Step 1013, dividing the target video frame into blocks to obtain image blocks.

[0074] Step 1014: convert the image block into an initial video feature vector through linear projection.

[0075] Through the above preprocessing of the video to be retrieved, the video data is converted into a set of structured and information-rich numerical vectors (initial video feature vectors), laying a solid foundation for the subsequent processing and analysis of the video encoder. These steps not only improve the efficiency and feasibility of data processing, but also provide the possibility for in-depth mining of rich information and features in the video content.

[0076] Step 102, inputting the initial video feature vector into a target video encoder, the target video encoder includes a convolutional neural network, a recurrent neural network and a selective sequence model.

[0077] Convolutional Neural Network (CNN) is a deep learning model specifically designed to process data with a grid-like structure (such as images). CNN automatically extracts image features through structures such as convolutional layers, pooling layers, and fully connected layers, and performs well in tasks such as classification and detection. Recurrent Neural Network (RNN) is a deep learning model for processing sequence data. RNN can capture the temporal dependencies in sequence data through its cyclic structure. When generating output, RNN depends on the previous hidden state and current input, without recalculating previous states, unlike Transformer. This structure enables RNN to reason quickly, the context length can be infinitely extended in theory, and the memory usage is stable.

[0078] The target video encoder in the embodiment of the present invention is a video encoder composed of the Mamba architecture. The Mamba model is based on linear time series modeling of the selected state space. The model is a mixture of convolutional neural network and recurrent neural network models. Therefore, it has both the parallel computing advantages of convolutional neural networks and the long-range perception ability of recurrent neural networks, which can effectively solve the problem of low computing efficiency of Transformer on long sequences. Compared with Transformer, the Mamba model is more suitable for extracting video features and establishing global relationships between features of different frames.

[0079] Step 103, obtaining a global feature vector of the initial video feature vector through a convolutional neural network, and obtaining a temporal feature vector of the initial video feature vector through a recurrent neural network.

[0080] The convolutional neural network in the embodiment of the present invention is good at capturing local features in image data, such as edges, textures, etc., and by stacking multiple convolutional layers, the receptive field can be gradually expanded, thereby establishing a more comprehensive global feature representation, which is very important for understanding the content of each frame of the video. Therefore, the global feature vector of the initial video feature vector is obtained by the convolutional neural network. The recurrent neural network is particularly suitable for processing sequence data, and its cyclic structure can capture dynamic changes and long-term dependencies in the time series. Applying RNN to video feature extraction can effectively understand the temporal relationship between video frames and capture dynamic information in the video. Therefore, the temporal feature vector of the initial video feature vector is obtained by the recurrent neural network.

[0081] Step 104: The global feature vector and the temporal feature vector are fused to obtain a first video feature vector.

[0082] When the embodiment of the present invention fuses two vectors, the fusion method may be simple concatenation, weighted summation, etc. It may also be other methods, which are not specifically limited in the present invention.

[0083] Step 105: configuring different weights for different first video feature vectors through a selective sequence model.

[0084] In the embodiment of the present invention, the selective sequence model may be an SSM. Generally, an SSM includes two parts: a state update equation and an output equation. The input signal (x(t)), the hidden state (h(t)) and the output signal (y(t)) are interconnected by three continuous parameter matrices A, B and C, and the formula is:

[0085] h(t)=Ah(t-1)+Bx(t)

[0086] y(t)=Ch(t)

[0087] Where h(t-1) is the hidden state of the previous time step, h(t) is the hidden state of the current time step, x(t) is the input signal, y(t) is the output signal, and A, B, and C are fixed parameter matrices with preset settings.

[0088] Because A, B, and C are constant, the selective sequence model processes all parts of the input in exactly the same way. For the input video, the SSM will spend the same "energy" to process each frame of the video, and the model cannot focus more on information that is more critical to the current task. Based on this, the embodiment of the present invention makes an improvement by setting a target parameter matrix that can be adjusted according to the input parameters, so that the selective sequence model can dynamically adjust its state according to the current input, selectively propagate or ignore information, and the corresponding adjusted formula is as follows:

[0089] h t =S A (x t )h t-1 +S B (x t )x t

[0090] y t =S C (x t )h t

[0091] Among them, h(t-1) is the hidden state value of the previous time step, h(t) is the hidden state value of the current time step, x(t) is the input signal, y(t) is the output signal, S A (x t ), S B (x t ) and S C (x t ) is the parameter matrix adjusted according to x(t). Through the above settings, the parameter matrix can be adjusted according to the input content, and then different weights (which can also be understood as attention) are assigned to the input vectors, which can make the model more focused on the information that is more critical to the current task, reduce unnecessary information processing, avoid wasting computing resources, and not only improve the accuracy and efficiency of feature extraction, but also help reduce the complexity and computing cost of the model.

[0092] Further, step 105, such as Figure 3 As shown:

[0093] Step 1051, pre-set a target parameter matrix for the selective sequence model, wherein the target parameter matrix is ​​adjusted according to the input parameters, and the target parameter matrix includes a first parameter matrix, a second parameter matrix and a third parameter matrix.

[0094] Step 1052, obtaining the hidden state value of the previous time step in the selective sequence model.

[0095] Step 1053, obtaining the hidden state value of the current time step in the selective sequence model through the hidden state value of the previous time step, the first video feature vector, the first parameter matrix and the second parameter matrix.

[0096] Step 1054, obtaining the first video feature vector configured with different weights through the hidden state value of the current time step and the third parameter matrix.

[0097] The target video encoder in the above steps combines the advantages of CNN and RNN, which can not only capture rich spatial information, but also understand the changes of video content over time. At the same time, through the application of selective sequence models, the processing efficiency and feature extraction accuracy are further improved.

[0098] It should be noted that after preprocessing and video encoding of the video to be retrieved, it is necessary to calculate the similarity between the first video feature vector carrying weights and the text feature vector to determine the correlation between the text to be retrieved and the video to be retrieved. Preprocessing is also required before calculating the similarity. First, the vectors output by the text encoder and the video encoder (i.e., the first video feature vector carrying weights and the text feature vector) are linearly projected in blocks. By converting high-dimensional feature vectors into smaller blocks, the complexity of the calculation is reduced and the subsequent processing is also convenient. Linear projection aims to maintain the expressive power of the features while reducing the dimension of the data and improving processing efficiency. Then, a category vector is assigned to each vector block. The category vector here is used to indicate whether the current block is from text or video (for example, 0 represents text and 1 represents video). This step provides additional modal information for the subsequent model, enabling the model to distinguish and correctly process data from different sources. The processed text and video feature vectors are then input into the Transformer decoder. This step realizes multimodal fusion and uses the powerful self-attention mechanism of Transformer to capture the relationship between different features. In the Transformer decoder, the feature vectors of text and video are deeply integrated through the interaction of self-attention layers. The model can identify and emphasize the features that are most important for similarity calculation, while suppressing irrelevant information. Finally, the model outputs the similarity results between text and video. The specific steps include:

[0099] Divide the text feature vector of the text to be retrieved and the first video feature vector carrying the weight in the video to be retrieved into blocks to obtain vector blocks;

[0100] Add a category vector to the vector block, where the category vector includes a text category vector and a video category vector;

[0101] The vector block with the added category vector is input into the decoder, and the similarity between the text to be retrieved and the video to be retrieved is output.

[0102] Through the above steps, the similarity between the text to be retrieved and the video to be retrieved can be effectively calculated. Specifically, the feature vectors of the text and video are first divided into blocks, and then the category vector is added to each vector block, and finally the similarity is calculated through the decoder model. This method can comprehensively consider the characteristics of text and video, so as to more accurately evaluate the correlation between them.

[0103] Furthermore, after obtaining the similarity, the target parameter matrix in the selective sequence model can also be adjusted according to the similarity. Based on this, an embodiment of the present invention constructs a similarity matrix and applies symmetric cross entropy loss to optimize the model. When constructing the similarity matrix, first, a batch (Batch) is given, which contains B video-text pairs. Generate a similarity matrix: The model will generate a B×B dimensional similarity matrix for this batch. In this matrix, each element represents the similarity score between a pair of videos and texts. Ideally, the diagonal elements of the similarity matrix, that is, the similarity score between each video and its corresponding text, should be the highest. Based on the similarity matrix, we use symmetric cross entropy loss to calculate two types of loss functions, namely the video-to-text loss L1 and the text-to-video loss L2. The total loss function is the sum of these two types of losses L=L1+L2. The loss sum is used as the objective function to calculate the gradient of the target parameter matrix. Using gradient descent or other optimization algorithms, the target parameter matrix is ​​updated to minimize the loss. The specific steps include:

[0104] Get several matched video-text pairs and similarity scores of the video-text pairs.

[0105] Based on the similarity scores, a similarity matrix is ​​generated.

[0106] Through the similarity matrix, the first loss from video to text and the second loss from text to video are obtained.

[0107] Get the sum of the first loss and the second loss.

[0108] Through the loss and adjust the target parameter matrix in the selective sequence model.

[0109] Through the above steps, the system can effectively calculate the video-to-text and text-to-video losses, and adjust the target parameter matrix in the selective sequence model through the total loss, thereby improving the performance of the model.

[0110] In summary, compared with the problems existing in the Transformer encoder in the existing Clip4clip model, the target encoder is used in the embodiment of the present invention. The target encoder can process the spatiotemporal information in the video data more naturally by combining the design of the convolutional network and the RNN. The convolution layer focuses on capturing the spatial features within each frame, while the RNN processes the temporal sequence relationship between frames. This structural design enables the Mamba architecture to extract the spatiotemporal characteristics in the video content more efficiently and directly. Moreover, the target encoder cleverly finds a balance between local (through convolution operations) and global (using RNN processing) processing, so that the temporal relationship between features in large-scale video data is effectively established without increasing the calculation of the time dimension, which greatly saves computing resources. In addition, the selective sequence model of the target encoder in the embodiment of the present invention will also assign different attention weights to the input vector, for example, give priority to the picture frames adjacent to the current frame (given a larger weight), and assign a smaller weight to the frames farther away, which can not only capture the key frames or dynamics in the video content more accurately and efficiently, but also reduce the computational complexity, improve the processing speed and accuracy, and especially show higher efficiency in identifying the temporal relationship and key content in the video sequence. In addition, because the target encoder has fully established the timing and global relationship of the video frames, when calculating the similarity of the video and text feature vectors, only the category vector is added before each feature vector after linear block projection, so as to more effectively fuse the multimodal features without adding additional position vectors or time vectors, thereby reducing the computational requirements and complexity of the model. The technical fields to which the method of the embodiment of the present invention directly belongs or is directly applied may include multimodal large models, artificial intelligence, computer vision, text video retrieval, etc.

[0111] In an embodiment of the present invention, an initial video feature vector of a video to be retrieved is obtained; the initial video feature vector is input into a target video encoder, the target video encoder includes a convolutional neural network, a recurrent neural network and a selective sequence model; a global feature vector of the initial video feature vector is obtained through the convolutional neural network, and a temporal feature vector of the initial video feature vector is obtained through the recurrent neural network; a first video feature vector is obtained by fusing the global feature vector and the temporal feature vector; different weights are configured for different first video feature vectors through the selective sequence model, and by using the convolutional neural network and the recurrent neural network of the target encoder, the temporal relationship between features in large-scale video data can be effectively established without increasing the time dimension calculation, thereby greatly saving computing resources, and the spatiotemporal characteristics in the video content can be extracted more efficiently and directly. The selective sequence model can further assign different weights to the input vector, thereby more accurately and efficiently capturing key frames or dynamics in the video content, reducing unnecessary calculations, and greatly improving the efficiency and accuracy of video encoding.

[0112] Figure 4 : is a structural diagram of a video data processing device provided by an embodiment of the present invention, and the device may include:

[0113] The first acquisition module 201 is used to acquire an initial video feature vector of a video to be retrieved.

[0114] The input module 202 is used to input the initial video feature vector into the target video encoder, and the target video encoder includes a convolutional neural network, a recurrent neural network and a selective sequence model.

[0115] The second acquisition module 203 is used to acquire the global feature vector of the initial video feature vector through a convolutional neural network, and acquire the temporal feature vector of the initial video feature vector through a recurrent neural network.

[0116] The fusion module 204 is used to fuse the global feature vector and the temporal feature vector to obtain a first video feature vector.

[0117] The weight configuration module 205 is used to configure different weights for different first video feature vectors through a selective sequence model.

[0118] Optionally, the video data processing device further includes:

[0119] The third acquisition module is used to acquire the text to be retrieved.

[0120] The word segmentation processing module is used to perform word segmentation on the search text to obtain the target word segmentation.

[0121] The fourth acquisition module is used to acquire the digital code corresponding to the target word segmentation according to a preset vocabulary.

[0122] The first output module is used to input the digital code into the text editor and output the text feature vector of the text to be retrieved.

[0123] Optionally, the first acquisition module 201 specifically includes:

[0124] The first acquisition submodule is used to acquire video frames of the video to be retrieved.

[0125] The extraction submodule is used to extract target video frames from video frames at preset intervals.

[0126] The second acquisition submodule is used to divide the target video frame into blocks to obtain image blocks.

[0127] The conversion submodule is used to convert the image block into the initial video feature vector through linear projection.

[0128] Optionally, the video data processing device further includes:

[0129] The block processing module is used to block the text feature vector of the text to be retrieved and the first video feature vector carrying the weight in the video to be retrieved to obtain vector blocks.

[0130] The vector adding module is used to add category vectors to the vector blocks, where the category vectors include text category vectors and video category vectors.

[0131] The second output module is used to input the vector block with the added category vector into the decoder, and output the similarity between the text to be retrieved and the video to be retrieved.

[0132] The fifth acquisition module is used to acquire a number of matched video-text pairs and similarity scores of the video-text pairs.

[0133] The generation module is used to generate a similarity matrix according to the similarity scores.

[0134] The sixth acquisition module is used to obtain a first loss from video to text and a second loss from text to video through a similarity matrix.

[0135] The seventh acquisition module is used to obtain the loss sum of the first loss and the second loss.

[0136] The adjustment module is used to adjust the target parameter matrix in the selective sequence model through the loss.

[0137] Optionally, the weight configuration module 205 specifically includes:

[0138] The matrix setting submodule is used to pre-set a target parameter matrix for the selective sequence model, wherein the target parameter matrix is ​​adjusted according to the input parameters, and the target parameter matrix includes a first parameter matrix, a second parameter matrix and a third parameter matrix.

[0139] The third acquisition submodule is used to obtain the hidden state value of the previous time step in the selective sequence model.

[0140] The fourth acquisition submodule is used to acquire the hidden state value of the current time step in the selective sequence model through the hidden state value of the previous time step, the first video feature vector, the first parameter matrix and the second parameter matrix.

[0141] The fifth acquisition submodule is used to acquire the first video feature vector configured with different weights through the hidden state value of the current time step and the third parameter matrix.

[0142] In an embodiment of the present invention, an initial video feature vector of a video to be retrieved is obtained; the initial video feature vector is input into a target video encoder, the target video encoder includes a convolutional neural network, a recurrent neural network and a selective sequence model; a global feature vector of the initial video feature vector is obtained through the convolutional neural network, and a time series feature vector of the initial video feature vector is obtained through the recurrent neural network; a first video feature vector is obtained by fusing the global feature vector and the time series feature vector; different weights are configured for different first video feature vectors through the selective sequence model, and by using the convolutional neural network and the recurrent neural network of the target encoder, the time relationship between features in large-scale video data can be effectively established without increasing the time dimension calculation, which greatly saves computing resources and can more efficiently and directly extract the spatiotemporal characteristics in the video content. The selective sequence model can further assign different weights to the input vector, so as to more accurately and efficiently capture the key frames or dynamics in the video content, reduce unnecessary calculations, and thus greatly improve the efficiency and accuracy of video encoding.

[0143] The present invention also provides an electronic device, such as Figure 5 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0144] Memory 303, used for storing computer programs;

[0145] The processor 301 is used to execute the program stored in the memory 303 to implement the following steps:

[0146] Obtaining an initial video feature vector of the video to be retrieved;

[0147] Inputting the initial video feature vector into a target video encoder, wherein the target video encoder includes a convolutional neural network, a recurrent neural network and a selective sequence model;

[0148] Acquire a global feature vector of the initial video feature vector through the convolutional neural network, and acquire a temporal feature vector of the initial video feature vector through the recurrent neural network;

[0149] The global feature vector and the temporal feature vector are fused to obtain a first video feature vector;

[0150] Different weights are configured for different first video feature vectors through the selective sequence model.

[0151] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into a physical address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0152] The communication interface is used for communication between the above terminal and other devices.

[0153] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0154] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0155] The present invention also provides a readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the video data processing method of the aforementioned embodiment.

[0156] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0157] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other equipment. According to the above description, it is obvious that the structure required for constructing such a system is. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific language is for disclosing the best mode of the present invention.

[0158] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0159] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all of the features of the individual embodiments disclosed above. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0160] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0161] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the present invention. The present invention may also be implemented as a device or apparatus program for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0162] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0163] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0165] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0166] It should be noted that the various data-related processes in the embodiments of the present application are all carried out in compliance with the corresponding data protection laws and policies of the country where the device is located, and with the authorization given by the owner of the corresponding device.

Claims

1. A video data processing method, characterized in that: The method comprises: Obtaining an initial video feature vector of the video to be retrieved; Inputting the initial video feature vector into a target video encoder, wherein the target video encoder includes a convolutional neural network, a recurrent neural network and a selective sequence model; Acquire a global feature vector of the initial video feature vector through the convolutional neural network, and acquire a temporal feature vector of the initial video feature vector through the recurrent neural network; The global feature vector and the temporal feature vector are fused to obtain a first video feature vector; Different weights are configured for different first video feature vectors through the selective sequence model.

2. The method according to claim 1, characterized in that: Before obtaining the initial video feature vector of the video to be retrieved, the method further includes: Get the text to be retrieved; Performing word segmentation processing on the text to be searched to obtain target word segments; According to a preset vocabulary, obtain the digital code corresponding to the target word segmentation; The digital code is input into a text editor, and a text feature vector of the text to be retrieved is output.

3. The method according to claim 1, characterized in that: The step of obtaining an initial video feature vector of a video to be retrieved includes: Get the video frame of the video to be retrieved; Extracting target video frames from the video frames at preset intervals; Divide the target video frame into blocks to obtain image blocks; The image block is converted into an initial video feature vector through linear projection.

4. The method according to claim 2, characterized in that: After configuring different weights for different first video feature vectors by using the selective sequence model, the method further includes: Divide the text feature vector of the text to be retrieved and the first video feature vector carrying the weight in the video to be retrieved into blocks to obtain vector blocks; Adding a category vector to the vector block, wherein the category vector includes a text category vector and a video category vector; The vector block with the added category vector is input into a decoder, and the similarity between the text to be retrieved and the video to be retrieved is output.

5. The method according to claim 4, characterized in that: After inputting the vector block with the added category vector into a decoder and outputting the similarity between the text to be retrieved and the video to be retrieved, the method further includes: Obtaining a plurality of matched video-text pairs and similarity scores of the video-text pairs; generating a similarity matrix according to the similarity scores; Through the similarity matrix, a first loss from video to text and a second loss from text to video are obtained.

6. The method according to claim 5, characterized in that: After obtaining the first loss from video to text and the second loss from text to video through the similarity matrix, the method further includes: obtaining a loss sum of the first loss and the second loss; The target parameter matrix in the selective sequence model is adjusted by the loss and.

7. The method according to claim 1, characterized in that: The configuring different weights for different first video feature vectors by using the selective sequence model includes: Pre-setting a target parameter matrix for the selective sequence model, wherein the target parameter matrix is ​​adjusted according to the input parameters, and the target parameter matrix includes a first parameter matrix, a second parameter matrix and a third parameter matrix; Obtaining a hidden state value of a previous time step in the selective sequence model; Obtaining the hidden state value of the current time step in the selective sequence model through the hidden state value of the previous time step, the first video feature vector, the first parameter matrix and the second parameter matrix; The first video feature vector configured with different weights is obtained through the hidden state value of the current time step and the third parameter matrix.

8. A video data processing device, characterized in that: The device comprises: A first acquisition module, used to acquire an initial video feature vector of a video to be retrieved; An input module, used for inputting the initial video feature vector into a target video encoder, wherein the target video encoder includes a convolutional neural network, a recurrent neural network and a selective sequence model; A second acquisition module is used to acquire a global feature vector of the initial video feature vector through the convolutional neural network, and to acquire a temporal feature vector of the initial video feature vector through the recurrent neural network; A fusion module, used for fusing the global feature vector and the temporal feature vector to obtain a first video feature vector; A weight configuration module is used to configure different weights for different first video feature vectors through the selective sequence model.

9. An electronic device, characterized in that: include: A transceiver, a memory, a processor, and a program stored on the memory and executable on the processor; The processor is used to read the program in the memory to implement the steps in the video data processing method as described in any one of claims 1-7.

10. A readable storage medium for storing a program, characterized in that: When the stored program is executed by a processor, the steps in the video data processing method according to any one of claims 1 to 7 are implemented.

Images