Video grounding method and apparatus

Abstract

The present application provides a video grounding method and apparatus in the field of computer vision, capable of using unannotated video data to perform training to reduce annotation costs and significantly reduce the manual labor required for video grounding, and capable of achieving zero-shot video grounding and demonstrating very strong generalization capabilities. The method comprises: acquiring a video data set, wherein the video data set comprises video data, and the video data comprises a plurality of image frames; separately acquiring first video features and first text features of the video data, wherein the first video features are features extracted from the video data, and the first text features comprise a plurality of words describing each image frame in the video data; segmenting the first video features to obtain features of a plurality of video segments; mapping the plurality of words to the plurality of video segments to obtain a text description corresponding to each video segment; and on the basis of the text description corresponding to each video segment, training a video grounding model to obtain a trained video grounding model.

Description

Video positioning method and device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 21, 2023, with application number 202311232024.5 and application name “A Video Positioning Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing, and in particular to a video positioning method and device. Background Art

[0003] The goal of the natural language-based temporal video grounding task is to, given a query text and a video, find the video clip that best matches the query text. Existing supervised temporal video grounding techniques typically require a densely labeled video training set, which incurs significant data annotation costs.

[0004] Summary of the Invention

[0005] The present application provides a video positioning method and device, which can use unlabeled video data for training, reduce the labeling cost, greatly reduce the manpower expenditure required for video positioning, and can achieve zero-sample video positioning with very strong generalization ability.

[0006] In view of this, in a first aspect, the present application provides a video positioning method, comprising: first, obtaining a video data set, the video data set including video data, the video data including multiple frames of images; then, respectively obtaining a first video feature and a first text feature of the video data, the first video feature being a feature extracted from the video data, the first text feature including multiple words describing each frame of image in the video data; then, segmenting the first video feature to obtain features of multiple video clips; and mapping the multiple words to the multiple video clips to obtain a text description corresponding to each video clip; training a video positioning model according to the text description corresponding to each video clip to obtain a trained video positioning model, the video positioning model being used to output information of the corresponding video clip based on the input text.

[0007] Therefore, in the embodiments of this application, the video features and text features corresponding to the video data can be extracted separately, and the video features and text features can be mapped on a word-by-word basis to obtain the text description corresponding to each video segment. The video data with the corresponding text description can then be used for video localization, thereby obtaining a video localization model that can locate video segments based on the input text. This eliminates the need for manual annotation of training samples and reduces annotation costs.

[0008] In a possible implementation, the aforementioned video data set may also include metadata corresponding to the video data, and the metadata includes a text description corresponding to the video data, and obtaining the first video feature and the first text feature of the video data includes: feature extraction can be performed through a pre-trained video feature extraction network and a text feature extraction network, respectively. For example, features can be extracted from the video data through a video feature extraction network to obtain the first video feature, and features can be extracted from the text description through a text feature extraction network to obtain the first text feature.

[0009] In the implementation manner of the present application, video features and text features can be extracted respectively through a pre-trained video feature extraction network and a text feature extraction network, thereby obtaining more accurate video features and text features.

[0010] In a possible implementation, the aforementioned metadata includes at least one of the following: an audio track, subtitles, or an optical character recognition (OCR) result.

[0011] In the embodiments of this application, the text description can come from audio tracks, subtitles, or OCR, etc. Optionally, multiple frames of images in the video data can be recognized to obtain text related to the video. Therefore, for video data with different text descriptions, the required text description can be extracted, that is, the annotated video data can be obtained, thereby achieving automatic annotation of samples to facilitate subsequent supervised learning.

[0012] In a possible implementation, the aforementioned segmenting of the first video feature to obtain multiple video segments may include: clustering the first video feature to obtain multiple video segments.

[0013] In the implementation manner of the present application, video features may be classified by clustering, so that features of the same category are grouped into the same video segment.

[0014] Optionally, the video features may be segmented by other methods, such as boundary detection, sliding window, etc.

[0015] In a possible implementation, the aforementioned mapping of multiple words to multiple video clips to obtain a text description corresponding to each video clip includes: obtaining similarities between the multiple words and each video clip; and obtaining a text description corresponding to each video clip based on the similarities.

[0016] In this embodiment of the application, when mapping words to video clips, the text description corresponding to each video clip can be determined by the similarity between each word and the video clip, thereby determining a text description that is more suitable for each video clip. This is equivalent to reorganizing the text description of each video clip at the word granularity, resulting in more accurate labeling of each video clip.

[0017] In a possible embodiment, the aforementioned multiple words include at least one noun and / or at least one verb. Before obtaining the similarity between the multiple words and each video clip, the above method may further include: aligning the nouns and / or verbs in the multiple words with each frame in the video data respectively; the aforementioned obtaining the similarity between the multiple words and each video clip may specifically include: obtaining the similarity between the nouns and / or verbs aligned with each frame.

[0018] In the implementation manner of the present application, before selecting words adapted for each video clip, alignment of text and video frames is performed to improve the efficiency of subsequent text and video reorganization.

[0019] In a possible implementation, the aforementioned training of the video positioning model according to the text description corresponding to each video clip may include: iteratively training the video positioning model according to the text description corresponding to each video clip; wherein any iterative training process includes: taking the video data and the first text description as inputs of the video positioning model updated in the previous iteration, and obtaining an output result, the output result including information of the video clip corresponding to the first text description, and the text description corresponding to the video data including the first text description; obtaining a loss value between the output result and the information of the second video clip, the second video clip being a video clip corresponding to the first text description among multiple video clips; updating the video positioning model updated in the previous iteration according to the loss value, and obtaining the video positioning model updated in the current iteration.

[0020] In the implementation mode of the present application, during the training process, text and video data can be used as inputs of the video localization model, and the loss value between the information of the predicted video segment output by the video localization model and the information of the video segment corresponding to the text is calculated, and the model is updated according to the loss value, thereby realizing supervised learning.

[0021] In a possible implementation, the video localization model obtained by the previous iterative update can be used to: extract features from video data to obtain second video features, such as by using a pre-trained video feature extraction network to extract features from video data; extract features from the first text description to obtain second text features, such as by; fuse the second video features with the second text features through an attention mechanism to obtain fused features; and use the fused features as input to the output layer to obtain output results.

[0022] In the embodiments of the present application, the video localization model can fuse video features and text features through an attention mechanism, and can more effectively align text descriptions and video features through the attention mechanism. Furthermore, when extracting text features and video features, a pre-trained video feature extraction network and / or text feature extraction network can be selected to extract more effective features, thereby improving model convergence efficiency.

[0023] In a possible implementation, the aforementioned fusing of the second video feature and the second text feature through the attention mechanism to obtain a fused feature may include: extracting context information from the first video feature to obtain a context feature; and fusing the context feature and the second text feature based on the attention mechanism to obtain a fused feature.

[0024] In the implementation manner of the present application, contextual information in the video features can also be extracted to obtain contextual features that can more accurately describe the features of the video data. In subsequent fusion, the contextual features and the second text features can be fused to obtain more accurate combined features.

[0025] In the second aspect, the present application provides a video positioning method, including: first, obtaining video data and query text; then using the video data and query text as input of a trained video positioning model, and outputting information of the query video, where the query video is a video clip in the video data corresponding to the query text. The trained video positioning model is used to extract features from the video data and the query text respectively to obtain a third video feature and a third text feature, and the third video feature and the third text feature are fused based on an attention mechanism, and the fused features are used as input of an output layer to output information of the query video.

[0026] In the implementation manner of the present application, the video positioning model can achieve zero-sample positioning of the input video data and query text, that is, there is no need to use the same or the same type of video for video positioning, and has very strong generalization.

[0027] Optionally, the training process of the aforementioned trained video positioning model can refer to the training process in the aforementioned first aspect or any optional implementation manner of the first aspect, and will not be repeated here.

[0028] In a third aspect, the present application provides a video positioning device, comprising:

[0029] An acquisition module, configured to acquire a video data set, wherein the video data set includes video data;

[0030] a feature extraction module, configured to obtain a first video feature and a first text feature of the video data, wherein the first text feature includes a plurality of words describing each frame of the video data;

[0031] A segmentation module, configured to segment the first video feature to obtain multiple video segments;

[0032] A reorganization module is used to map multiple words to multiple video clips to obtain a text description corresponding to each video clip;

[0033] The training module is used to train the video positioning model according to the text description corresponding to each video clip to obtain a trained video positioning model. The video positioning model is used to output the information of the corresponding video clip based on the input text.

[0034] In one possible embodiment, the aforementioned video data set also includes metadata corresponding to the video data, and the metadata includes a text description corresponding to the video data. The feature extraction module is specifically used to: extract features from the video data through a video feature extraction network to obtain a first video feature; and extract features from the text description through a text feature extraction network to obtain a first text feature.

[0035] In a possible implementation, the aforementioned metadata includes at least one of the following: an audio track, subtitles, or an optical character recognition (OCR) result.

[0036] In a possible implementation, the aforementioned segmentation module is specifically configured to cluster the first video feature to obtain a plurality of video segments.

[0037] In a possible implementation, the aforementioned reorganization module is specifically configured to: obtain similarities between multiple words and each video segment; and obtain a text description corresponding to each video segment based on the similarities.

[0038] In a possible implementation, the aforementioned multiple words include at least one noun and / or at least one verb, and the aforementioned device may further include:

[0039] an alignment module for aligning nouns and / or verbs in the plurality of words with each frame in the video data before obtaining similarities between the plurality of words and each video segment;

[0040] The reorganization module is specifically used to obtain the similarity between nouns and / or verbs aligned with each frame.

[0041] In a possible implementation, the aforementioned training module is specifically configured to iteratively train the video localization model based on the text description corresponding to each video clip; wherein any iterative training process includes:

[0042] The video data and the first text description are used as inputs of the video positioning model updated in the previous iteration to obtain an output result, which includes information about the video segment corresponding to the first text description, and the first text description is a text description corresponding to the video data; a loss value between the output result and the information about the second video segment is obtained, and the second video segment is a video segment corresponding to the first text description among multiple video segments; the video positioning model updated in the previous iteration is updated according to the loss value to obtain a video positioning model updated in the current iteration.

[0043] In one possible implementation, the video localization model updated in the previous iteration is specifically used to: extract features from video data to obtain second video features; extract features from the first text description to obtain second text features; fuse the second video features with the second text features through an attention mechanism to obtain fused features; and use the fused features as input to the output layer to output the output results.

[0044] In a possible implementation, the video localization model updated in the previous iteration is specifically used to extract context information from the first video feature to obtain a context feature; and fuse the context feature and the second text feature based on the attention mechanism to obtain a fused feature.

[0045] In a fourth aspect, the present application provides a video positioning device, comprising:

[0046] Input module, used to obtain video data and query text;

[0047] The positioning module is used to take the video data and the query text as the input of the trained video positioning model and output the information of the query video. The query video is the video clip in the video data corresponding to the query text. The trained video positioning model is used to extract features from the video data and the query text respectively to obtain a third video feature and a third text feature. The third video feature and the third text feature are fused based on the attention mechanism, and the fused features are used as the input of the output layer to output the information of the query video.

[0048] In a possible implementation, the aforementioned trained video positioning model is a model obtained by training according to the aforementioned first aspect or any optional implementation of the first aspect.

[0049] The effects achieved by any optional implementation of the third and fourth aspects can be found in the description of any optional implementation of the first or second aspects above, and will not be repeated here.

[0050] In a fifth aspect, embodiments of the present application provide a video positioning device comprising: a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor invokes program code in the memory to execute the processing-related functions of any of the methods described in the first aspect. Optionally, the video positioning device may be a chip.

[0051] In the sixth aspect, an embodiment of the present application provides a video positioning device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing as described in the first aspect or any optional embodiment of the first aspect.

[0052] In a seventh aspect, embodiments of the present application provide a video positioning device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor invokes program code in the memory to execute the processing-related functions of any of the methods described in the second aspect. Optionally, the video positioning device may be a chip.

[0053] In the eighth aspect, an embodiment of the present application provides a video positioning device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing as described in the second aspect or any optional embodiment of the second aspect.

[0054] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute a method in any optional implementation of the first or second aspect above.

[0055] In a tenth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute a method in any optional implementation of the first or second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] FIG1 is a schematic diagram of a system architecture provided by this application;

[0057] FIG2 is a schematic diagram of another system architecture provided by this application;

[0058] FIG3 is a schematic diagram of an application scenario of video positioning provided by this application;

[0059] FIG4 is a flow chart of a video positioning method provided by the present application;

[0060] FIG5 is a flow chart of another video positioning method provided by the present application;

[0061] FIG6 is a flow chart of another video positioning method provided by the present application;

[0062] FIG7 is a flow chart of another video positioning method provided by the present application;

[0063] FIG8 is a flow chart of another video positioning method provided by the present application;

[0064] FIG9 is a flow chart of another video positioning method provided by the present application;

[0065] FIG10 is a schematic structural diagram of a video positioning device provided by the present application;

[0066] FIG11 is a schematic structural diagram of a video positioning device provided by the present application;

[0067] FIG12 is a schematic structural diagram of another video positioning device provided by the present application;

[0068] FIG13 is a schematic structural diagram of another video positioning device provided by the present application;

[0069] FIG14 is a schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.

[0071] First, to facilitate understanding, the relevant terms and concepts involved in this application are introduced.

[0072] (1) Neural Network

[0073] A neural network can be composed of neural units, which can be represented by x s The output of the operation unit with the intercept 1 as input can be shown as formula (1-1):

[0074] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s is x sThe weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0075] (2) Deep Neural Networks

[0076] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple intermediate layers. Based on the location of the different layers, the neural network within a DNN can be divided into three categories: input layer, intermediate layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and all layers in between are intermediate layers. Each layer is fully connected, meaning that any neuron in layer i is connected to any neuron in layer i+1.

[0077] Although DNN looks complicated, the work of each layer is actually not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, w is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient w as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0078] In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

[0079] It's important to note that the input layer has no W parameter. In deep neural networks, more intermediate layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0080] (3) Convolutional Neural Networks

[0081] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which can be composed of a rectangular arrangement of neurons. Neurons in the same feature plane share weights, which are referred to as convolution kernels. Shared weights can be understood as extracting image information in a position-independent manner. Convolution kernels can be initialized as matrices of random size, and during CNN training, they can learn to acquire reasonable weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the CNN, thereby reducing the risk of overfitting.

[0082] Generally, a convolutional neural network or a convolutional layer can be used for feature extraction. The feature extraction steps mentioned below in this application, such as the video encoder mentioned below, can be performed by a CNN or a neural network including a convolutional layer.

[0083] (4) Multilayer perceptron (MLP)

[0084] A multilayer perceptron, also known as a multilayer perceptron, is a feedforward artificial neural network model that maps inputs to a single output.

[0085] (5) Transformer

[0086] The transformer structure is a feature extraction network that includes an encoder and a decoder (similar to a convolutional neural network). Of course, in some cases, the transformer structure can also include a decoder without an encoder.

[0087] Encoder: Performs feature learning in the global receptive field through self-attention, such as pixel features.

[0088] Decoder: Learns the features of the required modules, such as the features of the output box, through self-attention and cross-attention.

[0089] (6) Self-attention model

[0090] It refers to effectively encoding a sequence of data (such as the natural language "Your phone is great.") into several multi-dimensional vectors to facilitate numerical operations. The multi-dimensional vectors incorporate the similarity information between each element in the sequence, and this similarity is called self-attention.

[0091] (7) Attention Mechanism

[0092] It can quickly extract important features from sparse data. The attention mechanism occurs between the encoder and decoder, or rather, between the input sentence and the generated sentence. In contrast, the self-attention mechanism in the self-attention model occurs within the encoding matrix or within the output sequence, extracting connections between distant words within the same sentence, such as syntactic features (phrase structure). The self-attention mechanism, through QKV, provides an effective modeling approach for capturing global contextual information. Assume the input is Q (query), and the context is stored in the form of key-value pairs (K, V). The attention mechanism is essentially a mapping function from the query to a series of key-value pairs (key, value). The attention function can be described as mapping a query to a series of (key-value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in the form of (K, V), attention performs addressing by calculating the similarity between Q and K. The calculated similarity between Q and K reflects the importance of the extracted V value, i.e., the weight. The weighted sum of these values ​​then yields the final feature value.

[0093] The calculation of attention is mainly divided into three steps. The first step is to calculate the similarity between the query and each key to obtain the weight. Common similarity functions include dot product, concatenation, perceptron, etc. Then the second step is generally to use a softmax function (on the one hand, it can be normalized to obtain a probability distribution where the sum of all weight coefficients is 1. On the other hand, the characteristics of the softmax function can be used to highlight the weights of important elements) to normalize these weights. Finally, the weights and the corresponding key values ​​are weighted and summed to obtain the final eigenvalue. The specific calculation formula can be as follows:

[0094] Where d is the dimension of the matrix QK.

[0095] Attention includes self-attention and cross-attention. Self-attention can be considered a special type of attention, where the QKV input is consistent. Cross-attention, on the other hand, uses inconsistent QKV input. Attention uses the similarity between features (e.g., inner product) as weights to integrate the query features as the updated value of the current feature. Self-attention extracts attention based on the attention of the feature map itself.

[0096] For convolution, the kernel setting limits the size of the receptive field, often requiring multiple layers of stacking to focus on the entire feature map. The advantage of self-attention is its global focus, enabling it to capture the global spatial information of the feature map through simple queries and assignments. A special feature of self-attention in the query key value (QKV) model is that the inputs corresponding to the QKVs are consistent.

[0097] (8) Loss function

[0098] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make the prediction lower, and continue to adjust until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function (loss function) or objective function (objective function), which is an important equation used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible. The loss function can usually include loss functions such as mean square error, cross entropy, logarithm, exponential, etc. For example, the mean square error can be used as a loss function, defined as The specific loss function can be selected according to the actual application scenario.

[0099] (9) Backpropagation algorithm

[0100] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0101] (10) Supervised Learning

[0102] Supervised learning, also known as supervised learning, is a training method / learning method in machine learning. It refers to the process of training a model by letting the machine learn a large amount of labeled sample data, so that the model can obtain corresponding output based on the input.

[0103] Generally, supervised learning requires a large amount of labeled data. The more labeled data, the better the machine learning algorithm, and the more capable it is at selecting the correct answer from the input data. In other words, supervised learning involves finding the relationship between input and output. For example, taking an image or text as input and outputting classification features or segmentation features as output.

[0104] (11) Neural machine translation

[0105] Neural machine translation is a typical task in natural language processing. Given a sentence in a source language, it outputs its corresponding sentence in a target language. In common neural machine translation models, words in both the source and target language sentences are encoded into vector representations. The connections between words and sentences are calculated in this vector space to perform the translation task.

[0106] (12) Pre-trained language model (PLM)

[0107] It is a natural language sequence encoder that encodes each word in a natural language sequence into a vector representation for prediction tasks. PLM training consists of two stages: pre-training and fine-tuning. In the pre-training stage, the model is trained on language model tasks on large-scale unsupervised text to learn word representations. In the fine-tuning stage, the model is initialized using the parameters learned in the pre-training stage and trained in fewer steps on downstream tasks such as text classification or sequence labeling. This allows the semantic information obtained from pre-training to be successfully transferred to downstream tasks.

[0108] (13) Automated Machine Learning (AutoML)

[0109] This involves designing a series of advanced control systems to operate machine learning models, enabling them to automatically learn appropriate parameters and configurations without human intervention. In deep neural network-based learning models, automated computational learning primarily involves network architecture search and global parameter setting. Network architecture search, which uses data to generate the most appropriate neural network architecture for a problem, offers significant training complexity and performance improvements.

[0110] (14) Audio Track

[0111] An audio track can be understood as an audio channel in audio data. Each audio track defines the properties of the audio track, such as the timbre, timbre library, number of channels, input / output ports, volume, etc.

[0112] (15) Subtitles

[0113] Subtitles refer to the textual content displayed in television, film, and stage productions, including dialogue. They also generally refer to the text added to film and television productions in post-production. The explanatory text and various other text that appear at the bottom of a movie or television screen, such as the film's title, cast list, lyrics, dialogue, explanatory text, character introductions, place names, and year, are all called subtitles.

[0114] (16) Optical Character Recognition (OCR) refers to the process in which an electronic device (such as a scanner or digital camera) examines characters printed on paper, determines their shape by detecting dark and light patterns, and then uses character recognition methods to translate the shape into computer text. For example, for printed characters, the text in a paper document is converted into a black and white dot matrix image file using an optical method, and the text in the image is converted into a text format through recognition software for further editing and processing by word processing software. How to debug or use auxiliary information to improve recognition accuracy is the most important topic of OCR, and the term ICR (Intelligent Character Recognition) was also created for this reason. The main indicators for measuring the performance of an OCR system include: rejection rate, error recognition rate, recognition speed, user interface friendliness, product stability, ease of use and feasibility, etc.

[0115] (17) Model convergence: After multiple rounds of iteration, the error between the model's predicted value and the actual value is less than a pre-set smaller value.

[0116] The method provided in the embodiments of the present application can be executed on a server or on a terminal device. The terminal device can be a mobile phone with image processing capabilities, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC), a personal digital assistant (PDA), a personal computer (PC), a camera, a camcorder, a smart watch, a wearable device (WD), or an autonomous vehicle, etc., and the embodiments of the present application are not limited thereto.

[0117] The following describes the system architecture provided by the embodiments of the present application.

[0118] Referring to Figure 1 , an embodiment of the present application provides a system architecture 200 . As shown in system architecture 200 , a data acquisition device 260 can be used to collect training data. After data acquisition device 260 collects the training data, it stores the training data in database 230 . Training device 220 then trains a target model / rule 201 based on the training data maintained in database 230 . This target model can be the encoding model mentioned in this application.

[0119] The following describes how the training device 220 obtains the target model / rule 201 based on the training data. Exemplarily, the training device 220 performs self-supervised learning based on the data, thereby completing the training of the target model / rule 201. A detailed description is provided in the training method below.

[0120] The target model / rule 201 in the embodiment of the present application can specifically be a neural network. It should be noted that in actual applications, the training data maintained in the database 230 does not necessarily come from the collection of the data acquisition device 260, and may also be received from other devices. It should also be noted that the training device 220 does not necessarily train the target model / rule 201 entirely based on the training data maintained by the database 230, and may also obtain training data from the cloud or other places for video positioning. The above description should not be used as a limitation on the embodiments of the present application.

[0121] The target model / rule 201 trained by the training device 220 can be applied to different systems or devices, such as the execution device 210 shown in FIG2 . The execution device 210 can be a terminal such as a smart car, a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), an in-vehicle terminal, a television, etc., or a server or cloud. In FIG2 , the execution device 210 is configured with a transceiver 212 , which can include an input / output (I / O) interface or other wireless or wired communication interface for data exchange with external devices. Taking the I / O interface as an example, a user can input data into the I / O interface through the client device 240 .

[0122] When the execution device 210 preprocesses the input data, or when the computing module 212 of the execution device 210 performs calculations and other related processing, the execution device 210 can call the data, code, etc. in the data storage system 250 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 250.

[0123] Finally, the I / O interface 212 returns the processing result to the client device 240 so as to provide it to the user.

[0124] It is worth noting that the training device 220 can generate corresponding target models / rules 201 based on different training data for different goals or different tasks. The corresponding target models / rules 201 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0125] In the scenario shown in FIG1 , the user can manually input data, which can be performed through the interface provided by transceiver 212. Alternatively, client device 240 can automatically send input data to transceiver 212. If user authorization is required for client device 240 to automatically send input data, the user can set the corresponding permissions in client device 240. The user can view the results output by execution device 210 on client device 240, which can be presented in a display, sound, action, or other specific form. Client device 240 can also serve as a data acquisition terminal, collecting input data from transceiver 212 and output results from transceiver 212 as new sample data and storing them in database 230. Alternatively, data collection can be performed without client device 240, with transceiver 212 directly storing the input data from transceiver 212 and output results from transceiver 212 as new sample data in database 230.

[0126] It is worth noting that Figure 2 is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210.

[0127] As shown in FIG1 , a target model / rule 201 is obtained through training by a training device 220 . In an embodiment of the present application, the target model / rule 201 may be a trained or updated video positioning model in the present application.

[0128] For example, in scenarios where the methods provided herein are applied to a cloud or server, the system architecture for applying the methods provided herein can be shown in Figure 2. In this system architecture 300, a server cluster 310 is implemented by one or more servers, optionally in conjunction with other computing devices, such as data storage systems and routers. Server cluster 310 can use data in data storage system 250 or invoke program code in data storage system 250 to implement the steps of the methods provided herein.

[0129] Users can operate their respective user devices (e.g., local device 101 and local device 102) to interact with the server cluster 310. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0130] Each user's local device can interact with the server cluster 310 through a communication network of any communication mechanism / communication standard, and the communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof. Specifically, the communication network can include a wireless network, a wired network, or a combination of a wireless network and a wired network. The wireless network includes but is not limited to: a fifth-generation mobile communication technology (5th-Generation, 5G) system, a long-term evolution (long term evolution, LTE) system, a global system for mobile communication (global system for mobile communication, GSM) or a code division multiple access (code division multiple access, CDMA) network, a wideband code division multiple access (wideband code division multiple access, WCDMA) network, wireless fidelity (wireless fidelity, WiFi), Bluetooth (bluetooth), Zigbee protocol (Zigbee), radio frequency identification technology (radio frequency identification, RFID), long range (Lora) wireless communication, near field communication (NFC) in any one or more combinations. The wired network can include a fiber optic communication network or a network composed of coaxial cables, etc.

[0131] In another implementation, one or more aspects of the execution device 210 may be implemented by each local device. For example, the local device 101 may provide local data or feedback calculation results to the execution device 210 .

[0132] It should be noted that all functions of the execution device 210 can also be implemented by the local device. For example, the local device 101 implements the functions of the execution device 210 and provides services to its own user, or provides services to the user of the local device 102.

[0133] The goal of the natural language-based temporal video grounding task is: given a query text and a video, the model finds the video segment in the video that best matches the query text based on the query text.

[0134] For example, as shown in Figure 3, the input is a video. Given the query text "a man crossing the road," the output is the time starting and ending points of the video segment that is most relevant to the query text. Zero-shot video temporal localization refers to the use of training data containing only the video, without the query text. The main difficulty lies in ensuring that the video temporal localization model achieves good localization results on test data when the training data lacks manual annotation.

[0135] Video temporal localization can be used in practical application scenarios such as data growth, website / app retrieval, and video creation. Existing supervised video temporal localization solutions require densely labeled video training sets, which incurs significant data annotation costs. Zero-shot video temporal localization techniques also suffer from low model generalization and cannot be applied to datasets in domains different from the training set.

[0136] For example, LocVTP is a video multimodal pre-training solution for video temporal localization tasks. The entire model mainly consists of a visual encoder, a text encoder, and a Context Warping Head. The framework follows the "pre-training + fine-tuning" paradigm. In order to complete the downstream video temporal localization task, three loss functions are designed in the pre-training stage. Such as coarse-grained contrastive learning loss, fine-grained contrastive learning loss, and time-aware contrastive learning loss. The visual encoder and text encoder are pre-trained in the video localization stage. In the test stage, the pre-trained visual encoder and text encoder are used to extract the video and query text features of the downstream video temporal localization dataset. The training data of the downstream dataset is then used to fine-tune the video temporal localization model. Finally, the fine-tuned video temporal localization model is used to complete the test of the downstream test data.

[0137] However, LocVTP has limitations in both its methods and data. At the methodological level, it follows the traditional "pre-training + fine-tuning" paradigm, which can only extract features from video and text. It requires fine-tuning the video temporal localization model on downstream datasets, making zero-shot testing impossible and difficult to migrate to different datasets. At the data level, the image-text and video-text pair data used in current video pre-training methods are suitable for visual-text matching tasks such as classification, retrieval, and question-answering, but not for temporal localization tasks.

[0138] For example, PSVL is a zero-shot video temporal localization solution. PSVL mainly includes training and testing phases. In the training phase, the temporal event proposal module is used to extract video clips that may contain events and the corresponding timestamps. The pseudo-query generation module is then used to generate nouns and verbs for the video clips through a large-scale text vocabulary and a target detection model to form a text description of the video clip. Finally, the extracted video clips, timestamps, and text descriptions are used to train a video temporal localization model (NLVL Model). In the testing phase, for the test set of video temporal localization data, the text description is separated into nouns and verbs through part-of-speech tagging to form a new text description. The video and text description are then input into the trained video temporal localization model (NLVL Model) to generate a predicted timestamp.

[0139] The PSVL method uses videos from downstream datasets to train a video temporal localization model, and generates text descriptions for video clips based on an existing target detection model. As a result, the text diversity is low (for example, the text comes from 1,600 categories), resulting in low generalization and difficulty adapting to test datasets in different domains.

[0140] Therefore, the present application provides a video positioning method that can utilize unlabeled video data for training, reduce the labeling cost, and greatly reduce the manpower overhead required for video positioning.

[0141] The following describes the method flow provided in this application in combination with the aforementioned system architecture and application scenarios.

[0142] The video localization method provided in this application can be divided into a training phase and an inference phase. During the training phase, unlabeled video data can be used to train video localization. During the inference phase, the model trained in the training phase can be used for inference. The training and inference phases can be deployed on the same device or on different devices, and this application does not limit this. The training and inference phases are described below.

[0143] 1. Training Phase

[0144] Referring to FIG4 , a flow chart of a video positioning method provided by the present application is described as follows.

[0145] 401. Obtain a video data set.

[0146] The video data set may include video data, wherein the video data may include multiple frames of images.

[0147] Optionally, in order to facilitate subsequent acquisition of text related to the video, the video data set may also include video metadata, which may include but is not limited to information such as audio tracks, subtitles or OCR results, so as to facilitate subsequent extraction of text related to the video.

[0148] The video data may be received data or data read locally, and may be used for subsequent video positioning.

[0149] 402. Obtain a first video feature and a first text feature of the video data.

[0150] After obtaining the video data set, feature extraction can be performed. Specifically, features can be extracted from the video data to obtain video features, which are referred to as first video features for ease of distinction. Feature extraction can also be performed on the text corresponding to the video data to obtain text features, which are referred to as first text features for ease of distinction. This first text feature can include multiple texts, and this application uses the example of multiple words for illustrative purposes.

[0151] Specifically, video features can be extracted from each frame of the video data, and text features can be extracted from a variety of ways. For example, if the video data set also includes an audio track, speech recognition can be performed on the data in the audio channel corresponding to the audio track to obtain the text corresponding to the speech of the audio track, and then features can be extracted from the text to obtain text features. For another example, if the video data set also includes subtitles, if a separate subtitle file can be generated for the video data, the subtitle file can be translated to obtain the subtitle text corresponding to the video, and then features can be extracted from the subtitle text to obtain text features. For another example, if the video data set also includes OCR results, text recognition can be performed on the OCR results to obtain text used to describe the video data or the text included in the video data, and feature extraction can be performed on the text to obtain text features.

[0152] In a possible implementation, each frame of image in the video data may be recognized to obtain text included in each frame of image, and features may be extracted from the text to obtain text features.

[0153] Therefore, in the embodiments of the present application, the text included in the video data or the text describing the video data can be extracted through various approaches, which can be applied to various scenarios and has a very strong generalization capability.

[0154] Specifically, when performing feature extraction, feature extraction can be performed using a pre-trained feature extraction network, such as a pre-cloud-connected visual encoder and text encoder, thereby extracting usable features from the input data.

[0155] 403. Segment the first video feature to obtain features of multiple video segments.

[0156] After the first video feature is extracted, the first video feature may be segmented to obtain a plurality of feature segments, that is, features of a plurality of video segments.

[0157] Specifically, there are many ways to segment video data, such as segmenting according to a pre-set length to obtain multiple video segments, or segmenting after clustering, dividing features of the same category into features of the same video segment to obtain multiple video segments, etc. The appropriate video segmentation method can be selected according to the actual application scenario.

[0158] 404. Map the multiple words to the multiple video clips to obtain a text description corresponding to each video clip.

[0159] After obtaining multiple video clips, the video clips can be aligned with the text in the text features to obtain a text description corresponding to each video clip.

[0160] Usually, when extracting text features and video features, the extracted video features and text features usually do not reflect the dimension of representing time sequence. Therefore, the text features and video features can be aligned, or it can be understood as mapping multiple words in the text features to the corresponding video clips to obtain the text description corresponding to each video clip.

[0161] Alternatively, similarities between multiple words in the text features and each video clip can be calculated. For example, Euclidean distance or cosine similarity can be calculated to represent the similarity between the words and each video clip, and then the corresponding word for each video clip can be determined based on the similarity. For example, the N words with the highest similarity can be selected for each video clip, where N is a positive integer, to obtain a text description corresponding to each video clip.

[0162] 405. Train the video positioning model according to the text description corresponding to each video clip to obtain a trained video positioning model.

[0163] After determining the text description corresponding to each video clip, the text description corresponding to each video clip can be used as a label for each video clip to perform supervised learning on the video positioning model to obtain a trained video positioning model.

[0164] In the implementation of this application, a pre-trained model can be used to generate labels, i.e., text descriptions, for each video clip, eliminating the need for manual annotation and reducing video localization costs. Furthermore, the reorganized video clip descriptions are more diverse than existing methods, enabling the pre-trained model to have strong generalization capabilities and adapt to downstream datasets from different domains. This model can also perform zero-shot video temporal localization tests, which are infeasible for existing video pre-training models, and exhibits stronger generalization performance than existing video temporal localization models.

[0165] In one possible implementation, when training a video positioning model, the video data can be used as the input of the video positioning model, and the information of the text description corresponding to each video segment in the video data can be used to calculate the loss value of the output result of the video positioning model, thereby realizing the training of the video positioning model. Taking one of the iterations as an example, the aforementioned video data and the text to be queried (referred to as the first text description for ease of distinction) can be used as training input to obtain an output result. The output result may include information about the video segment corresponding to the first text description, such as timestamp or length. For ease of distinction, the video segment can be referred to as the first video segment. The loss value between the output result and the information of the second video segment is calculated. The second video segment is the information of the video segment corresponding to the first description text.

[0166] Optionally, in one iteration, a pre-trained feature extraction network can be set in the video positioning model to extract video features and text features, which can be referred to as second video features and second text features for ease of distinction. For example, a pre-trained video feature extraction network can be used to extract features from video data to obtain second video features, and a pre-trained text encoder can be used to encode or extract features from the first text description to obtain second text features; then the second video features and the second text features can be fused through an attention mechanism to obtain fused features, and the fused features can be used as input to the output layer of the video positioning model to obtain the output result, i.e., information of the video segment corresponding to the first text description, such as timestamp or video length. Therefore, in an embodiment of the present application, the video positioning model can fuse video features and text features through an attention mechanism, thereby achieving more accurate fusion between video features and text features through the attention mechanism.

[0167] Optionally, before fusing the second video feature and the second text feature through the attention mechanism, the second video feature can also be encoded through the context encoding module, or the context feature in the second video feature can be extracted. When the second video feature and the second text feature are fused through the attention mechanism, the context feature and the second text feature can be fused, so that the features based on the contextual semantics in the video feature can be fused with the text features, so that the obtained fused features can more accurately describe the features included in each video clip.

[0168] The above describes the process of the training phase in the method provided by the present application. The following describes the process of the training phase in the method provided by the present application in further detail in combination with specific application scenarios.

[0169] Refer to FIG5 , which is a flowchart of another video positioning method provided by the present application.

[0170] First, we obtain unannotated video data and video metadata and use them as input to the video segment generation module, which outputs video segments with text descriptions. We then use the video segments with text descriptions to train the initial video localization model, resulting in a trained video localization model.

[0171] More specifically, the training phase can be divided into multiple parts, such as the video clip generation part and the video positioning model update phase shown in FIG5 .

[0172] The steps of the video clip generation part can be shown in Figure 6, which can be specifically divided into a video processing channel and a text processing channel as shown in Figure 6. Then, text-video recombining can be performed to output a video clip with text description.

[0173] The update phase is illustrated in Figure 7. The video localization model takes unlabeled video data and training text descriptions (referred to as training text for clarity) as input, and outputs a prediction value, which is the information about the video segment corresponding to the text description, as predicted by the video localization model. A boundary loss function is then calculated based on the predicted value and the information about the video segment corresponding to the training text, and the video localization model is updated based on the predicted value.

[0174] The following describes each stage in conjunction with the flowcharts shown in FIG6 and FIG7 .

[0175] (1) Video processing channel

[0176] First, the video data is used as the input of the visual encoder, which outputs the extracted video features.

[0177] The visual encoder can be used to extract features from input video data and output the extracted video features. Alternatively, the visual encoder can be called a video feature extraction network or a video feature extractor. The visual encoder can be a pre-trained network for video feature extraction, thereby outputting features that accurately describe the video.

[0178] Subsequently, the video features are divided into features corresponding to multiple video segments, and the timestamps of the corresponding video segments are output. For example, a dual-encoder multimodal model, Contrastive Language-Image Pre-Training (CLIP), pre-trained on large-scale image-text pairs, can be used. This allows for simultaneous image and text feature extraction using the CLIP model. Of course, other pre-trained models can also be used; this is merely an example.

[0179] The video feature segmentation method can specifically adopt boundary detection, clustering (such as K-means feature clustering) or sliding window method for segmentation. In this application, the segmentation is performed by feature clustering as an example.

[0180] For video features, a feature clustering method can be used to cluster the video features into features corresponding to several video segments, and obtain the features of the video segments and the video segment timestamps.

[0181] Specifically, the extracted video frame features {F1, F2, ..., F p ,…,F P} and frame index {1,2,…,p,…,P}, and use K-means clustering algorithm to perform feature clustering. In order to encourage adjacent frames to be in the same cluster with a greater probability, the temporal continuity of the video segments can be guaranteed instead of obtaining scattered clustering results. The video frame features and frame index vectors can be connected as the input of K-means to obtain the video segments {v1,v2,…,v n} and video clip timestamp That is, each video segment is located by means of frame indexing, so as to know which frames are adjacent frames, thereby improving the temporal continuity of the video segment.

[0182] (2) Text processing channel

[0183] The text processing channel receives video metadata as input, which may include but is not limited to the audio track, subtitles, or OCR corresponding to the video data. The subtitles can be sourced from a subtitle file corresponding to the video data or from image recognition of video frames. The choice depends on the application scenario.

[0184] For example, the text description of the entire video can be generated through the text description generation step, such as denoted as l, and the size of the dataset is denoted as N, so there are N pairs of videos and text descriptions: (V i ,C i ),i∈{1,2,…,N}. For each pair of video and text description, the video consists of P frames {f1,f2,…,f p ,…,f P}, the text description is decomposed into Q words {w1,w2,…,w q ,…,w Q}.

[0185] (3) Text-Video Recombination

[0186] The corresponding noun features, verb features, and video clip features can be used to reconstruct the text description, that is, the text description corresponding to the entire video data can be composed of words. The text description can then be aligned with the video data to obtain the text description corresponding to each video clip.

[0187] For example, after part-of-speech tagging, the text description is divided into Q (positive integer) words, which may include I (positive integer) candidate nouns {noun1, noun2, ..., noun i ,…,noun I} and J candidate verbs {verb1,verb2,…,verb j ,…,verb J}.

[0188] The CLIP model can be used to provide image-text alignment to help select nouns and verbs that have a strong match with the video clip. Each word is evaluated and sorted based on its similarity score with the video clip (such as cosine similarity, dot product similarity, etc.). Then, the top N1 nouns are selected from the candidate nouns, such as the top N1 nouns with the highest relevance, and the top N2 verbs are selected from the candidate verbs, such as the top N2 verbs with the highest relevance. Finally, the selected verbs and nouns are connected to form the final video clip text description C.

[0189] (4) Model update

[0190] After obtaining the video segment carrying the text description and the corresponding timestamp, the video segment carrying the text description and the corresponding timestamp can be used to perform video positioning.

[0191] Specifically, the video localization model can use a pre-trained visual encoder to encode videos without manual annotations, and a pre-trained text encoder to encode text descriptions. These visual encoder and text encoder can be the same as or different from the aforementioned video and text processing pipelines, and can be adjusted based on the actual application scenario. When updating the model, the visual encoder and text encoder do not need to be updated.

[0192] Then, the context encoding module is used to model the global context of the perceived video. For example, the context encoding module can extract and fuse contextual information from video features to obtain video features that can more accurately describe each video frame by combining features of adjacent video frames.

[0193] Subsequently, the text features and video features can be fused through a cross-attention module. This cross-attention module fuses the video features and text features through attention methods such as word-guided attention, video-guided attention, or multimodal cross-attention, and outputs more accurate fused features.

[0194] Subsequently, the fused features are input into a multi-layer perceptron to output the video prediction timestamp.

[0195] Then, the loss value between the predicted timestamp and the true timestamp, that is, the value of the boundary loss function, can be calculated, and the video positioning model can be reversely updated according to the value to obtain the updated video positioning model. Among them, when the video positioning model is reversely updated, the context encoding module, cross-attention module, and multi-layer perceptron in the video positioning model can be updated specifically. Optionally, the visual encoder and the text encoder can be updated or not, which can be determined according to the actual application scenario, and this application does not limit this. For example, in order to improve training efficiency, the visual encoder and text encoder in the video positioning model can adopt a pre-trained model, so the pre-trained visual encoder and text encoder can be used in the model update stage.

[0196] Therefore, in the embodiments of this application, video data is collected from multiple sources, eliminating the need for manual annotation to generate large-scale training data for video localization. This allows for zero-shot video temporal localization testing, demonstrating enhanced generalization performance compared to existing video temporal localization models. Compared to methods that generate textual descriptions of video clips solely from visual signals, the textual descriptions of video clips generated by the method provided in this application are more accurate. Furthermore, the reconstructed video clip descriptions are more diverse, enabling the pre-trained model to possess strong generalization capabilities and adapt to downstream datasets from diverse domains.

[0197] 2. Reasoning Stage

[0198] During the training phase, the initial video localization model is trained to produce a trained video localization model. During the inference phase, the trained video localization model is used for inference. The inference and training phases can be deployed on the same device or on different devices.

[0199] As shown in FIG8 , a flow chart of a video positioning method provided by the present application is as follows.

[0200] 801. Obtain video data and query text.

[0201] The video data may include multiple frames of images, and the query text may be text corresponding to the video segment to be queried.

[0202] Optionally, the video data and query text can be obtained from user input data. For example, if the trained video positioning model can be deployed on a terminal, the user can directly enter the video data and query text into the terminal. If the trained video positioning model is deployed on a server or cloud platform, the user can enter the data through the client, and the server or cloud platform can obtain the video data and query text from the received user input data.

[0203] In some possible scenarios, video data can also be stored locally, and the query text can be obtained only from the user input data. For example, if the trained video positioning model is deployed on the terminal and the video data is stored on the terminal, the user can simply enter the query text on the terminal.

[0204] 802. Use the video data and the query text as input to the trained video localization model, and output the query video clip.

[0205] After obtaining the video data and query text, the video data and query text can be used as input of the video positioning model. The video positioning model can identify and output the video segment corresponding to the query text in the video data. Specifically, the information of the video segment can be output, such as the timestamp, time window, video revelation moment and duration of the video segment.

[0206] The trained video positioning model can refer to the description of the aforementioned training stage and will not be repeated here.

[0207] Therefore, in the embodiment of the present application, the video positioning model trained in the aforementioned training phase can be used to perform video positioning, such as querying the video clip corresponding to the text. This can directly generate the predicted timestamp without fine-tuning the model, improving prediction efficiency and enhancing user experience.

[0208] For example, the video localization process can be illustrated in Figure 9. The trained video temporal localization network takes the video data and the corresponding query text as input, and outputs predicted timestamps, which are the start and end time points of the video segment most relevant to the query text. This eliminates the need for fine-tuning the model and enables zero-shot video localization.

[0209] In addition, the method provided in this application can be applied to a variety of scenarios, such as but not limited to surveillance video positioning, industrial positioning detection, video retrieval, data increment, video creation, etc. Some possible application scenarios are introduced as examples below.

[0210] Scenario 1: Data Increment

[0211] The method provided in this application can automatically annotate video data with rich metadata information. The video clip generation module with descriptions automatically annotates easily available video data with timestamps and clip descriptions, increases the training set data required for machine learning video positioning, and improves the performance of the machine learning model.

[0212] Scenario 2: Video Retrieval

[0213] The video positioning model obtained by the method provided in this application can be deployed on various cloud platforms or terminals. For example, the video positioning model can be deployed on cloud platforms or terminals, and video retrieval can be performed through websites, apps, etc. For example, users can enter query text on the website or app to obtain videos corresponding to the query text. For example, a user can enter "dog, white" to retrieve a large number of videos related to "dog, white".

[0214] Scene 3: Video Creation

[0215] The video positioning model obtained by the method provided in this application can be applied to automated video editing or generation of wonderful video clips, etc., to provide users with a better experience when using video creation, games and other related software.

[0216] Scenario 4: Surveillance Video Positioning

[0217] For example, in massive amounts of urban traffic surveillance videos, locating certain events involving bad behavior (such as robbery, fighting) or traffic incidents can help quickly lock the time, location, and people involved in the incident, thereby improving work efficiency.

[0218] Scenario 4: Industrial Detection and Positioning

[0219] The video positioning model obtained through the method provided in this application can achieve zero-sample video positioning. Since industrial inspection data is relatively sparse, the model provided in this application has strong generalization and can be adapted to industrial inspection data. For example, it can analyze videos from an assembly line in real time to locate key events such as product defects and machine failures, thereby improving manufacturing line efficiency.

[0220] The above describes the method provided by the present application. The following describes the structure of the device provided by the present application for executing the above method.

[0221] Referring to FIG10 , which is a schematic structural diagram of a video positioning device provided by the present application, the video positioning device is described as follows.

[0222] An acquisition module 1001 is configured to acquire a video data set, where the video data set includes video data;

[0223] A feature extraction module 1002 is configured to obtain a first video feature and a first text feature of the video data, wherein the first text feature includes a plurality of words describing each frame of the video data;

[0224] A segmentation module 1003 is configured to segment the first video feature to obtain multiple video segments;

[0225] a reorganization module 1004 for mapping the plurality of words to the plurality of video clips to obtain a text description corresponding to each video clip;

[0226] The training module 1005 is used to train the video positioning model according to the text description corresponding to each video clip to obtain a trained video positioning model. The video positioning model is used to output information of the corresponding video clip based on the input text.

[0227] In a possible implementation, the aforementioned video data set also includes metadata corresponding to the video data, and the metadata includes a text description corresponding to the video data. The feature extraction module 1002 is specifically used to: extract features from the video data through a video feature extraction network to obtain a first video feature; and extract features from the text description through a text feature extraction network to obtain a first text feature.

[0228] In a possible implementation, the aforementioned metadata includes at least one of the following: an audio track, subtitles, or an optical character recognition (OCR) result.

[0229] In a possible implementation, the aforementioned segmentation module 1003 is specifically configured to cluster the first video features to obtain a plurality of video segments.

[0230] In a possible implementation, the aforementioned reorganization module 1004 is specifically configured to: obtain similarities between multiple words and each video segment; and obtain a text description corresponding to each video segment based on the similarities.

[0231] In a possible implementation, the aforementioned multiple words include at least one noun and / or at least one verb, and the aforementioned device may further include:

[0232] an alignment module 1006 for aligning nouns and / or verbs in the plurality of words with each frame in the video data before obtaining similarities between the plurality of words and each video segment;

[0233] The reorganization module 1004 is specifically configured to obtain similarities between nouns and / or verbs aligned with each other in each frame.

[0234] In a possible implementation, the aforementioned training module 1005 is specifically configured to iteratively train the video localization model based on the text description corresponding to each video clip. Any iterative training process includes:

[0235] The video data and the first text description are used as inputs of the video positioning model updated in the previous iteration to obtain an output result, where the output result includes information about the video segment corresponding to the first text description, and the first text description is a text description corresponding to the video data; a loss value between the output result and the information about the second video segment is obtained, where the second video segment is a video segment corresponding to the first text description among multiple video segments; the video positioning model updated in the previous iteration is updated according to the loss value to obtain a video positioning model updated in the current iteration.

[0236] In one possible implementation, the video localization model updated in the previous iteration is specifically used to: extract features from video data to obtain second video features; extract features from the first text description to obtain second text features; fuse the second video features with the second text features through an attention mechanism to obtain fused features; and use the fused features as input to the output layer to output the output results.

[0237] In a possible implementation, the video localization model updated in the previous iteration is specifically used to extract context information from the first video feature to obtain a context feature; and fuse the context feature and the second text feature based on the attention mechanism to obtain a fused feature.

[0238] Referring to FIG11 , a structural diagram of another video positioning device provided by the present application is described below.

[0239] Input module 1101, used to obtain video data and query text;

[0240] The positioning module 1102 is used to take the video data and the query text as inputs of the trained video positioning model and output the information of the query video. The query video is a video segment in the video data corresponding to the query text. The trained video positioning model is used to extract features from the video data and the query text respectively to obtain a third video feature and a third text feature. The third video feature and the third text feature are fused based on the attention mechanism, and the fused features are used as inputs of the output layer to output the information of the query video.

[0241] In a possible implementation, the aforementioned trained video positioning model is a model obtained by training through the implementations corresponding to the aforementioned Figures 4 to 9 .

[0242] Please refer to FIG12 , which is a structural diagram of another video positioning device provided in this application, as described below.

[0243] The video positioning device may include a processor 1201 and a memory 1202. The processor 1201 and the memory 1202 are interconnected via a line. The memory 1202 stores program instructions and data.

[0244] The memory 1202 stores program instructions and data corresponding to the steps in FIG. 4 to FIG. 7 .

[0245] The processor 1201 is configured to execute the method steps performed by the video positioning device shown in any of the embodiments in FIG. 4 to FIG. 7 .

[0246] Optionally, the video positioning device may further include a transceiver 1203 for receiving or sending data.

[0247] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a program, which, when executed on a computer, enables the computer to execute the steps of the method described in the embodiments shown in the aforementioned Figures 4 to 7 .

[0248] Optionally, the video positioning device shown in the aforementioned FIG. 12 is a chip.

[0249] An embodiment of the present application also provides a video positioning device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to execute the method steps performed by the video positioning device shown in any of the embodiments in Figures 4 to 7 above.

[0250] Please refer to FIG13 , which is a structural diagram of another video positioning device provided in this application, as described below.

[0251] The training device may include a processor 1301 and a memory 1302. The processor 1301 and the memory 1302 are interconnected via a circuit. The memory 1302 stores program instructions and data.

[0252] The memory 1302 stores program instructions and data for the steps executed by the video positioning device in Figures 8 and 9.

[0253] The processor 1301 is configured to execute the method steps executed by the video positioning apparatus in FIG. 8-FIG . 9 .

[0254] Optionally, the video positioning device may further include a transceiver 1303 for receiving or sending data.

[0255] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a program, which, when executed on a computer, enables the computer to execute the steps of the method described in the embodiments shown in the aforementioned Figures 8-9.

[0256] Optionally, the video positioning device shown in the aforementioned FIG. 13 is a chip.

[0257] An embodiment of the present application also provides a video positioning device. The training device can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is used to execute the method steps shown in any of the embodiments in Figure 6 above.

[0258] The present application also provides a digital processing chip. The digital processing chip integrates circuitry and one or more interfaces for implementing the functions of the aforementioned processor 1201, processor 1301, or both processor 1201 and processor 1301. When the digital processing chip integrates memory, the digital processing chip can perform the method steps of any one or more of the aforementioned embodiments. When the digital processing chip does not integrate memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the video positioning device or the actions performed by the video positioning device in the aforementioned embodiments based on program code stored in the external memory.

[0259] An embodiment of the present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the steps of the method described in the embodiments shown in the aforementioned Figures 4-7 or Figures 8-9.

[0260] The video positioning device or video positioning device provided in the embodiment of the present application can be a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit to enable the chip in the server to execute the method described in the embodiment shown in Figures 4 to 7 or Figures 8 to 9 above. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0261] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0262] For example, see Figure 14, which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 140. NPU 140 is mounted on the host CPU (host CPU) as a coprocessor and assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 1403, which is controlled by controller 1404 to extract matrix data from memory and perform multiplication operations.

[0263] In some implementations, arithmetic circuit 1403 includes multiple processing engines (PEs). In some implementations, arithmetic circuit 1403 is a two-dimensional systolic array. Arithmetic circuit 1403 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 1403 is a general-purpose matrix processor.

[0264] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1402 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1401 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1408.

[0265] Unified memory 1406 is used to store input and output data. Weight data is directly transferred to weight memory 1402 via direct memory access controller (DMAC) 1405. Input data is also transferred to unified memory 1406 via DMAC.

[0266] The bus interface unit (BIU) 1410 is used for interaction between the AXI bus, the DMAC, and the instruction fetch buffer (IFB) 1409 .

[0267] The bus interface unit 1410 (BIU) is used for the instruction fetch memory 1409 to obtain instructions from the external memory, and is also used for the storage unit access controller 1405 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0268] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1406 or transfer weight data to the weight memory 1402 or transfer input data to the input memory 1401.

[0269] The vector calculation unit 1407 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0270] In some implementations, the vector calculation unit 1407 can store the processed output vector to the unified memory 1406. For example, the vector calculation unit 1407 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1403, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1407 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1403, for example, for use in a subsequent layer in a neural network.

[0271] An instruction fetch buffer 1409 connected to the controller 1404 is used to store instructions used by the controller 1404;

[0272] Unified memory 1406, input memory 1401, weight memory 1402, and instruction fetch memory 1409 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0273] The operations of each layer in the recurrent neural network can be performed by the operation circuit 1403 or the vector calculation unit 1407.

[0274] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the program of the method of Figures 4-7 or Figures 8-9 above.

[0275] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0276] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., including a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0277] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0278] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a server, or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0279] The terms "first," "second," "third," "fourth," and the like in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0280] Finally, it should be noted that the above is only a specific implementation method of the present application, but the protection scope of the present application is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A video positioning method, characterized in that: include: Acquire a video data set, wherein the video data set includes video data; Acquire a first video feature and a first text feature of the video data, wherein the first text feature includes a plurality of words describing each frame of the image in the video data, and the first video feature is a feature extracted from the video data; Segmenting the first video feature to obtain features of multiple video clips; Mapping the multiple words to the features of the multiple video clips to obtain a text description corresponding to each video clip; The video positioning model is trained according to the text description corresponding to each video clip to obtain a trained video positioning model, and the video positioning model is used to output the information of the corresponding video clip based on the input text.

2. The method according to claim 1, characterized in that The video data set also includes metadata corresponding to the video data, and the metadata includes a text description corresponding to the video data. The obtaining of the first video feature and the first text feature of the video data includes: Extracting features from the video data through a video feature extraction network to obtain the first video features; Features are extracted from the text description through a text feature extraction network to obtain the first text feature.

3. The method according to claim 2, characterized in that The metadata includes at least one of the following: an audio track, a subtitle, or an optical character recognition (OCR) result.

4. The method according to any one of claims 1 to 3, characterized in that The first video feature is segmented to obtain a plurality of video segments, including: The first video features are clustered to obtain the multiple video clips.

5. The method according to claim 1, characterized in that The mapping of the multiple words to the multiple video clips to obtain a text description corresponding to each video clip includes: Obtaining similarities between the plurality of words and the features of each video clip; A text description corresponding to each video clip is obtained according to the similarity.

6. The method according to claim 5, characterized in that The multiple words include at least one noun and / or at least one verb. Before obtaining the similarity between the multiple words and the features of each video clip, the method further includes: Aligning nouns and / or verbs in the plurality of words with each frame in the video data respectively; The obtaining of the similarity between the plurality of words and the feature of each video clip comprises: The similarity between the feature of each frame and the noun and / or verb aligned with each frame is obtained.

7. The method according to any one of claims 1 to 6, characterized in that The step of training the video positioning model according to the text description corresponding to each video clip includes: Iteratively training the video localization model according to the text description corresponding to each video clip; Among them, any iterative training process includes: Using the video data and the first text description as inputs of the video localization model updated in the last iteration, obtaining an output result, wherein the output result includes information of the video segment corresponding to the first text description, and the text description corresponding to the video data includes the first text description; Obtaining a loss value between the output result and information of a second video segment, where the second video segment is a video segment corresponding to the first text description among the multiple video segments; The video positioning model updated in the previous iteration is updated according to the loss value to obtain the video positioning model updated in the current iteration.

8. The method according to claim 7, characterized in that The video positioning model updated in the last iteration is used for: Extracting features from the video data to obtain second video features; Extract features from the first text description to obtain second text features; fusing the second video feature and the second text feature through an attention mechanism to obtain a fused feature; The fused features are used as input of the output layer, and the output result is output.

9. The method according to claim 8, characterized in that The fusing the second video feature and the second text feature through the attention mechanism to obtain a fused feature includes: Extracting context information from the first video feature to obtain a context feature; The context feature and the second text feature are fused based on an attention mechanism to obtain the fused feature.

10. A video positioning method, characterized in that: include: Get video data and query text; The video data and the query text are used as inputs of a trained video positioning model, and information of the query video is output. The query video is a video clip in the video data corresponding to the query text. The trained video positioning model is used to extract features from the video data and the query text respectively to obtain a third video feature and a third text feature. The third video feature and the third text feature are fused based on an attention mechanism, and the fused features are used as inputs of an output layer to output information of the query video.

11. The method according to claim 10, characterized in that The trained video positioning model is a model obtained by training according to any one of claims 1 to 9.

12. A video positioning device, characterized in that: include: An acquisition module, used for acquiring a video data set, wherein the video data set includes video data; A feature extraction module, used to obtain a first video feature and a first text feature of the video data, wherein the first text feature includes a plurality of words describing each frame of the video data, and the first video feature is a feature extracted from the video data; A segmentation module, used for segmenting the first video feature to obtain features of multiple video clips; A reorganization module, used for mapping the multiple words with the features of the multiple video clips to obtain a text description corresponding to each video clip; The training module is used to train the video positioning model according to the text description corresponding to each video clip to obtain a trained video positioning model, and the video positioning model is used to output the information of the corresponding video clip based on the input text.

13. The device according to claim 12, characterized in that The video data set also includes metadata corresponding to the video data, and the metadata includes a text description corresponding to the video data. The feature extraction module is specifically used to: Extracting features from the video data through a video feature extraction network to obtain the first video features; Features are extracted from the text description through a text feature extraction network to obtain the first text feature.

14. The device according to claim 13, characterized in that The metadata includes at least one of the following: an audio track, a subtitle, or an optical character recognition (OCR) result.

15. The device according to any one of claims 12 to 14, characterized in that The segmentation module is specifically used to cluster the first video features to obtain the multiple video clips.

16. The device according to claim 12, characterized in that The recombination module is specifically used for: Obtaining similarities between the multiple words and each video clip; A text description corresponding to each video clip is obtained according to the similarity.

17. The device according to claim 16, characterized in that The plurality of words include at least one noun and / or at least one verb, and the apparatus further includes: an alignment module, configured to align nouns and / or verbs in the plurality of words with each frame in the video data before obtaining the similarity between the plurality of words and each video segment; The aforementioned reorganization module is specifically used to obtain the similarity between the nouns and / or verbs aligned with each frame.

18. The device according to any one of claims 12 to 17, characterized in that The training module is specifically used for: Iteratively training the video localization model according to the text description corresponding to each video clip; Among them, any iterative training process includes: Using the video data and the first text description as inputs of the video localization model updated in the last iteration, obtaining an output result, wherein the output result includes information of a video segment corresponding to the first text description, and the first text description is a text description corresponding to the video data; Obtaining a loss value between the output result and information of a second video segment, where the second video segment is a video segment corresponding to the first text description among the multiple video segments; The video positioning model updated in the previous iteration is updated according to the loss value to obtain the video positioning model updated in the current iteration.

19. The device according to claim 18, characterized in that The video positioning model updated in the last iteration is used for: Extracting features from the video data to obtain second video features; Extract features from the first text description to obtain second text features; fusing the second video feature and the second text feature through an attention mechanism to obtain a fused feature; The fused features are used as input of the output layer, and the output result is output.

20. The device according to claim 19, characterized in that The video positioning model updated in the last iteration is specifically used for: Extracting context information from the first video feature to obtain a context feature; The context feature and the second text feature are fused based on an attention mechanism to obtain the fused feature.

21. A video positioning device, characterized in that: include: Input module, used to obtain video data and query text; A positioning module is used to use the video data and the query text as inputs of a trained video positioning model to output information of the query video, wherein the query video is a video clip in the video data corresponding to the query text, and the trained video positioning model is used to extract features from the video data and the query text respectively to obtain a third video feature and a third text feature, fuse the third video feature and the third text feature based on an attention mechanism, and use the fused features as inputs of an output layer to output information of the query video.

22. The device according to claim 21, characterized in that The trained video positioning model is a model obtained by training according to any one of claims 1 to 9.

23. A video positioning device, characterized in that: The method comprises a processor, wherein the processor is coupled to a memory, wherein the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the method according to any one of claims 1 to 9 is implemented.

24. A video positioning device, characterized in that: The method comprises a processor, the processor is coupled to a memory, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the method according to any one of claims 10 to 11 is implemented.

25. A computer-readable storage medium comprising a program, which, when executed by a processing unit, performs the method according to any one of claims 1 to 9 or 10 to 11.

26. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 or 10 to 11 is implemented.

Images