Video retrieval method based on deep neural network model and multiple example learning

Abstract

The application relates to the field of computer vision processing, in particular to a video retrieval method based on a deep neural network model and multi-example learning, which comprises the following steps: obtaining initial features by pre-training a query text, extracting I 3D-RGB features, ROI features and connection features from a video; updating frame-level visual features and word-level text features; constructing a graph for training, learning word-level text features by using a graph attention network; calculating the residual error of the word-level text features and the word-level text features, and taking the mean value of the residual error as a sentence-level text feature; performing segment dimension average operation on the frame-level visual features to obtain pipeline-level visual features; calculating the alignment score of the sentence-level text features and the pipeline-level visual features, constructing positive sample pairs and negative sample pairs, and training a video retrieval network; and the application constructs a graph neural network by acquiring discriminative features in multiple query texts through deep learning features, so as to provide text features with more representation meanings and multi-modal alignment supervision signals under weak supervision settings.

Description

Technical Field

[0001] The present invention relates to the field of computer vision processing, and more specifically to a video retrieval method based on a deep neural network model and multi-instance learning. Background Art

[0002] With the popularization of the Internet and the rapid development of artificial intelligence, information is presented in more diverse forms, and is mainly concentrated in the most intuitive and rich images, voices, videos, and texts. Therefore, the interdisciplinary research in the fields of computer vision and natural language processing, as well as the processing of "video", the most widely disseminated and popular form of information dissemination, has become a new development direction.

[0003] Given a query, i.e. a natural language expression, visual localization aims to locate the corresponding region or fragment through a comprehensive understanding of the visual content. First, the target is queried in the image based on category, attribute and context. In real-world scenarios, visual localization in space and time is a more practical, reasonable and challenging task.

[0004] In the research task of aligning visual areas with query text in videos, in order to make the alignment effect more realistic, a large amount of labels and area annotation information is often required during the training process. The annotation of this information greatly increases the consumption of manpower, financial and material resources. Therefore, in terms of technical implementation and practical application, more scenarios tend to solve this problem through weak supervision. Under the setting of weak supervision, aligning visual areas with text with less known label information has become the key to the video query positioning task.

[0005] Existing research work based on weakly supervised learning only uses a single video-query-text pair for training when processing the alignment of visual and textual information, while ignoring the scenario where multiple queryable objects appear simultaneously in a video. Obviously, this scene information can provide semantic relationships, visual overlap and alignment relationships between multiple queries, and can provide important supervisory signals for the training of video retrieval models under weak supervision settings.

[0006] In summary, the main challenges of existing video localization methods are summarized as follows: 1. Semantic understanding: In-depth semantic understanding of video content is required to accurately identify and match natural language descriptions; 2. Modal difference and information density: There is a modal difference between the visual information and language description of the video, and these two types of information need to be effectively integrated; 3. Utilization of contextual information: Activities in videos usually do not appear independently, they often have internal semantic associations and temporal interdependence, which puts forward requirements for the modeling of video context information.

[0007] Therefore, this application proposes a video retrieval method based on a deep neural network model and multi-instance learning. Summary of the Invention

[0008] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a video retrieval method based on a deep neural network model and multi-instance learning, which can provide more representative text features and multimodal alignment supervision signals under a weak supervision setting, and can achieve good results with a small amount of labeled data.

[0009] The technical solution adopted by the present invention to solve its technical problem is:

[0010] A video retrieval method based on a deep neural network model and multi-instance learning, characterized by:

[0011] S1: Given a video and its corresponding query text, the query text is defined as , The number of texts, is a sequence of video frames, is the number of frames in the video; the query text obtains initial features through the BERT-Base pre-training model ; Extract I3D-RGB features and ROI features from the video, connect the I3D-RGB features and ROI features to obtain the connection features ;

[0012] S2: Connect features Update features through the visual Transformer network to obtain frame-level visual features ; Initial features of query text Update features through the text Transformer network to obtain word-level text features ,

[0013] ;

[0014] ;

[0015] in, , is the maximum length of a description sentence, is the embedding dimension, is the dimension of the concatenated features; 、 、 、 、 、 are different learnable transformation matrices; 、 is the input value;

[0016] S3: word-level text features of multiple query texts Define edge and node features to obtain a graph for construction training;

[0017] S4: Input the training graph into the graph attention network to learn the attention weights between different nodes and obtain word-level text features containing rich mutual relationship information ;

[0018] S5: Calculate the residual of word-level text features and character-level text features , ;

[0019] And the average value of the residuals is used as the sentence-level text feature , ;

[0020] Frame-level visual features Perform average operation on segment dimension to obtain pipeline-level visual features ;

[0021] S6: Calculate the alignment scores of sentence-level text features and pipeline-level visual features through the multimodal fusion module ; Based on alignment score Construct positive sample pairs and negative sample pairs;

[0022] S7: Input the positive sample pairs and negative sample pairs into the pre-trained video retrieval network for training. During the training process, the mutual exclusion disambiguation loss function and the multi-instance ranking loss function are combined to obtain the total loss of the video retrieval network. The loss function gradient descent method is then used to pass the loss back to the video retrieval network to update the structural parameters until the total loss of the video retrieval network is less than the set loss convergence limit, completing the training.

[0023] S8: Input the query text to be retrieved and the corresponding video into the trained video retrieval network, so as to locate the segment corresponding to the query text in the video.

[0024] Preferably, a further technical solution of the present invention is:

[0025] Preferably, in step S1, the video frame sequence is first input into the visual feature encoder, and the Faster-RCNN network in the visual feature encoder extracts the center coordinate dimension vector sequence ,in is a 4-dimensional vector; then, the SORT algorithm is used to extract candidate spatiotemporal pipeline sequences , where each is a continuous sequence of bounding boxes; then, using Extract I3D-RGB features and ROI features, and divide each spatiotemporal pipeline evenly into The average value of each segment is used as the segment feature, and the segment features of the I3D-RGB feature and the segment features of the ROI feature are connected to obtain the connection feature .

[0026] Preferably, the process of defining edge and node features in step S3 is as follows: ;in, , yes The sum of the number of words in the sentences, is the concatenation operator;

[0027] Graph Vertices It is defined as,

[0028] ;

[0029] Graph Construction Edge It is defined as,

[0030] ;

[0031] in, Include vertices, Include Vertices, operators exist Create edges between each vertex in and For each vertex in Represented as an adjacency matrix .

[0032] Preferably, step S4, word-level text features The calculation process is as follows. First, the vertex is calculated The importance score of the feature to vertex j ,

[0033] ;

[0034] in, are the parameters of the graph attention network, is the Leaky-ReLU function, is the parameterized weight vector of the feedforward neural network;

[0035] Then calculate the word-level text features , .

[0036] Preferably, in step S6, the alignment score of the sentence-level text feature and the pipeline-level visual feature is calculated When, first, element-wise addition and element-wise multiplication operations are performed on the sentence-level text feature and the pipeline-level visual feature; then, the element-wise addition operation result, the element-wise multiplication operation result, the sentence-level text feature, and the pipeline-level visual feature are connected to obtain a fusion feature, so as to obtain the alignment score between the first sentence and the first spatiotemporal pipeline in the video . ;

[0037] ;

[0038] ;

[0039] .

[0040] Preferably, in step S6, the similarity score between the sentence descriptions of the video and is defined as the maximum value of the spatiotemporal pipeline alignment score of the first query text and the video ,

[0041] ;

[0042] wherein the video contains spatiotemporal pipelines; in the training batch, the positive sample pair is defined, wherein represents the first query text in the query texts corresponding to a video; at the same time, and are negative sample pairs. Preferably, in step S7, the multi-instance ranking loss function is defined as

[0043] ,

[0044] ;

[0045] wherein is a boundary parameter, and the target of ranking is to expand the similarity score between the matching query text and the spatiotemporal pipeline pair and the similarity score between the non-matching query text and the spatiotemporal pipeline pair;

[0046] The mutually exclusive disambiguation loss function is defined. First, the alignment score is converted into a probability by a Softmax function, ​​​​​

[0047] ;

[0048] Definition of the present invention It is The probability distribution of query texts is then used to force the distribution of query texts to be different.

[0049] ;

[0050] ;

[0051] The total loss calculation formula of the video retrieval network is defined as:

[0052] ;

[0053] in, It is a trade-off hyperparameter and defines the maximum value of KL divergence. ; If the KL divergence is greater than , it will be clipped to , to prevent gradient explosion during optimization.

[0054] This paper constructs a scenario where multiple query text localization tasks are performed simultaneously on a single video. In this scenario, a model architecture based on the attention mechanism and graph neural network is proposed. Compared with the existing technology, its outstanding features are:

[0055] 1. The present invention can obtain discriminative features from multiple query texts through deep learning features to construct graph neural network nodes. The graph neural network can capture the rich semantic relationships between multiple query texts, mine the global structural information and node neighborhood information of the graph, and provide more representative text features and multimodal alignment supervision signals under weak supervision settings. Therefore, the present invention can achieve good results with a small amount of labeled data.

[0056] 2. The present invention combines the mutually exclusive disambiguation loss function and the multi-instance ranking loss function to obtain the total loss of the video retrieval network and train the video retrieval network; the multi-instance ranking loss function is based on a predefined margin. The similarity distance between aligned query texts and spatiotemporal pipeline pairs is enlarged; the mutually exclusive disambiguation loss function is designed to penalize the model when multiple query texts are based on the same spatiotemporal pipeline proposal. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a network architecture diagram of a video retrieval method based on a deep neural network model and multi-instance learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described below in conjunction with specific embodiments. The purpose is only to better understand the content of the present invention. Therefore, the examples given do not limit the scope of protection of the present invention.

[0059] like Figure 1 As shown, this embodiment provides a video retrieval method based on a deep neural network model and multi-instance learning, which is characterized by:

[0060] S1: Given a video and its corresponding query text, the query text is defined as , The number of texts, is a sequence of video frames, is the number of frames in the video. The query text gets the initial features through the BERT-Base pre-training model , is the maximum length of a description sentence, is the embedding dimension; for videos, the video frame sequence is first input into the visual feature encoder, and the center coordinate dimension vector sequence is extracted through the Faster-RCNN network pre-trained on MSCOCO ,in is a 4-dimensional vector; then, the SORT algorithm is used to extract candidate spatiotemporal pipeline sequences , where each is a continuous sequence of bounding boxes; then, using Extract I3D-RGB features and frame-level ROI features, and divide each spatiotemporal pipeline evenly into The average value of each segment is used as the segment feature, and the segment features of the I3D-RGB feature and the segment features of the ROI feature are connected to obtain the connection feature ,in, In this embodiment, the block size of the I3D-RGB feature is 16 and the frequency is 8, and the dimensions of the I3D-RGB feature and the ROI feature are 1024 and 2048 respectively.

[0061] In this embodiment, the VidSTG dataset and the YouCook2-BoundingBox dataset are used for training. In the video, it provides at least one subject and object. Since we only focus on declarative sentences, we remove all interrogative sentences from the VidSTG dataset and delete videos that contain only one declarative sentence. YouCook2-BoundingBox is a word / phrase-based object dataset. Each video contains multiple clips, and each clip includes multiple phrases / words and a whole sentence to describe the clip. We connect the phrases / words with their corresponding description sentences as the object sentence description in each clip, and regard all these object sentence descriptions as video descriptions. We select 15 spatiotemporal pipeline proposals for each video, that is, the corresponding spatiotemporal pipeline proposals of the VidSTG dataset. =15, the YouCook2-BoundingBox dataset corresponds to =30; segment number Set to 12.

[0062] S2: Connect features Update features through the visual Transformer network to obtain frame-level visual features ; Initial features of query text Update features through the text Transformer network to obtain word-level text features ,

[0063] ;

[0064] ;

[0065] in, , ; 、 、 、 、 、 are different learnable transformation matrices; 、 is the input value; in this embodiment, for each sentence, we set the dimensions of Q, K, and V to 256 and the number to 2 in the self-attention layer of the transformer encoder.

[0066] S3: word-level text features of multiple query texts Define edge and node features to obtain a graph for construction training.

[0067] The process of defining edge and node features is to input ;in, , yes The sum of the number of words in the sentences, is the concatenation operator;

[0068] Graph Vertices It is defined as,

[0069] ;

[0070] Graph Construction Edge It is defined as,

[0071] ;

[0072] in, Include vertices, Include Vertices, operators exist Create edges between each vertex in and For each vertex in Represented as an adjacency matrix .

[0073] S4: Input the training graph into the graph attention network to learn the attention weights between different nodes and obtain word-level text features containing rich mutual relationship information .

[0074] Word-level text features The calculation process is as follows. First, the vertex is calculated The importance score of the feature to vertex j ,

[0075] ;

[0076] in, are the parameters of the graph attention network, is the Leaky-ReLU function, is the parameterized weight vector of the feedforward neural network;

[0077] Then calculate the word-level text features ,

[0078] .

[0079] S5: Calculate the residual of word-level text features and character-level text features ,

[0080] ;

[0081] And the average value of the residuals is used as the sentence-level text feature ,

[0082] ;

[0083] frame-level visual features average operation is performed on the segment dimension to obtain pipeline-level visual features .

[0084] S6: Calculate the alignment score of the sentence-level text feature and the pipeline-level visual feature through the multi-modal fusion module ; based on the alignment score construct a positive sample pair and a negative sample pair.

[0085] When calculating the alignment score of the sentence-level text feature and the pipeline-level visual feature , first perform element-wise addition and element-wise multiplication operations on the sentence-level text feature and the pipeline-level visual feature;

[0086] ;

[0087] ;

[0088] Then, the element-wise addition operation result, the element-wise multiplication operation result, the sentence-level text feature and the pipeline-level visual feature are connected to obtain a fusion feature, so as to obtain the alignment score between the first sentence in the video and the first spatiotemporal pipeline ; ;

[0089] .

[0090] In the multi-query video retrieval scene, the ground video in multiple sentences can be abstracted as a multiple instance learning (MIL) problem, and the ranking loss is

[0091] The present application regards the video as a bag containing multiple instances, that is, candidate spatiotemporal pipelines, and finds the matching pipeline of each sentence, and the similarity score between the sentence description of the video and in the present application is defined as the maximum value of the spatiotemporal pipeline alignment score of the first query text and the video , ,

[0092] ;

[0093] wherein the video contains ​​​spatiotemporal pipeline; multi-instance ranking loss function: In weakly supervised learning, in order to make full use of the weak supervision information between video and text pairs for training, this paper focuses on defining positive sample pairs and negative sample pairs, and designs a multi-instance ranking loss function. In the training batch, we define is a positive sample pair, where Indicates that a video corresponds to The first query text; at the same time, and is a negative sample pair, ≠ 、 .

[0094] S7: Input the positive sample pairs and negative sample pairs into the pre-trained video retrieval network for training. During the training process, the mutually exclusive disambiguation loss function and the multi-instance ranking loss function are combined to obtain the total loss of the video retrieval network. The loss function gradient descent method is then used to pass it back to the video retrieval network to update the structural parameters until the total loss of the video retrieval network is less than the set loss convergence extreme value, completing the training.

[0095] In step S7, in the multi-query video retrieval scenario, the multi-instance ranking loss function is defined as ,

[0096] ;

[0097] in, Is a boundary parameter, the goal of sorting is to expand the similarity score between the matching query text and spatiotemporal pipe pairs and the unmatched query text and spatiotemporal pipe pairs;

[0098] Define the mutually exclusive disambiguation loss function. First, the alignment score Converted to probability through Softmax function,

[0099] ;

[0100] Definition of the present invention It is The probability distribution of query texts is then used to force the distribution of query texts to be different.

[0101] ;

[0102] ;

[0103] The total loss calculation formula of the video retrieval network is defined as:

[0104] ;

[0105] in, It is a trade-off hyperparameter and defines the maximum value of KL divergence. ; If the KL divergence is greater than , it will be clipped to , to prevent gradient explosion during optimization.

[0106] When training the video retrieval network model, we use Adam as the optimizer, with a learning rate of 2e-5 and a batch size of 16; the trade-off The value is set to 0.02 and the crop value is 15. At the same time, the PyTorch-1.8.1 framework and NVIDIA-RTX-2080Ti-GPU are used to implement it.

[0107] S8: Input the query text to be retrieved and the corresponding video into the trained video retrieval network, so as to locate the segment corresponding to the query text in the video.

[0108] The main invention points of the present invention are as follows:

[0109] 1. The present invention constructs a scenario in which multiple query positioning tasks are performed simultaneously on a single video, and proposes a model architecture based on the attention mechanism and graph neural network in this scenario; the architecture includes a graph neural network semantic feature learning part between multiple query texts and a visual text bimodal feature alignment learning based on the attention mechanism. This structure is not involved in previous inventions; among them, deep learning features can obtain the discriminative features in social comment texts to construct graph neural network nodes; graph neural networks can capture rich semantic relationships between multiple query texts, mine the global structural information and node neighborhood information of the graph, and provide more representative text features and multimodal alignment supervision signals under weak supervision settings. Therefore, the present invention can achieve good results with a small amount of labeled data.

[0110] 2、The application finds new supervision signals and multi-modal feature learning methods (including: multi-query text input calculation at the same time, constructing edge and node features of a graph neural network according to multiple query texts, and visual mutual exclusion relationship between multiple query text results corresponding to the same video) in the multi-query positioning setting based on multi-instance learning in the weakly supervised learning scene, introduces a graph neural network to extract fine-grained query semantic information, and further aligns and fuses the updated text features and visual features, finally learns more optimal text and visual features for matching and sorting; the multi-query network structure and training method integrating the graph neural network and the attention mechanism are important invention points; the application supports multiple retrieval texts for simultaneous query during reasoning, improves the query result accuracy by modeling the fine-grained relationship between multiple queries, improves the time efficiency of multiple single query operations on the video, and thus comprehensively realizes the accuracy and reasoning speed improvement brought by the graph neural network and GPU parallel computing, and improves the retrieval performance of the multi-query video positioning task.

[0111] 3、The application combines a mutual exclusion disambiguation loss function and a multi-instance ranking loss function to obtain the total loss of the video retrieval network and train the video retrieval network; the multi-instance ranking loss function expands the similarity distance between the aligned query text and the spatiotemporal pipeline pair by a predefined margin The mutual exclusion disambiguation loss function aims to punish the model when multiple query texts are based on the same spatiotemporal pipeline proposal.

[0112] The above description is only a preferred embodiment of the application, and does not limit the scope of the application, and any equivalent changes made according to the content of the specification and drawings of the application are included in the scope of the application.

Claims

1. A video retrieval method based on a deep neural network model and multi-instance learning, characterized in that: Here are the steps: S1: Given a video and its corresponding query text, the query text is defined as , The number of texts, is a sequence of video frames, is the number of frames in the video; The query text obtains initial features through the BERT-Base pre-training model ; Extract I3D-RGB features and ROI features from the video, connect the I3D-RGB features and ROI features to obtain the connection features ; S2: Connect features Update features through the visual Transformer network to obtain frame-level visual features ; Initial features of query text Update features through the text Transformer network to obtain word-level text features , ； ； in, , is the maximum length of a description sentence, is the embedding dimension, is the dimension of the concatenated features; 、 、 、 、 、 are different learnable transformation matrices; 、 is the input value; S3: word-level text features of multiple query texts Define edge and node features to obtain a graph for construction training; S4: Input the training graph into the graph attention network to learn the attention weights between different nodes and obtain word-level text features containing rich mutual relationship information ; S5: Calculate the residual of word-level text features and character-level text features , ; And the average value of the residuals is used as the sentence-level text feature , ; Frame-level visual features Perform average operation on the segment dimension to obtain pipeline-level visual features ; S6: Calculate the alignment scores of sentence-level text features and pipeline-level visual features through the multimodal fusion module ; Based on alignment score Construct positive sample pairs and negative sample pairs; S7: Input the positive sample pairs and negative sample pairs into the pre-trained video retrieval network for training. During the training process, the mutual exclusion disambiguation loss function and the multi-instance ranking loss function are combined to obtain the total loss of the video retrieval network. The loss function gradient descent method is then used to pass the loss back to the video retrieval network to update the structural parameters until the total loss of the video retrieval network is less than the set loss convergence limit, completing the training. S8: Input the query text to be retrieved and the corresponding video into the trained video retrieval network, so as to locate the segment corresponding to the query text in the video.

2. The video retrieval method based on a deep neural network model and multi-instance learning according to claim 1, characterized in that: In step S1, the video frame sequence is first input into the visual feature encoder, and the Faster-RCNN network in the visual feature encoder extracts the center coordinate dimension vector sequence ,in is a 4-dimensional vector, T represents the number of frames of the video; then, the SORT algorithm is used to extract candidate spatiotemporal pipeline sequences , where each is a continuous sequence of bounding boxes; then, using Extract I3D-RGB features and ROI features, and divide each spatiotemporal pipeline evenly into The average value of each segment is used as the segment feature, and the segment features of the I3D-RGB feature and the segment features of the ROI feature are connected to obtain the connection feature .

3. The video retrieval method based on a deep neural network model and multi-instance learning according to claim 1, characterized in that: The process of defining edge and node features in step S3 is as follows: ;in, , yes The sum of the number of words in the sentences, is the concatenation operator; Graph Vertices It is defined as, ； Graph Construction Edge It is defined as, ； in, Include vertices, Include Vertices, operators exist Create edges between each vertex in and Each vertex in Represented as an adjacency matrix .

4. The video retrieval method based on a deep neural network model and multi-instance learning according to claim 3 is characterized in that: Step S4: word-level text features The calculation process is as follows. First, the vertex is calculated The importance score of the feature to vertex j , ； in, are the parameters of the graph attention network, is the Leaky-ReLU function, is the parameterized weight vector of the feedforward neural network; Then calculate the word-level text features .

5. The video retrieval method based on a deep neural network model and multi-instance learning according to claim 1, characterized in that: In step S6, the alignment scores of sentence-level text features and pipeline-level visual features are calculated. When , the sentence-level text features and pipeline-level visual features are firstly added and multiplied element by element; then, the results of the element-by-element addition and multiplication, the sentence-level text features, and the pipeline-level visual features are connected to obtain the fusion features to obtain the first Sentences and alignment scores between spatiotemporal pipelines ; ； ； 。 6. The video retrieval method based on a deep neural network model and multi-instance learning according to claim 1, characterized in that: In step S6, the video and The similarity score between the sentence descriptions Defined as Query text and video Spatiotemporal pipeline alignment score The maximum value of ； Among them, video Include spatiotemporal pipelines; in the training batch, define is a positive sample pair.

7. The video retrieval method based on a deep neural network model and multi-instance learning according to claim 6, characterized in that: In step S7, the multi-instance ranking loss function is defined as , ； in, Is a boundary parameter, the goal of sorting is to expand the similarity score between the matching query text and spatiotemporal pipe pairs and the unmatched query text and spatiotemporal pipe pairs; and is a negative sample pair; Define the mutually exclusive disambiguation loss function. First, the alignment score Converted to probability through Softmax function, ； definition It is The probability distribution of query texts is then used to force the distribution of query texts to be different. ； ； The total loss calculation formula of the video retrieval network is defined as: ； in, It is a trade-off hyperparameter and defines the maximum value of KL divergence. ; If the KL divergence is greater than , it will be clipped to , to prevent gradient explosion during optimization.

Images