Video retrieval method based on deep neural network model and multi-instance learning

Abstract

The invention relates to the field of computer vision processing, in particular to a video retrieval method based on a deep neural network model and multi-instance learning, which comprises the following steps: pre-training a query text to obtain an initial feature, and extracting an I 3D-RGB feature, an ROI feature and a connection feature from a video; updating to obtain frame-level visual features and word-level text features; constructing a training graph, and performing graph attention network learning to obtain word-level text features; calculating feature residual errors of the word-level text and the word-level text, wherein the mean value of the residual errors is sentence-level text features; performing segment dimension average operation on the frame-level visual features to obtain pipeline-level visual features; calculating alignment scores of the sentence-level text features and the pipeline-level visual features, constructing a positive sample pair and a negative sample pair, and training a video retrieval network; according to the method, discriminability features in a multi-query text are obtained through deep learning features to construct the graph neural network, so that text features with more representation meanings and multi-modal alignment supervision signals are provided under weak supervision setting.

Description

Technical Field

[0001] The present invention relates to the field of computer vision processing, and in particular to a video retrieval method based on a deep neural network model and multi-instance learning. Background Art

[0002] With the popularization of the Internet and the rapid development of artificial intelligence, information is presented in more diverse forms, and is mainly concentrated in the most intuitive and rich images, voices, videos, and texts. Therefore, the interdisciplinary research of computer vision and natural language processing, as well as the processing of "video", the most widely disseminated and most popular form of information dissemination, has become a new development direction.

[0003] Given a query, i.e. a natural language expression, visual localization aims to locate the corresponding region or fragment through a comprehensive understanding of the visual content; first, the target is queried in the image based on category, attribute and context. In real-world scenarios, visual localization in space and time is a more practical, reasonable and challenging task.

[0004] In the research task of aligning the visual area with the query text in the video, in order to make the alignment effect more realistic, a large amount of labels and area annotation information is often required in the training process. The annotation of this information greatly increases the consumption of manpower, financial and material resources; therefore, in terms of technical implementation and practical application, more scenarios tend to solve this problem through weak supervision; and under the setting of weak supervision, aligning the visual area with the text with less known label information has become the key to the video query positioning task.

[0005] Existing research works based on weakly supervised learning only use single video-query-text pair information for training when processing the alignment of visual and textual information, while ignoring the scenes where multiple queryable objects appear simultaneously in a video; obviously, this scene information can provide semantic relationships as well as visual overlap and alignment relationships between multiple queries, and can provide important supervisory signals for the training of video retrieval models under weak supervision settings.

[0006] In summary, the main challenges of existing video localization methods are summarized as follows: 1. Semantic understanding: In-depth semantic understanding of video content is required in order to accurately identify and match natural language descriptions; 2. Modal difference and information density: There is a modal difference between the visual information and language description of the video, and these two types of information need to be effectively integrated; 3. Utilization of contextual information: Activities in videos usually do not appear independently, they often have internal semantic associations and temporal interdependence, which puts forward requirements for the modeling of video context information.

[0007] Therefore, this application proposes a video retrieval method based on a deep neural network model and multi-instance learning. Summary of the invention

[0008] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a video retrieval method based on a deep neural network model and multi-instance learning, which can provide more representative text features and multimodal alignment supervision signals under a weak supervision setting and can achieve good results with a small amount of labeled data.

[0009] The technical solution adopted by the present invention to solve the technical problem is:

[0010] A video retrieval method based on a deep neural network model and multi-instance learning, characterized by:

[0011] S1: Given a video and its corresponding query text, the query text is defined as M The number of texts, is a video frame sequence, T is the number of video frames; the query text obtains the initial feature F through the BERT-Base pre-training model s ; Extract I3D-RGB features and ROI features from the video, connect the I3D-RGB features and ROI features to obtain the connection feature F p ;

[0012] S2: Connect feature F p The feature is updated through the visual Transformer network to obtain the frame-level visual feature O p ; Initial feature F of query text s The feature is updated through the text Transformer network to obtain the word-level text feature O s ,

[0013]

[0014] in, N s is the maximum length of a descriptive sentence, d s is the embedding dimension, is the dimension of the concatenated features; are different learnable transformation matrices; K and V are input values;

[0015] S3: By analyzing the word-level text features of multiple query texts s Define edge and node features to obtain a graph for construction training;

[0016] S4: Input the training graph into the graph attention network to learn the attention weights between different nodes and obtain word-level text features containing rich mutual relationship information

[0017] S5: Calculate the residual of word-level text features and character-level text features

[0018] And the average of the residuals is used as the sentence-level text feature

[0019] Frame-level visual features O p Perform average operation on the segment dimension to obtain pipeline-level visual features

[0020] S6: Calculate the alignment score e of sentence-level text features and pipeline-level visual features through the multimodal fusion module i,j ; Based on the alignment score e i,j Construct positive sample pairs and negative sample pairs;

[0021] S7: Input the positive sample pairs and the negative sample pairs into the pre-trained video retrieval network for training. During the training process, the mutually exclusive disambiguation loss function and the multi-instance ranking loss function are combined to obtain the total loss of the video retrieval network, and the loss function gradient descent method is used to return it to the video retrieval network to update the structural parameters until the total loss of the video retrieval network is less than the set loss convergence extreme value, and the training is completed;

[0022] S8: Input the query text to be retrieved and the corresponding video into the trained video retrieval network, so as to locate the segment corresponding to the query text in the video.

[0023] Preferably, a further technical solution of the present invention is:

[0024] Preferably, in step S1, the video frame sequence is first input into a visual feature encoder, and the Faster-RCNN network in the visual feature encoder extracts a center coordinate dimension vector sequence where b t is a 4-dimensional vector; then, the SORT algorithm is used to extract K candidate spatiotemporal pipeline sequences Among them, each p k is a continuous sequence of bounding boxes; then, use Extract I3D-RGB features and ROI features, and divide each spatiotemporal pipeline evenly into N p The average value of each segment is used as the segment feature, and the segment features of the I3D-RGB feature and the segment features of the ROI feature are connected to obtain the connection feature

[0025] Preferably, the process of defining edge and node features in step S3 is as follows: the input is in, is the sum of the number of words in M ​​sentences, || is the concatenation operator;

[0026] Graph Vertices Defined as,

[0027]

[0028] Graph Construction Edge Defined as,

[0029]

[0030] Among them, O i Include Vertex, O j Include Vertices, operators exist Create edges between each vertex in and For each vertex in Represented as an adjacency matrix

[0031] Preferably, step S4, word-level text features The calculation process is as follows: First, the importance score of the feature of vertex (i, j) to vertex j is calculated

[0032]

[0033] in, is the parameter of the graph attention network, σ is the Leaky-ReLU function, is the parameterized weight vector of the feedforward neural network;

[0034] Then calculate the word-level text features

[0035] Preferably, in step S6, the alignment score e of the sentence-level text features and the pipeline-level visual features is calculated. i,j When , the sentence-level text features and the pipeline-level visual features are firstly added and multiplied element by element; then, the results of the element-by-element addition and multiplication, the sentence-level text features, and the pipeline-level visual features are connected to obtain the fusion features to obtain the alignment score e between the i-th sentence and the j-th spatiotemporal pipeline in the video. i,j ;

[0036]

[0037] Preferably, in step S6, we use the video and sentence descriptions i The similarity score between the i-th query text and the video is defined as The spatiotemporal pipeline alignment score i，j The maximum value of

[0038]

[0039] Among them, video Contains K spatiotemporal pipelines; in the training batch, define is a positive sample pair, where s i Indicates the corresponding The i-th query text among the query texts; at the same time, and is a negative sample pair.

[0040] Preferably, in step S7, the multi-instance ranking loss function is defined as

[0041]

[0042] Among them, Δ is a boundary parameter, and the ranking goal is to expand the similarity score between the matching query text and spatiotemporal pipe pairs and the unmatched query text and spatiotemporal pipe pairs;

[0043] Define the mutually exclusive disambiguation loss function. First, the alignment score e i,j Converted to probability through Softmax function,

[0044]

[0045] The present invention defines e′ i,: is the probability distribution of the i-th query text; then the Kullback-Leibler divergence is used to force the distribution of the query text to be different;

[0046] L mutex =-∑ i ∑ j≠i KL(e′ i,: ||e′ j,: );

[0047]

[0048] The total loss calculation formula of the video retrieval network is defined as:

[0049]

[0050] Among them, λ is a trade-off hyperparameter and defines a maximum value of KL divergence If the KL divergence is greater than It will be clipped to This is to prevent gradient explosion during optimization.

[0051] The present invention constructs a scenario where multiple query text localization tasks are performed on a single video at the same time, and proposes a model architecture based on the attention mechanism and graph neural network in this scenario. Compared with the existing technology, its outstanding features are:

[0052] 1. The present invention can obtain the discriminative features in multiple query texts through deep learning features to construct graph neural network nodes; the graph neural network can capture the rich semantic relationships between multiple query texts, mine the global structural information of the graph, node neighborhood information, etc., and provide more representative text features and multimodal alignment supervision signals under weak supervision settings. Therefore, the present invention can achieve good results with a small amount of labeled data;

[0053] 2. The present invention combines the mutually exclusive disambiguation loss function and the multi-instance ranking loss function to obtain the total loss of the video retrieval network and train the video retrieval network; the multi-instance ranking loss function expands the similarity distance between the aligned query text and the spatiotemporal pipeline pair by a predefined margin Δ; the mutually exclusive disambiguation loss function is intended to penalize the model when multiple query texts are based on the same spatiotemporal pipeline proposal. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a network architecture diagram of a video retrieval method based on a deep neural network model and multi-instance learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention is further described below in conjunction with specific embodiments, the purpose of which is only to provide a better understanding of the content of the present invention. Therefore, the examples given do not limit the protection scope of the present invention.

[0056] like Figure 1 As shown, this embodiment provides a video retrieval method based on a deep neural network model and multi-instance learning, which is characterized by:

[0057] S1: Given a video and its corresponding query text, the query text is defined as M The number of texts, is the video frame sequence, T is the number of video frames. The query text obtains initial features through the BERT-Base pre-training model N s is the maximum length of a descriptive sentence, d s is the embedding dimension; for videos, the video frame sequence is first input into the visual feature encoder, and the center coordinate dimension vector sequence is extracted through the Faster-RCNN network pre-trained on MSCOCO where b t is a 4-dimensional vector; then, the SORT algorithm is used to extract K candidate spatiotemporal pipeline sequences Among them, each p k is a continuous sequence of bounding boxes; then, use Extract I3D-RGB features and frame-level ROI features, and evenly divide each spatiotemporal pipeline into N p The average value of each segment is used as the segment feature, and the segment features of the I3D-RGB feature and the segment features of the ROI feature are connected to obtain the connection feature Among them, N p is the dimension of the concatenated feature. In this embodiment, the block size of the I3D-RGB feature is 16 and the frequency is 8, and the dimensions of the I3D-RGB feature and the ROI feature are 1024 and 2048 respectively.

[0058] In this embodiment, the VidSTG dataset and YouCook2-BoundingBox dataset are used for training. In the video, it provides no less than one subject and object. Since we only focus on declarative sentences, we remove all interrogative sentences from the VidSTG dataset and remove videos that only contain one declarative sentence. YouCook2-BoundingBox is a word / phrase-based object dataset. Each video contains multiple clips, and each clip includes multiple phrases / words and a whole sentence to describe the clip; we connect the phrases / words with their corresponding description sentences as the object sentence description in each clip, and regard all these object sentence descriptions as video descriptions. We select 15 spatiotemporal pipeline proposals for each video, that is, K=15 for the VidSTG dataset and K=30 for the YouCook2-BoundingBox dataset; segment number N p Set to 12.

[0059] S2: Connect feature F p The feature is updated through the visual Transformer network to obtain the frame-level visual feature O p ; Initial feature F of query text s The feature is updated through the text Transformer network to obtain the word-level text feature O s ,

[0060]

[0061] in, are different learnable transformation matrices; K and V are input values; in this embodiment, for each sentence, we set the dimensions of Q, K, and V to 256 and the number one to 2 in the self-attention layer of the transformer encoder.

[0062] S3: By analyzing the word-level text features of multiple query texts s Define edge and node features to obtain a graph for construction training.

[0063] The process of defining edge and node features is to input in, is the sum of the number of words in M ​​sentences, || is the concatenation operator;

[0064] Graph Vertices Defined as,

[0065]

[0066] Graph Construction Edge Defined as,

[0067]

[0068] Among them, O i Include Vertex, O j Include Vertices, operators exist Create edges between each vertex in and For each vertex in Represented as an adjacency matrix

[0069] S4: Input the training graph into the graph attention network to learn the attention weights between different nodes and obtain word-level text features containing rich mutual relationship information

[0070] Word-level text features The calculation process is as follows: First, the importance score of the feature of vertex (i, j) to vertex j is calculated

[0071]

[0072] in, is the parameter of the graph attention network, σ is the Leaky-ReLU function, is the parameterized weight vector of the feedforward neural network;

[0073] Then calculate the word-level text features

[0074]

[0075] S5: Calculate the residual of word-level text features and character-level text features

[0076]

[0077] And the average of the residuals is used as the sentence-level text feature

[0078]

[0079] Frame-level visual features O p Perform average operation on the segment dimension to obtain pipeline-level visual features

[0080] S6: Calculate the alignment score e of sentence-level text features and pipeline-level visual features through the multimodal fusion module i,j ; Based on the alignment score e i,j Construct positive and negative sample pairs.

[0081] Calculate the alignment score e of sentence-level text features and pipeline-level visual features i,j When , the sentence-level text features and pipeline-level visual features are firstly added and multiplied element by element;

[0082]

[0083] Then, the element-by-element addition operation results, element-by-element multiplication operation results, sentence-level text features, and pipeline-level visual features are concatenated to obtain the fused features to obtain the alignment score e between the i-th sentence and the j-th spatiotemporal pipeline in the video. i,j ;

[0084]

[0085] In the multi-query video retrieval scenario, the grounded videos in multiple sentences can be abstracted as a multiple instance learning (MIL) problem guided by a ranking loss.

[0086] The present invention regards the video as a package, which contains multiple instances, namely candidate spatiotemporal pipelines, and finds the matching pipeline for each sentence. and sentence descriptions i The similarity score between the i-th query text and the video is defined as The spatiotemporal pipeline alignment score i,j The maximum value of

[0087]

[0088] Among them, video Contains K spatiotemporal pipelines; Multi-instance ranking loss function: In weakly supervised learning, in order to make full use of the weakly supervised information between video and text pairs for training, the present invention focuses on defining positive sample pairs and negative sample pairs, and designs a multi-instance ranking loss function. In the training batch, define is a positive sample pair, where s i Indicates the corresponding The i-th query text among the query texts; at the same time, and is a negative sample pair, s′≠s i ,

[0089] S7: Input the positive sample pairs and negative sample pairs into the pre-trained video retrieval network for training. During the training process, the mutually exclusive disambiguation loss function and the multi-instance ranking loss function are combined to obtain the total loss of the video retrieval network. The loss function gradient descent method is used to pass it back to the video retrieval network to update the structural parameters until the total loss of the video retrieval network is less than the set loss convergence extreme value, and the training is completed.

[0090] In step S7, in the multi-query video retrieval scenario, the multi-instance ranking loss function is defined as

[0091]

[0092] Among them, Δ is a boundary parameter, and the ranking goal is to expand the similarity score between the matching query text and spatiotemporal pipe pairs and the unmatched query text and spatiotemporal pipe pairs;

[0093] Define the mutually exclusive disambiguation loss function. First, the alignment score e i,j Converted to probability through Softmax function,

[0094]

[0095] The present invention defines e′ i,: is the probability distribution of the i-th query text; then the Kullback-Leibler divergence is used to force the distribution of the query text to be different;

[0096] L mutex =-∑ i ∑ j≠i KL(e′ i,: ||e′ j,: );

[0097]

[0098] The total loss calculation formula of the video retrieval network is defined as:

[0099]

[0100] Among them, λ is a trade-off hyperparameter and defines a maximum value of KL divergence If the KL divergence is greater than It will be clipped to This is to prevent gradient explosion during optimization.

[0101] When training the video retrieval network model, we use Adam as the optimizer, with a learning rate of 2e-5 and a batch size of 16; the trade-off The value is set to 0.02 and the crop value is 15. At the same time, PyTorch-1.8.1 framework and NVIDIA-RTX-2080Ti-GPU are used to implement it.

[0102] S8: Input the query text to be retrieved and the corresponding video into the trained video retrieval network, so as to locate the segment corresponding to the query text in the video.

[0103] The main invention points of the present invention are as follows:

[0104] 1. The present invention constructs a scenario in which multiple query positioning tasks are performed on a single video at the same time, and proposes a model architecture based on an attention mechanism and a graph neural network in this scenario; the architecture includes a graph neural network semantic feature learning part between multiple query texts and a visual text bimodal feature alignment learning based on an attention mechanism, which is a structure not involved in previous inventions; wherein, deep learning features can obtain discriminative features in social comment texts to construct graph neural network nodes; graph neural networks can capture rich semantic relationships between multiple query texts, mine the global structural information and node neighborhood information of the graph, and provide more representative text features and multimodal alignment supervision signals under weak supervision settings. Therefore, the present invention can achieve good results with a small amount of labeled data.

[0105] 2. In the scenario of weakly supervised learning, the present invention has found a new supervision signal and multimodal feature learning method under the multi-query positioning setting based on multi-example learning (including: simultaneous input and calculation of multiple query texts, construction of edge and node features of graph neural network according to multiple query texts, and visual mutually exclusive relationship between multiple query text results corresponding to the same video). The present invention introduces graph neural network to extract fine-grained query semantic information, and further aligns, fuses and updates the updated text features with the visual features, and finally learns better text and visual features for matching and sorting; the multi-query network structure and training method integrating graph neural network and attention mechanism are important invention points; the present invention supports simultaneous query of multiple retrieval texts during reasoning, and improves the accuracy of query results by modeling the fine-grained relationship between multiple queries, while improving the time efficiency of multiple single query operations on the video, thereby comprehensively realizing the accuracy and reasoning speed improvement brought by graph neural network and GPU parallel computing, and improving the retrieval performance of multi-query video positioning tasks.

[0106] 3. The present invention combines the mutually exclusive disambiguation loss function and the multi-instance ranking loss function to obtain the total loss of the video retrieval network and train the video retrieval network; the multi-instance ranking loss function expands the similarity distance between the aligned query text and the spatiotemporal pipeline pair by a predefined margin Δ; the mutually exclusive disambiguation loss function is intended to penalize the model when multiple query texts are based on the same spatiotemporal pipeline proposal.

[0107] The above description is only the preferred embodiment of the present invention, and does not limit the scope of rights of the present invention. All equivalent changes made by using the contents of the present invention specification and its drawings are included in the scope of rights of the present invention.

Claims

1. A video retrieval method based on a deep neural network model and multi-instance learning, characterized in that: Here are the steps: S1: Given a video and its corresponding query text, the query text is defined as M The number of texts, is the video frame sequence, T is the number of video frames; The query text obtains the initial feature F through the BERT-Base pre-training model s ; Extract I3D-RGB features and ROI features from the video, connect the I3D-RGB features and ROI features to get the connection feature F p ; S2: Connect feature F p The feature is updated through the visual Transformer network to obtain the frame-level visual feature O p ; Initial feature F of query text s The feature is updated through the text Transformer network to obtain the word-level text feature O s , in, N s is the maximum length of a descriptive sentence, d s is the embedding dimension, is the dimension of the concatenated features; are different learnable transformation matrices; K and V are input values; S3: By analyzing the word-level text features of multiple query texts s Define edge and node features to obtain a graph for construction training; S4: Input the training graph into the graph attention network to learn the attention weights between different nodes and obtain word-level text features containing rich mutual relationship information S5: Calculate the residual of word-level text features and character-level text features And the average of the residuals is used as the sentence-level text feature Frame-level visual features O p Perform average operation on the segment dimension to obtain pipeline-level visual features S6: Calculate the alignment score e of sentence-level text features and pipeline-level visual features through the multimodal fusion module i,j ; Based on the alignment score e i,j Construct positive sample pairs and negative sample pairs; S7: Input the positive sample pairs and the negative sample pairs into the pre-trained video retrieval network for training. During the training process, the mutually exclusive disambiguation loss function and the multi-instance ranking loss function are combined to obtain the total loss of the video retrieval network, and the loss function gradient descent method is used to return it to the video retrieval network to update the structural parameters until the total loss of the video retrieval network is less than the set loss convergence extreme value, and the training is completed; S8: Input the query text to be retrieved and the corresponding video into the trained video retrieval network, so as to locate the segment corresponding to the query text in the video.

2. The video retrieval method based on deep neural network model and multi-instance learning according to claim 1, characterized in that: In step S1, the video frame sequence is first input into the visual feature encoder, and the Faster-RCNN network in the visual feature encoder extracts the center coordinate dimension vector sequence where b t is a 4-dimensional vector; then, the SORT algorithm is used to extract K candidate spatiotemporal pipeline sequences Among them, each p k is a continuous sequence of bounding boxes; then, use Extract I3D-RGB features and ROI features, and divide each spatiotemporal pipeline evenly into N p The average value of each segment is used as the segment feature, and the segment features of the I3D-RGB feature and the segment features of the ROI feature are connected to obtain the connection feature 3. The video retrieval method based on deep neural network model and multi-instance learning according to claim 1, characterized in that: The process of defining edge and node features in step S3 is as follows: in, is the sum of the number of words in M ​​sentences, || is the concatenation operator; Graph Vertices Defined as, Graph Construction Edge Defined as, Among them, O i Include Vertex, O j Include Vertices, operators exist Create edges between each vertex in and For each vertex in Represented as an adjacency matrix 4. The video retrieval method based on deep neural network model and multi-instance learning according to claim 3 is characterized in that: Step S4: word-level text features The calculation process is as follows: First, the importance score of the feature of vertex (i, j) to vertex j is calculated in, is the parameter of the graph attention network, σ is the Leaky-ReLU function, is the parameterized weight vector of the feedforward neural network; Then calculate the word-level text features 5. The video retrieval method based on deep neural network model and multi-instance learning according to claim 1, characterized in that: In step S6, the alignment score e of the sentence-level text features and pipeline-level visual features is calculated i,j When , the sentence-level text features and the pipeline-level visual features are firstly added and multiplied element by element; then, the results of the element-by-element addition and multiplication, the sentence-level text features, and the pipeline-level visual features are connected to obtain the fusion features to obtain the alignment score e between the i-th sentence and the j-th spatiotemporal pipeline in the video. i,j ; 6. The video retrieval method based on deep neural network model and multi-instance learning according to claim 1, characterized in that: In step S6, we will video and sentence descriptions i The similarity score between the i-th query text and the video is defined as The spatiotemporal pipeline alignment score i,j The maximum value of Among them, video Contains K spatiotemporal pipelines; in the training batch, define is a positive sample pair, where s i Indicates the corresponding The i-th query text among the query texts; at the same time, and is a negative sample pair.

7. The video retrieval method based on deep neural network model and multi-instance learning according to claim 6 is characterized in that: In step S7, the multi-instance ranking loss function is defined as Among them, Δ is a boundary parameter, and the ranking goal is to expand the similarity score between the matching query text and spatiotemporal pipe pairs and the unmatched query text and spatiotemporal pipe pairs; Define the mutually exclusive disambiguation loss function. First, the alignment score e i,j Converted to probability through Softmax function, The present invention defines e' i,: is the probability distribution of the i-th query text; then the Kullback-Leibler divergence is used to force the distribution of query texts to be different; L mutex =-∑ i ∑ j≠i KL(e′ i,: ||e′ j,: ); The total loss calculation formula of the video retrieval network is defined as: Among them, λ is a trade-off hyperparameter and defines a maximum value of KL divergence If the KL divergence is greater than It will be clipped to This is to prevent gradient explosion during optimization.

Images