Video time sequence positioning method and device based on relation perception

Abstract

The invention discloses a video time sequence positioning method and device based on relation perception, and the method comprises the steps: firstly carrying out the feature extraction of a video and a text through a pre-trained video encoder and a pre-trained text encoder, and mapping the features to a public feature space; then, a context-rich feature sequence is generated through information interaction in the modality; and then, carrying out inter-modal information interaction through a word-level feature fusion module. On the basis, a foreground frame detector and a relation signal generator are used for respectively calculating a fine-granularity correlation score and a coarse-granularity correlation score between the video and the text. Then, the multi-granularity discriminator synthesizes the two types of correlations to generate a multi-granularity correlation score for providing correlation feedback; and finally, related feedback is introduced in a video time sequence positioning process through a fragment prediction head of relation perception, so that the model can cope with the situation that no related fragment exists in the video possibly.

Description

Technical Field

[0001] The present invention relates to the field of video timing positioning, and in particular to a video timing positioning method and device based on relationship perception. Background Art

[0002] With the advent of the big data era, digital social media platforms, online shopping platforms, and short video sharing platforms have developed rapidly. Everyone can upload, share, and display their works on these digital platforms through the mobile Internet. As the most important communication medium, the number of videos is growing exponentially. Faced with such a large amount of video content, how to accurately locate the video clips that users need or are interested in through video timing positioning technology is of great research significance.

[0003] Although video temporal localization technology has made significant progress in recent years, it still has great limitations. At present, the ideal video temporal localization task must meet an assumption that there must be short video clips matching the text description in a given long video. However, in real application scenarios, the above assumption is not always true, because there may not be clips related to the query text in a given long video. Therefore, it is crucial to study the extended task of the video temporal localization task, the video temporal localization task with relevant feedback. This extended task can bridge the gap between traditional video temporal localization tasks and real-world applications, and help ensure that the technology of video temporal localization can keep pace with actual application needs, so that the positioning system can actively cope with the complexity and uncertainty of the real world, further improve user experience and broaden the scope of application of the technology. At the same time, solving this challenge will promote the development of video temporal localization technology to a higher level of adaptability and intelligence, enabling it to play a key role in a wider range of scenarios.

[0004] The current video temporal localization methods are not applicable to the video temporal localization task of relevant feedback. When there is no segment related to the query text in the video, the traditional video temporal localization method will still predict the start and end time of the segment and output the localized segment. The essential reason is that these traditional methods cannot determine whether the text and video are relevant and cannot give corresponding relevant feedback. However, the most convenient way to make the traditional video temporal localization method applicable to the video temporal localization task of relevant feedback involves two stages. First, a situation discrimination network model needs to be trained separately to determine whether the text and video are relevant and output relevant feedback. Subsequently, the samples judged to be relevant are input into the above-mentioned traditional video temporal localization model for video temporal localization. However, although this integrated method has a clear structure, it needs to train two models separately, resulting in redundant use of computing resources, increasing storage requirements, and is not ideal in terms of overall reasoning efficiency. Therefore, it is urgent to design a multi-task method that can both feedback the relevance between text and video and accurately locate the relevant samples in time. Summary of the invention

[0005] The present invention aims to address the deficiencies of the prior art and to propose a video timing positioning method and device based on relationship perception.

[0006] The objective of the present invention is achieved through the following technical solution: a video timing positioning method based on relationship perception, the method comprising the following steps:

[0007] S1, obtaining training data of video and text representation in video temporal positioning;

[0008] S2. Construct a video temporal positioning model based on relationship perception. The processing flow of the model includes:

[0009] S2.1, input the video to be located and the text expression into the pre-trained video feature encoder and text feature encoder respectively, to obtain a frame-level video feature sequence and a word-level text feature sequence;

[0010] S2.2, and use a fully connected layer to map the two obtained feature sequences to the same dimension, and then perform information interaction between the feature vectors within the modality to obtain a frame-level video feature sequence and a word-level text feature sequence rich in contextual information;

[0011] S2.3. Use the self-attention mechanism to aggregate the word-level feature sequences rich in contextual information to obtain the sentence-level feature vector of the query text.

[0012] S2.4. The frame-level video feature sequence and word-level text feature sequence rich in contextual information are fused through a word-level feature fusion module to obtain a video feature sequence fused with word-level text information.

[0013] S2.5, using a feature concatenation and fusion method, the sentence-level text feature vector is fused with the video feature sequence fused with word-level text information to obtain a video feature sequence fused with text information;

[0014] S2.6, using the video feature sequence fused with text information through the foreground frame detector module, obtain the fine-grained correlation discrimination score and foreground frame prediction score sequence between the query text and the input video;

[0015] S2.7, using the sentence-level feature vector and the video feature sequence fused with text information through the relational signal generator module, obtain the coarse-grained correlation discrimination score and relational signal latent vector describing the relational information between the text and the video clip;

[0016] S2.8, inputting the fine-grained correlation discrimination score and the coarse-grained correlation discrimination score into the multi-grained discriminator module to obtain the multi-grained correlation discrimination score;

[0017] S2.9, using the foreground frame prediction score sequence, the foreground content in the video feature sequence integrated with word-level text information is enhanced, and then the relational signal latent vector is concatenated as the input sequence to pass through the relation-aware segment prediction head module to obtain the start boundary probability distribution and the end boundary probability distribution of the predicted segment;

[0018] S3, based on the relation-aware video temporal localization model, the training data is combined with the loss function of the foreground frame detector, the relation signal generator and the relation-aware segment prediction head for multi-task learning;

[0019] S4. Use the trained relationship-aware video temporal localization model to obtain the correlation between the video to be located and the text description, and perform video temporal localization on the video to be located and the text description that have the correlation.

[0020] Furthermore, the calculation process of the word-level feature fusion module in S2.4 includes:

[0021] S2.4.1. Use trilinear function to calculate the common similarity matrix between the frame-level video feature sequence and the word-level text feature sequence rich in context information.

[0022] S2.4.2, normalize the common similarity matrix row by row to obtain the attention weight from video to text;

[0023] S2.4.3. Normalize the common similarity matrix column by column to obtain the attention weight from text to video.

[0024] S2.4.4, weighting the word-level text feature sequence using the attention weight from the video to text direction to obtain a video feature sequence reconstructed from the text information;

[0025] S2.4.5. Use the attention weight from text to video to weight the frame-level video feature sequence to obtain a text feature sequence reconstructed from the video features;

[0026] S2.4.6. Use the reconstructed text feature sequence and the attention weight from the video to the text direction to obtain the text-aware video feature sequence;

[0027] S2.4.7. Through a layer of feedforward network, the frame-level video feature sequence rich in contextual information, the video feature sequence reconstructed by text features, and the text-aware video feature sequence are fused to obtain a video feature sequence that integrates word-level text information.

[0028] Furthermore, the calculation process of the foreground detector module in S2.6 includes:

[0029] S2.6.1. The video feature sequence fused with word-level text information is predicted through a fully connected layer to obtain a sequence of similarity scores between the text and each frame of the video. The sigmoid function is then used to map the similarity scores to between 0 and 1 to obtain a sequence of foreground frame prediction scores.

[0030] S2.6.2. Calculate the maximum value in the foreground frame prediction score sequence to obtain the fine-grained correlation discriminant score between the query text and the input video.

[0031] Furthermore, the calculation process of the relation signal generator in S2.7 includes:

[0032] S2.7.1. Input the video feature sequence into a fully connected layer to obtain a similarity score sequence between the text and each frame of the video. Then, the similarity score sequence is normalized using a softmax function to obtain a relative correlation score between the text and each frame. Finally, the relative correlation score is used to perform a weighted sum operation on the video feature sequence to obtain a segment-level video feature vector.

[0033] S2.7.2. The sentence-level feature vector and the segment-level video feature vector are concatenated and fused to obtain the relational signal latent vector describing the relational information between the text and the video segment, namely:

[0034] S2.7.3. Input the relation signal latent vector into the fully connected layer with the sigmoid function to obtain the coarse-grained correlation discrimination score between the query text and the input video.

[0035] Furthermore, the calculation of the multi-granularity discriminator in S2.8 includes:

[0036] The fine-grained correlation discriminant score fg The coarse-grained correlation discrimination score scor cg Input to the multi-granularity discriminator module to obtain the multi-granularity correlation discrimination score P v ,Right now:

[0037] P v =avg(score fg ,score cg )

[0038] Among them, avg(·) is the operation of calculating the mean.

[0039] Furthermore, the relationship-aware segment prediction head calculation process in S2.9 specifically includes:

[0040] S2.9.1. Using the foreground frame to predict the score sequence S f , for the video feature sequence V q The foreground frame feature vector in the image is strengthened, and the background frame feature vector irrelevant to the text is weakened to obtain the video feature sequence after the foreground frame is enhanced. Right now:

[0041]

[0042] Among them, ⊙ represents element-by-element multiplication;

[0043] S2.9.2. Concatenate the relation signal latent vector g to the foreground frame enhanced video feature sequence Before the first video feature vector of

[0044] S2.9.3. Input feature sequence of the video Input into the stacked two-layer LSTM network, and after passing through the first layer of LSTM network, the time series feature sequence of the predicted start boundary is obtained. Then the time series feature sequence H s Input to the second layer LSTM network to obtain the time series feature sequence of the predicted end boundary Right now:

[0045]

[0046] in, Indicates that in the time series feature sequence H s / e The tth time series feature vector in, Indicated in The tth video feature vector in LSTM * Represents the *th layer LSTM network.

[0047] S2.9.4. Set the start and end timing feature sequence H s , H e Video feature sequences Concatenate and input into the corresponding feedforward network to obtain the boundary probability distribution Right now:

[0048]

[0049] Among them, P s is the starting boundary probability distribution, P e is the end boundary probability distribution, FFN start is the feed-forward network of the starting time sequence, FFN end is a feed-forward network that ends the time series.

[0050] Furthermore, the loss function of the foreground frame detector, the relation signal generator and the relation-aware segment prediction head in S3 is Specifically:

[0051]

[0052] Among them, β and γ are hyperparameters used to balance the three losses; is the loss function for the relation-aware fragment prediction head, is the loss function of the relation signal generator, is the loss function of the foreground frame detector;

[0053] The loss function of the relation-aware fragment prediction head is for:

[0054]

[0055] Among them, Y s and Y e are the start boundary indices expressed in one-hot vector form (a s ) tag and end boundary index (a e ) label; P s is the starting boundary probability distribution, P e is the end boundary probability distribution

[0056] Loss function for relational signal generator for:

[0057]

[0058] Among them, yh The true value label of whether there is any correlation between the video and the query text; score cg is the coarse-grained relevance discriminant score between the query text and the input video;

[0059] The loss function of the foreground frame detector is for:

[0060]

[0061] Among them, s fi is the foreground score sequence S f The i-th element in , n is the length of the sequence, y fi is a binary sequence The i-th element in the binary sequence Y f is the true value label of each frame in the video sequence as a foreground frame or a background frame;

[0062] Furthermore, in S4, the calculation of the correlation between the video to be located and the text description includes:

[0063] Given a video and a query text, the trained model is used to obtain the multi-granularity correlation discrimination score P between the video and the query text. v , and compare it with the threshold k to determine whether the video and the query text are relevant;

[0064] The video temporal positioning of the video to be positioned and the text description having correlation includes:

[0065] Predict the start boundary index of the relevant video segment and the end boundary index The specific process is to calculate the starting boundary probability distribution P s and the end boundary probability distribution P e The two-dimensional index of the maximum value in the probability joint distribution matrix is ​​the starting boundary index of the predicted related video segment and the end boundary index Right now:

[0066]

[0067] According to another aspect of the specification, a video timing positioning device based on relationship perception is provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, a video timing positioning method based on relationship perception as described is implemented.

[0068] According to another aspect of the specification, a computer-readable storage medium is provided, on which a program is stored. When the program is executed by a processor, the video timing positioning method based on relationship perception is implemented.

[0069] Beneficial effects of the present invention: The present invention proposes a video timing positioning method based on relationship perception, which designs a foreground frame detector and a relationship signal generator, respectively measures the fine-grained and coarse-grained correlations between the input video and the query text from the video frame scale and the video segment scale, and designs a multi-granularity discriminator to combine the correlation discrimination capabilities of the above two granularities to obtain a multi-granularity correlation discrimination score, thereby achieving efficient correlation feedback. At the same time, a relationship-aware segment prediction head is further designed to further integrate relevant feedback into the video timing positioning process, thereby taking into account the situation that there may be no relevant segments in the video, so that the model can handle the situation in real application scenarios where there may be no segments related to the query text in a given long video. At the same time, the foreground frame detector, relationship signal generator, multi-granularity discriminator and relationship-aware segment prediction head module designed by the present invention can be embedded in the traditional video timing positioning method as a plug-and-play plug-in, so that they can be efficiently upgraded and directly applied to the video timing positioning task with relevant feedback that is more in line with real needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of a video timing positioning method based on relationship perception provided by an embodiment of the present invention;

[0071] Figure 2 A schematic diagram of a binary sequence generation method provided by an embodiment of the present invention;

[0072] Figure 3 A schematic diagram of the structure of a video timing positioning device based on relationship perception provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0074] like Figure 1 As shown, the present invention provides a video timing positioning method based on relationship perception, comprising:

[0075] (1) Use a pre-trained video feature encoder, such as the I3D model, to extract features from the video and obtain a frame-level video feature sequence. Where n is the length of the video feature sequence, d v is the feature dimension of the video feature vector.

[0076] (2) Use a pre-trained text feature encoder, such as the GloVe300d model, to extract word-level text feature sequences of the query text. Where m is the length of the text feature sequence, d q is the feature dimension of the word vector.

[0077] (3) Use the fully connected layer to combine the frame-level video feature sequence V obtained in step (1) and step (2) org and word-level text feature sequence Q org Mapped to the same dimension d, we get a frame-level video feature sequence in a common feature space and word-level text feature sequences Right now:

[0078] V=FC(V org )

[0079] Q=FC(Q org )

[0080] Where FC represents the fully connected layer.

[0081] (4) Use the Transformer encoding layer to perform information interaction between the intra-modal feature vectors of the frame-level video feature sequence V and the word-level text feature sequence Q in the common feature space obtained in step (3) to obtain a frame-level video feature sequence rich in contextual information. and word-level text feature sequences Right now:

[0082] V′=Transformer(V)

[0083] Q′=Transformer(Q)

[0084] Where Transformer represents a single-layer Transformer encoding layer.

[0085] (5) Use the self-attention mechanism to aggregate the word-level feature sequence Q′ rich in context information obtained in step (4) to obtain the sentence-level feature vector of the query text This sentence-level feature vector contains the complete semantic information of the text, namely:

[0086] h Q =Q ′T softmax(Q′W)

[0087] in, It is a learnable weight matrix that represents the importance of each word in the sentence.

[0088] (6) Using the frame-level video feature sequence V′ and word-level text feature sequence Q′ rich in context information obtained in step (4) through a word-level feature fusion module, a video feature sequence fused with word-level text information is obtained.

[0089] (6-1) Use the trilinear function to calculate the common similarity matrix between the context-rich frame-level video feature sequence V′ and the word-level text feature sequence Q′ obtained in step (4) Right now:

[0090] s ij =α.V′ i: ,Q′ j: /

[0091] in, Represents the i-th video feature vector V′ in the public similarity matrix S i: and the jth text feature vector Q′ j: The similarity score between two feature vectors V′ i: and Q′ j: The trilinear function of the similarity between them is:

[0092] α(v,q)=W0[v;q;v⊙q]

[0093] Where v represents the input video feature vector, q represents the input text feature vector, Is a trainable weight matrix, ⊙ is element-by-element multiplication, and represents the concatenation of vectors along the feature dimension.

[0094] (6-2) Normalize the common similarity matrix S obtained in step (6-1) row by row to obtain the attention weight from video to text The process of row-by-row normalization can be expressed as:

[0095]

[0096] in represents the correlation between each text feature vector and the i-th video feature vector, and softmax represents the normalization function.

[0097] (6-3) Normalizing the common similarity matrix S obtained in step (6-1) column by column can obtain the attention weight from text to video direction The process of column-by-column normalization can be expressed as:

[0098]

[0099] in represents the correlation between each video feature vector and the i-th text feature vector, and softmax represents the normalization function.

[0100] (6-4) Use the attention weight A from video to text obtained in step (6-2) v2q The word-level text feature sequence Q′ obtained in step (4) is weighted to obtain the video feature sequence reconstructed from the text information Right now:

[0101] C v =A v2q Q′

[0102] (6-5) Use the attention weight A from text to video obtained in step (6-3) q2v The frame-level video feature sequence V′ obtained in step (4) is weighted to obtain the text feature sequence reconstructed from the video features Right now:

[0103] C q2v =A q2v ·V′

[0104] (6-6) Use the reconstructed text feature sequence C obtained in step (6-5) q2v The attention weight A from video to text obtained in step (6-2) v2q Weighted to obtain text-aware video feature sequence Right now:

[0105] C q =A v2q ·C q2v

[0106] (6-7) Through a simple feed-forward network, the frame-level video feature sequence V′ rich in context information obtained in step (4) and the video feature sequence C reconstructed by text features obtained in step (6-4) are fused v And the text-aware video feature sequence C obtained in step (6-6) q , and obtain the video feature sequence that integrates word-level text information Right now:

[0107] V q =FFN([V′;C v ; V′⊙C v ; V′⊙C q ])

[0108] Among them, FFN(·) is a feed-forward network; ⊙ is element-by-element multiplication, and [;] means concatenating the vectors along the dimension of the feature.

[0109] (7) Using the feature concatenation and fusion method, the sentence-level text feature vector h obtained in step (5) is Q The video feature sequence obtained by step (6-7) that integrates word-level text information Perform feature fusion, namely:

[0110] V′ q =FC([h Q ;v q1 ,h Q ;v q2 …h Q ;v qn ])

[0111] in, Represents the video feature sequence V q The i-th feature vector in , [;] means concatenating two vectors along the feature dimension, and FC(·) means a fully connected layer.

[0112] (8) Using the video feature sequence V′ obtained in step (7) to integrate word-level text information q Through the foreground frame detector module, we get the fine-grained correlation discrimination score between the query text and the input video, and use the foreground detection loss function L fore Perform supervised learning on the foreground frame detector module.

[0113] (8-1) The video feature sequence V′ obtained in step (7) by integrating word-level text information q After a fully connected layer is used for prediction, a sequence of similarity scores between the text and each frame of the video is obtained. Then, the sigmoid function is used to map the similarity scores to between 0 and 1 to obtain a sequence of foreground frame prediction scores. Right now:

[0114] S f =sigmoid.FC(V′ q ) /

[0115] Among them, FC(·) represents a fully connected layer.

[0116] (8-2) Calculate the foreground frame prediction score sequence S obtained in step (8-1) f The maximum value among them is used to obtain the fine-grained correlation discrimination score between the query text and the input video. fg =max(S f ).

[0117] (9) Using the sentence-level feature vector h obtained in step (5) Q , the video feature sequence V′ obtained by step (7) by integrating word-level text information qThrough the relational signal generator module, the relational signal latent vector describing the relationship information between the text and the video clip and the coarse-grained correlation discrimination score between the query text and the input video are obtained. At the same time, the relational signal loss function is used to perform supervised learning on the relational signal generator module.

[0118] (9-1) The video feature sequence V′ obtained in step (7) q The input is sent to a fully connected layer to obtain a sequence of similarity scores between the text and each frame of the video. The similarity score sequence is then normalized using the softmax function to obtain the relative correlation score between the text and each frame. Finally, the relative correlation score is used to quantize the video feature sequence V′. q Perform weighted sum operation to obtain the segment-level video feature vector Right now:

[0119] h V =softmax.FC(V′ q ) / ·V′ q

[0120] (9-2) The sentence-level feature vector h obtained in step (5) Q and the segment-level video feature vector h V Perform feature splicing and fusion to obtain the relational signal latent vector describing the relationship between the text and the video clip Right now:

[0121] g=FC([h Q ;h V ])

[0122] (9-3) Input the relational signal latent vector g obtained in step (9-2) into the fully connected layer with the sigmoid function to obtain the coarse-grained correlation discrimination score score between the query text and the input video cg ,Right now:

[0123] score cg =sigmoid(FC(g))

[0124] The loss function of the relation signal generator uses the relation signal segment loss function The relation signal latent vector g generated by the relation signal generator module can correctly learn the semantic relationship between the query text and the video clip, that is:

[0125]

[0126] Among them, y h is the true value label of whether there is any correlation between the video and the query text.

[0127] (10) The fine-grained correlation discrimination score obtained in step (8-2) fg And the coarse-grained correlation discrimination score obtained in step (9-3) cg Input to the multi-granularity discriminator module to obtain the multi-granularity correlation discrimination score P v ,Right now:

[0128] P v =avg(score fg ,score cg )

[0129] Among them, avg(·) is the operation of calculating the mean.

[0130] (11) Using the foreground frame prediction score sequence S obtained in step (8-1) f , for the video feature sequence V obtained in step (6-7) q The foreground content in is enhanced, and then concatenated with the relational signal latent vector g obtained in step (9-2) as the input sequence and passed through the relation-aware segment prediction head module to obtain the start boundary probability distribution and the end boundary probability distribution of the predicted segment.

[0131] (11-1) Use the foreground frame prediction score sequence S obtained in step (8-1) f , for the video feature sequence V obtained in step (6-7) q The foreground frame feature vector in the image is strengthened, and the background frame feature vector irrelevant to the text is weakened to obtain the video feature sequence after the foreground frame is enhanced. Right now:

[0132]

[0133] Among them, ⊙ represents element-by-element multiplication.

[0134] (11-2) Splice the relationship signal latent vector g obtained in step (9-2) to the foreground frame enhanced video feature sequence obtained in step (11-1) Before the first video feature vector of

[0135] (11-3) The video feature sequence obtained in step (11-2) Input into the stacked two-layer LSTM network, and after passing through the first layer of LSTM network, the time series feature sequence of the predicted start boundary is obtained. Then the time series feature sequence H s Input to the second layer LSTM network to obtain the time series feature sequence of the predicted end boundary Right now:

[0136]

[0137] in, Indicates that in the time series feature sequence H s / e The tth time series feature vector in, Indicated in The tth video feature vector in LSTM * Represents the *th layer LSTM network.

[0138] (11-4) The start and end sequence feature sequences H obtained in step (11-3) are s , H e Respectively with the video feature sequence obtained in step (11-2) Concatenate and input into the corresponding feedforward network to obtain the boundary probability distribution Right now:

[0139]

[0140] Among them, P s is the starting boundary probability distribution, P e is the end boundary probability distribution, FFN start is the feed-forward network of the starting time sequence, FFN end is a feed-forward network that ends the time series.

[0141] (12) The loss function of the foreground frame detector, the relation signal generator, and the relation-aware segment prediction head is used together for multi-task learning, which has both the relevance discrimination capability and the video temporal localization capability, namely:

[0142]

[0143] Among them, β and γ are hyperparameters. In this implementation case, β = 6, γ = 6, which are used to balance the importance of the three losses.

[0144] The loss function of the relation-aware fragment prediction head is It enables the relationship-aware segment prediction head module to learn how to accurately locate the start and end times of related segments, specifically:

[0145]

[0146] Among them, Y s and Y e are the start boundary indices expressed in one-hot vector form (a s ) tag and end boundary index (a e ) label; P s is the starting boundary probability distribution, P eis the end boundary probability distribution

[0147] Loss function for relational signal generator The relation signal latent vector g generated by the relation signal generator module can correctly learn the semantic relationship between the query text and the video clip, that is:

[0148]

[0149] Among them, y h The true value label of whether there is any correlation between the video and the query text; score cg is the coarse-grained relevance discriminant score between the query text and the input video;

[0150] The loss function of the foreground frame detector is It enables the foreground frame detector module to correctly learn the difference between the foreground frame features and the background frame features, namely:

[0151]

[0152] in, is the foreground score sequence S f The i-th element in , n is the length of the sequence, y fi is a binary sequence The i-th element in the binary sequence Y f is the true value label of each frame in the video sequence as a foreground frame or a background frame, which is generated as follows Figure 2 As shown, for a video feature sequence of length n, a s and a e They represent the start and end boundary indexes of the target segment respectively, 1 represents the foreground frame between the boundary indexes, and 0 represents the background frame outside the boundary indexes.

[0153] (13) Using the model trained in step (12), relevant feedback is provided on the correlation between the video and the text, and video temporal positioning is performed for the video and text input sample pairs with correlation.

[0154] (13-1) Given a video and a query text, the multi-granularity correlation discrimination score P between the video and the query text is obtained through the model trained in step (12) v , and compare it with the threshold k to determine whether the video is relevant to the query text, and thus make relevant feedback, namely:

[0155]

[0156] Here, k is a threshold value, and in this implementation case, k=0.5.

[0157] (13-2) For the videos and query texts that are determined to be relevant in step (13-1), further video timing positioning is required to predict the start boundary index of the relevant video segments. and the end boundary index The specific process is to calculate the starting boundary probability distribution P obtained in step (11-4) s and the end boundary probability distribution P e The two-dimensional index of the maximum value in the probability joint distribution matrix is ​​the starting boundary index of the predicted related video segment and the end boundary index Right now:

[0158]

[0159] The present invention uses two traditional video temporal positioning task datasets to reconstruct a video temporal positioning task dataset with relevant feedback, and conducts a large number of relevant experiments to verify the usefulness of the present invention. "Accuracy", "Rank@n, IoU = μ" and "mIoU" are used as performance indicators. Among them, "Accuracy" is used to measure the accuracy of the correlation situation judgment, "Rank@n, IoU = μ" means that among the top-n moments located, the query text has at least one moment with an intersection-over-union (IoU) greater than μ with the true value. "mIoU" is the average intersection-over-union (IoU) of all test samples. In the experiment, n = 1, μ = {0.3, 0.5, 0.7}. Since the reconstructed test set contains samples that are not relevant, and there are no clips related to the query text in the videos of these samples, the present invention redefines the calculation method of the intersection-over-union (IoU), which is specifically divided into four cases:

[0160] (1) When the prediction is not relevant but it is actually relevant, the IoU is 0;

[0161] (2) When the prediction and the actual are consistent and have no correlation, the IoU is 1;

[0162] (3) When the prediction is relevant but it is not actually relevant, the IoU is 0;

[0163] (4) When the prediction and the actual are consistent and are correlated, IoU is the intersection over union ratio of the predicted relevant fragment moment to the true value moment.

[0164] At this time, the IoU comprehensively considers the accuracy of the model in determining whether there are retrieval results and its precision in retrieving fine-grained video clips.

[0165] Table 1 summarizes the performance comparison of the RaTSG model designed by the present invention and other baseline models on the Charades-RF and ActivityNet-RF datasets. The results show that the model proposed by the present invention has significant advantages in various performance indicators and has a smaller number of parameters.

[0166] Table 2 embeds the foreground frame detector, relation signal generator, multi-granularity discriminator and relation-aware segment prediction head module designed by the present invention into the traditional video timing positioning EAMAT method as a plug-and-play plug-in, so that it can be directly applied to the video timing positioning task with relevant feedback. The results show that the module proposed by the present invention can be used as a plug-in to efficiently upgrade the traditional timing positioning method.

[0167] Table 1 Performance comparison of RaTSG and other baseline models on Charades-RF and ActivityNet-RF datasets.

[0168]

[0169]

[0170] Table 2 The main components of the present invention are plug-and-play and can be directly embedded into the traditional video timing positioning method to achieve efficient upgrading.

[0171] method Acc R1@0.3 R1@0.5 R1@0.7 QUR EAMAT 50.00 37.12 30.59 20.86 27.27 EAMAT++ 71.94 63.55 59.17 51.96 56.23 EAMAT+Ours 76.37 67.47 62.02 54.83 59.58

[0172] Corresponding to the aforementioned embodiment of a video timing positioning method based on relationship perception, the present invention also provides an embodiment of a video timing positioning device based on relationship perception.

[0173] See also Figure 3 A video timing positioning device based on relationship perception provided by an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a video timing positioning method based on relationship perception in the above embodiment.

[0174] An embodiment of a video timing positioning device based on relationship perception provided by the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 3As shown, it is a hardware structure diagram of a video timing positioning device based on relationship perception provided by the present invention, in which any device with data processing capability is located, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0175] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0176] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0177] An embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a video timing positioning method based on relationship perception in the above embodiment is implemented.

[0178] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0179] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the video timing positioning method based on relationship perception is implemented.

[0180] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0181] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A video temporal positioning method based on relationship perception, characterized in that: The method comprises the following steps: S1, obtaining training data of video and text representation in video temporal positioning; S2. Construct a video temporal positioning model based on relationship perception. The processing flow of the model includes: S2.1, input the video to be located and the text expression into the pre-trained video feature encoder and text feature encoder respectively, to obtain a frame-level video feature sequence and a word-level text feature sequence; S2.2, and use a fully connected layer to map the two obtained feature sequences to the same dimension, and then perform information interaction between the feature vectors within the modality to obtain a frame-level video feature sequence and a word-level text feature sequence rich in contextual information; S2.

3. Use the self-attention mechanism to aggregate the word-level feature sequences rich in contextual information to obtain the sentence-level feature vector of the query text. S2.4, the frame-level video feature sequence and the word-level text feature sequence rich in context information are fused through a word-level feature fusion module to obtain a video feature sequence fused with word-level text information; S2.5, using a feature concatenation and fusion method, the sentence-level text feature vector is fused with the video feature sequence fused with word-level text information to obtain a video feature sequence fused with text information; S2.6, using the video feature sequence fused with text information through the foreground frame detector module, obtain the fine-grained correlation discrimination score and foreground frame prediction score sequence between the query text and the input video; S2.7, using the sentence-level feature vector and the video feature sequence fused with text information through the relational signal generator module, obtain the coarse-grained correlation discrimination score and relational signal latent vector describing the relational information between the text and the video clip; S2.8, inputting the fine-grained correlation discrimination score and the coarse-grained correlation discrimination score into the multi-grained discriminator module to obtain the multi-grained correlation discrimination score; S2.9, using the foreground frame prediction score sequence, the foreground content in the video feature sequence integrated with word-level text information is enhanced, and then the relational signal latent vector is concatenated as the input sequence to pass through the relation-aware segment prediction head module to obtain the start boundary probability distribution and the end boundary probability distribution of the predicted segment; S3, based on the relation-aware video temporal localization model, the training data is combined with the loss function of the foreground frame detector, the relation signal generator and the relation-aware segment prediction head for multi-task learning; S4. Use the trained relationship-aware video temporal localization model to obtain the correlation between the video to be located and the text description, and perform video temporal localization on the video to be located and the text description that have the correlation.

2. The method for video timing positioning based on relationship perception according to claim 1, characterized in that: The calculation process of the word-level feature fusion module in S2.4 includes: S2.4.

1. Use trilinear function to calculate the common similarity matrix between the frame-level video feature sequence and the word-level text feature sequence rich in context information. S2.4.2, normalize the common similarity matrix row by row to obtain the attention weight from video to text; S2.4.

3. Normalize the common similarity matrix column by column to obtain the attention weight from text to video. S2.4.4, weighting the word-level text feature sequence using the attention weight from the video to text direction to obtain a video feature sequence reconstructed from the text information; S2.4.

5. Use the attention weight from text to video to weight the frame-level video feature sequence to obtain a text feature sequence reconstructed from the video features; S2.4.

6. Use the reconstructed text feature sequence and the attention weight from the video to the text direction to obtain the text-aware video feature sequence; S2.4.

7. Through a layer of feedforward network, the frame-level video feature sequence rich in contextual information, the video feature sequence reconstructed by text features, and the text-aware video feature sequence are fused to obtain a video feature sequence that integrates word-level text information.

3. The method for video timing positioning based on relationship perception according to claim 1, characterized in that: The calculation process of the foreground detector module in S2.6 includes: S2.6.

1. The video feature sequence fused with word-level text information is predicted through a fully connected layer to obtain a sequence of similarity scores between the text and each frame of the video. The sigmoid function is then used to map the similarity scores to between 0 and 1 to obtain a sequence of foreground frame prediction scores. S2.6.

2. Calculate the maximum value in the foreground frame prediction score sequence to obtain the fine-grained correlation discriminant score between the query text and the input video.

4. The method for video timing positioning based on relationship perception according to claim 1, characterized in that: The calculation process of the relation signal generator in S2.7 includes: S2.7.

1. Input the video feature sequence into a fully connected layer to obtain a similarity score sequence between the text and each frame of the video. Then, the similarity score sequence is normalized using a softmax function to obtain a relative correlation score between the text and each frame. Finally, the relative correlation score is used to perform a weighted sum operation on the video feature sequence to obtain a segment-level video feature vector. S2.7.

2. The sentence-level feature vector and the segment-level video feature vector are concatenated and fused to obtain the relational signal latent vector describing the relational information between the text and the video segment, namely: S2.7.

3. Input the relation signal latent vector into the fully connected layer with the sigmoid function to obtain the coarse-grained correlation discrimination score between the query text and the input video.

5. The method for video timing positioning based on relationship perception according to claim 1, characterized in that: The calculation of the multi-granularity discriminator in S2.8 includes: The fine-grained correlation discriminant score fg and step coarse-grained correlation discrimination score score cg Input to the multi-granularity discriminator module to obtain the multi-granularity correlation discrimination score P v ,Right now: P v =avg(mother-in-law fg mother-in-law cg ) Among them, avg(·) is the operation of calculating the mean.

6. The method for video timing positioning based on relationship perception according to claim 1, characterized in that: The relationship-aware segment prediction head calculation process in S2.9 specifically includes: S2.9.

1. Using the foreground frame to predict the score sequence S f , for the video feature sequence V q The foreground frame feature vector in the image is strengthened, and the background frame feature vector irrelevant to the text is weakened to obtain the video feature sequence after the foreground frame is enhanced. Right now: Among them, ⊙ represents element-by-element multiplication; S2.9.

2. Concatenate the relation signal latent vector g to the foreground frame enhanced video feature sequence Before the first video feature vector of S2.9.

3. Input feature sequence of the video Input into the stacked two-layer LSTM network, and after passing through the first layer of LSTM network, the time series feature sequence of the predicted start boundary is obtained. Then the time series feature sequence H s Input to the second layer LSTM network to obtain the time series feature sequence of the predicted end boundary Right now: in, Indicates that in the time series feature sequence H s / e The tth time series feature vector in, Indicated in The tth video feature vector in LSTM * Represents the *th layer LSTM network; S2.9.

4. Set the start and end timing feature sequence H s , H e Video feature sequences Concatenate and input into the corresponding feedforward network to obtain the boundary probability distribution Right now: Among them, P s is the starting boundary probability distribution, P e is the end boundary probability distribution, FFN start is the feed-forward network of the starting time sequence, FFN end is a feed-forward network that ends the time series.

7. The method for video timing positioning based on relationship perception according to claim 1, characterized in that: The loss function of the foreground frame detector, the relation signal generator and the relation-aware segment prediction head in S3 is Specifically: Among them, β and γ are hyperparameters used to balance the three losses; is the loss function for the relation-aware fragment prediction head, is the loss function of the relation signal generator, is the loss function of the foreground frame detector; The loss function of the relation-aware fragment prediction head is for: Among them, Y s and Y e are the start boundary indices expressed in one-hot vector form (a s ) tag and end boundary index (a e ) label; P s is the starting boundary probability distribution, P e is the end boundary probability distribution Loss function for relational signal generator for: Among them, y h The true value label of whether there is any correlation between the video and the query text; score cg is the coarse-grained relevance discriminant score between the query text and the input video; The loss function of the foreground frame detector is for: in, is the foreground score sequence S f The i-th element in , n is the sequence length, is a binary sequence The i-th element in the binary sequence Y f It is the true value label of each frame in the video sequence as a foreground frame or a background frame.

8. The method for video timing positioning based on relationship perception according to claim 1, characterized in that: In S4, the calculation of the correlation between the video to be located and the text description includes: Given a video and a query text, the trained model is used to obtain the multi-granularity correlation discrimination score P between the video and the query text. v , and compare it with the threshold k to determine whether the video and the query text are relevant; The video temporal positioning of the video to be positioned and the text description having correlation includes: Predict the start boundary index of the relevant video segment and the end boundary index The specific process is to calculate the starting boundary probability distribution P s and the end boundary probability distribution P e The two-dimensional index of the maximum value in the probability joint distribution matrix is ​​the starting boundary index of the predicted related video segment and the end boundary index Right now:

9. A video timing positioning device based on relationship perception, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a video timing positioning method based on relationship perception as described in any one of claims 1-8 is implemented.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a video timing positioning method based on relationship perception as described in any one of claims 1 to 8 is implemented.

Images