A multimodal video language learning method based on video depth information

Abstract

This invention discloses a multimodal video language learning method based on video depth information, comprising the following steps: S1: extracting first feature information of the original video and second feature information of the text in the original video; S2: using the DAST module based on the first feature information to obtain final video features; the final video features are video features that include video depth information and consider the temporal correlation of frames at different scales; S3: using the final video features and the second feature information to design a head network to adapt to different cross-modal task requirements. Incorporating video depth information, this invention can better extract and model the geometric relationships between different individual objects in a video.

Description

Technical Field

[0001] The present invention relates to the technical field of video language learning, and in particular to a multimodal video language learning method based on video depth information. Background Art

[0002] In video language tasks, a large amount of text annotations describe the geometric relationships (including shapes, relative positions, and so on) between objects in the video. On four commonly used video question answering and video retrieval datasets (MSVD, MSR-VTT, MSVD-QA, and MSRVTT-QA), the proportion of such annotations ranges from 19.6% to 45.0%. Previous models often perform poorly on these geometry-related tasks.

[0003] Video and language learning have received unprecedented attention. In recent years, many video language tasks have been proposed, including automatic generation of video descriptions, video question answering, and video-to-text matching. Typically, these tasks use an encoder to extract video features from a fixed video sequence, and then adopt task-specific module design to promote alignment between the two modalities of video and language, enabling the model to better understand multimodal semantic information. In recent years, research on video language learning has mainly focused on various pre-trained video feature extraction models, and some work has also studied object-level representations to improve language generation and prediction using fine-grained objects and actions. However, these widely used video representations often lack geometric information, which can easily lead to poor performance of the model on videos with cluttered scenes. Summary of the Invention

[0004] The purpose of the present invention is to provide a multimodal video language learning method based on video depth information, so as to better extract and model the geometric relationships between different individual objects in the video in combination with video depth information; in addition, in order to ensure computational efficiency, the present invention also designs a sparse attention mechanism module with linear computational complexity for each transformer module to learn the two-dimensional representation of depth perception.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] The present invention provides a multimodal video language learning method based on video depth information, the multimodal video language learning method based on video depth information comprising:

[0007] S1: extracting first feature information of an original video and second feature information of text in the original video;

[0008] S2: Based on the first feature information, using the DAST module, obtain a final video feature; the final video feature is a video feature that contains video depth information and considers the temporal correlation of frames at different scales;

[0009] S3: Design a head network using the final video features and the second feature information to adapt to different cross-modal task requirements.

[0010] Optionally, in S1, the first feature information is an appearance feature of a video frame, a motion feature of a video frame, and a depth feature in the video, and the depth feature in the video is obtained by:

[0011] DepthFormer is used to extract depth information from the original video, and a residual neural network is used to extract depth features from the depth information.

[0012] Optionally, the S2 includes:

[0013] Using the first feature information as input information of the DAST module;

[0014] performing a multi-scale feature operation on the first feature information to obtain multi-scale features of the first feature information;

[0015] The features of different scales are expanded and the attention is calculated to obtain the final video features.

[0016] Optionally, the DAST module includes four constituent groups, a first constituent group includes two stacked DSCA submodules and a fusion submodule, the two stacked DSCA submodules and the fusion submodule are arranged in sequence, the first constituent group and the second constituent group are constructed to have the same structure, and the first constituent group and / or the second constituent group performs a multi-scale feature operation on the first feature information;

[0017] The third constituent group includes a sequentially arranged expansion submodule and two stacked DSCA submodules, the third constituent group and the fourth constituent group are constructed to have the same structure, and the third constituent group and / or the fourth constituent group perform expansion operations and attention calculations on features of different scales;

[0018] A bottleneck layer is provided between the second constituent group and the third constituent group;

[0019] The output results of the second constituent group and the third constituent group are residually connected, and the output results of the first constituent group and the fourth constituent group are residually connected.

[0020] Optionally, the DSCA submodule adopts a sparse attention calculation method, that is, for each frame, only the depth map features within a local time range are calculated. The local time range is defined as:

[0021]

[0022] in, represents the deep feature set of the local frame sampled at the current time t, Represents the depth feature of the sampling frame within the local time range, r represents the time of the sampling frame within the local time range, t represents the time of the current frame, and h represents the radius threshold of the local sampling.

[0023] Optionally, the DSCA submodule adopts a multi-head attention mechanism. For each head, the DSCA submodule uses different mapping heads to learn different features. For the mth head, the DSCA submodule uses the depth feature as the key and value, and the frame feature of the video as the query. The attention score of each frame feature and the depth map features of the adjacent frames is calculated using the following formula:

[0024]

[0025] in, represents the video frame feature at the current time t, m represents the mth attention head, Represents the scaling factor, k and V t Represents key and value, V m with U m They all represent the learnable parameters of the mapping, and the final A mqk is the attention map, Represents the deep feature set of the local frame sampled at the current time t.

[0026] Optionally, after obtaining the attention scores of the key and query, it is necessary to perform a weighted summation of the values ​​according to the attention graph. The specific method is shown in the following formula:

[0027]

[0028] Among them, Zi t A represents the video features of the t-th frame after passing through the i-th layer of DAST, guided by deep features and weighted by multi-head attention. mqk represents the attention map, W m With W m ' represent learnable parameters, M represents the number of heads in the multi-head attention, Represents the deep feature set of the local frame sampled at the current time t, v represents the value of the local frame deep feature used to calculate the attention value, and m represents the mth attention head.

[0029] Optionally, in S3, the head network includes an attention enLSTM, a language deLSTM, and a multi-layer perceptron network arranged in sequence, wherein the multi-layer perceptron network includes a first linear layer and a first softmax layer, and is applied to the video description task. The input of the attention enLSTM is the final video feature. For each time step t, the enLSTM receives the feature from the last hidden layer of the deLSTM. Video features with depth information after DAST encoding and the previously generated word embedding w t-1 , generate a new feature with historical information and current frame information Then deLSTM will combine this feature with the video feature to generate new features Finally, a multi-layer perceptron network is used to generate the probability distribution p of the next word t , the specific methods are shown in the following three formulas:

[0030]

[0031]

[0032]

[0033] The loss function L of the head network CAP as follows:

[0034]

[0035] Among them, P θ represents the model parameters, Indicates the current s-th generated word, represents the true first s-1 words, S represents the true overall text length, and s represents the index.

[0036] Optionally, in S3, the head network includes a second linear layer, an ELU activation function layer, a third linear layer, and a second softmax layer, which are arranged in sequence and applied to the video question answering task. The second feature and the final video feature are concatenated as the input of the second linear layer. The head network obtains the probability of each answer in the answer set through the second softmax layer. The specific method is:

[0037]

[0038]

[0039] Where A represents the answer set, W2 and W3 represent the learnable parameters of the second and third linear layers respectively, b2 and b3 represent the learnable offsets of the second and third linear layers respectively, and o represents the feature after the fusion of video and text. represents the video features output from DAST, q represents the text features, and p represents the output probability. A vector space representing the size of the answer set A;

[0040] The loss function L of the head network QA for:

[0041]

[0042] Among them, y i Represents the one-hot vector of the i-th correct answer, i represents the answer index, p i represents the probability of the ith answer.

[0043] Optionally, in S3, the head network includes a fourth linear layer and a fifth linear layer arranged in parallel, and a cosine similarity layer. The inputs of the fourth linear layer and the fifth linear layer are the second feature and the final video feature, respectively, and the output is used as the input of the cosine similarity layer. The specific method is as follows:

[0044]

[0045] f(s)=W5S+b5,

[0046]

[0047] Where f(v) represents, W4 represents the learnable parameters of the fourth linear layer, represents the video features output from DAST, b4 represents the learnable offset of the fourth linear layer, W5 represents the learnable parameter of the fifth linear layer, S represents the text features, b5 represents the learnable offset of the fifth linear layer, and sim(v,s) represents the similarity between the text features and the video features;

[0048] The loss function of the head network is:

[0049] L match =max(0,Δ+sim(v,s - )-sim(v,s))+max(0,Δ+sim(v - ,s)-sim(v,s))

[0050] sim(v,s -) represents the similarity between the video v and the negative sample text, sim(v - ,s) represents the similarity between text s and negative sample video, sim(v,s) represents the similarity between positive sample video and text, Δ represents the regularization term, s - With v - Represent the negative sample text and negative sample video in a batch respectively.

[0051] The present invention has the following beneficial effects:

[0052] The present invention clearly recognizes that current work on video language learning has failed to pay enough attention to the problem of video depth information. It proposes a video language learning method that can extract and utilize video depth information, and proposes the use of sparse attention to effectively reduce the amount of computation. In addition, the hierarchical model structure proposed in the present invention effectively maintains the temporal coherence of video features. As a result, the present invention achieves optimal performance on multiple traditional multimodal tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Flowchart of the multimodal video language learning method based on video depth information of the present invention;

[0054] Figure 2 This is a framework diagram of the multimodal video language learning method based on video depth information of the present invention;

[0055] Figure 3 It is a structural diagram of the DSCA submodule;

[0056] Figure 4 This is the network structure diagram of the video description task head of the present invention;

[0057] Figure 5 This is the head network structure diagram of the video question answering task of the present invention;

[0058] Figure 6 This is the head network structure diagram of the video text matching task of the present invention;

[0059] Figure 7 This is a comparison example of DAST and the baseline model in the video text matching task;

[0060] Figure 8 An example diagram showing the comparison between DAST and the baseline model on the video description task. DETAILED DESCRIPTION

[0061] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0062] The present invention provides a multimodal video language learning method based on video depth information, referring to Figure 1 As shown, the multimodal video language learning method based on video depth information includes:

[0063] S1: extracting first feature information of an original video and second feature information of text in the original video;

[0064] In the present invention, the first feature information is the appearance feature of the video frame, the motion feature of the video frame and the depth feature in the video, and the depth feature in the video is obtained by:

[0065] DepthFormer is used to extract depth information from the original video, and a residual neural network (ResNet) is used to extract depth features from the depth information.

[0066] The appearance features of the video frames and the motion features of the video frames are extracted respectively by two-dimensional convolutional neural networks and three-dimensional convolutional neural networks.

[0067] S2: Based on the first feature information, using the DAST module, obtain a final video feature; the final video feature is a video feature that contains video depth information and considers the temporal correlation of frames at different scales;

[0068] Specifically, the first feature information is used as input information of the DAST module;

[0069] performing a multi-scale feature operation on the first feature information to obtain multi-scale features of the first feature information;

[0070] The features of different scales are expanded and the attention is calculated to obtain the final video features.

[0071] Alternatively, refer to Figure 2 As shown, the DAST module includes four constituent groups, the first constituent group includes two stacked DSCA submodules and a fusion submodule, the two stacked DSCA submodules and the fusion submodule are arranged in sequence, the first constituent group and the second constituent group are constructed to have the same structure, and the first constituent group and / or the second constituent group performs a multi-scale feature operation on the first feature information;

[0072] The third constituent group includes a sequentially arranged expansion submodule and two stacked DSCA submodules, the third constituent group and the fourth constituent group are constructed to have the same structure, and the third constituent group and / or the fourth constituent group perform expansion operations and attention calculations on features of different scales;

[0073] A bottleneck layer is provided between the second constituent group and the third constituent group;

[0074] The output results of the second constituent group and the third constituent group are residually connected, and the output results of the first constituent group and the fourth constituent group are residually connected.

[0075] Alternatively, the DSCA submodule structure diagram can be found in Figure 3 As shown in Figure 2, the DSCA submodule uses a sparse attention calculation method, that is, for each frame, only the depth map features within a local time range are calculated. The local time range is defined as:

[0076]

[0077] in, represents the deep feature set of the local frame sampled at the current time t, Represents the depth feature of the sampling frame within the local time range, r represents the time of the sampling frame within the local time range, t represents the time of the current frame, and h represents the radius threshold of the local sampling.

[0078] Optionally, the DSCA submodule adopts a multi-head attention mechanism. For each head, the DSCA submodule uses different mapping heads to learn different features. For the mth head, the DSCA submodule uses the depth feature as the key and value, and the frame feature of the video as the query. The attention score of each frame feature and the depth map features of the adjacent frames is calculated using the following formula:

[0079]

[0080] in, represents the video frame feature at the current time t, m represents the mth attention head, Represents the scaling factor, k and V t Represents key and value, V m with U m They all represent the learnable parameters of the mapping, and the final A mqk is the attention map, Represents the deep feature set of the local frame sampled at the current time t.

[0081] Optionally, after obtaining the attention scores of the key and query, it is necessary to perform a weighted summation of the values ​​according to the attention graph. The specific method is shown in the following formula:

[0082]

[0083] in, A represents the video features of the t-th frame after passing through the i-th layer of DAST, guided by deep features and weighted by multi-head attention. mqk represents the attention map, W m and W′ m Both represent learnable parameters, M represents the number of heads in the multi-head attention. Represents the deep feature set of the local frame sampled at the current time t, v represents the value of the local frame deep feature used to calculate the attention value, and m represents the mth attention head.

[0084] Finally, in order to prevent the gradient from disappearing and to maintain information transfer, the original input feature V is added to the obtained feature Z, that is, a residual connection is added; at the same time, the obtained features are layer normalized to ensure the stability of data distribution during training. The specific operations are shown in the following two formulas:

[0085]

[0086]

[0087] in, Represents the video features after residual connection, It represents the video features of the t-th frame after passing through the i-th layer of DAST, guided by deep features and weighted by multi-head attention. represents the video features of the original input DSCA, represents the video features of the final DSCA output, PWFF() represents the feedforward neural network, and LN() represents layer normalization.

[0088] S3: Design a head network using the final video features and the second feature information to adapt to different cross-modal task requirements.

[0089] Alternatively, refer to Figure 4 As shown, the head network includes an attention enLSTM, a language deLSTM, and a multi-layer perceptron network arranged in sequence. The multi-layer perceptron network includes a first linear layer and a first softmax layer, which is applied to the video description task. The input of the attention enLSTM is the final video feature. For each time step t, the enLSTM receives the feature from the last hidden layer of the deLSTM. Video features with depth information after DAST encoding and the previously generated word embedding w t-1 , generate a new feature with historical information and current frame information Then deLSTM will combine this feature with the video feature to generate new features Finally, a multi-layer perceptron network is used to generate the probability distribution p of the next word t , the specific methods are shown in the following three formulas:

[0090]

[0091]

[0092]

[0093] The loss function L of the head network CAP as follows:

[0094]

[0095] Among them, P θ represents the model parameters, Indicates the current s-th generated word, represents the true first s-1 words, S represents the true overall text length, and s represents the index.

[0096] Alternatively, refer to Figure 5 As shown, the head network includes a second linear layer, an ELU activation function layer, a third linear layer, and a second softmax layer, which are arranged in sequence and applied to the video question answering task. The second feature and the final video feature are spliced ​​as the input of the second linear layer. The head network obtains the probability of each answer in the answer set through the second softmax layer. The specific method is:

[0097]

[0098]

[0099] Where A represents the answer set, W2 and W3 represent the learnable parameters of the second and third linear layers respectively, b2 and b3 represent the learnable offsets of the second and third linear layers respectively, and o represents the feature after the fusion of video and text. represents the video features output from DAST, q represents the text features, and p represents the output probability. A vector space representing the size of the answer set A;

[0100] The loss function L of the head network QA for:

[0101]

[0102] Among them, y i Represents the one-hot vector of the i-th correct answer, i represents the answer index, p i represents the probability of the ith answer.

[0103] Alternatively, refer to Figure 6 As shown, the head network includes a fourth linear layer and a fifth linear layer arranged in parallel, and a cosine similarity layer. The inputs of the fourth linear layer and the fifth linear layer are the second feature and the final video feature, respectively, and the output is used as the input of the cosine similarity layer. The specific method is as follows:

[0104]

[0105] f(s)=W5S+b5,

[0106]

[0107] Where f(v) represents, W4 represents the learnable parameters of the fourth linear layer, represents the video features output from DAST, b4 represents the learnable offset of the fourth linear layer, W5 represents the learnable parameter of the fifth linear layer, S represents the text features, b5 represents the learnable offset of the fifth linear layer, and sim(v,s) represents the similarity between the text features and the video features;

[0108] The loss function of the head network is:

[0109] L match =max(0,Δ+sim(v,s - )-sim(v,s))+max(0,Δ+sim(v - ,s)-sim(v,s))

[0110] sim(v,s - ) represents the similarity between the video v and the negative sample text, sim(v - ,s) represents the similarity between text s and negative sample video, sim(v,s) represents the similarity between positive sample video and text, Δ represents the regularization term, s - With v - Represent the negative sample text and negative sample video in a batch respectively.

[0111] Example 2

[0112] The effects of the present invention will be described below with reference to examples:

[0113] The experimental conditions are: system: Ubuntu 18.04, programming language: Python 3.8, deep learning framework: Pytorch 11.6, GPU: NVIDIA GeForce RTX 3090.

[0114] Input a video and uniformly sample 35 frames of the video. For the input text, first convert it all to lowercase and then limit its length to less than 35 words.

[0115] Experiment 1: For the video description task, the model is fed with a video of a young man running on a track and is required to output the correct description, such as Figure 8 shown.

[0116] Experiment 2: For the video text matching task, the model is fed with a video of two men eating, and is required to match the correct corresponding description, such as Figure 7 shown.

[0117] Experimental results analysis:

[0118] from Figure 8 It can be seen that DSAT can well capture the relationship between the characters in the video in the geometric space, and thus output "after", and the generated video annotation will be more accurate; Figure 5 It can be seen that the model has matched the text that more accurately describes the video clip.

[0119] The following is a comparative analysis of this method with other methods:

[0120] For the video question answering task, the datasets used were MSVD and MSR-VTT. According to the standard, the MSVD dataset is divided into a training set of 1200 clips, a validation set of 100 video clips, and a test set of 670 video clips. Each video clip has 25 corresponding questions and answers. MSR-VTT, on the other hand, uses a training set, validation set, and test set split in a ratio of 6573 / 497 / 2990. The two currently top-performing methods, B2A and IGV, were selected. As shown in Table 1, our method outperforms the current best method (IGV) by 3.3 percentage points on the MSVD-QA dataset and by 1.1 percentage points on the MSRVTT-QA dataset.

[0121] Table 1 Performance comparison of DAST and other video question answering methods

[0122] method MSVD-QA MSRVTT-QA B2A 37.2 36.9 IGV 40.8 38.3 DSAT 44.1 39.4

[0123] For the video description task, MSVD and MSRVTT were also used. The results are shown in Table 2. On the four evaluation metrics of video description, DAST outperforms the current best method RMN on both datasets:

[0124] Table 2 Comparison of DAST with other video description methods

[0125]

[0126] The four evaluation indicators in Table 2 are BLEU-4, METEOR, ROUGE-L and CIDEr.

[0127] For video-to-text matching, there are two tasks: retrieving text from video and retrieving video from text. The dataset used is MSR-VTT, and the evaluation metrics are Recall@1, Recall@5, Recall@10, Med r, and SumR. HGR and Dual Encoding, the currently optimal methods for video-to-text matching, were selected. The comparison results are shown in Table 3:

[0128] Table 3 Comparison of DAST with other video text matching methods

[0129]

[0130]

[0131] As can be seen from the results in Table 3, DAST outperforms the previous best model in all evaluation indicators.

[0132] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal video language learning method based on video depth information, characterized in that: The multimodal video language learning method based on video depth information includes: S1: extracting first feature information of an original video and second feature information of text in the original video; S2: Based on the first feature information, using the DAST module, obtain a final video feature; the final video feature is a video feature that contains video depth information and considers the temporal correlation of frames at different scales; S3: Designing a head network using the final video feature and the second feature information to adapt to different cross-modal task requirements; The DAST module includes four constituent groups, the first constituent group includes two stacked DSCA submodules and a fusion submodule, the two stacked DSCA submodules and the fusion submodule are arranged in sequence, the first constituent group and the second constituent group are constructed to have the same structure, and the first constituent group and / or the second constituent group performs a multi-scale feature operation on the first feature information; The third constituent group includes a sequentially arranged expansion submodule and two stacked DSCA submodules, the third constituent group and the fourth constituent group are constructed to have the same structure, and the third constituent group and / or the fourth constituent group perform expansion operations and attention calculations on features of different scales; A bottleneck layer is provided between the second constituent group and the third constituent group; The output results of the second constituent group and the third constituent group are residually connected, and the output results of the first constituent group and the fourth constituent group are residually connected.

2. The multimodal video language learning method based on video depth information according to claim 1 is characterized in that In S1, the first feature information is the appearance feature of the video frame, the motion feature of the video frame, and the depth feature in the video. The depth feature in the video is obtained by: DepthFormer is used to extract depth information from the original video, and a residual neural network is used to extract depth features from the depth information.

3. The multimodal video language learning method based on video depth information according to claim 1, characterized in that: The S2 includes: Using the first feature information as input information of the DAST module; performing a multi-scale feature operation on the first feature information to obtain multi-scale features of the first feature information; The features of different scales are expanded and the attention is calculated to obtain video features containing depth information.

4. The multimodal video language learning method based on video depth information according to claim 1, characterized in that: The DSCA submodule adopts the method of calculating sparse attention, that is, for each frame, only the depth map features within a local time range are calculated. The local time range is defined as: in, represents the deep feature set of the local frame sampled at the current time t, represents the deep features of the sampled frames within the local time range, r Represents the time of the sampling frame within the local time range, t Indicates the time of the current frame, h Indicates the radius threshold for local sampling.

5. The multimodal video language learning method based on video depth information according to claim 1, characterized in that: The DSCA submodule adopts a multi-head attention mechanism. For each head, the DSCA submodule uses different mapping heads to learn different features. For the mth head, the DSCA submodule uses the depth feature as the key and value, and the frame feature of the video as the query. The following formula is used to calculate the attention score of each frame feature and the depth map features of the adjacent frames: in, Indicates the current time t The video frame features, m Indicates the m An attention head, represents the scaling factor, k and Represents key and value respectively, and They all represent the learnable parameters of the mapping, and the final is the attention map, Indicates that according to the current time t A set of deep features of sampled local frames.

6. The multimodal video language learning method based on video depth information according to claim 5, characterized in that: After obtaining the attention scores of the key and query, it is necessary to perform a weighted summation of the values ​​according to the attention graph. The specific method is shown in the following formula: in, Indicates the t Frame passes through i Layer DAST, video features guided by deep features and weighted by multi-head attention, represents the attention map, and Both represent learnable parameters, M Represents the number of heads in the multi-head attention. Indicates that according to the current time t The deep feature set of the sampled local frame, Indicates the value of the local frame depth feature used to calculate the attention value, m Indicates the m An attention head.

7. The multimodal video language learning method based on video depth information according to claim 1, characterized in that: In S3, the head network includes attentions set in sequence enLSTM ,language deLSTM and a multilayer perceptron network comprising a first linear layer and a first softmax layer, applied to the video description task, the attention enLSTM The input is the final video feature, for each time step t , enLSTM Accept from deLSTM Features from the last hidden layer , Video features with depth information after DAST encoding and the previously generated word embedding , generate a new feature with historical information and current frame information ,Then deLSTM This feature will be combined with the video feature to generate a new feature , and finally generate the probability distribution of the next word through a multi-layer perceptron network , the specific methods are shown in the following three formulas: The loss function of the head network is as follows: in, represents the model parameters, Indicates the current s Generated words, Indicates the real front s- 1 word, S Indicates the actual overall text length, s Represents an index.

8. The multimodal video language learning method based on video depth information according to claim 1, characterized in that: In S3, the head network includes a second linear layer, an ELU activation function layer, a third linear layer, and a second softmax Layer, applied to the video question answering task, the second feature and the final video feature are spliced ​​as the input of the second linear layer, and the head network passes through the second softmax The probability of each answer in the answer set is obtained by the following method: in A represents the answer set, and denote the learnable parameters of the second and third linear layers, and denote the learnable offsets of the second and third linear layers, o Represents the features after video and text fusion, represents the video features output from DAST, Represents text features, represents the probability of output, Represents the answer set A vector space of size; The loss function of the head network is for: in, Indicates the i The one-hot vector of the correct answer, i represents the answer index, Indicates the i The probability of an answer.

9. The multimodal video language learning method based on video depth information according to claim 1, characterized in that: In S3, the head network includes a fourth linear layer and a fifth linear layer arranged in parallel, and a cosine similarity layer. The inputs of the fourth linear layer and the fifth linear layer are the second feature and the final video feature, respectively, and the output is used as the input of the cosine similarity layer. The specific method is as follows: in, express, represents the learnable parameters of the fourth linear layer, represents the video features output from DAST, represents the learnable offset of the fourth linear layer, represents the learnable parameters of the fifth linear layer, Represents text features, represents the learnable offset of the fifth linear layer, Indicates the similarity between text features and video features; The loss function of the head network is: Represents a video v Similarity with negative sample text, Represents text s Similarity with negative sample videos, Represents the similarity of positive sample video text pairs, represents the regularization term, and Represent the negative sample text and negative sample video in a batch respectively.

Images