A video segment retrieval method and system based on cross-modal correspondence matching and dataset debiasing

Abstract

The present application belongs to the technical field of computer vision and pattern recognition, and particularly relates to a video segment retrieval method and system based on cross-modal corresponding matching and dataset debiasing, and the specific steps of the method are as follows: selecting a feature extraction network; coarse-grained perception; cross-modal corresponding matching; dataset debiasing; selecting representation generation; fine-grained feature encoder; conditional interaction module; and selecting comparison module. The present application takes the segment retrieval as a multiple-choice reading comprehension task, and integrates human reading strategies into the framework, so that better performance is achieved. The present application corrects the wrong matching result based on cross-modal corresponding matching, solves the problem of wrong prediction caused by unbalanced data distribution through dataset debiasing, so as to obtain more accurate video segment retrieval result.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and pattern recognition, and particularly relates to a video segment retrieval method and system based on cross-modal correspondence matching and data set debiasing. BACKGROUND

[0002] In recent years, a large number of videos on the Internet contain a variety of human activities, which is conducive to our perception of the real world. This phenomenon has inspired many popular tasks, such as action recognition, video retrieval, etc. However, most of the online videos are long, untrimmed, and contain irrelevant content, so there is an urgent need for automatic video content analysis methods. Temporal action localization is a task of detecting action instances in untrimmed videos, which ignores the fact that videos are usually accompanied by sentence descriptions. Gao et al. and Hendricks et al. therefore fully utilize the annotated language information and propose the segment retrieval task. Given an untrimmed video and a sentence description, the goal of this task is to retrieve the video moments that are most relevant to the semantic description. This is a meaningful but challenging task because it requires simultaneous understanding of complex video content and sentence context.

[0003] Existing works on this task are generally divided into two categories: two-stage models and one-stage models. Two-stage methods usually follow a proposal-then-ranking procedure and achieve better performance than one-stage methods. Most two-stage models learn self-modal relationships and cross-modal interactions in a coarse-grained manner. They tend to use two independent feature encoders to capture intra-modal information and then utilize attention mechanisms to explore inter-modal information. However, these two-stage methods have two serious drawbacks: (1) they ignore detailed intra-modal and inter-modal information, which needs to be fully understood through a more in-depth reasoning stage. (2) To achieve high recall, two-stage models need to densely generate candidate segments, which is very computationally expensive and time-consuming. One-stage methods avoid the need for predefined candidate segments, which can obtain segments with flexible lengths and perform more efficiently. They directly predict the likelihood of each frame, regardless of whether it is a boundary frame. However, single-stage methods usually perform poorly because they ignore rich segment information, which is important for precise localization.

[0004] Therefore, in order to solve the above problems, a video segment retrieval method and system based on cross-modal correspondence matching and data set debiasing are proposed. SUMMARY

[0005] The purpose of the present application is to solve the problem of ignoring the detailed information within and between modalities in video segment retrieval, and the problem of needing a two-stage model to densely generate candidate segments, and to propose a multi-granularity perception network that combines human reading strategies, including article question re-reading, enhanced article question alignment and selection comparison, so that the model can thoroughly understand the video and query content, thereby achieving good performance.

[0006] The technical solution of the present application solves the technical problem:

[0007] A video segment retrieval method based on cross-modal corresponding matching and dataset debiasing, comprising the following steps:

[0008] Step 1. Select a feature extraction network

[0009] For the input video, select a pre-trained network to extract the features of each segment, and for a given language query, use a pre-trained 300-dimensional vector training model Glove model to encode each word;

[0010] Step 2. Coarse-grained perception

[0011] On the basis of step 1 network, the video and sentence embeddings are fed into a coarse-grained encoder, and a common attention mechanism is used to combine them to obtain a preliminary alignment feature;

[0012] Step 3. Cross-modal corresponding matching

[0013] Use the natural language processing spacy toolkit to separate the nouns and verbs in a sentence, and decompose the video into temporal and spatial features, match the nouns and spatial features, and match the verbs and temporal features;

[0014] Step 4. Dataset debiasing

[0015] Determine the end position of the video instance in the video dataset, if it is less than one-third of the video length, the instance position is biased forward, and one-half of all the biased forward instances are extracted and shifted backward by one-half of the video length;

[0016] Step 5. Selective representation generation

[0017] On the basis of step 2 network, use the preliminary alignment video features to generate content-level and boundary-level segment features;

[0018] Step 6. Fine-grained feature encoder

[0019] On the basis of step 2 network, follow the re-reading strategy used in reading comprehension and deeply encode the video and query features to obtain fine-grained intra-modal features;

[0020] Step 7. Condition interaction module

[0021] On the basis of the network in step 6, the segment features at the content level and the boundary level are combined with the features passing through the fine-grained encoder to obtain an enhanced aligned segment feature;

[0022] Step 8. Selection comparison module

[0023] On the basis of the network in step 7, a selection comparison module is further applied to help distinguish similar segments, and finally a selection sorter is used to retrieve the most relevant video segments in the candidate set.

[0024] The specific operation of step 2 is as follows:

[0025] On the basis of the network selected in step 1, for the extracted video features, a time one-dimensional convolution layer and an average pooling layer are applied to map the features of each segment to the ideal video dimension C and video length T;

[0026] A bidirectional gate recurrent unit GRU is applied to capture the time dependence in the video segment, and for the text features, a bidirectional GRU is sequentially fed to capture the context information in the sentence;

[0027] A multi-modal common attention mechanism is applied to capture the inter-modal context.

[0028] The specific operation of step 3 is as follows:

[0029] First, use the spacy toolkit to separate the verbs and nouns in a sentence, obtain the verb mask and noun mask, then multiply the text features by the verb mask and noun mask to obtain the verb text features and noun text features, then perform position encoding on the features passing through the video encoder to obtain video timing features, subtract the timing features from the total features to obtain video spatial features, and finally calculate an MSE loss between the verb text features and the video timing features, and the noun text features and the video spatial features;

[0030] The language query is q n represents the nth word in the sentence, and L represents the length of the sentence;

[0031] Use the spacy toolkit to analyze the part of speech of the words in the sentence Q to obtain the verb mask mask verb ∈R L×1 and the noun mask mask noun ∈R L×1 , R represents the set of real numbers, and R L×1 represents a real matrix with dimensions Lx1;

[0032] In the coarse-grained feature encoder, the text features passing through the query encoder are The video features passing through the video encoder are Then the text features are multiplied by the verb mask to obtain the verb text features, and the text features are multiplied by the noun mask to obtain the noun text features, R L×C represents a real matrix with dimensions LxC, R C×T represents a real matrix with dimensions CxT.

[0033]

[0034]

[0035] is the verb text feature, is the noun text feature, and represents element-wise multiplication.

[0036] The video features are positionally encoded, and one-dimensional convolution is applied to obtain fine-grained time sequence information feature representation:

[0037]

[0038] P represents the time sequence information feature, PE represents the position encoding, ReLU(·) represents the activation function, BatchNorm1d(·) represents the batch normalization, and Conv1d(·) represents the one-dimensional convolution.

[0039] The video features are subtracted from the time sequence information features to obtain the spatial information features:

[0040]

[0041] S is the spatial information feature.

[0042] After obtaining the verb text features and the time sequence information features, and the noun text features and the spatial information features, they are respectively transformed to the same dimension for loss calculation:

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049] for the transformed verb text feature, for the transformed noun text feature, MSE is the mean square loss function, Loss pv for the loss between the time information feature and the verb text feature, Loss sn for the loss between the spatial information feature and the noun text feature, Sum(·) represents the addition function, softmax(·) represents the normalized exponential function, Linear(·) represents the linear layer, R C represents a real matrix with dimension C;

[0050] Therefore, the total loss of the cross-modal corresponding matching module is:

[0051] L cmm =Loss pv +Losssn.

[0052] The specific operation of step 4 is as follows:

[0053] From step 2, the length of the video is T, and the position of the instance segment in the video is g=(g s ,g e ), if:

[0054] g e <αT

[0055] The position of this instance in the video is considered to be biased to the front, and this segment needs to be moved backward, and the position after displacement is represented as:

[0056]

[0057] Only the total number of instances with positions in the front of the data set is displaced by 1 / 2, so as to achieve uniform distribution of data positions;

[0058] In the above, α is a proportionality coefficient, set α to 1 / 3, γ is a proportionality coefficient, set γ to 1 / 2, g is the position of the instance segment, g s is the start time of the instance segment, g e is the end time of the instance segment, is the position of the instance after displacement;

[0059] Get the original position of the instance g=(g s ,g e ) and the position after displacement After that, the features of the video encoder are shifted accordingly, and the features at position g are moved to position

[0060]

[0061] is a new feature formed after a positional displacement, is a feature vector representation before displacement, is a feature vector representation after displacement.

[0062] The specific operation of step 5 is as follows:

[0063] On the basis of the base network selected in step 2, a two-dimensional time graph is constructed to generate candidate segments, the vertical axis and the horizontal axis represent the start and end segment indexes respectively, and each block in the time graph represents a candidate segment. Following the sparse sampling strategy, the lower triangular part of the two-dimensional time graph is invalid, and part of the upper triangular is masked. The time correlation in the video is fully captured, and the content-level segment features and boundary-level segment features are generated by using the MaxPooling operation and the addition operation respectively.

[0064] The specific operation of step 6 is as follows:

[0065] On the basis of the base network selected in step 2, given the video features of the perception query, a residual feedforward network is designed to explore fine-grained visual clues, and the coarse-grained features are used as the initial perception. For query encoding, three different convolution kernel sizes of one-dimensional time convolution are applied to capture token-level and phrase-level features. Token represents word level, and phrase represents phrase level. Then, the three features are combined and fed into a fully connected layer, and then added to the input query feature through a residual link. The fine-grained feature encoder follows the common process from coarse to fine in reading comprehension, and obtains fine-grained video features and query features.

[0066] The specific operation of step 7 is as follows:

[0067] On the basis of the base network selected in step 6, after obtaining the fine-grained intra-modal features, they are combined with the segment features to further enhance the inter-modal feature representation. A symmetric interaction is established between the video and the sentence. In this module, the query perception branch and the video perception branch are used to learn the query perception segment representation and the video perception segment representation respectively by using the designed gate mechanism. In the query perception branch, MaxPooling is first applied on the query representation to find the token with the largest contribution in each feature dimension, and a fully connected layer is used to map it to the segment-level feature space. The boundary-level segment features are combined with the transferred query features, and the sigmoid activation function is used as a gate function to capture the semantic related boundary information. The gated query features and the content-level segment features are further aggregated to explore the query-related information contained in the candidate segment. The video perception branch is processed in a similar way to obtain the video perception segment features. Finally, the two conditional features are integrated to obtain fine-grained alignment segment features.

[0068] Step 8 is specifically operated as follows:

[0069] On the basis of the step 7 base network selection, first, the fine-grained aligned feature map Content-level feature map A C And boundary-level feature map A B Combined, the relevance of the two is integrated, then a skip connection is used, and a convolution operation with ReLU activation function is applied to obtain enhanced fusion features, and four convolution blocks are stacked on the fusion features, each convolution block is composed of a group convolution layer, a batch normalization and a ReLU activation function, after the fusion of the segment features and the comparison information of all candidate segments, the complex temporal correlation between different candidate segments is captured; a retrieval score is generated for each candidate segment, and it is sorted to make a decision; after capturing the relationship between different candidate segments, the relationship-aware features are sent to the convolution layer, and a Sigmoid nonlinear function is used to generate the matching score of all candidate segments, and finally a selection sorter is used to retrieve the most relevant video segments in the candidate set.

[0070] Video loss exists in the video retrieval process, and the definition of video loss is as follows:

[0071] First, the training sample is a three-tuple (V, Q, A), including an input video V, an input language query Q and a real segment set A, for each candidate segment Calculate the IoU score o s between it and the real segment (g e , g i ), and adjust the IoU score o min with two thresholds θ max and θ i , represents the start time of the i-th candidate segment, represents the end time of the i-th candidate segment; g s represents the start time of the instance segment, g e represents the end time of the instance segment; IoU represents the intersection over union; θ min represents the minimum threshold set, θ max represents the maximum threshold set;

[0072]

[0073] y i is used as a supervision label, which is the adjusted IoU score. Finally, an alignment loss is used to align the predicted confidence score and the adjusted IoU, which is defined as follows:

[0074]

[0075] p i is the output score of answer choice a i , subscript i represents the i-th, a i represents the i-th candidate segment, p i represents the output score of the i-th candidate segment, N A is the total number of valid candidate segments;

[0076] Therefore, the total loss is:

[0077] L = L cmm + L align

[0078] L cmm is the cross-modal correspondence matching loss, L align is the alignment loss.

[0079] The application also provides a video segment retrieval system based on cross-modal correspondence matching and dataset debiasing, characterized by comprising the following modules:

[0080] Coarse-grained perception module: combining video and sentence through a coarse-grained encoder to obtain a preliminary alignment feature;

[0081] Cross-modal correspondence matching module: error correction of the matching result is achieved by separating nouns and verbs using the spacy toolkit;

[0082] Dataset debiasing module: according to the end position of the instance in the video, it is judged whether the prediction error is caused by unbalanced data distribution, and the error is corrected;

[0083] Selection representation generation module: a two-dimensional time graph is constructed to generate candidate segments, and content-level and boundary-level segment features are generated to comprehensively capture the time correlation in the video;

[0084] Fine-grained feature encoder module: fine-grained visual clues are explored through the design of a residual feedforward network, and fine-grained intra-modal features are obtained through a fine-grained feature encoder;

[0085] Conditional interaction module: the content-level and boundary-level segment features are combined with the features obtained through the fine-grained encoder in the previous module to obtain an enhanced alignment segment feature;

[0086] Selection comparison module: helps to distinguish similar segments, and retrieves the most relevant video segments in the candidate set through a selection sorter.

[0087] The above technical solution has the following advantages or beneficial effects:

[0088] 1) Existing methods globally match video features and text features, which may lead to incorrect matching results. However, this invention avoids incorrect matching results by matching video features and query features locally through cross-modal correspondence matching, and can achieve correct matching results.

[0089] 2) In existing datasets, most instances are concentrated at the beginning of the video, causing the predicted segments to also be positioned forward after model training, resulting in inaccurate predictions. By partially shifting the instances positioned forward in the dataset, the distribution of instances in the dataset becomes more uniform, solving the dataset bias problem and thus obtaining more accurate predictions. Attached Figure Description

[0090] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0091] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0092] Figure 2 This is an example of the Charades-STA dataset in Embodiment 1 of the present invention. Detailed Implementation

[0093] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0094] Example 1

[0095] like Figure 1 As shown, a video segment retrieval method based on cross-modal correspondence matching and data solution bias includes the following steps:

[0096] Step 1. Select a feature extraction network

[0097] For the input video, features of each segment are extracted by selecting a pre-trained network. For a given language query, a pre-trained 300-dimensional vector-trained GloVe model is used to encode each word.

[0098] Step 2. Coarse-grained perception

[0099] Based on the network in step 1, the embeddings of video and sentences are fed into a coarse-grained encoder, and a common attention mechanism is used to combine them to obtain a preliminary alignment feature.

[0100] Step 3. Cross-modal correspondence matching

[0101] Using natural language processing spacy toolkit to separate the nouns and verbs in a sentence, and decompose the video into temporal and spatial features, match the nouns and spatial features, and match the verbs and temporal features;

[0102] Step 4. Dataset debiasing

[0103] Determine the end position of the video instance in the video dataset. If it is less than one third of the video length, the instance is biased forward. Extract half of the biased forward instances and shift them back by half of the video length.

[0104] Step 5. Representation generation

[0105] On the basis of the network in step 2, use the preliminarily aligned video features to generate content-level and boundary-level segment features;

[0106] Step 6. Fine-grained feature encoder

[0107] On the basis of the network in step 2, follow the re-reading strategy used in reading comprehension and deeply encode video and query features to obtain fine-grained intra-modal features;

[0108] Step 7. Conditional interaction module

[0109] On the basis of the network in step 6, combine the content-level and boundary-level segment features with the features after the fine-grained encoder to obtain an enhanced aligned segment feature;

[0110] Step 8. Selection comparison module

[0111] On the basis of the network in step 7, further apply a selection comparison module to help distinguish similar segments, and finally retrieve the most relevant video segments in the candidate set through a selection sorter.

[0112] The specific operation of step 2 is as follows:

[0113] On the basis of the network selected in step 1, for the extracted video features, apply a time one-dimensional convolution layer and average pooling layer to map the features of each segment to the ideal video dimension C and video length T;

[0114] Apply a bidirectional gated recurrent unit (GRU) to capture the temporal dependencies in the video segment. For the text features, sequentially feed them into the bidirectional GRU to capture the context information in the sentence.

[0115] Apply a multi-modal common attention mechanism to capture the inter-modal context.

[0116] The specific operation of step 3 is as follows:

[0117] First, the spacy toolkit is used to separate the verbs and nouns in a sentence, and after obtaining the verb mask and noun mask, the verb text feature and noun text feature are obtained by multiplying the text feature with the verb mask and noun mask respectively, then the position encoding is performed on the feature obtained by the video encoder to obtain the video timing feature, the total feature is subtracted from the timing feature to obtain the video space feature, and finally an MSE loss is calculated between the verb text feature and the video timing feature, and between the noun text feature and the video space feature;

[0118] The language query is q n represents the nth word in the sentence, and L represents the length of the sentence;

[0119] The part of speech of the words in the sentence Q is analyzed using the spacy toolkit to obtain the verb mask mask verb ∈R L×1 and the noun mask mask noun ∈R L×1 , R represents the set of real numbers, and R L×1 represents a real matrix with a dimension of Lx1;

[0120] In the coarse-grained feature encoder, the text feature obtained by the query encoder is The video feature obtained by the video encoder is Then the text feature is multiplied by the verb mask to obtain the verb text feature, and the text feature is multiplied by the noun mask to obtain the noun text feature, R L×C represents a real matrix with a dimension of LxC, and R C×T represents a real matrix with a dimension of CxT;

[0121]

[0122]

[0123] is the verb text feature, is the noun text feature, and ⊙ represents element-wise multiplication;

[0124] The video feature is positionally encoded, and a one-dimensional convolution is applied to obtain fine-grained timing information feature representation:

[0125]

[0126] P represents the timing information feature, PE represents the position encoding, ReLU(·) represents the activation function, BatchNorm1d(·) represents the batch normalization, and Conv1d(·) represents the one-dimensional convolution;

[0127] The spatial information feature is obtained by subtracting the timing information feature from the video feature:

[0128]

[0129] S is spatial information feature;

[0130] After the verb text feature and the time sequence information feature, the noun text feature and the spatial information feature are obtained, they are transformed to the same dimension for loss calculation:

[0131]

[0132]

[0133]

[0134]

[0135]

[0136]

[0137] For the transformed verb text feature, For the transformed noun text feature, MSE is the mean square loss function, Loss pv Loss is the loss between the time sequence information feature and the verb text feature, Loss sn Loss is the loss between the spatial information feature and the noun text feature, Sum(·) represents the addition function, softmax(·) represents the normalized exponential function, Linear(·) represents the linear layer, R C represents a real matrix with dimension C;

[0138] Therefore, the total loss of the cross-modal corresponding matching module is:

[0139] L cmm =Loss pv +Loss sn .

[0140] The specific operation of step 4 is as follows:

[0141] From step 2, the length of the video is T, and the position of the instance segment in the video is g=(g s ,g e ), if:

[0142] g e <αT

[0143] then it is considered that the position of this instance in the video is biased to the front, and this segment needs to be moved backward, and the position after the displacement is represented as:

[0144]

[0145] Only half of the total number of instances that are centrally located in the data are shifted to achieve a more uniform data distribution.

[0146] In the above, α is the scaling factor, set to 1 / 3; γ is the scaling factor, set to 1 / 2; and g is the instance fragment position. s g is the start time of the instance fragment. e This is the end time of the instance segment. This represents the instance position after displacement;

[0147] Get the original position of the instance g = (g s ,g e ) and position after displacement Then, the features after passing through the video encoder are processed. Perform the corresponding feature shift, moving the feature at position g to position g.

[0148]

[0149] These are new features formed after positional displacement. This represents the eigenvectors before the displacement. This represents the eigenvectors after displacement.

[0150] The specific steps for step 5 are as follows:

[0151] Based on the basic network selection in step 2, a two-dimensional time map is constructed to generate candidate segments. The vertical and horizontal axes represent the start and end segment indices, respectively. Each block in the time map represents a candidate segment. Following the sparse sampling strategy, the lower triangular part of the two-dimensional time map is invalid, and some upper triangular parts are masked to fully capture the temporal correlation in the video. Max pooling and addition operations are used to generate content-level segment features and boundary-level segment features, respectively.

[0152] The specific steps for step 6 are as follows:

[0153] On the basis of step 2, the video features of the given perception query are given, and a residual feedforward network is designed to explore fine-grained visual clues and take coarse-grained features as the initial perception. For query encoding, three different convolution kernel sizes are applied to capture token-level and phrase-level features. Token represents word-level, and phrase represents phrase-level. Then, the three features are combined and fed into a fully connected layer, and a residual link is added to the input query feature. The fine-grained feature encoder follows the common process from coarse to fine in reading comprehension to obtain fine-grained video and query features.

[0154] The specific operation of step 7 is as follows:

[0155] On the basis of step 6, after obtaining fine-grained intra-modal features, they are combined with segment features to further enhance inter-modal feature representation. A symmetric interaction is established between video and sentence. In this module, there are two branches: query perception branch and video perception branch. The designed gate mechanism is used to learn query perception segment representation and video perception segment representation respectively. In the query perception branch, MaxPooling is first applied to the query representation to find the token with the largest contribution in each feature dimension. A fully connected layer is used to map it to the segment-level feature space. The boundary-level segment feature is combined with the transferred query feature, and the sigmoid activation function is used as the gate function to capture the boundary information related to semantics. The gated query feature and the content-level segment feature are further aggregated to explore the query-related information contained in the candidate segment. The video perception branch is processed in a similar way to obtain the video perception segment feature. Finally, the two conditional features are integrated to obtain fine-grained aligned segment features.

[0156] The specific operation of step 8 is as follows:

[0157] On the basis of step 7, the fine-grained aligned feature map is first divided into content-level feature map A and boundary-level feature map A. Content-level feature map A C and boundary-level feature map A BThe algorithm combines these features to integrate their correlations, then uses a skip connection and applies convolutional operations with ReLU activation to obtain enhanced fused features. A total of four convolutional blocks are stacked on the fused features to perceive more contextual information from adjacent candidate segments. Each convolutional block consists of a group convolutional layer, a batch normalization, and a ReLU activation function. After fusing the segment features of all candidate segments with comparative information, the algorithm captures the complex temporal correlations between different candidate segments. A retrieval score is generated for each candidate moment and they are ranked to make a decision. After capturing the relationships between different candidate moments, the relationship-aware features are fed into a convolutional layer, and a sigmoid nonlinear shape function is used to generate matching scores for all candidate segments. Finally, a selection sorter is used to retrieve the most relevant video segment in the candidate set.

[0158] Video loss occurs during video retrieval, and the definition of video loss is as follows:

[0159] The training sample is a triple (V, Q, A), consisting of an input video V, an input language query Q, and a set of ground truth segments A. For each candidate segment in the answer set... Calculate it and the real fragment (g) s ,g e IoU score between ) i And using two thresholds θ min and θ max To adjust the IoU score i , Indicates the start time of the i-th candidate segment. Indicates the end time of the i-th candidate segment; g s Indicates the start time of the instance fragment, g e Indicates the end time of the instance segment; IoU represents the intersection-union ratio; θ min θ represents the set minimum threshold. max This indicates the maximum threshold value set.

[0160]

[0161] y i The adjusted IoU score is used as the supervision label. Finally, an alignment loss is used to align the predicted confidence score with the adjusted IoU, defined as follows:

[0162]

[0163] p i The answer is a. i The output score is given by the index i, where i represents the i-th score. i Let p represent the i-th candidate segment.i is the output score of the i-th candidate segment, N A is the total number of valid candidate segments;

[0164] Thus the total loss is:

[0165] L = L cmm + L align

[0166] L cmm is the cross-modal correspondence matching loss, L align is the alignment loss.

[0167] The comparison of the experimental effects of the present application and other methods on the Charades-STA dataset, the TACoS dataset and the ActivityNet Captions dataset is shown in the following table:

[0168]

[0169]

[0170] The methods used in the table are CTRL (Cross-modal Temporal Regression Localizer), ACRN (Attentive Cross-Modal Retrieval Network), QSPN (query-guided segment proposal network), CMIN (Cross-modal inter-action networks), 2D-TAN (2D Temporal Adjacent Network), DPIN (Dual path interaction network), FIAN (Fine-grained iterative attention network), BPNet (Boundary Proposal Network), and RaNet (Relation-aware Network).

[0171] On the temporal segment retrieval task, the proposed method is compared with some published works, where CPNet (Contextual Pyramid Network), DRN (Dense Regression Network), CI-MHA (Cross Interaction Multi-headed Attention Network) and VSLNet (video span localizing network) belong to one-stage models without proposal, and other works belong to two-stage models based on proposal. The "Rank@n, IoU=m" is used as the evaluation indicator, and "Rank@n, IoU=m" is defined as the percentage of at least one of the top n selected segments whose IoU is greater than m, and Rank represents the selection order. For the TACoS dataset, the results are reported with n e {1, 5} and m e {0.3, 0.5}, and for the Charades-STA dataset and the ActivityNet Captions dataset, the results are reported with n e {1, 5} and m e {0.5, 0.7}. Overall, the proposed method is superior to the latest method on three challenging benchmarks, ranking first or second in all evaluation indicators.

[0172] By Figure 2 It can be seen that the present application can accurately retrieve the most relevant segments to the language query, even if these segments are visually similar.

[0173] Embodiment 2

[0174] The present application also provides a video segment retrieval system based on cross-modal correspondence matching and dataset debiasing, comprising the following modules:

[0175] Coarse-grained perception module: combine the video and sentence through a coarse-grained encoder to obtain a preliminary alignment feature;

[0176] Cross-modal correspondence matching module: correct the error matching result by separating nouns and verbs using the spacy toolkit;

[0177] Dataset debiasing module: determine whether the prediction error is caused by unbalanced data distribution according to the end position of the instance in the video, and correct the error;

[0178] Selection representation generation module: generate candidate segments by constructing a two-dimensional time graph, and generate content-level and boundary-level segment features to comprehensively capture the time correlation in the video;

[0179] Fine-grained feature encoder module: explore fine-grained visual cues by designing a residual feed-forward network, and obtain fine-grained intra-modal features through a fine-grained feature encoder;

[0180] Conditional interaction module: combine the content-level and boundary-level segment features with the features obtained through the fine-grained encoder in the previous module to obtain an enhanced aligned segment feature;

[0181] Selection comparison module: helps to distinguish similar segments and retrieves the most relevant video segments in the candidate set through a selection sorter.

[0182] Embodiment 3

[0183] In the embodiments of the present application, a computer readable storage medium storing a computer program is also provided, wherein when the computer program is executed by a processor, the video segment retrieval method is realized.

[0184] Embodiment 4

[0185] In the embodiments of the present application, a computing device is also provided, comprising: at least one processor; at least one memory storing a computer program, when the computer program is executed by the at least one processor, the video segment retrieval method is realized.

[0186] The present application realizes a video segment retrieval method and system based on cross-modal corresponding matching and data de-set bias through the above technical solutions. By cross-modal corresponding matching, the local matching video features and the query features are avoided to obtain correct matching results. By partially displacing the instances at the front position in the data set, the instance distribution position in the data set is uniform, the data set bias problem is solved, and more accurate prediction results can be obtained.

[0187] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A video segment retrieval method based on cross-modal correspondence matching and data solution bias, characterized by: Includes the following steps: Step 1. Select a feature extraction network For the input video, features of each segment are extracted by selecting a pre-trained network. For a given language query, a pre-trained 300-dimensional vector-trained GloVe model is used to encode each word. Step 2. Coarse-grained perception Based on the network in step 1, the embeddings of video and sentences are fed into a coarse-grained encoder, and a common attention mechanism is used to combine them to obtain a preliminary alignment feature. Step 3. Cross-modal correspondence matching The Spacy toolkit for natural language processing was used to separate nouns and verbs in a sentence, and the video was decomposed into temporal and spatial features. Nouns were matched with spatial features, and verbs were matched with temporal features. The specific steps for step 3 are as follows: First, the spacy toolkit is used to separate verbs and nouns in a sentence to obtain verb masks and noun masks. Then, the verb masks and noun masks are multiplied by the text features to obtain verb text features and noun text features, respectively. Next, the features passed through the video encoder are positionally encoded to obtain video temporal features. The total features are subtracted from the temporal features to obtain video spatial features. Finally, an MSE loss is calculated between verb text features and video temporal features, and between noun text features and video spatial features. Language query is , This represents the nth word in the sentence. Represents the length of the sentence; Analyze sentences using the spacy toolkit Determine the part of speech of a word and obtain the verb mask. and noun mask R represents the set of real numbers. The dimension is A real number matrix; In a coarse-grained feature encoder, the text features after querying the encoder are: The video features after passing through the video encoder are Then, the text features are multiplied by the verb mask to obtain the verb text features, and the text features are multiplied by the noun mask to obtain the noun text features. The dimension is Real number matrices, The dimension is A real number matrix; = ⊙ = ⊙ For verb text features, For noun text features, ⊙ represents element-wise multiplication; The video features are positionally encoded, and one-dimensional convolution is applied to obtain fine-grained temporal information feature representations: Represents time-series information characteristics, Represents position code, This represents the activation function. Indicates batch normalization. This represents one-dimensional convolution; Subtracting temporal information features from video features yields spatial information features: Spatial information features; After obtaining verb text features and temporal information features, and noun text features and spatial information features, we transform them to the same dimension for loss calculation: The mean squared loss function, The loss is between temporal information features and verb text features. The loss between spatial information features and noun text features, This represents an addition function. Represents the normalized exponential function, Indicates a linear layer. The dimension is A real number matrix; Therefore, the total loss of the cross-modal correspondence matching module is: ； Step 4. Data Decomposition Bias Determine the end position of video instances in the video dataset. If it is less than one-third of the video length, the instance is positioned too far forward. Extract half of all instances that are positioned too far forward and shift them backward by half the video length. Step 5. Select to generate representation Based on the network in step 2, the preliminarily aligned video features are used to generate content-level and boundary-level segment features; Step 6. Fine-grained feature encoder Building upon the network from step 2, fine-grained intramodal features are obtained by employing rereading strategies in reading comprehension and deeply encoding video and query features. Step 7. Conditional Interaction Module Based on the network in step 6, the content-level and boundary-level fragment features are combined with the features processed by the fine-grained encoder to obtain an enhanced aligned fragment feature. Step 8. Select the comparison module Building upon the network in step 7, a selection comparison module is further applied to help distinguish similar segments, and finally a selection sorter is used to retrieve the most relevant video segments in the candidate set.

2. The video segment retrieval method based on cross-modal correspondence matching and data decomposition bias according to claim 1, characterized in that: The specific steps for step 2 are as follows: Based on the basic network selection in step 1, for the extracted video features, a temporal one-dimensional convolutional layer and an average pooling layer are applied to map the features of each segment to the ideal video dimension C and video length T; A bidirectional gated recurrent unit (GRU) is applied to capture the temporal dependencies in video clips. For text features, they are sequentially fed into the bidirectional GRU to capture the contextual information in sentences. A multimodal joint attention mechanism is applied to capture the context between modalities.

3. The video segment retrieval method based on cross-modal correspondence matching and data decomposition bias according to claim 1, characterized in that: The specific steps for step 4 are as follows: From step 2, we can obtain the length of the video as T, and the position of the instance segment in the video is represented as... ,if: This means that the instance is positioned too far forward in the video and needs to be moved backward. The new position is represented as follows: Only half of the total number of instances that are centrally located in the data are shifted to achieve a more uniform data distribution. In the above, Set as the scaling factor It is 1 / 3. Set as the scaling factor It is 1 / 2. For instance fragment position, The start time of the instance fragment. This is the end time of the instance segment. This represents the instance position after displacement; Get the original location of the instance Position after displacement Then, the features after passing through the video encoder are processed. Perform the corresponding feature shift to change the position. Features on the position : These are new features formed after positional displacement. This represents the eigenvectors before the displacement. This represents the eigenvectors after displacement.

4. The video segment retrieval method based on cross-modal correspondence matching and data decomposition bias according to claim 1, characterized in that: The specific steps for step 5 are as follows: Based on the basic network selection in step 2, a two-dimensional time map is constructed to generate candidate segments. The vertical and horizontal axes represent the start and end segment indices, respectively. Each block in the time map represents a candidate segment. Following the sparse sampling strategy, the lower triangular part of the two-dimensional time map is invalid, and some upper triangular parts are masked to fully capture the temporal correlation in the video. Max pooling and addition operations are used to generate content-level segment features and boundary-level segment features, respectively.

5. The video segment retrieval method based on cross-modal correspondence matching and data decomposition bias according to claim 1, characterized in that: The specific steps for step 6 are as follows: Building upon the basic network selection in step 2, given the video features of the perceptual query, a residual feedforward network is designed to explore fine-grained visual cues for interactive purposes. Coarse-grained features are used as initial perception. For query encoding, three temporal one-dimensional convolutions with different kernel sizes are applied to capture token-level and phrase-level features, where token represents word-level and phrase represents phrase-level. These three features are then combined and fed into a fully connected layer, and finally added to the input query features using a residual connection. This fine-grained feature encoder follows the common coarse-to-fine process in reading comprehension to obtain fine-grained video and query features.

6. The video segment retrieval method based on cross-modal correspondence matching and data decomposition bias according to claim 1, characterized in that: Step 7 is performed as follows: Building upon the basic network selection in step 6, after obtaining fine-grained intra-modal features, these are combined with segment features to further enhance inter-modal feature representation and establish symmetrical interaction between videos and sentences. This step is divided into two branches: a query-aware branch and a video-aware branch. The designed gating mechanism is used to learn query-aware segment representations and video-aware segment representations, respectively. In the query-aware branch, MaxPooling is first applied to the query representation to find the token that contributes the most in each feature dimension, and then a fully connected layer is used to map it to the segment-level feature space. The boundary-level segment features are combined with the transitioned query features, and the sigmoid activation function is used as a gating function to capture semantically relevant boundary information. The gated query features and content-level segment features are further aggregated to explore query-related information contained in candidate segments. The video-aware branch is processed in a similar way to obtain video-aware segment features. Finally, these two conditional features are integrated to obtain fine-grained aligned fragment features.

7. The video segment retrieval method based on cross-modal correspondence matching and data decomposition bias according to claim 1, characterized in that: Step 8 is performed as follows: Based on the basic network selection in step 7, the fine-grained aligned feature maps are first... Content-level feature maps and boundary level feature maps The features are combined to integrate their correlations, and then a skip connection is used to apply convolutional operations with ReLU activation to obtain enhanced fused features. A total of four convolutional blocks are stacked on the fused features to perceive more contextual information from adjacent candidate segments. Each convolutional block consists of a group convolutional layer, a batch normalization, and a ReLU activation function. After fusing the segment features of all candidate segments with comparative information, the complex temporal correlations between different candidate segments are captured. A retrieval score is generated for each candidate segment and they are ranked to make a decision. After capturing the relationships between different candidate moments, the relationship-aware features are fed into a convolutional layer, and a sigmoid non-linear shape function is used to generate matching scores for all candidate segments. Finally, a selection sorter is used to retrieve the most relevant video segment in the candidate set.

8. The video segment retrieval method based on cross-modal correspondence matching and data decomposition bias according to claim 1, characterized in that: Video loss occurs during video retrieval, and the definition of video loss is as follows: The training sample is a triplet. Including an input video A single input language query and a real collection of clips For each candidate fragment in the answer set Calculate its relationship with the real fragment. IoU score between And using two thresholds and To adjust the IoU score , Indicates the start time of the i-th candidate segment. Indicates the end time of the i-th candidate segment; Indicates the start time of the instance fragment. Indicates the end time of the instance segment; IoU represents the intersection-union ratio; This indicates the set minimum threshold. This indicates the maximum threshold value set. The adjusted IoU score is used as the supervision label. Finally, an alignment loss is used to align the predicted confidence score with the adjusted IoU, defined as follows: Answer selection The output score is given by the index i, where i represents the i-th score. This represents the i-th candidate segment. This represents the output score of the i-th candidate segment. It represents the total number of valid candidate segments; Therefore, the total loss is: For cross-modal correspondence matching loss, For alignment loss.

9. A video segment retrieval system based on cross-modal correspondence matching and data decomposition bias, executing the video segment retrieval method based on cross-modal correspondence matching and data decomposition bias as described in any one of claims 1 to 8, characterized in that: Includes the following modules: Coarse-grained perception module: Combines video and sentences through a coarse-grained encoder to obtain a preliminary alignment feature; Cross-modal correspondence matching module: Corrects erroneous matching results by separating nouns and verbs using the spacy toolkit; Data set decomposition bias module: Determines whether the prediction error is caused by data imbalance based on the end position of the instance in the video, and corrects the error; Selection Representation Generation Module: Generates candidate segments by constructing a two-dimensional time map, generating content-level and boundary-level segment features to comprehensively capture the temporal correlations in the video; Fine-grained feature encoder module: Explores fine-grained visual cues by designing a residual feedforward network, and obtains fine-grained intramodal features through a fine-grained feature encoder; Conditional Interaction Module: Combines content-level and boundary-level fragment features with the features from the fine-grained encoder in the previous module to obtain an enhanced aligned fragment feature. The selection comparison module helps distinguish similar segments and uses a selection sorter to retrieve the most relevant video segments in the candidate set.

Images