Cross-modal video segment retrieval method based on pre-training language model adaptation network

Abstract

The application discloses a cross-modal video text segment retrieval method of a pre-training language model adaptation network. The steps of the application are as follows: 1, data preprocessing and data set division, 2, using a pre-trained network to extract features from a video, 3, constructing a video feature sequence and a candidate segment feature sequence, 4, constructing a text feature sequence, 5, constructing a pre-training language model, 6, constructing a segment retrieval adaptation network, 7, a loss function, 8, training a model, and 9, network prediction value calculation. The application proposes a cross-modal video text segment retrieval method based on a pre-training language model adaptation network, simultaneously adopts two kinds of representation methods of a video frame and a candidate segment, realizes the simultaneous interaction of text features, frame features and video features in one forward propagation, fully excavates the potential connection between video and text information, and has great performance improvement compared with a traditional two-stage mode adopting frame regression and candidate segment sorting.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and natural language processing. This invention proposes a cross-modal video moment retrieval method based on a pre-trained language model adaptation network. Background Technology

[0002] Multimodal retrieval refers to returning samples from other modalities that are semantically related to the query sample, given a query example in one modality. Unlike single-modal research, multimodal tasks often require the participation of multiple modalities and simultaneous analysis of these modalities to obtain the final result. Therefore, multimodal models first need to correctly understand the content of each individual modality. Secondly, since information from different modalities often has significant semantic gaps, multimodal models need to analyze the features and relationships of multiple modalities to effectively model and interact with multimodal information. In real life, we often encounter information from multiple modalities simultaneously when receiving information; therefore, compared to single-modal tasks, multimodal tasks are more suitable for the needs of real-world scenarios.

[0003] Cross-modal video-text retrieval is a multimodal task involving both video and text modalities, and is an interdisciplinary research project between computer vision and natural language processing. This task aims to enable the model to automatically locate the specific video segment most relevant to a given text description within a complete video, based on the text description provided by the user. The input to this task consists of video and text information. First, the information from both modalities needs to be encoded to obtain feature representations. Second, the semantic relevance between the two modalities needs to be analyzed and predicted. Cross-modal video-text retrieval has a wide range of practical applications in life. For example, (1) in entertainment, one application scenario is that during video playback, users can automatically jump to video segments of interest by inputting sentences, avoiding the tedious manual selection process and greatly improving the user experience. (2) Another entertainment application scenario is that users can use text to describe scenes of interest, and the machine can automatically and quickly find multiple video segments related to the given text and create a personal video collection, adding to the user's enjoyment of watching videos. (3) In terms of intelligent security, the system can be set up to quickly retrieve illegal and disciplinary behaviors, such as "a person wearing a helmet is beating a passerby" or "a person in red is robbing someone and then fleeing on a bicycle." The corresponding time sequence location can be obtained by directly inputting the video into the model. This application can reduce the time complexity and tediousness of manually reviewing surveillance, and more efficiently ensure social stability and harmony.

[0004] In summary, cross-modal video segment retrieval is a topic worthy of in-depth research. This patent aims to explore this topic from several key aspects of the task, address the difficulties and key points of current methods, and form a complete cross-modal video segment retrieval system.

[0005] The labels for scene graphs typically come from different annotators, exhibiting a high degree of freedom and lacking unified, fixed semantic information. Furthermore, images in natural scenes often feature diverse themes, complex and varied content, and significant differences between targets, posing a substantial challenge to scene graph generation.

[0006] For the task of cross-modal video segment retrieval, the research focuses on exploring the potential correlations between video and text modalities, that is, mining the semantic relevance between information through the interaction of multimodal information. However, compared with image-text localization, video modalities have very obvious temporal correlations and contain rich action information. Therefore, cross-modal video segment retrieval tasks must also consider the challenges of representing the spatiotemporal evolution of video clips. In summary, there are two main difficulties:

[0007] (1) How to effectively model target contextual information: Video modalities have strong temporal correlations, typically exhibiting strong correlations between adjacent temporal positions and word order. Videos can be understood as a combination of images with temporal correlations. For example, if the current event is "a basketball in the air," then the previous event was "throwing the ball," which could be "the basketball bouncing and falling." Therefore, fully utilizing the temporal information in videos can lead to a better understanding of them. Thus, how to model video temporal information is a crucial issue. Text modalities also have strong sequential correlations, with rich contextual information between words. How to obtain the correlation at each step of the text is also an important question. However, current methods, by focusing excessively on intermodal interactions, lose the fine-grained information of the individual modalities themselves.

[0008] (2) How to model the semantic information of the two modalities at a deeper level to better obtain information that is helpful for cross-modal alignment: Most existing methods model video context information and text context information separately, and then stack them deep within the interaction modules to align cross-modal information. A potential problem with this approach is that useful information for cross-modal interaction may be lost when modeling a single modality. For example, in describing "a person picks up a water glass for the second time," text modal information modeling may weaken the emphasis on the semantic information of "second time," resulting in a large number of "a person picks up a water glass" segments being retrieved in subsequent cross-modal information interactions. Similarly, in describing "a person wearing a white hat raises their right hand," video modal information modeling may weaken the emphasis on the semantic information of "right hand," leading to the inability to align the given description with the target segment pair in subsequent cross-modal information interactions. Therefore, we need a way to simultaneously model intra-modal and inter-modal information in a single model, capture the correlation between different modalities, explore intra-modal features and correlations, and improve the ability to align cross-modal information.

[0009] [1]Devlin J,Chang MW,Lee K,et al.Bert:Pre-training of deepbidirectional transformers for language understanding[J].arXiv preprintarXiv:1810.04805,2018. [2]Liu Y,Ott M,Goyal N,et al.Roberta:A robustly optimized bertpretraining approach[J].arXiv preprint arXiv:1907.11692,2019.

[0010] [3]Tran D,Bourdev L,Fergus R,et al.Learning spatiotemporal features with 3dconvolutional networks[C] / / Proceedings of the IEEE internationalconference on computer vision.2015:4489-4497.

[0011] [4]Carreira J, Zisserman A. Quo vadis, action recognition? a new model and the kinetics dataset[C] / / proceedings of the IEEE Conference on ComputerVision and Pattern Recognition.2017:6299-6308. Summary of the Invention

[0012] The problem this invention aims to solve is that existing cross-modal video segment retrieval methods generally employ a two-stage approach: first, candidate segments are generated through regression of video frame feature sequences; then, a score is calculated through the interaction of candidate segment features and text features, and the candidate segments are ranked based on the score. This two-stage approach has the following problems: 1. Regardless of whether it's the interaction between video frame features and text features or the interaction between candidate segment features and text features, the cross-modal interaction modules are shallowly stacked, and there is no deep interaction between the information of the two different modal features, failing to fully explore the potential semantic connections between the two modal features. 2. Each stage only involves the interaction of one type of video feature and text feature, severing the connection between video frame features and candidate segment features, failing to fully explore the potential semantic connections among video frame features, candidate segment features, and text features.

[0013] To address the aforementioned issues, this invention proposes a cross-modal video segment retrieval method based on a pre-trained language model adaptation network. This method mainly comprises two aspects: 1. Concatenating video frame feature sequences and text feature sequences together and simultaneously modeling them using a pre-trained language model. Utilizing the reasoning capabilities obtained from the pre-trained language model, the connections between the contextual information of the two modalities are fully explored. Typically, pre-trained language models deeply stack self-attention modules, thereby achieving deep interaction between modalities. 2. Based on the adaptation network method for candidate segment features, interaction between candidate segment features and intermediate layer features of the model can be interspersed within the pre-trained language model. Considering the input of the pre-trained language model described in point 1, frame-by-frame and word-by-word interaction between candidate segment features, video frame features, and text features is achieved, fully exploring the potential discriminative information between different candidate segments, which is more beneficial for interaction with text features.

[0014] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0015] Step 1: Data preprocessing and dataset partitioning

[0016] First, the video is preprocessed, and then the dataset is divided.

[0017] Step 2: Extract features from the video using a pre-trained network.

[0018] For an input video, it is divided into multiple video segments with a fixed duration. Then, a network is used to extract semantic features from each video segment. Combining the features of all video segments yields the complete features of the input video.

[0019] Step 3: Construct video feature sequences and candidate segment feature sequences

[0020] Since the duration of the input videos varies, the feature lengths extracted in step 2 also vary. Therefore, we need to uniformly sample the original video features to ensure that the feature sequence length input into the model for each video is consistent, i.e., the input model feature sequence is... Simultaneously, based on the video feature sequence V, candidate segment features f at different time scales are constructed, and all generated candidate segment features are concatenated to obtain the candidate segment feature sequence F. ad Finally, based on the video duration T and the input feature sequence length N... v The correspondence between the time boundary of the target segment and the start and end numbers of the feature sequence is calculated.

[0021] Step 4: Constructing the text feature sequence

[0022] The input text is segmented into words, then transformed into word vectors containing semantic information using a pre-trained word vector model. All the resulting word vectors are then concatenated to obtain the feature sequence of the input text.

[0023] Step 5: Construct a pre-trained language model

[0024] The text feature sequence Q and the video feature sequence V are concatenated to form the final input feature sequence. The feature sequence is input into a pre-trained language model, which fuses intra-modal and inter-modal information from text and video using a self-attention mechanism, and infers the video segment sequence corresponding to its semantic content from the text.

[0025] Step 6: Construct a fragment retrieval adaptation network

[0026] The feature sequence is input into the fragment retrieval adaptation network. The fragment retrieval adaptation network interacts with the pre-trained language model in each stage and step 5 to semantically align the candidate fragment features with text features and video features.

[0027] Step 7: Loss Function

[0028] The video feature sequence output in step 5 is used to calculate the time span loss function, and the candidate segment feature sequence output in step (6) is recombined into a two-dimensional adjacent time relationship graph for calculating the loss function of the candidate segment.

[0029] Step 8: Train the model

[0030] Based on the loss value generated by the loss function in step 7, the backpropagation algorithm is used to backpropagate the gradients of the model parameters in steps 5 and 6, continuously optimizing them until the entire network model converges.

[0031] Step 9: Calculate Network Predictions

[0032] Based on the two-dimensional adjacent time relationship graph obtained in step 7, a series of two-dimensional convolutions are performed to obtain the score corresponding to each candidate segment. The scores of all candidate segments are sorted, and the candidate segment with the highest score is selected as the prediction result.

[0033] Step 1 is implemented as follows:

[0034] Considering the different video sources and varying video encoding frame rates during dataset acquisition, the dataset videos are first preprocessed to unify the video encoding frame rate f. v Then, the dataset is split, with 70% of the samples used for training and the remaining 30% used for testing.

[0035] Step 2 involves using a pre-trained network to extract features from the video, as detailed below:

[0036] For a frame rate of f per second v Given an input video of total duration T seconds, use the pre-trained behavior recognition network C3D. [3] / I3D [4] Feature extraction is then performed. First, the network samples the video every s frames, and the output of the last layer of the network is extracted as the video's features. By concatenating the features from all video segments, a complete original feature sequence can be obtained. in d v This indicates the dimension of the network output features.

[0037] Step 3, which involves constructing the video feature sequence and candidate segment feature sequence, is as follows:

[0038] Considering that different videos have different durations, the lengths of the video features extracted in step 2 will also differ. Therefore, a uniform video feature length N is set. v , for V o Perform uniform sampling to satisfy Simultaneously, considering the consistency of the intermediate layer feature dimensions with the pre-trained language model in step 5, dimension mapping is performed through a feedforward neural network.

[0039]

[0040] For the start and end of a video event τs(e) and the corresponding serial number a in the video feature sequence s(e) The conversion formula is as follows

[0041] a s(e) = <τ s(e) / T × N v > (Formula 2)

[0042] τ s(e) = a s(e) / N v × T (Formula 3)

[0043] Where, <·> represents the rounding operation.

[0044] According to the unified video feature V, the features of the candidate segments corresponding to the serial numbers (i, j) can be obtained through the max-pooling operation The corresponding formula is as follows

[0045] f i,j = MAXPOOL(v i , v i+1 , …, v j ) (Formula 4)

[0046] Where, 1 ≤ i < j ≤ N. Considering all combinations that satisfy 1 ≤ i < j ≤ N, N 2 / 2 candidate segment features will be generated. From the perspective of computational efficiency, all the generated candidate segments are sparsely sampled according to a certain rule, and finally M (M << N 2 / 2) candidate segment features are obtained. Thus, the candidate segment feature sequence can be expressed as follows

[0047] F ad = [f 1 , f 2 , …, f M (Formula 5)

[0048] Meanwhile, according to the rule shown in Formula 6, we can convert the candidate segment feature sequence into a two-dimensional adjacent time relationship graph

[0049] F map [i, j, :] = f i,j (Formula 6) The construction of the text feature sequence described in Step 4 is as follows

[0050] The event description text S contains N w words. After word segmentation, it can be expressed as S = Where [cls], [seq] are the sentence start symbol and end symbol of the pre-trained language model. Then, the word segmentation result is mapped through the word vectors of the pre-trained language model to obtain the text feature sequence Where L represents the length of the sequence after including the start and end symbols, and d s This represents the intermediate layer feature dimension of the pre-trained language model.

[0051] Step 5, which involves constructing a pre-trained language model, is detailed below:

[0052] The pre-trained language model consists of N lm The system consists of layers of self-attention modules, with the input F to the l-th layer of self-attention modules. l The output is F l+1 The calculation process is as follows:

[0053] F′=LN(F l +Attention(F l ,F l ,F l )) (Formula 7)

[0054] F l+1 =LN(F′+FFN(F′)) (Formula 8)

[0055] Where LN represents layer normalization operation, FFn represents feedforward neural network, and Attention represents multi-head attention module.

[0056] The input to the pre-trained language model is a concatenation of text feature sequences and video feature sequences. That is, F0 = F lm .

[0057] Typically, BERT can be considered as a pre-trained language model here. [1] Or ROBERTA [2] .

[0058] Step 6, which involves constructing the fragment retrieval adaptation network, is as follows:

[0059] The pre-trained language model in step 5 can be divided into N groups based on the k-layer self-attention modules. lm / k groups of self-attention modules. For each group of modules, two adaptation networks are inserted at its input and output to obtain the candidate fragment feature sequence F. ab and the feature sequence F of the pre-trained language model lm To interact, such as Figure 2 As shown.

[0060] 6-1. Specifically, when the i-th group of modules is input, the adaptation network needs to input the candidate segment feature sequence. Semantic information injection Obtain the features after interaction Then The input is fed into the self-attention module of the pre-trained language model for computation, as described in the following formula.

[0061]

[0062] Where, γ i It is a learnable vector used to control the degree of influence of the adaptation network on the input features.

[0063] 6-2. When the i-th module outputs, the adaptation network needs to incorporate the features. Semantic information injection Features obtained after interaction Obtained through a feedforward neural network The input for the (i+1)th group of module adaptation network is described by the following formula.

[0064]

[0065]

[0066] The loss function described in step 7 is as follows:

[0067] 7-1. For the input video sequence of the pre-trained language model, calculate the start and end time indices p of the predicted target segment. s(e) The start and end time sequence of the actual target segment y s(e) The difference between them is calculated using the cross-entropy loss function, with the specific formula as follows:

[0068]

[0069] 7-2. For the candidate segment feature sequence, generate a two-dimensional adjacent time relationship graph F according to formula 6. map Calculate F map Generate rating p i and candidate segment confidence score y i The gap between them. The candidate segment confidence score is calculated based on the time intersection ratio (TLU) between the candidate segment and the target segment. i The calculation yields the following formula:

[0070]

[0071] Where t min ,t max The threshold value set.

[0072] The cross-entropy loss function is used here, and the specific formula is as follows:

[0073]

[0074] 7-3. The total loss of the model is given by the following formula.

[0075] L=λL span +L map (Formula 23)

[0076] Here, λ is a hyperparameter used to balance the two losses.

[0077] The training model described in step 8 is as follows:

[0078] Based on the loss value generated by the loss function in step 7, the backpropagation algorithm is used to backpropagate gradients to the parameters of the adapted network model in step (6), continuously optimizing the parameters of the adapted network model until the entire network model converges. Throughout the process, the pre-trained language model participates in gradient backpropagation but does not update its parameters, i.e., the pre-trained language model parameters are frozen.

[0079] The network prediction values ​​mentioned in step 9 are as follows:

[0080] Based on the two-dimensional adjacent time relationship diagram F obtained in step 7-2, map A score is generated for each candidate segment, the scores are sorted, and the candidate segment with the highest score is selected as the final prediction target segment.

[0081] The essential feature of this invention is that the algorithm proposed in this invention changes the two-stage mode of cross-modal video segment retrieval tasks that previously used video frame regression and candidate segment sorting. The proposed method, based on a pre-trained language model adaptation network, simultaneously uses both video frames and candidate segments as representation methods, and achieves information fusion between text and video frames, as well as word-by-word and frame-by-frame interaction between candidate segments and text and video frames in one forward propagation. Attached Figure Description

[0082] Figure 1 : Sparse sampling rules for candidate segments;

[0083] Figure 2 : An adaptation network insertion method based on a pre-trained language model. Detailed Implementation

[0084] The detailed parameters of the present invention will be further described below.

[0085] like Figure 1 As shown in Figure 2, this invention provides a cross-modal video segment retrieval method based on a pre-trained language model adaptation network.

[0086] Step (1) is implemented as follows:

[0087] Considering that the video sources are different and the video coding frame rates are different during data set collection, the videos in the data set are first preprocessed to unify the video coding frame rate to 30 fps. Then, the data set is divided, and 70% of the samples are selected for training, and the remaining 30% of the samples are used for testing.

[0088] The step (2) of using the pre-trained network to extract features from the video is as follows:

[0089] For an input video with a frame rate of f per second v = 30 and a total duration of T seconds, the trained action recognition network C3D [3] / I3D [4] is used for feature extraction. First, the network samples the video every 24 frames, and the output of the last layer of the network is extracted as the feature of the video The features of all video segments are concatenated to obtain a complete original feature sequence where

[0090] The step (3) of constructing the video feature sequence and the candidate segment feature sequence is as follows:

[0091] Considering that the video time lengths are different and the lengths of the video features extracted in the corresponding step (2) are also different, a unified video feature length N v = 32 is set, and V o is evenly sampled to satisfy At the same time, considering that it is consistent with the intermediate layer feature dimension of the pre-trained language model in step (5), it is mapped through a feed-forward neural network to obtain

[0092] According to the unified video feature V, the features of the candidate segments corresponding to the serial numbers (i, j) can be obtained through the max pooling operation Considering all combinations that satisfy 1 ≤ i < j ≤ 32, 32×32 / 2 = 512 candidate segment features will be generated. For computational efficiency, all the generated candidate segments are sparsely sampled according to a certain rule (such as Figure 1 shown), and finally M = 280 candidate segment features are obtained. At the same time, according to the rule shown in formula 5, we can convert the candidate segment feature sequence into a two-dimensional adjacent time relationship graph

[0093] The step (4) of constructing the text feature sequence is as follows: <​​​​​Where [cls] and [seq] are the sentence start and end symbols of the pre-trained language model. Then, the word segmentation results are mapped to word vectors from the pre-trained language model to obtain text features. Where L represents the length of the sequence after adding the start and end symbols.

[0095] Step (5) involves constructing a pre-trained language model, as detailed below:

[0096] The pre-trained language model consists of 12 layers of self-attention modules, and its input is a concatenation of text feature sequences and video feature sequences. That is, F0 = F lm .

[0097] Typically, BERT can be considered as a pre-trained language model here. [1] Or ROBERTA [2] .

[0098] The construction of the fragment retrieval adaptation network in step (6) is as follows:

[0099] The pre-trained language model in step (5) can be divided into 12 / 3 = 4 groups of self-attention modules, grouped into a set of k=3 layers. For each group of modules, two adaptation networks are inserted at its input and output to obtain the candidate segment feature sequence F. ad and the feature sequence F of the pre-trained language model lm To interact, such as Figure 2 As shown.

[0100] The loss function described in step (7) is as follows:

[0101] 7-1. For the input video sequence of the pre-trained language model, calculate the start and end time indices p of the predicted target segment. s(e) The start and end time sequence of the actual target segment y s(e) The difference between them is determined using the cross-entropy loss function.

[0102] 7-2. For the candidate segment feature sequence, generate a two-dimensional adjacent time relationship graph F according to formula 6. map Calculate F map Generate rating p i and candidate segment confidence score y i The gap between them. The candidate segment confidence score is calculated based on the time intersection ratio (TLU) between the candidate segment and the target segment. i The calculation yields a threshold t. min =0.5,t max =1, here we use the cross-entropy loss function.

[0103] 7-3. The total loss of the model is set to λ = 0.01, which is used as a hyperparameter to balance the two losses.

[0104] The training model described in step (8) is as follows:

[0105] Based on the loss value generated by the loss function in step (7), the backpropagation algorithm is used to backpropagate gradients to the parameters of the adapted network model in step (6), continuously optimizing the parameters of the adapted network model until the entire network model converges. Throughout the process, the pre-trained language model participates in gradient backpropagation but does not update its parameters, i.e., the pre-trained language model parameters are frozen.

[0106] The network prediction value mentioned in step (9) is as follows:

[0107] The two-dimensional adjacent time relationship diagram F obtained from step (7) 7-2 is as follows: map A score is generated for each candidate segment, the scores are sorted, and the candidate segment with the highest score is selected as the final prediction target segment.

Claims

1. A cross-modal video segment retrieval method based on a pre-trained language model adaptation network, characterized in that... Includes the following steps: Step 1: Data preprocessing and dataset partitioning The video is preprocessed and the dataset is divided. Step 2: Extract features from the video using a pre-trained network. For an input video, it is divided into multiple video segments according to a fixed time length. For each video segment, the semantic features are extracted using a network. The features of all video segments are combined to obtain the complete features of the input video. Step 3: Construct video feature sequences and candidate segment feature sequences Uniform sampling is performed on the original video features to ensure that the feature sequence length input into the model remains consistent for each video, i.e., the input model feature sequence is... , Indicates the first Features corresponding to each video segment Indicates a fixed number of video segments; Based on video feature sequences Construct candidate segment features at different time scales Concatenate all generated candidate segment features to obtain the candidate segment feature sequence. ; Based on video length and the length of the input feature sequence The correspondence between the time boundary of the target segment and the start and end numbers of the feature sequence is calculated. Step 4: Constructing the text feature sequence The input text is segmented into words, then transformed into word vectors containing semantic information using a pre-trained word vector model. All the resulting word vectors are then concatenated to obtain the feature sequence of the input text. , Indicates the first Features corresponding to each word Indicates the length of the text feature sequence; Step 5: Construct a pre-trained language model Text feature sequence Video feature sequences Concatenate them into the final input feature sequence ;Will Input to A pre-trained language model with multiple layers, where each layer takes the output features of the previous layer as input, ultimately yielding an output feature sequence. ; Step 6: Construct a fragment retrieval adaptation network Candidate segment feature sequence The input segment retrieval adaptation network interacts with the pre-trained language model in each stage and step 5 to semantically align candidate segment features with text features and video features. Step 7: Loss Function The portion of the video frame corresponding to the feature sequence output in step 5 is used for the calculation of the time span loss function, and the candidate segment feature sequence output in step 6 is recombined into a two-dimensional adjacent time relationship graph for the calculation of the candidate segment loss function. Step 8: Train the model Based on the loss value generated by the loss function in step 7, the backpropagation algorithm is used to backpropagate the gradients of the model parameters in steps 5 and 6 to continuously optimize the learning process. Step 9: Calculate Network Predictions Based on the two-dimensional adjacent time relationship graph obtained in step 7, a series of two-dimensional convolutions are performed to obtain the score corresponding to each candidate segment. The scores of all candidate segments are sorted, and the candidate segment with the highest score is selected as the prediction result. Step 6, which involves constructing the fragment retrieval adaptation network, includes the following steps: according to The layer self-attention modules are grouped together, and the pre-trained language model in step 5 is divided into... Grouped self-attention modules; for each group of modules, two adaptation networks are inserted at its input and output to obtain candidate fragment feature sequences. and pre-trained language model feature sequences Interact; 6-1. In the first When inputting into the group module, the adaptation network needs to process the candidate fragment feature sequences. Semantic information injection Obtain the features after interaction Then The input is fed into the self-attention module of the pre-trained language model for computation, as described in the following formula. ； in, It is a learnable vector used to control the degree of influence of the adaptation network on the input features; 6-2. In the When the group module outputs, the adaptation network needs to include the features. Semantic information injection Features obtained after interaction , Obtained through a feedforward neural network For the The input of the group module adapts to the network, and the formula is described as follows: ； 。 2. The cross-modal video segment retrieval method based on a pre-trained language model adaptation network as described in claim 1, characterized in that... Step 2 uses a pre-trained network to extract features from the video, including the following steps: For a frame rate of 1 second Total duration is The input video is 1 second long, and features are extracted using the pre-trained behavior recognition network C3D / I3D. Network every The video is sampled frame by frame, and the output of the last layer of the network is extracted as the video features. By concatenating the features of all video segments, a complete original feature sequence is obtained. ,in , This indicates the dimension of the network output features.

3. The cross-modal video segment retrieval method based on a pre-trained language model adaptation network as described in claim 2, characterized in that... Step 3, which involves constructing the video feature sequence and candidate segment feature sequence, includes the following steps: Considering that different videos have different durations, the corresponding video feature lengths extracted in step 2 will also differ. Therefore, a uniform video feature length should be set. ,right Perform uniform sampling to satisfy Simultaneously, considering the consistency of the intermediate layer feature dimensions with the pre-trained language model in step 5, dimension mapping is performed through a feedforward neural network. ； Regarding the start and end of the video event and the corresponding sequence number in the video feature sequence The conversion formula is as follows: ； ； in, Indicates the rounding operation; Based on the unified video characteristics The corresponding sequence number is obtained through max pooling operation. Features of candidate fragments The formula is as follows: ； in, Sparse sampling is performed on all generated candidate fragments according to certain rules to obtain... ( There are 10 candidate segment features; the candidate segment feature sequence is represented as follows: ； At the same time, according to the rules shown in Formula 6, the candidate segment feature sequence is converted into a two-dimensional adjacent time relationship graph. ; Official 6: .

4. The cross-modal video segment retrieval method based on a pre-trained language model adaptation network as described in claim 3, wherein step 4 involves constructing a text feature sequence. Includes the following steps: Event description text Include Each word, after word segmentation, is represented as: ,in These are the sentence start and end symbols for a pre-trained language model; The word segmentation results are mapped to word vectors using a pre-trained language model to obtain the text feature sequence. ,in This indicates the length of the sequence after adding the start and end symbols. This represents the intermediate layer feature dimension of the pre-trained language model.

5. The cross-modal video segment retrieval method based on a pre-trained language model adaptation network as described in claim 4, wherein step 5, constructing the pre-trained language model, includes the following steps: Pre-trained language models are The self-attention module consists of layers, the first layer. The input to the layer self-attention module is The output is The calculation process is as follows: ； ； in, Presentation layer normalization operation, This represents a feedforward neural network. This indicates a multi-head attention module; The input to the pre-trained language model is a concatenation of text feature sequences and video feature sequences. ,Right now .

6. The cross-modal video segment retrieval method based on a pre-trained language model adaptation network as described in claim 1, wherein the loss function in step 7 includes the following steps: 7-1. For the input video sequence of the pre-trained language model, calculate the start and end time numbers of the predicted target segment. , The start and end time sequence numbers of the actual target segment , The difference between them is calculated using the cross-entropy loss function, with the specific formula as follows: ； 7-2. For the candidate segment feature sequence, generate a two-dimensional adjacent time relationship graph according to Formula 6. ,calculate Generate rating and candidate segment confidence score The gap between them; The candidate segment confidence score is calculated based on the time intersection-union ratio between the candidate segment and the target segment. The calculation yields the following formula: ； in The threshold value set; The cross-entropy loss function is used, and the specific formula is as follows: ； 7-3. The total loss of the model is given by the following formula. ； in, It is a hyperparameter used to balance the two losses.

7. The cross-modal video segment retrieval method based on a pre-trained language model adaptation network as described in claim 6, wherein step 8, training the model, includes the following steps: The training process employs different optimization methods for the pre-trained language model and the adaptation network. For the pre-trained language model, the frozen parameters are frozen during training, and the model participates in gradient backpropagation but does not update the learned parameters. For the adaptation network, the parameters are updated during training, the model participates in gradient backpropagation, and the learned parameters are updated.

8. The cross-modal video segment retrieval method based on a pre-trained language model adaptation network as described in claim 7, wherein the network prediction value in step 9 includes the following steps: Based on the two-dimensional adjacent time relationship diagram obtained in step 7.2 A score is generated for each candidate segment, the scores are sorted, and the candidate segment with the highest score is selected as the final prediction target segment.

Images