A video segment retrieval method based on fine-grained modal relation perception network

Abstract

This invention relates to the fields of computer vision and pattern recognition technology, specifically to a video segment retrieval method based on a fine-grained modality relationship-aware network. The steps are as follows: extracting semantically enhanced text query features and video features; mining fine-grained modality relationship-aware information; concatenating the interacted boundary graph and content graph along the channel dimension to obtain a fusion graph; mining the relationships between proposals using convolutional blocks on the fusion graph; and obtaining the final prediction score through a prediction module; constructing a loss function based on the fine-grained modality relationship-aware network; constraining the network training with the loss function; sorting the predicted scores of the entire network output prediction graph from largest to smallest to obtain the video segment retrieval results. This invention can mine fine-grained modality relationship-aware information, thereby obtaining robust query-aware video features and video-aware text features, resulting in more accurate retrieval results.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and pattern recognition technology, and in particular to a video segment retrieval method based on a fine-grained modal relationship sensing network. Background Technology

[0002] With the exponential growth of online video content, locating video segments in undressed videos has become increasingly urgent. To address this challenge, researchers have proposed Temporal Action Localization (TAL) methods. However, existing TAL techniques are limited in their ability to accurately locate action segments that meet specific conditions, often resulting in located video segments that do not fully satisfy those conditions. For example, in the case of the action "running," it is possible to simultaneously locate segments depicting "one person running" and "two people running." To more accurately locate the desired video segments under specific conditions, Video Segment Retrieval (VMR) has been introduced. This method retrieves video segments semantically consistent with the text query and has attracted researchers' attention. A common VMR approach is to first generate proposals, then calculate the similarity between these proposals and the text query, and then rank the similarities to obtain the final retrieved segments. While some methods utilize multi-scale sliding windows to generate proposals, densely sampled sliding windows may not be the most efficient solution. Therefore, researchers have developed a segmented proposal network to generate proposals, aiming to reduce the number of proposals. Compared to the methods mentioned above, anchor-based methods do not require proposal generation at the input stage, making them a more efficient strategy.

[0003] While previous video segment retrieval methods have achieved promising results, some unresolved issues remain. For example, the exploration of fine-grained modal relationship awareness information within and across modalities is still incomplete. Furthermore, many existing methods rely on simple dot product operations for text queries and video interactions, neglecting subtle relationship information within and between modalities.

[0004] Therefore, in order to solve these problems, this invention proposes a video segment retrieval method based on a fine-grained modal relation sensing network. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a video segment retrieval method based on a fine-grained modal relationship-aware network. This invention can mine fine-grained modal relationship-aware information, thereby obtaining robust query-aware video features and video-aware text features, and achieving more accurate retrieval results.

[0006] The technical solution to the problem solved by this invention is: a video segment retrieval method based on a fine-grained modality relation-aware network, comprising the following steps: A video segment retrieval method based on a fine-grained modality relation-aware network includes the following steps: S1. Feature Extraction: Mining information from video modalities and text query modalities to obtain semantically enhanced text query features. and video features ; S2. Mining fine-grained modality-aware information: enhancing text query features with semantic relationships. and video features The input is fed into the cross-similarity weighting module and the intra-modal weighting module, and the video features are processed based on the results. and video features The weighted values ​​are then input into the context-aware module to obtain video features with fine-grained modal relationship awareness information. and text query features ; S3. Constructing a fine-grained modality relationship-aware network to mine proposal relationships: Enhance video features based on fine-grained modality relationship-aware information, and then generate a boundary map and a content map using the enhanced video features. Each position in the map represents a proposal. Then, the proposals are enhanced through an attention mechanism. The content map is then interacted with the text query features enhanced by fine-grained modality relationship-aware information, and the boundary map is interacted with the video features enhanced by fine-grained modality relationship-aware information. Finally, the interacted boundary map and content map are concatenated along the channel dimension to obtain a fusion map. After obtaining the fusion map, convolutional blocks are used to further mine the relationships between proposals, and the final prediction score is obtained through a prediction module. S4. Construct a loss function based on a fine-grained modality relationship awareness network: Calculate the intersection-union ratio (CIU) of each proposal and the real segment, scale the CIU to obtain the supervision signal of the corresponding proposal, and then calculate the loss function using cross-entropy loss. S5. Obtain video segment retrieval results: Train the constrained fine-grained modality relationship perception network using the loss function constructed in step S4. Sort the predicted scores of the prediction graph output by the fine-grained modality relationship perception network from largest to smallest, and obtain the proposal with the largest intersection-union ratio with the current text query, which is the retrieval result of the video segment.

[0007] Step S1 is as follows: For video segment retrieval tasks, a video is divided into continuous and non-overlapping units, each containing 16 frames. These divided video units are then input into a video modality encoder, which includes an I3D network and a GRU network. Each video unit is first input into the I3D network for video feature extraction, resulting in a 1024×T video feature set. , The number of video units is represented by the output of the I3D network, which is then fed into the GRU network to extract contextual information from the video and obtain features. ; For the text query modality encoder, sentences from the video segment to be retrieved are input into the GloVe word2vec model, which converts each word in the text into a tensor to obtain the text query features. The dimensions of the text query features are... , This represents the number of text words in the text query. The text query features are input into a GRU network to obtain the text query features after extracting contextual information. ; Features that enhance video context information Text query features enhanced with contextual information Next, a semantic graph encoder is used to mine semantic information within the video and text query, representing each unit in the video and each word in the query sentence as a node in the graph. Then, the k-nearest neighbor algorithm (KNN) is used to identify adjacent segments and words. A convolutional network is used to aggregate information from each segment and its neighbors, as well as each word and its neighbors. The specific calculation is as follows: , , in, This represents a 2D convolution operation with 3 kernels of size 1. It is the number of neighboring nodes. This represents the text query features after semantic relationship enhancement. This represents video features enhanced with semantic relationships.

[0008] Step S2 is as follows: Text query features enhanced with semantic relations Video features enhanced with semantic relationships As input to the fine-grained cross-modal interaction module, it is fed into the cross-similarity weighting module and the intra-modal weighting module; In the cross-similarity weighted module, matrix multiplication is used to calculate the similarity matrix. : , in Each element in Indicates the first The word and the first The similarity of each video unit is calculated, and then the similarity between the query and each segment, as well as the similarity between the video and each word, are obtained through the similarity between words and segments. , , in Each element in the table represents the similarity between each video unit and the entire text query. Each element in the table represents the similarity of each word to the entire video. express Activation function This indicates that the similarity matrix is ​​applied along dimension 𝑖. The mean calculation, through the cross-similarity weighting module, obtains fine-grained cross-modal relationship information, and then further enhances query and video features through the above information and intra-modal weighting; In the intramodal weighting module, the text query features are enhanced with semantic relations. Video features enhanced with semantic relationships As input, for text query features, a multilayer perceptron and a softmax function are used to obtain the weight of each word, thereby extracting relational information within the text modality. For video features, a convolutional layer followed by a softmax function is used to obtain the weight of each video unit, thereby mining relational information within the video modality. The specific calculation is as follows: , , in This represents the weight of each word. This represents the weight of each video unit; After obtaining the cross-similarity weights and intra-modal weights, an adaptive fusion module is used for fusion. The obtained cross-similarity weights and intra-modal weights are then applied to the query and video features, respectively. The calculation process is as follows: , , in This represents the weighted text query features. This represents the weighted video features; Then, the context-aware module is used to extract fine-grained contextual information from text queries and video features, converting the given weighted video features into keys and values: , , , in, and This represents two keys with different receptive fields in video feature transformation. This represents the value converted from video features. Indicates the kernel size. This represents the convolution, batch normalization, and ReLU activation function operations on the keys in video feature transformation. The convolution, batch normalization, and ReLU activation function operations represent the median of video feature transformations; In video feature transformation, two keys with different receptive fields are used. and The input video features are concatenated separately, and two corresponding weights are obtained through an attention module. and The specific calculations are as follows: , , in, This indicates that it includes 1D convolution operations with a kernel of 1, batch normalization, ReLU activation function, and Attention module for functions Indicates a splicing operation; Then through weight and weight The output of the context-aware module is obtained, and the specific calculation is as follows: , in, This indicates a splicing operation. This represents the output of the context-aware module. This represents the value converted from video features; For text queries, convert the given weighted text query features into keys and values: , , , in, and This represents two keys with different receptive fields in text query feature transformation. This represents the value converted from the text query features. Indicates the kernel size. This represents the convolution, batch normalization, and ReLU activation function operations on the keys in text query feature transformation. The convolution, batch normalization, and ReLU activation function operations represent the median of text query feature transformations. Transforming text query features into two keys with different receptive fields and The input text query features are concatenated separately, and two corresponding weights are obtained through an attention module. and The specific calculations are as follows: , , in, This indicates that it includes 1D convolution operations with a kernel of 1, batch normalization, ReLU activation function, and Attention module for functions Indicates a splicing operation; Then through weight and weight The output of the context-aware module is obtained, and the specific calculation is as follows: , in, This indicates a splicing operation. This represents the output of the context-aware module. Finally, the video features were obtained. and text query features .

[0009] Step S3 is as follows: Video features containing fine-grained intra-modal and inter-modal information Inputting the data into the proposal module generates 2D-Maps, which include boundary maps. and content images , , , Representing feature dimension, Indicate the start and end indices of the candidate time points for the boundary graph. For each proposal in the content graph, a summation operation is performed at the corresponding start and end positions, followed by clipping to generate candidates. For each proposal, max pooling is performed on the corresponding video segments from the start to the end of the editing process to obtain candidates. After obtaining the boundary graph and content graph, the proposal relationship module is used to fuse the boundary graph and content graph and extract the relationship information between proposals. Attention is then used to enhance the information in the boundary graph and content graph. The specific calculation is as follows: , , in, This indicates a global average pooling operation. This represents the sigmoid activation function. and These represent the enhanced content graph and boundary graph, respectively. Text query features Interact with content images and leverage video features Interact with the boundary map to obtain the fused map. The specific calculations are as follows: , , , in, This indicates a splicing operation. This represents the sigmoid activation function. and These represent the upper dimensions of the query feature and the video feature, respectively. Maximum and average operations, and These represent the content graph after interaction with the query and the boundary graph after interaction with the video, respectively. After obtaining the fusion graph, convolutional blocks are used to mine the relationships between different proposals. The specific calculation is as follows: , in, This represents a convolutional block containing two 2D convolution operations with kernels of 3 and padding of 1, batch normalization, and ReLU function operations. This represents the fusion diagram after fully exploring the relationships between proposals. Finally, the prediction module is used to obtain the prediction: , in, This represents the final predicted graph. This represents a 1D convolution operation with a kernel of 1.

[0010] Step S4 is as follows: For supervisory labels The processing uses a scaled Intersection over Union (IoU) as the supervision signal. For each proposal, the IoU score is... The score of the IoU crossover ratio calculated based on the real time. Then compare the scores of the intersection and IoU. Scaling is performed, and the calculation is as follows: , in and Indicates two thresholds, Indicates a supervisory label; Then based on the supervision label Constructing the loss function The specific calculations are as follows: , in, This represents the prediction graph for the i-th proposal. It refers to the number of proposals.

[0011] Step S5 is as follows: The fine-grained modality perception network is trained by setting the learning rate, batch size, number of training iterations, and threshold in the loss function to obtain supervised labels. The fine-grained modality perception network obtains the backpropagation gradient through backpropagation of the loss function in step S4. Then, the Adam optimizer is used to update the network parameters based on the gradient from backpropagation. Finally, the predicted scores of the prediction map output by the fine-grained modality perception network are sorted from largest to smallest to obtain the retrieval results of the video segments.

[0012] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. The above technical solutions have the following advantages or beneficial effects: Through the designed fine-grained cross-modal interaction module, this invention can utilize intra-modal and inter-modal relationship information to weight video features and text query features. Furthermore, a context-aware module further mines context-aware information, deeply exploring fine-grained intra-modal and inter-modal relationship information, thereby obtaining more robust video-aware text query features and text-aware video features. Through the designed proposal relationship module, this invention can weight proposal information to obtain importance information with different weights. Convolutional blocks can better mine the relationships between proposals, thereby obtaining more accurate video segment retrieval results. Attached Figure Description The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0013] Figure 1 This is a schematic diagram of the structure of the method of the present invention.

[0014] Figure 2 This is a schematic diagram illustrating the visualization results of the present invention. Detailed Implementation

[0015] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, the components and arrangements of specific examples are described below.

[0016] Example 1 A video segment retrieval method based on a fine-grained modality relation-aware network includes the following steps: S1. Feature Extraction: Mining information from video modalities and text query modalities to obtain semantically enhanced text query features. and video features ; S2. Mining fine-grained modality-aware information: enhancing text query features with semantic relationships. and video features The input is fed into the cross-similarity weighting module and the intra-modal weighting module, and the video features are processed based on the results. and video features The weighted values ​​are then input into the context-aware module to obtain video features with fine-grained modal relationship awareness information. and text query features ; S3. Constructing a fine-grained modality relationship-aware network to mine proposal relationships: Enhance video features based on fine-grained modality relationship-aware information, and then generate a boundary map and a content map using the enhanced video features. Each position in the map represents a proposal. Then, the proposals are enhanced through an attention mechanism. The content map is then interacted with the text query features enhanced by fine-grained modality relationship-aware information, and the boundary map is interacted with the video features enhanced by fine-grained modality relationship-aware information. Finally, the interacted boundary map and content map are concatenated along the channel dimension to obtain a fusion map. After obtaining the fusion map, convolutional blocks are used to further mine the relationships between proposals, and the final prediction score is obtained through a prediction module. S4. Construct a loss function based on a fine-grained modality relationship awareness network: Calculate the intersection-union ratio (CIU) of each proposal and the real segment, scale the CIU to obtain the supervision signal of the corresponding proposal, and then calculate the loss function using cross-entropy loss. S5. Obtain video segment retrieval results: Train the constrained fine-grained modality relationship perception network using the loss function constructed in step S4. Sort the predicted scores of the prediction graph output by the fine-grained modality relationship perception network from largest to smallest, and obtain the proposal with the largest intersection-union ratio with the current text query, which is the retrieval result of the video segment.

[0017] Step S1 is as follows: For video segment retrieval tasks, a video is divided into continuous and non-overlapping units, each containing 16 frames. These divided video units are then input into a video modality encoder, which includes an I3D network and a GRU network. Each video unit is first input into the I3D network for video feature extraction, resulting in a 1024×T video feature set. , The number of video units is represented by the output of the I3D network, which is then fed into the GRU network to extract contextual information from the video and obtain features. ; For the text query modality encoder, sentences from the video segment to be retrieved are input into the GloVe word2vec model, which converts each word in the text into a tensor to obtain the text query features. The dimensions of the text query features are... , This represents the number of text words in the text query. The text query features are input into a GRU network to obtain the text query features after extracting contextual information. ; Features that enhance video context information Text query features enhanced with contextual information Next, a semantic graph encoder is used to mine semantic information within the video and text query, representing each unit in the video and each word in the query sentence as a node in the graph. Then, the k-nearest neighbor algorithm (KNN) is used to identify adjacent segments and words. A convolutional network is used to aggregate information from each segment and its neighbors, as well as each word and its neighbors. The specific calculation is as follows: , , in, This represents a 2D convolution operation with 3 kernels of size 1. It is the number of neighboring nodes. This represents the text query features after semantic relationship enhancement. This represents video features enhanced with semantic relationships.

[0018] Step S2 is as follows: Text query features enhanced with semantic relations Video features enhanced with semantic relationships As input to the fine-grained cross-modal interaction module, it is fed into the cross-similarity weighting module and the intra-modal weighting module; In the cross-similarity weighted module, matrix multiplication is used to calculate the similarity matrix. : , in Each element in Indicates the first The word and the first The similarity of each video unit is calculated, and then the similarity between the query and each segment, as well as the similarity between the video and each word, are obtained through the similarity between words and segments. , , in Each element in the table represents the similarity between each video unit and the entire text query. Each element in the table represents the similarity of each word to the entire video. express Activation function This indicates that the similarity matrix is ​​applied along dimension 𝑖. The mean calculation, through the cross-similarity weighting module, obtains fine-grained cross-modal relationship information, and then further enhances query and video features through the above information and intra-modal weighting; In the intramodal weighting module, the text query features are enhanced with semantic relations. Video features enhanced with semantic relationships As input, for text query features, a multilayer perceptron and a softmax function are used to obtain the weight of each word, thereby extracting relational information within the text modality. For video features, a convolutional layer followed by a softmax function is used to obtain the weight of each video unit, thereby mining relational information within the video modality. The specific calculation is as follows: , , in This represents the weight of each word. This represents the weight of each video unit; After obtaining the cross-similarity weights and intra-modal weights, an adaptive fusion module is used for fusion. The obtained cross-similarity weights and intra-modal weights are then applied to the query and video features, respectively. The calculation process is as follows: , , in This represents the weighted text query features. This represents the weighted video features; Then, the context-aware module is used to extract fine-grained contextual information from text queries and video features, converting the given weighted video features into keys and values: , , , in, and This represents two keys with different receptive fields in video feature transformation. This represents the value converted from video features. Indicates the kernel size. This represents the convolution, batch normalization, and ReLU activation function operations on the keys in video feature transformation. The convolution, batch normalization, and ReLU activation function operations represent the median of video feature transformations; In video feature transformation, two keys with different receptive fields are used. and The input video features are concatenated separately, and two corresponding weights are obtained through an attention module. and The specific calculations are as follows: , , in, This indicates that it includes 1D convolution operations with a kernel of 1, batch normalization, ReLU activation function, and Attention module for functions Indicates a splicing operation; Then through weight and weight The output of the context-aware module is obtained, and the specific calculation is as follows: , in, This indicates a splicing operation. This represents the output of the context-aware module. This represents the value converted from video features; For text queries, convert the given weighted text query features into keys and values: , , , in, and This represents two keys with different receptive fields in text query feature transformation. This represents the value converted from the text query features. Indicates the kernel size. This represents the convolution, batch normalization, and ReLU activation function operations on the keys in text query feature transformation. The convolution, batch normalization, and ReLU activation function operations represent the median of text query feature transformations. Transforming text query features into two keys with different receptive fields and The input text query features are concatenated separately, and two corresponding weights are obtained through an attention module. and The specific calculations are as follows: , , in, This indicates that it includes 1D convolution operations with a kernel of 1, batch normalization, ReLU activation function, and Attention module for functions Indicates a splicing operation; Then through weight and weight The output of the context-aware module is obtained, and the specific calculation is as follows: , in, This indicates a splicing operation. This represents the output of the context-aware module. Finally, the video features were obtained. and text query features .

[0019] Step S3 is as follows: Video features containing fine-grained intra-modal and inter-modal information Inputting the data into the proposal module generates 2D-Maps, which include boundary maps. and content images , , , Representing feature dimension, Indicate the start and end indices of the candidate time points for the boundary graph. For each proposal in the content graph, a summation operation is performed at the corresponding start and end positions, followed by clipping to generate candidates. For each proposal, max pooling is performed on the corresponding video segments from the start to the end of the editing process to obtain candidates. After obtaining the boundary graph and content graph, the proposal relationship module is used to fuse the boundary graph and content graph and extract the relationship information between proposals. Attention is then used to enhance the information in the boundary graph and content graph. The specific calculation is as follows: , , in, This indicates a global average pooling operation. This represents the sigmoid activation function. and These represent the enhanced content graph and boundary graph, respectively. Text query features Interact with content images and leverage video features Interact with the boundary map to obtain the fused map. The specific calculations are as follows: , , , in, This indicates a splicing operation. This represents the sigmoid activation function. and These represent the upper dimensions of the query feature and the video feature, respectively. Maximum and average operations, and These represent the content graph after interaction with the query and the boundary graph after interaction with the video, respectively. After obtaining the fusion graph, convolutional blocks are used to mine the relationships between different proposals. The specific calculation is as follows: , in, This represents a convolutional block containing two 2D convolution operations with kernels of 3 and padding of 1, batch normalization, and ReLU function operations. This represents the fusion diagram after fully exploring the relationships between proposals. Finally, the prediction module is used to obtain the prediction: , in, This represents the final predicted graph. This represents a 1D convolution operation with a kernel of 1.

[0020] Step S4 is as follows: For supervisory labels The processing uses a scaled Intersection over Union (IoU) as the supervision signal. For each proposal, the IoU score is... The score of the IoU crossover ratio calculated based on the real time. Then compare the scores of the intersection and IoU. Scaling is performed, and the calculation is as follows: , in and Indicates two thresholds, Indicates a supervisory label; Then based on the supervision label Constructing the loss function The specific calculations are as follows: , in, This represents the prediction graph for the i-th proposal. It refers to the number of proposals.

[0021] Step S5 is as follows: The fine-grained modality perception network is trained by setting the learning rate, batch size, number of training iterations, and threshold in the loss function to obtain supervised labels. The fine-grained modality perception network obtains the backpropagation gradient through backpropagation of the loss function in step S4. Then, the Adam optimizer is used to update the network parameters based on the gradient from backpropagation. Finally, the predicted scores of the prediction map output by the fine-grained modality perception network are sorted from largest to smallest to obtain the retrieval results of the video segments.

[0022] Example 2 like Figure 2 The visualization results shown are obtained using the method of this invention. GT represents the start and end times of the actual video segments to be retrieved; baseline represents the start and end times of the video segments retrieved by the network architecture after removing the semantic graph encoder, fine-grained cross-modal interaction module, and proposal relationship module; FMRN represents the start and end times of the video segments retrieved by the method of this patent. Figure 2 The three visualization results show that, compared to the baseline, the video segments retrieved by this patented method are closer to the actual video segments that should be retrieved, thus verifying the effectiveness of this patented method.

[0023] Example 3 As shown in Table 1, the performance of our method and existing video segment retrieval methods was compared on the Charades-STA and TACoS datasets. When only the segment with the highest predicted score Recall@1 is retained, our method achieves the best results under different IoU thresholds, proving the effectiveness of our invention.

[0024] Table 1. Performance comparison of existing video segment retrieval methods and the method of this invention on the Charades-STA and TACoS datasets. The parameters in Table 1 have the following meanings: CTRL, 2D-TAN, CBP, RANet, IA-Net, FVMR, VLG-Net, MMN, HCLNet, MGPN, and CRNet are existing video segment retrieval methods, where R@1, m represents the value assigned to the Intersection over Union (IoU). Charades-STA dataset: The Charades-STA dataset contains 12,408 video-text query pairs in the training set and 3,720 in the test set. The Charades-STA dataset is designed based on the Charades dataset. The TACoS dataset contains 127 videos collected from kitchen scenes, with 10,146, 4,589, and 4,083 video-text query pairs in the training, validation, and test sets, respectively.

[0025] Although the specific embodiments of the invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the invention. Based on the technical solutions of the invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the invention.

Claims

1. A video segment retrieval method based on a fine-grained modality relation-aware network, characterized in that, Includes the following steps: S1. Feature Extraction: Mining information from video modalities and text query modalities to obtain semantically enhanced text query features. and video features ; S2. Mining fine-grained modality-aware information: enhancing text query features with semantic relationships. and video features The input is fed into the cross-similarity weighting module and the intra-modal weighting module, and the video features are processed based on the results. and video features The weighted values ​​are then input into the context-aware module to obtain video features with fine-grained modal relationship awareness information. and text query features ; S3. Constructing a fine-grained modality relationship-aware network to mine proposal relationships: Enhance video features based on fine-grained modality relationship-aware information, and then generate a boundary map and a content map using the enhanced video features. Each position in the map represents a proposal. Then, the proposals are enhanced through an attention mechanism. The content map is then interacted with the text query features enhanced by fine-grained modality relationship-aware information, and the boundary map is interacted with the video features enhanced by fine-grained modality relationship-aware information. Finally, the interacted boundary map and content map are concatenated along the channel dimension to obtain a fusion map. After obtaining the fusion map, convolutional blocks are used to further mine the relationships between proposals, and the final prediction score is obtained through a prediction module. Step S3 is as follows: Video features containing fine-grained intra-modal and inter-modal information Inputting the data into the proposal module generates 2D-Maps, which include boundary maps. and content images , , , Representing feature dimension, Indicate the start and end indices of the candidate time points for the boundary graph. For each proposal in the content graph, a summation operation is performed at the corresponding start and end positions, followed by clipping to generate candidates. For each proposal, max pooling is performed on the corresponding video segments from the start to the end of the editing process to obtain candidates. After obtaining the boundary graph and content graph, the proposal relationship module is used to fuse the boundary graph and content graph and extract the relationship information between proposals. Attention is then used to enhance the information in the boundary graph and content graph. The specific calculation is as follows: ， ， in, This indicates a global average pooling operation. This represents the sigmoid activation function. and These represent the enhanced content graph and boundary graph, respectively. Text query features Interact with content images and leverage video features Interact with the boundary map to obtain the fused map. The specific calculations are as follows: ， ， ， in, This indicates a splicing operation. This represents the sigmoid activation function. and These represent the upper dimensions of the query feature and the video feature, respectively. Maximum and average operations, and These represent the content graph after interaction with the query and the boundary graph after interaction with the video, respectively. After obtaining the fusion graph, convolutional blocks are used to mine the relationships between different proposals. The specific calculation is as follows: ， in, This represents a convolutional block containing two 2D convolution operations with kernels of 3 and padding of 1, batch normalization, and ReLU function operations. This represents the fusion diagram after fully exploring the relationships between proposals. Finally, the prediction module is used to obtain the prediction: ， in, This represents the final predicted graph. This represents a 1D convolution operation with a kernel of 1. S4. Construct a loss function based on a fine-grained modality relationship awareness network: Calculate the intersection-union ratio (CIU) of each proposal and the real segment, scale the CIU to obtain the supervision signal of the corresponding proposal, and then calculate the loss function using cross-entropy loss. S5. Obtain video segment retrieval results: Train the constrained fine-grained modality relationship perception network using the loss function constructed in step S4. Sort the predicted scores of the prediction graph output by the fine-grained modality relationship perception network from largest to smallest, and obtain the proposal with the largest intersection-union ratio with the current text query, which is the retrieval result of the video segment.

2. The video segment retrieval method based on a fine-grained modal relation sensing network according to claim 1, characterized in that, Step S1 is as follows: For video segment retrieval tasks, a video is divided into continuous and non-overlapping units, each containing 16 frames. These divided video units are then input into a video modality encoder, which includes an I3D network and a GRU network. Each video unit is first input into the I3D network for video feature extraction, resulting in a 1024×T video feature set. , The number of video units is represented by the output of the I3D network, which is then fed into the GRU network to extract contextual information from the video and obtain features. ; For the text query modality encoder, sentences from the video segment to be retrieved are input into the GloVe word2vec model, which converts each word in the text into a tensor to obtain the text query features. The dimensions of the text query features are... , This represents the number of text words in the text query. The text query features are input into a GRU network to obtain the text query features after extracting contextual information. ; Features that enhance video context information Text query features enhanced with contextual information Next, a semantic graph encoder is used to mine semantic information within the video and text query, representing each unit in the video and each word in the query sentence as a node in the graph. Then, the k-nearest neighbor algorithm (KNN) is used to identify adjacent segments and words. A convolutional network is used to aggregate information from each segment and its neighbors, as well as each word and its neighbors. The specific calculation is as follows: ， ， in, This represents a 2D convolution operation with 3 kernels of size 1. It is the number of neighboring nodes. This represents the text query features after semantic relationship enhancement. This represents video features enhanced with semantic relationships.

3. The video segment retrieval method based on a fine-grained modal relation sensing network according to claim 2, characterized in that, Step S2 is as follows: Text query features enhanced with semantic relations Video features enhanced with semantic relationships As input to the fine-grained cross-modal interaction module, it is fed into the cross-similarity weighting module and the intra-modal weighting module; In the cross-similarity weighted module, matrix multiplication is used to calculate the similarity matrix. : ， in Each element in Indicates the first The word and the first The similarity of each video unit is calculated, and then the similarity between the query and each segment, as well as the similarity between the video and each word, are obtained through the similarity between words and segments. ， ， in Each element in the table represents the similarity between each video unit and the entire text query. Each element in the table represents the similarity of each word to the entire video. express Activation function Indicates along the dimension Applied to similarity matrix The mean calculation, through the cross-similarity weighting module, obtains fine-grained cross-modal relationship information, and then further enhances query and video features through the above information and intra-modal weighting; In the intramodal weighting module, the text query features are enhanced with semantic relations. Video features enhanced with semantic relationships As input, for text query features, a multilayer perceptron and a softmax function are used to obtain the weight of each word, thereby extracting relational information within the text modality. For video features, a convolutional layer followed by a softmax function is used to obtain the weight of each video unit, thereby mining relational information within the video modality. The specific calculation is as follows: ， ， in This represents the weight of each word. This represents the weight of each video unit; After obtaining the cross-similarity weights and intra-modal weights, an adaptive fusion module is used for fusion. The obtained cross-similarity weights and intra-modal weights are then applied to the query and video features, respectively. The calculation process is as follows: ， ， in This represents the weighted text query features. This represents the weighted video features; Then, the context-aware module is used to extract fine-grained contextual information from text queries and video features, converting the given weighted video features into keys and values: ， ， ， in, and This represents two keys with different receptive fields in video feature transformation. This represents the value converted from video features. Indicates the kernel size. This represents the convolution, batch normalization, and ReLU activation function operations on the keys in video feature transformation. The convolution, batch normalization, and ReLU activation function operations represent the median of video feature transformations; In video feature transformation, two keys with different receptive fields are used. and The input video features are concatenated separately, and two corresponding weights are obtained through an attention module. and The specific calculations are as follows: ， ， in, This indicates that it includes 1D convolution operations with a kernel of 1, batch normalization, ReLU activation function, and Attention module for functions Indicates a splicing operation; Then through weight and weight The output of the context-aware module is obtained, and the specific calculation is as follows: ， in, This indicates a splicing operation. This represents the output of the context-aware module. This represents the value converted from video features.

4. The video segment retrieval method based on a fine-grained modality relation-aware network according to claim 3, characterized in that: For text queries, convert the given weighted text query features into keys and values: ， ， ， in, and This represents two keys with different receptive fields in text query feature transformation. This represents the value converted from the text query features. Indicates the kernel size. This represents the convolution, batch normalization, and ReLU activation function operations on the keys in text query feature transformation. The convolution, batch normalization, and ReLU activation function operations represent the median of text query feature transformations. Transforming text query features into two keys with different receptive fields and The input text query features are concatenated separately, and two corresponding weights are obtained through an attention module. and The specific calculations are as follows: ， ， in, This indicates that it includes 1D convolution operations with a kernel of 1, batch normalization, ReLU activation function, and Attention module for functions Indicates a splicing operation; Then through weight and weight The output of the context-aware module is obtained, and the specific calculation is as follows: ， in, This indicates a splicing operation. This represents the output of the context-aware module; Finally, the video features were obtained. and text query features .

5. The video segment retrieval method based on a fine-grained modality relation-aware network according to claim 4, characterized in that, Step S4 is as follows: For supervisory labels The processing uses a scaled Intersection over Union (IoU) as the supervision signal. For each proposal, the IoU score is... The score of the IoU crossover ratio calculated based on the real time. Then compare the scores of the intersection and IoU. Scaling is performed, and the calculation is as follows: ， in and Indicates two thresholds, Indicates a supervisory label; Then based on the supervision label Constructing the loss function The specific calculations are as follows: ， in, This represents the prediction graph for the i-th proposal. It refers to the number of proposals.

6. The video segment retrieval method based on a fine-grained modality relation-aware network according to claim 5, characterized in that, Step S5 is as follows: The fine-grained modality perception network is trained by setting the learning rate, batch size, number of training iterations, and threshold in the loss function to obtain supervised labels. The fine-grained modality perception network obtains the backpropagation gradient through backpropagation of the loss function in step S4. Then, the Adam optimizer is used to update the network parameters based on the gradient from backpropagation. Finally, the predicted scores of the prediction map output by the fine-grained modality perception network are sorted from largest to smallest to obtain the retrieval results of the video segments.

Images