A video sentence retrieval method based on boundary-sensitive multi-scale candidate interaction

Abstract

The present application relates to the technical field of computer vision and pattern recognition, and specifically designs a video sentence retrieval method based on boundary-sensitive multi-scale candidate interaction, and the steps are as follows: extracting features through a pre-training model; modeling of video and text time sequence information; preliminary interaction of inter-modal information; generation of candidate rectangle features; fine-grained coding within different modalities; second interaction of modal information; and interaction between candidate rectangle features.The present application proposes a method of fusing boundary features and context features, which enables the model to utilize boundary features to understand complex video and text information, adopts a multi-scale candidate rectangle interaction module, which enables the model to better capture the relationship between candidate rectangles, which can serve as supplementary information for candidate rectangles to help the model correct previously ignored errors.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and pattern recognition technology, and in particular to a video sentence retrieval method based on boundary-sensitive multi-scale candidate interaction. Background Technology

[0002] With the advent of the video era, the internet is flooded with videos such as surveillance footage, movie clips, and short videos. As a subtask in the video domain, the TSGV method has attracted widespread attention from the academic community. As a multimodal task involving both video and text, it presents significant challenges. The model not only needs to understand the semantic information described by the video and text separately, but also needs to map the information from the two modalities together. Current methods first extract corresponding features from the original video and text using pre-trained models, then use different strategies to fuse the features from the two modalities, and finally predict the final result based on the fused features. During the fusion process, some methods divide the video into different candidate moments, then predict based on the fused features of the candidate moments and text features, and finally select the final result from multiple fusion results. Alternatively, some methods skip generating candidate moments and directly predict the answer based on the features generated by the modal interaction. Current methods can be divided into two types based on whether candidate moments are generated. Methods using candidate moments generally first generate corresponding candidate moments based on video features, then use text queries to find the candidate moments that are most semantically similar to the text. Methods that do not use candidate moments directly obtain the final timestamp. Methods for generating candidate moments have evolved from initially manually generated candidate moments, to the MCN method which segments the video into six 5-second segments and then combines these segments to obtain the final candidate moments. Later, the CTRL method used a multi-scale sliding window to generate candidate moments, employing simple addition and multiplication or direct concatenation for multimodal fusion, and finally using an MLP to obtain the matching probability between each candidate moment and the text. The SLTA method captures object-level interaction information. The ACL method identifies active concept enhancement features from both video and text. The QSPN method uses text features before generating candidate moments. The 2D-TAN method represents video features as a two-dimensional feature map through sparse sampling. The subsequent SMIN method uses simple pooling operations to separate moment features into content features and boundary features.

[0003] Existing methods do not fully consider the internal structure of video segments when solving this problem. As a localization task, finding the correct boundaries is crucial. Therefore, the selection and use of candidate moments are essential. Current candidate moment selection can cover most possible segments, making the utilization of candidate moments even more important. To fully utilize the information from candidate moments, this paper proposes a video sentence retrieval method based on boundary-sensitive multi-scale candidate interaction. Compared with existing methods, this invention places greater emphasis on boundary features. This invention further refines the boundary features and interacts these refined boundary features with content features and text features respectively. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention develops a video sentence retrieval method based on boundary-sensitive multi-scale candidate interaction. This invention can fully integrate boundary features and contextual features and perform multi-scale fusion of candidate features, resulting in a highly effective retrieval method.

[0005] The technical solution of this invention to solve the technical problem is as follows: A video sentence retrieval method based on boundary-sensitive multi-scale candidate interaction includes the following steps: S1. Extract video features from the video using I3D and C3D networks, and extract features of each word in the text using the GloVe tool; S2. Establish a temporal information model based on video features and word features, and aggregate the temporal information from video features and word features to obtain global temporal information; S3. Perform initial interaction on the global temporal information to obtain the video features and text features after interaction; S4. Generate candidate matrix features based on the video features in S3, generate 2D mapping features for representation from the candidate matrix features, and then visualize the 2D mapping features to obtain a two-dimensional feature map. S5. Re-encode the video features and text features obtained from the initial interaction to obtain fine-grained features of the video and phrase features of the text. S6. Perform a second information interaction, combine the information obtained from the first interaction with the candidate matrix features to obtain more granular video intermodal information and text intermodal information, then fuse the two intermodal information to obtain complementary fusion features, and then splice the complementary fusion features to obtain the final fusion features. S7. The final fused features are processed by multi-scale candidate matrix interaction, and then the prediction scores of each candidate matrix are obtained through calculation. The prediction scores of the candidate matrices are optimized by cross-entropy loss calculation to obtain the prediction scores of the most accurate candidate matrices. These scores are then used for text video retrieval to obtain the final results.

[0006] The video features obtained in step S1 are 256-dimensional for each segment and 300-dimensional for each word.

[0007] Step S2 is as follows: Global temporal information from video features and word features is aggregated into features of different modalities through a multi-layered bidirectional GRU network. The specific process is as follows: , , in, and These represent the video features and text features extracted using the C3D network and GloVe tool, respectively. and These represent video features and text features that aggregate global temporal information, respectively. This represents a commonly used bidirectional recurrent neural network that uses a gating mechanism to control the updating of weights and is able to capture temporal information in sequential data.

[0008] The specific operation of step S3 is as follows: The attention weights of video features and text features that aggregate global temporal information in both video and text modal modes are calculated using a co-attention method. The weights of highly correlated information are increased, while the weights of low-correlation information are decreased. The resulting weighted features are then multiplied by the video features aggregated with global temporal information in the video modal and the text features aggregated with global temporal information in the text modal, respectively, to obtain the video features and text features after preliminary interaction. The specific calculation process is as follows: , , in , These represent the video features and text features after initial interaction, respectively. This represents the normalized exponential function, through which the weights of attention are calculated. This represents the convolution operation. This represents video features that aggregate global temporal information. This represents text features that aggregate global temporal information.

[0009] Step S4 is as follows: Candidate matrix features are generated based on candidate matrix fragments. These candidate moment features are categorized into start boundary features, end boundary features, and content boundary features. Two-dimensional mapping features are then generated from these features for representation. The two-dimensional mapping features are then visualized, yielding two-dimensional feature maps corresponding to the three features, namely the start boundary two-dimensional feature map. 2D feature map of the end boundary and content boundary two-dimensional feature map The calculation process is as follows: , , , , , , in, Indicates the first The initial boundary features of candidate moments, Indicates the first The termination boundary features of candidate moments, Indicates the first Content boundary features of each candidate moment Indicates the first Video features of the starting positions of candidate moments Indicates the first Video features at the end positions of candidate moments Represents timestamp to timestamp Features between This indicates a max pooling operation. This represents the total number of candidate moments. Indicates the first One candidate matrix.

[0010] Step S5 is as follows: The video and text features obtained from the initial interaction are re-encoded. For the video features, they are passed through a linear layer... The activation function operation, combined with the original interactive video features, yields fine-grained video features. For the text features after interaction, convolution with different kernels is used to obtain phrase features of the text. Then, all the obtained phrase features are concatenated along the feature dimension to obtain the text's phrase features. The calculation process is as follows: , , in, and These represent the text features and video features after the initial interaction, respectively. Represents linear layer operations. This represents the activation function operation. , and These represent convolution operations with kernels of 1, 2, and 3, respectively. This indicates a dimension concatenation operation.

[0011] Step S6 is as follows: A second information exchange is performed to obtain fine-grained video features. and phrase features of the text Respectively with the initial boundary features End boundary features By combining these, more fine-grained video intermodal information and text intermodal information are obtained. Then, the obtained information is combined with content boundary features. The fusion yields four complementary fusion features. , , , The four complementary fusion features are then combined to obtain the final fusion feature. The specific calculations are as follows: , , , , , in, This indicates the first complementary fusion feature. This indicates the second complementary fusion feature. This represents the third complementary fusion feature. This represents the fourth complementary and fusion characteristic. Indicates the final fusion characteristics, Represents fine-grained features of the video. Phrases representing text features Represents the normalized exponential function, Represents linear layer operations. This indicates the average pooling operation.

[0012] Step S7 is as follows: The final fused features are evaluated through a multi-scale candidate matrix interaction method. Features are obtained through processing Then the features The predicted scores of each candidate moment are obtained after convolutional dimensionality reduction and the Sigmoid function. Cross-entropy loss is then applied to the predicted scores of the candidate moments. The calculation is used to optimize the results, and the predicted score is approximated with the Iou of the candidate moments. The candidate moment with the highest score is selected from the predicted scores of each candidate moment as the final result of the text video retrieval. The specific calculations are as follows: , in, This represents the activation function. This represents the convolution operation. Indicates a splicing operation; Then, interactions between candidate boxes are performed. The convolution operation includes three stacked convolutions, each containing four subgroups. The dilation factors of the first convolution in each group are 1, 2, and 2, respectively. The group size of all convolutions is 32. Each group of convolutions learns the relationships between candidate boxes at different scales, ultimately yielding three fused features. These three fused features are then concatenated to obtain the final fused feature. The calculation process is as follows: , , , , in, 1. 2 represents the first convolution with an expansion factor of 1 and 2, respectively. Convolution operations, specifically Concat, represent feature concatenation. Indicates the final fusion features; Finally, based on the obtained candidate matrix features, i.e., the fused features After convolution and The function obtains the predicted scores for each candidate matrix. The process is as follows: , For predicted scores Apply constraints by using the first Prediction scores of candidate matrices The intersection and union Iou of the video time segments corresponding to the candidate moments and the text in the video are approximated using the cross-entropy loss as follows: , in, Indicates the number of valid candidate matrices. Indicates the first The intersection and union of each candidate moment and the corresponding video time segment in the video text, Iou. Indicates the first The predicted scores of each candidate matrix.

[0013] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. The above technical solutions have the following advantages or beneficial effects: This invention addresses the text-based video retrieval problem by proposing a boundary-sensitive multi-scale candidate interaction-based video sentence retrieval method. This method employs a novel interaction mechanism between candidate moment information and other information. First, candidate moments are generated from video segment-level features that have already undergone coarse-grained interaction with the text. Then, these candidate moments are categorized into start boundary features, end boundary features, and content features based on their location. Finally, these three moment features are interacted with the video and text features extracted from coarse-grained to fine-grained features to obtain the final candidate moment features. For the relationships between candidate moments, multiple convolutions at different scales are used for interaction. This allows the method to utilize boundary features to understand complex video and text information. Furthermore, the invention provides a multi-scale candidate rectangle interaction module, enabling the model to better capture the relationships between candidate rectangles. This information can serve as supplementary information for the candidate rectangles, helping the model correct previously overlooked errors. Attached Figure Description

[0014] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0015] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0016] Figure 2 This diagram illustrates the comparison between the results of incorporating the technical solution of this invention into the benchmark model and the benchmark model. Figure 1 .

[0017] Figure 3 This diagram illustrates the comparison between the results of incorporating the technical solution of this invention into the benchmark model and the benchmark model. Figure 2 . Detailed Implementation

[0018] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, the components and arrangements of specific examples are described below.

[0019] Example 1 like Figure 1 As shown, a video sentence retrieval method based on boundary-sensitive multi-scale candidate interaction includes the following steps: S1. Extract video features from the video using I3D and C3D networks, and extract features of each word in the text using the GloVe tool; S2. Establish a temporal information model based on video features and word features, and aggregate the temporal information from video features and word features to obtain global temporal information; S3. Perform initial interaction on the global temporal information to obtain the video features and text features after interaction; S4. Generate candidate matrix features based on the video features in S3, generate 2D mapping features for representation from the candidate matrix features, and then visualize the 2D mapping features to obtain a two-dimensional feature map. S5. Re-encode the video features and text features obtained from the initial interaction to obtain fine-grained features of the video and phrase features of the text. S6. Perform a second information interaction, combine the information obtained from the first interaction with the candidate matrix features to obtain more granular video intermodal information and text intermodal information, then fuse the two intermodal information to obtain complementary fusion features, and then splice the complementary fusion features to obtain the final fusion features. S7. The final fused features are processed by multi-scale candidate matrix interaction, and then the prediction scores of each candidate matrix are obtained through calculation. The prediction scores of the candidate matrices are optimized by cross-entropy loss calculation to obtain the prediction scores of the most accurate candidate matrices. These scores are then used for text video retrieval to obtain the final results.

[0020] In this embodiment, each video feature obtained in step S1 is 256-dimensional. For text processing, a 300-dimensional GloVe model is used to encode each word in the text, ultimately resulting in a 300-dimensional word group.

[0021] In a specific implementation, based on step S1, the features of each word in the text and each segment in the video are obtained. There are temporal relationships between words and segments, therefore it is necessary to capture this temporal information to obtain more accurate features. In this embodiment, temporal information is modeled for the obtained video and text features respectively. For both video and text features, a multi-layer bidirectional GRU network is used to aggregate their respective temporal information. Since the video duration varies, the number of video feature segments obtained also varies. To achieve a unified representation, convolution and pooling are used to standardize the video features; the process is as follows: , , in, and These represent the video features and text features extracted using the C3D network and GloVe tool, respectively. and These represent video features and text features that aggregate global temporal information, respectively. This represents a commonly used bidirectional recurrent neural network that uses a gating mechanism to control the updating of weights and is able to capture temporal information in sequential data.

[0022] In a specific implementation, in step S3, attention weights are calculated for video features and text features that aggregate global temporal information in both video and text modes using a common attention method. The weights of highly correlated information are increased, and the weights of low-correlation information are decreased. Then, the obtained weighted features are multiplied by the video features that aggregate global temporal information in the video mode and the text features that aggregate global temporal information in the text mode, respectively, to obtain the video features and text features after preliminary interaction. The specific calculation process is as follows: , , in , These represent the video features and text features after initial interaction, respectively. Represents the normalized exponential function, This represents the convolution operation. Indicates linear layer operations. This represents the video features obtained from the encoding. This represents text features that aggregate global temporal information.

[0023] In a specific implementation, step S4 generates candidate matrix features based on candidate matrix fragments, categorizes these features into start boundary features, end boundary features, and content boundary features, and generates two-dimensional mapping features for representation. The two-dimensional mapping features are then visualized to obtain two-dimensional feature maps corresponding to the three features, namely the start boundary two-dimensional feature map. 2D feature map of the end boundary and content boundary two-dimensional feature map The calculation process is as follows: , , , , , , in, Indicates the first The initial boundary features of candidate moments, Indicates the first The termination boundary features of candidate moments, Indicates the first Content boundary features of each candidate moment Indicates the first Video features of the starting positions of candidate moments Indicates the first Video features at the end positions of candidate moments This indicates a max pooling operation. This represents the total number of candidate moments. Indicates the first One candidate moment, Indicates the first One candidate matrix.

[0024] In a specific implementation, step S5 re-encodes the video features and text features obtained from the initial interaction. For the video features, they are processed through a linear layer and a ReLU activation function, then added to the original video features to obtain fine-grained video features. For the text features, they are processed through convolutions with different kernels to obtain phrase features. All the resulting phrase features are then concatenated along the feature dimension. The calculation process is as follows: , , in, and These represent the text features and video features after the initial interaction, respectively. Linear represents the linear layer operation, ReLU represents the activation function operation, Conv1, Conv2, and Conv3 represent the convolution operations with kernels of 1, 2, and 3, respectively, and Concat represents the dimension concatenation operation.

[0025] In a specific implementation, a second information exchange is performed in step S6, where the obtained fine-grained video features are... and phrase features of the text Respectively with the initial boundary features End boundary features By combining the video intermodal information and text intermodal information, more fine-grained information is obtained, and then this information is combined with content boundary features. The fusion yields four complementary fusion features. , , , The four complementary fusion features are then combined to obtain the final fusion feature. The specific calculations are as follows: , , , , , in, This indicates the first complementary fusion feature. This indicates the second complementary fusion feature. This represents the third complementary fusion feature. This represents the fourth complementary and fusion characteristic. Indicates the final fusion characteristics, Represents fine-grained features of the video. Phrases representing text features Represents the normalized exponential function, Represents linear layer operations. This indicates the average pooling operation.

[0026] In a specific implementation, step S7 uses a multi-scale candidate matrix interaction method to refine the final fused features. Features are obtained through processing Then the features The predicted scores of each candidate matrix are obtained after convolutional dimensionality reduction and the Sigmoid function. Cross-entropy loss is then applied to the predicted scores of the candidate matrices. The results are optimized by calculating the predicted score to approximate the Iou of the candidate moments, and the candidate moment with the highest predicted score among all candidate moments is selected as the final result of the text video retrieval. Specifically as follows: Within the context of the same video, there are correlations between segmented candidate intervals. For example, if a person performs the same action twice consecutively, the segments corresponding to the first and second occurrences of the action will have similarities but also differences. If only a single candidate interval is considered, the model may struggle to determine which segment corresponds to the second occurrence of the action. Therefore, individual candidate intervals cannot be predicted independently; they need to interact. This interaction is crucial for the model to learn more discriminative features and obtain supplementary information. Before applying convolution, a residual connection is first implemented to extract the initial boundary features of the obtained candidate boxes. End boundary features and content features Fusion features in the feature dimension The connections are made, and then the fused features are obtained through a residual layer. The interaction process is as follows: , in, This represents the activation function. This represents the convolution operation. Indicates a splicing operation; Then, interactions between candidate boxes are performed. The convolution operation includes three stacked convolutions, each containing four subgroups. The dilation factors of the first convolution in each group are 1, 2, and 2, respectively. The group size of all convolutions is 32. Each group of convolutions learns the relationships between candidate boxes at different scales, ultimately yielding three fused features. These three fused features are then concatenated to obtain the final fused feature. The calculation process is as follows: , , , , in, 1. 2 represents the first convolution with an expansion factor of 1 and 2, respectively. Convolution operations, specifically Concat, represent feature concatenation. Indicates the final fusion features; Finally, based on the obtained candidate matrix features, i.e., the fused features After convolution and The function obtains the predicted scores for each candidate matrix. The process is as follows: , For predicted scores Apply constraints by using the first Prediction scores of candidate matrices The intersection and union Iou of the video time segments corresponding to the candidate moments and the text in the video are approximated using the cross-entropy loss as follows: , in, Indicates the number of valid candidate matrices. Indicates the first The intersection and union of each candidate moment and the corresponding video time segment in the video text, Iou. Indicates the first The predicted scores of each candidate matrix.

[0027] Example 2 This invention uses the I3D model for the Charades-STA dataset, uses the pre-trained C3D model to extract features from the TACoS dataset, and uses the pre-trained GloVe to encode words into 300-dimensional vectors for text encoding. To improve model speed, a hidden dimension of 256 is used. For TACoS, the sampling of candidate time steps is set to 128, while for Charades-STA, 64 sampling steps are used. During training, a learning rate of 10⁻³ was used. In the candidate interaction module, three sets of convolutions were used, each set containing four grouped convolutions. The inflation factors of the first convolution in each set were 1, 2, and 2, respectively, and the group size of all convolutions was 32. During the testing process, the video features are first extracted using the I3D or C3D model, and the text is encoded using the GloVe model. Then, the obtained features are processed by a bidirectional GRU to extract temporal information. After initial interaction between the two modalities, candidate moment features are obtained based on the features obtained from the initial interaction. Next, the video features and text features are processed by fine-grained encoding to obtain fine-grained information. Then, the fine-grained information is fused with the candidate moment information. The fused features of the candidate moments are interacted to learn the relationship between the candidate moments. Finally, prediction is performed based on the final fused features of the candidate moments to obtain a prediction score. Finally, the prediction scores are ranked, and the one with the highest prediction score is the final result. The method in this invention was compared with other methods on the Charades-STA dataset and the TACoS dataset. The experimental results are shown in Table 1 and Table 2.

[0028] Table 1 shows the comparison of the experimental results of the present invention with other methods on the Charades-STA dataset, and Table 2 shows the comparison of the experimental results of the present invention with other methods on the TACoS dataset.

[0029] Table 1 Comparison of the results of the method of this invention and other methods on the Charades-STA dataset. For the Charades-STA dataset, the results are 62.04 and 42.53 on R@1,Iou=0.5 and R@1,Iou=0.7, respectively, and 89.95 and 66.26 on R@1=5,Iou=0.5 and R@5,Iou=0.7, respectively.

[0030] Table 2 Comparison of the results of the method of the present invention with other methods on the TACOS dataset. For the TACoS dataset, the results are 48.86 and 37.24 on R@1,Iou=0.5 and R@1,Iou=0.7, respectively, and 69.58 and 57.74 on R@1=5,Iou=0.5 and R@5,Iou=0.7, respectively.

[0031] Example 3 To highlight the advantages of the method in this invention, the results are compared by incorporating the technical solution of this invention into a benchmark model and comparing them with those of the benchmark model. Figure 2 and Figure 3 As shown, given the search sentences "Query: The person puts down the bag." and "Query: Another person comes in takes the box away.", determine the video segment containing the given sentence. Figure 2 and Figure 3 It can be seen that, compared with the baseline model, the baseline model incorporating the technical solution of this invention can achieve results that are closer to the correct answer.

[0032] Although specific embodiments of the invention have been described in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the invention. Based on the technical solutions of the invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the invention.

Claims

1. A video sentence retrieval method based on boundary-sensitive multi-scale candidate interaction, characterized in that, Includes the following steps: S1. Extract video features from the video using I3D and C3D networks, and extract features of each word in the text using the GloVe tool; S2. Establish a temporal information model based on video features and word features, and aggregate the temporal information from video features and word features to obtain global temporal information; S3. Perform initial interaction on the global temporal information to obtain the video features and text features after interaction. The specific operation of step S3 is as follows: The attention weights of video features and text features that aggregate global temporal information in both video and text modal modes are calculated using a co-attention method. The weights of highly correlated information are increased, while the weights of low-correlation information are decreased. The resulting weighted features are then multiplied by the video features aggregated with global temporal information in the video modal and the text features aggregated with global temporal information in the text modal, respectively, to obtain the video features and text features after preliminary interaction. The specific calculation process is as follows: ， ， in , These represent the video features and text features after initial interaction, respectively. This represents the normalized exponential function, through which the weights of attention are calculated. This represents the convolution operation. This represents video features that aggregate global temporal information. This represents text features that aggregate global temporal information; S4. Generate candidate matrix features based on the video features in S3, generate 2D mapping features for representation from the candidate matrix features, and then visualize the 2D mapping features to obtain a two-dimensional feature map. Step S4 is as follows: Candidate matrix features are generated based on candidate matrix fragments. These candidate moment features are categorized into start boundary features, end boundary features, and content boundary features. Two-dimensional mapping features are then generated from these features for representation. The two-dimensional mapping features are then visualized, yielding two-dimensional feature maps corresponding to the three features, namely the start boundary two-dimensional feature map. 2D feature map of the end boundary and content boundary two-dimensional feature map The calculation process is as follows: ， ， ， ， ， ， in, Indicates the first The initial boundary features of candidate moments, Indicates the first The termination boundary features of each candidate moment, Indicates the first Content boundary features of each candidate moment Indicates the first Video features of the starting positions of candidate moments Indicates the first Video features at the end positions of candidate moments Represents timestamp to timestamp Features between This indicates a max pooling operation. This represents the total number of candidate moments. Indicates the first One candidate matrix; S5. Re-encode the video features and text features obtained from the initial interaction to obtain fine-grained features of the video and phrase features of the text. S6. Perform a second information interaction, combine the information obtained from the first interaction with the candidate matrix features to obtain more granular video intermodal information and text intermodal information, then fuse the two intermodal information to obtain complementary fusion features, and then splice the complementary fusion features to obtain the final fusion features. S7. The final fused features are processed by multi-scale candidate matrix interaction, and then the prediction scores of each candidate matrix are obtained through calculation. The prediction scores of the candidate matrices are optimized by cross-entropy loss calculation to obtain the prediction scores of the most accurate candidate matrices. These scores are then used for text video retrieval to obtain the final results.

2. The video sentence retrieval method based on boundary-sensitive multi-scale candidate interaction according to claim 1, characterized in that: The video features obtained in step S1 are 256-dimensional for each segment and 300-dimensional for each word.

3. The video sentence retrieval method based on boundary-sensitive multi-scale candidate interaction according to claim 2, characterized in that, Step S2 is as follows: Global temporal information from video features and word features is aggregated into features of different modalities through a multi-layered bidirectional GRU network. The specific process is as follows: ， ， in, and These represent the video features and text features extracted using the C3D network and GloVe tool, respectively. and These represent video features and text features that aggregate global temporal information, respectively. This represents a commonly used bidirectional recurrent neural network that uses a gating mechanism to control the updating of weights and is able to capture temporal information in sequential data.

4. The video sentence retrieval method based on boundary-sensitive multi-scale candidate interaction according to claim 3, characterized in that, Step S5 is as follows: The video and text features obtained from the initial interaction are re-encoded. For the video features, they are passed through a linear layer... The activation function operation, combined with the original interactive video features, yields fine-grained video features. For the text features after interaction, convolution with different kernels is used to obtain phrase features of the text. Then, all the obtained phrase features are concatenated along the feature dimension to obtain the text's phrase features. The calculation process is as follows: ， ， in, and These represent the text features and video features after the initial interaction, respectively. Represents linear layer operations. This represents the activation function operation. , and These represent convolution operations with kernels of 1, 2, and 3, respectively. This indicates a dimension concatenation operation.

5. The video sentence retrieval method based on boundary-sensitive multi-scale candidate interaction according to claim 4, characterized in that, Step S6 is as follows: A second information exchange is performed to obtain fine-grained video features. and phrase features of the text Respectively with the initial boundary features End boundary features By combining these, more fine-grained video intermodal information and text intermodal information are obtained. Then, the obtained information is combined with content boundary features. The fusion yields four complementary fusion features. , , , The four complementary fusion features are then combined to obtain the final fusion feature. The specific calculations are as follows: ， ， ， ， ， in, This indicates the first complementary fusion feature. This indicates the second complementary fusion feature. This represents the third complementary fusion feature. This represents the fourth complementary and fusion characteristic. Indicates the final fusion characteristics, Represents fine-grained features of the video. Phrases representing text features Represents the normalized exponential function, Represents linear layer operations. This indicates the average pooling operation.

6. The video sentence retrieval method based on boundary-sensitive multi-scale candidate interaction according to claim 5, characterized in that, Step S7 is as follows: The final fused features are evaluated through a multi-scale candidate matrix interaction method. Features are obtained through processing. Then the features The predicted scores of each candidate moment are obtained after convolutional dimensionality reduction and the Sigmoid function. Cross-entropy loss is then applied to the predicted scores of the candidate moments. The calculation is used to optimize the results, and the predicted score is approximated with the Iou of the candidate moments. The candidate moment with the highest score is selected from the predicted scores of each candidate moment as the final result of the text video retrieval. The specific calculations are as follows: ， in, This represents the activation function. This represents the convolution operation. Indicates a splicing operation; Then, interactions between candidate boxes are performed. The convolution operation includes three stacked convolutions, each containing four subgroups. The dilation factors of the first convolution in each group are 1, 2, and 2, respectively. The group size of all convolutions is 32. Each group of convolutions learns the relationships between candidate boxes at different scales, ultimately yielding three fused features. These three fused features are then concatenated to obtain the final fused feature. The calculation process is as follows: ， ， ， ， in, 1. 2 represents the first convolution with an expansion factor of 1 and 2, respectively. Convolution operations, specifically Concat, represent feature concatenation. Indicates the final fusion characteristics; Finally, based on the obtained candidate matrix features, i.e., the fused features After convolution and The function obtains the predicted scores for each candidate matrix. The process is as follows: ， For predicted scores To impose constraints, by... Prediction scores of candidate matrices The intersection and union Iou of the video time segments corresponding to the candidate moments and the text in the video are approximated using the cross-entropy loss as follows: ， in, Indicates the number of valid candidate matrices. Indicates the first The intersection and union of each candidate moment and the corresponding video time segment in the video text, Iou. Indicates the first The predicted scores of each candidate matrix.

Images