Video segment positioning system based on spatio-temporal semantic decomposition

Abstract

The application belongs to the technical field of multi-modal retrieval, and particularly relates to a video segment positioning system based on spatiotemporal semantic decomposition. The system comprises a video and text encoder module, a spatiotemporal semantic decomposition module, a multi-modal attention module and a segment boundary positioning module. The video and text encoder module respectively extracts features of the video and the text to generate features containing context information. The spatiotemporal semantic decomposition module divides the text into spatial words and temporal words according to the part of speech, and samples the video at different frame rates to obtain fast video stream and slow video stream. The multi-modal attention module first calculates the mutual attention between the spatial words and the slow video stream, fuses the result with the fast video stream, and then calculates the mutual attention between the temporal words and the fast video stream to obtain the fusion features of the two modalities. The segment boundary positioning module first calculates the frame of the end point of the target segment, and then regresses the start point frame in the frame before the predicted end point frame. The application can improve the discrimination ability for similar queries and improve the system performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-modal retrieval, and particularly relates to a video segment positioning system based on spatio-temporal semantic decomposition. BACKGROUND

[0002] With the rapid growth of video resources on the Internet, video understanding related tasks are attracting more and more researchers. As one of the problems worth exploring, the goal of video segment positioning is to locate the segment described by the query for a given long video without cutting and a query expressed in natural language. This task plays a crucial role in video content analysis and video retrieval. Since the query of the video segment positioning task can be any natural language sentence, its flexibility and usability far exceed the previous video segment detection, classification and other related tasks using fixed class labels.

[0003] The mainstream method of the video segment positioning task trains a classification model to determine whether each candidate interval matches the query or whether each frame is a start / end frame. For the processing of video and text, existing works generally first use independent feature encoders to encode the video / text, and then use attention mechanisms, graph neural networks and other modules to fuse video and text features to calculate the matching of text semantics in the video frame. In the above processing, existing works often treat video / text as a whole, ignoring the semantics of both in the spatio-temporal dimension. For the video segment positioning task, due to the integrity of the video, different text queries often contain the same spatial semantics such as objects and scenes, but different temporal semantics such as actions and displacements. The system decomposes video and text separately in space and time, and has a finer granularity than existing works when calculating multi-modal attention.

[0004] Early research mainly adopts the mode of first proposing candidate intervals and then calculating matching scores. In this mode, the system needs to first enumerate a series of candidate intervals of different scales and different positions, then calculate the matching score of each candidate interval with the query one by one, and finally select the interval with the highest matching score as the output. However, this two-stage method has the following disadvantages: (1) In order to make the prediction results cover all possible intervals as much as possible, the system needs to propose as many candidate intervals as possible, which consumes a lot of computing resources; (2) In order to balance the contradiction between the amount of calculation and the performance of the system, the distribution of candidate intervals needs to be designed according to the prior situation of the data set, so that fewer candidate intervals can cover more training samples. However, this design needs to be carefully designed for different data sets, which requires a certain amount of work and is not scalable. To overcome the shortcomings of the two-stage method, existing methods propose two alternative strategies: (1) Model the edge distribution of whether each frame is the starting point / endpoint, and take the frame with the highest probability as the predicted boundary; (2) Directly regress the normalized starting point and endpoint coordinates on the video features. Although these two strategies get rid of the shortcomings of the two-stage method, the performance is often not as good as the two-stage method because both strategies assume the independence between the starting point and the endpoint of the target segment. The two-step prediction strategy of the segment boundary positioning module in the system adopts the calculation process of first classifying the endpoint frame and then regressing the starting point frame, and explicitly establishes the conditional dependence between the starting point and the endpoint of the target segment, thereby inheriting the low computational complexity of the above two strategies and improving the positioning performance of the system. SUMMARY

[0005] The purpose of the present application is to provide a video segment positioning system based on spatiotemporal semantic decomposition to solve the problem that existing models cannot distinguish queries with the same spatial semantics in video segment positioning.

[0006] The application provides a video segment positioning system based on spatio-temporal semantic decomposition, comprising a video and text encoder module, a spatio-temporal semantic decomposition module, a multi-modal attention module and a segment boundary positioning module; the video and text encoder module respectively extracts features of the video and the text to generate features containing context information; the spatio-temporal semantic decomposition module divides the text into spatial words and temporal words according to the part of speech, and samples the video at different frame rates to obtain fast video stream and slow video stream; the multi-modal attention module first calculates the mutual attention between the spatial words and the slow video stream, fuses the result with the fast video stream, and then calculates the mutual attention between the temporal words and the fast video stream to obtain the fusion features of the two modalities; the segment boundary positioning module first calculates the frame of the end point of the target segment, and then regresses the start point frame in the frame before the predicted end point frame. The application improves the discrimination ability for similar queries by respectively calculating the spatio-temporal semantics of the video and the text, and taking the spatial semantics appearing in the video frame as the priori of positioning in the time dimension. The two-step prediction method used by the segment boundary positioning module explicitly establishes the conditional dependency between the start point and the end point of the target segment, further improving the system performance.

[0007] In the application, the video encoder module is composed of a backbone network and a Transformer encoder; and the text encoder module is composed of a backbone network, a Transformer encoder and a relation graph convolution network.

[0008] The backbone network in the video encoder module adopts a C3D and I3D deep convolutional neural network to extract video features, and extracts a 4,096-dimensional (C3D) or 1,024-dimensional (I3D) vector from the center frame image and its adjacent 31 frame images. The Transformer encoder in the video encoder module comprises two isomorphic computing units, which act on the features extracted by the backbone network, encode the position information and context information of the video frame into the features, and the output video feature dimension is Txd.

[0009] In the application, the backbone network in the text encoder module adopts a Glove word vector to extract text features. The Transformer encoder in the text encoder module has the same structure as the Transformer encoder in the video encoder module, but does not share parameters. The text features extracted by the backbone network are context-encoded by the Transformer encoder, and finally relation-encoded by the relation graph convolution with residual connection on the syntax dependency tree structure of the text. The process is represented as:

[0010]

[0011] Among them, Xi, l+1 represents the feature of the ith word in the (l+1)-th layer relation graph convolution; R represents the set of relations on the edge in the syntactic dependency tree structure; Xi, l represents the neighbor of the ith word in the syntactic dependency tree structure with the relation r on the edge; is the learnable parameter in the l-th layer relation graph convolution. Since there are 55 kinds of relations on the edge in the syntactic dependency tree structure, the number of relations doubles after considering the direction of the edge, which will introduce too many parameter matrices In the present application, the method of basis decomposition is used to reduce the number of parameters. The process is represented as:

[0012]

[0013] wherein Xi, l+1 represents the feature of the ith word in the (l+1)-th layer relation graph convolution; R represents the set of relations on the edge in the syntactic dependency tree structure; is the learnable linear combination coefficient.

[0014] In the present application, the spatio-temporal semantic decomposition module decomposes the video and the text into respective temporal semantics and spatial semantics. The spatio-temporal decomposition of the video takes the original frame rate video as a fast video stream, and performs sparse sampling on the fast video stream with T frames to obtain a slow video stream with T slow frames by selecting one frame every four frames. The spatio-temporal decomposition of the text first performs part-of-speech tagging on the text in the query, and takes the nouns and pronouns in the query as spatial words, and takes the verbs and adverbs as temporal words.

[0015] In the present application, the multi-modal attention module is used to calculate the multi-modal attention of the slow video stream and the spatial words as the prior of the multi-modal attention of the fast video stream and the temporal words. The process of calculating the multi-head attention between the two modalities can be represented as:

[0016]

[0017] MultiHead(X, Y) = [Attention 1...h (X, Y)]W O , #(4)

[0018] wherein W Q , W K , W V and W O are learnable linear transformation parameters; h represents the number of heads used by the multi-head attention; X represents the feature of the video modality, Y represents the feature of the text modality; d represents the dimension of the feature; and sigma represents the softmax function. The process of calculating the multi-modal attention of the slow video stream and the spatial words is represented as:

[0019]

[0020]

[0021] where Q s represents the spatial word; represents the calculation result of the slow video stream in the l-th layer multimodal attention; MultiHead represents the multi-head attention mechanism; LN represents the layer normalization; and MLP represents a two-layer perceptron. After the multimodal attention calculation of the slow video stream and the spatial word, the slow video stream is more focused on each frame and the corresponding spatial semantics in the text. To use the multimodal attention calculation result of the slow video stream and the spatial word as prior knowledge for positioning in the time dimension, the application uses the attention map of the normalized slow video stream in the last layer multimodal attention, and the calculation method is as follows:

[0022]

[0023] where N s represents the number of spatial words; A is the attention map of the slow video stream and the spatial word in the last layer multimodal attention; and A i represents the two-dimensional attention map in the i-th head in the multi-head attention, with the dimension of T slow ×N s . represents the j-th column of the two-dimensional attention map in the i-th head in the multi-head attention; and minmax represents the min-max normalization. The value corresponding to each frame in the normalized attention map a reflects the attention degree of the frame in the multimodal attention with the spatial word, and the higher the value, the more the spatial semantics such as objects and scenes described in the text appear in the frame. The system adds the features of the slow video stream output by the last layer multimodal attention and the fast video stream V fast , and multiplies the normalized attention map a and the fast video stream V fast . The process is represented as follows:

[0024]

[0025]

[0026]

[0027] where upsample represents the up-sampling operation; and represents the element-wise multiplication. represents a vector with all elements being 1. Since the number of frames of the slow video stream does not match that of the fast video stream, the application uses linear interpolation to up-sample the slow video stream to T fast frames before fusion. After fusing the features of the slow video stream and the attention map a, the application calculates the fast video stream V fast,fusedand temporal words, to obtain the final fused video and text spatio-temporal semantic feature V st , whose dimension is Txd.

[0028] In the present application, the segment boundary positioning module first predicts the end point of the segment according to the modeled end point edge distribution P of the segment The process is represented as:

[0029]

[0030] The end point edge distribution P is modeled using a two-layer perception machine. The perception machine acts independently on each position of the fused feature V st , and the probability distribution P is obtained by using softmax normalization on the outputs of all positions. The segment boundary positioning module further includes a regressor for modeling the conditional expectation of the start point, which outputs the expected value of the normalized start frame E[start|end=i] for the i-th frame, and predicts the start point The process is represented as:

[0031]

[0032] The regressor for the conditional expectation E[start|end=i] is a two-layer perception machine that acts independently on each position of the fused feature V st , and outputs the normalized start frame after sigmoid activation function. The segment boundary positioning module obtains significant performance improvement by explicitly modeling the conditional dependency between the start point and the end point of the segment.

[0033] The present application uses an end-to-end training method to train the final output of the model, i.e., the end point edge distribution P and the normalized start point conditional expectation E[start|end=i] with supervision signals. For the supervision signal design of the end point edge distribution P, existing works often use simple one-hot encoding, i.e., the label of the end point frame is 1, and the labels of other frames are 0. Due to the large number of frames in long videos and only one end point frame, this simple label setting method has a serious class imbalance problem, resulting in poor performance of the model. Some existing works use label smoothing techniques, i.e., change the one-hot encoded label to a Gaussian distribution label centered on the end point frame, so that several frames near the end point frame will have label values between 0 and 1. Although the label smoothing technique plays a regularization role and improves the generalization ability of the model, the assumed label distribution is independent of the video segment positioning task. The present application proposes to use the IoU of the predicted interval with the fixed start point and the real interval as the supervision information, and the calculation process of the label value of the i-th frame is represented as:

[0034]

[0035] where s is the true value of the start point; e is the true value of the end point. According to formula (13), the label value of the end point frame e is 1, which is consistent with the above two label setting modes. The label value of the frame after the start point frame is set as the IoU of the interval [s, i] and the true interval [s, e], and the label value becomes larger as i is closer to e from left to right, and becomes smaller as i is farther away from e when i is greater than e. Since the video segment positioning task uses the IoU based on the prediction interval and the true interval as the evaluation index, the label setting mode based on the interval IoU used by the present application is closer to the goal of the video segment positioning task than the above two label setting modes, and better performance is achieved. The training target of the end point edge distribution P is to minimize the binary cross entropy function:

[0036]

[0037] wherein, is the end point edge distribution without softmax normalization. For the conditional expectation E[start|end=i] of the normalized start point, the training target is to minimize the SmoothL1 loss function:

[0038]

[0039] Finally, the training target of the present application is the sum of the above two loss functions:

[0040] L=L end +L start

[0041] The video segment positioning system based on spatiotemporal semantic decomposition provided by the present application has a working process summarized as follows:

[0042] (I) First, according to the encoder module of the video and the text, the pre-trained backbone network is used to extract the features of the video and the text respectively, and then the position information and the context information are fused using the Transformer encoder; for the text query, the relationship graph convolution network is used to encode the relationship on the syntax dependency tree structure;

[0043] (II) The encoded video features and text features are used as the input of the spatiotemporal semantic decomposition module, the text is divided into spatial words and temporal words according to the part of speech, and the video is sampled at different frame rates to obtain fast video stream and slow video stream;

[0044] ​(Three) Taking the spatial words and the time words, the fast video stream and the slow video stream as the inputs of the multi-modal attention module, first calculating the multi-modal attention of the spatial words and the slow video stream, adding the fused features to the fast video stream, normalizing the attention map in the last layer of the multi-modal attention and multiplying the fast video stream, and then calculating the multi-modal attention of the time words and the fast video stream;

[0045] (Four) Taking the fused video features as the input of the segment boundary positioning module, predicting the end point of the segment according to the modeled segment end point distribution using the greedy strategy, and then regressing the start point frame in the frame before the predicted end point frame;

[0046] (Five) Using the IoU of the predicted interval and the real interval as the supervision information, and taking minimizing the binary cross entropy loss function and the SmoothL1 loss function as the target to train the network end to end.

[0047] The advantages of the present application include:

[0048] Firstly, aiming at the problem that the existing video segment positioning work cannot distinguish the queries with similar spatial semantics such as objects and scenes for the same video, the present application proposes a video segment positioning system based on spatio-temporal semantic decomposition, which realizes the decomposition of the spatio-temporal semantics of video and text and the fine-grained semantic matching;

[0049] Secondly, the multi-modal attention calculation of the spatial dimension first and then the time dimension is first proposed, the matching result of the spatial semantics is taken as the prior of positioning in the time dimension, and the frames in the video which do not appear the spatial semantics in the query are given a smaller weight; the multi-modal attention in the time dimension is based on the result of the spatial dimension multi-modal attention, which highlights the importance of the time dimension in the segment positioning;

[0050] Thirdly, the two-step prediction boundary calculation method is first proposed, which first calculates the edge distribution of the end point distribution and then regresses the conditional expectation of the start point, and by explicitly establishing the conditional dependence between the start point and the end point of the segment, the performance is greatly improved compared with the existing boundary calculation method;

[0051] Finally, the present application achieves the best video segment positioning result on the public datasets Charades-STA, ActivityNet-Captions and TACoS, and the positioning result has interpretability. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is the system diagram of the present application.

[0053] Figure 2 is the complete framework diagram of the model in the present application. DETAILED DESCRIPTION

[0054] As can be known from the background art, most previous studies encode features of videos and texts and match them in a multi-modal manner, taking videos / texts as a whole and ignoring their respective semantics in the spatial and temporal dimensions. In the video segment positioning task, queries about the same video often involve the same spatial semantics such as objects and scenes, while the temporal semantics such as actions and transformations contained in the queries play a more crucial role in positioning. To address this problem, the present application provides a video segment positioning system based on spatial-temporal semantic decomposition, which separately processes the spatial-temporal semantics of videos and texts. The system decomposes videos and texts according to their frame rates and part-of-speech tagging results, and performs multi-modal attention calculation in the spatial dimension first and then in the temporal dimension. The two-step prediction method in the present application significantly improves positioning accuracy by explicitly modeling the conditional dependency between the start and end points of the target segment compared to existing prediction methods.

[0055] The spirit and essence of the present application are further described below in conjunction with the accompanying drawings and examples.

[0056] As shown in Figure 1 The video segment positioning system based on spatial-temporal semantic decomposition of the present application includes video and text encoder modules, a spatial-temporal semantic decomposition module, a multi-modal attention module, and a segment boundary positioning module. The original video is converted into a series of video frames after frame extraction, and the video frames are encoded into feature vectors containing contextual features via the video encoder module. The original text is preprocessed and then encoded into feature vectors containing contextual features via the text encoder module. The video features are converted into fast video streams containing action information and slow video streams containing static object information after passing through the spatial-temporal semantic decomposition module. The text features are converted into temporal words containing action information and spatial words containing static object information after passing through the spatial-temporal semantic decomposition module. The multi-modal attention module first calculates the multi-modal attention between the spatial words and the slow video streams to obtain the activation values of each frame interacting with the spatial words. The activation values are used as priors in the spatial dimension and are fused with the fast video streams to further calculate the multi-modal attention between the temporal words and the fused video streams. Finally, the segment boundary positioning module predicts the start and end points of the positioned segment based on the features fused with spatial-temporal information.

[0057] In this embodiment, the backbone network in the video encoder module uses a C3D deep convolutional neural network. After the original video is decimated at a frame rate of 25 fps, a series of video image frames is obtained. The C3D network divides these video frames into 1 unit every 16 frames, and each unit is extracted into a 4096-dimensional vector containing action information after passing through the C3D network. In order to save computing overhead, principal component analysis method is further used to reduce each 4096-dimensional vector to 500 dimensions. The encoding dimension of the Transformer encoder in the video encoder module is 512 dimensions, the number of layers is 2 layers, and the number of heads in the multi-head attention is 8. For programming convenience, all videos will be sampled with the same video frame. Finally, the encoded video features containing context information have a dimension of Txd, where T = 128 and d = 512.

[0058] In this embodiment, the backbone network in the text encoder module uses a Glove word vector. After the original text is preprocessed by converting uppercase letters to lowercase letters, English word segmentation, removing stop words, etc., it becomes a word sequence. Each word is mapped to a 300-dimensional Glove word vector after passing through the Glove word table. Words not in the word table are uniformly mapped to a special word <unk>For coding convenience, all texts are uniformed to 45 words, if more than 45 words, the sentence is truncated, if less than 45 words, special words <pad>The padding is performed. The Transformer encoder in the text encoder module and the Transformer encoder in the video encoder module have the same parameter settings, but do not share weights. In this embodiment, the syntax dependency tree of the text is extracted using the open source natural language processing tool stanza of Stanford University. A syntax dependency tree is a directed acyclic graph structure, each node in the graph corresponds to a word in the sentence; each edge in the graph has a relationship label, and the syntax relationship between the two words connected by the edge. Relation encoding is performed on the syntax dependency tree using a relation graph convolution with residual connection. The process is represented as:

[0059]

[0060] wherein, represents the feature of the ith word in the (l+1)th layer relation graph convolution; R represents the set of relations on the edges in the syntax dependency tree structure; represents the neighbor of the ith word in the syntax dependency tree structure with the relationship r on the edge; is a learnable parameter in the lth layer relation graph convolution. The relation graph convolution network used in this embodiment has 2 layers, and the total number of relations |R| is 55, The dimension of is 512x512, that is, the dimension of the text feature remains unchanged in the calculation process of the graph convolution network.

[0061] In this embodiment, the spatio-temporal decomposition of the video is performed, and the original frame rate video is used as a fast video stream. The slow video stream with T slow frames is obtained by sparse sampling of every 4 frames in the fast video stream with T frames. In this embodiment, the spatio-temporal decomposition of the text is performed, and the text in the query is part-of-speech tagged using the open source natural language processing tool stanza of Stanford University. The words in the query that are tagged as nouns and pronouns are used as spatial words, and the verbs and adverbs are used as temporal words. If no verb or adverb is detected in a sentence, the gerund in the sentence will be extracted as a temporal word.

[0062] In this embodiment, the multi-modal attention module calculates the multi-modal attention of the slow video stream and the spatial word as the prior of the multi-modal attention of the fast video stream and the temporal word:

[0063]

[0064]

[0065] wherein, Q s represents the spatial word; denotes the computation result of the slow video stream in the l-th layer multi-modal attention; MultiHead denotes the multi-head attention mechanism; LN denotes layer normalization; MLP denotes a two-layer perceptron with a hidden layer dimension of 2048 and an output layer dimension of 512. In this embodiment, three layers of multi-modal attention are used. Then, the normalized attention map of the slow video stream in the last layer of multi-modal attention is calculated:

[0066]

[0067] The slow video stream output by the last layer of multi-modal attention is denoted as: Feature and fast video stream V fast is added, and the normalized attention map a is multiplied by the fast video stream V fast :

[0068]

[0069]

[0070]

[0071] wherein the upsampling operation upsample uses linear interpolation. Finally, the fast video stream V fast,fused is calculated, and the multi-modal attention between the time words is calculated to obtain the final fused video and text spatio-temporal semantic feature V st . The calculation of this part of the multi-modal attention has the same parameter settings as the multi-modal attention between the slow video stream and the spatial words, but has independent learnable parameters.

[0072] In this embodiment, the segment boundary positioning module first predicts the end point of the segment according to the modeled segment end edge distribution P This process is represented as:

[0073]

[0074] wherein the segment end edge distribution P is modeled by a two-layer perceptron with a hidden layer dimension of 128 and an output layer dimension of 1. The perceptron calculates the probability of each frame as the end point of the frame, and the final output is a discrete distribution with T possible values.

[0075] Then, for the i-th frame, the expected value E[start|end=i] of the normalized start point frame is predicted, and the start point is predicted The product of the expected value of the start point and the end point is output. This process is represented as:

[0076]

[0077] where the conditional expectation E[start|end=i] is modeled by a two-layer perceptron with 128 hidden units and 1 output unit. The input of the perceptron is the fused feature V st The feature vector of the i-th frame in the segment, and the output is the expectation of the start frame index given that the frame is the end frame. The segment boundary localization module obtains significant performance improvement by explicitly modeling the conditional dependency between the start and end frames of a segment.

[0078] In this embodiment, the dataset used is the open source video dataset ActivityNet-Captions. ActivityNet-Captions is an open source video dataset provided by Stanford University, which contains more than 20,000 videos in different fields, and the training set contains 37,421 segment-text pairs, and the test set contains 17,031 segment-text pairs. For the data division method, the official division provided by Stanford University is used.

[0079] In this embodiment, the positioning accuracy under different intersection-over-union thresholds is selected as the evaluation criterion. If the intersection-over-union of the predicted segment and the real segment is greater than the threshold, the sample is predicted correctly, otherwise it is predicted incorrectly. The threshold used is {0.5, 0.7}.

[0080] Through experiments on the validation set, the hyperparameters contained in the model proposed in the present application are set as follows. The optimizer uses the Adam optimizer, the initial learning rate is set to 0.0004, and the weight decay is set to 0.000006. The training batch size is set to 128. In this embodiment, the accuracy under the intersection-over-union threshold of 0.5 reaches 48.94%, which is 3.02% higher than the best model before, and the accuracy under the intersection-over-union threshold of 0.5 reaches 28.30%, which is 2.91% higher than the best model before.

[0081] In summary, the present application is directed to the problem of video segment localization, and in view of the fact that existing work ignores the semantic of video and text in the time and space dimensions, thus failing to distinguish similar queries, a novel video segment localization system based on spatiotemporal semantic decomposition is proposed. Through the four modules of video and text encoder, spatiotemporal semantic decomposition, multi-modal attention, and segment boundary localization, the video and text are separately decomposed in the time and space dimensions, and fine-grained multi-modal matching is performed in the time and space dimensions, respectively, and the boundary of the predicted segment is generated by the two-stage prediction method, so that the accuracy of video segment localization is greatly improved.

[0082] Although the present application has been disclosed with reference to the preferred embodiments, it is not intended to limit the present application, and any person skilled in the art can make possible changes and modifications to the technical solutions of the present application using the disclosed methods and technical contents without departing from the spirit and scope of the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application without departing from the technical solutions of the present application shall fall within the protection scope of the technical solutions of the present application.< / pad> < / unk>

Claims

1. A video segment positioning system based on spatio-temporal semantic decomposition, characterized in that, The encoder module of video and text, the spatio-temporal semantic decomposition module, the multi-modal attention module, and the segment boundary positioning module; the encoder module of video and text respectively extracts features of video and text to generate features containing context information; the spatio-temporal semantic decomposition module divides the text into spatial words and temporal words according to the part of speech, and samples the video at different frame rates to obtain fast video stream and slow video stream; the multi-modal attention module first calculates the mutual attention between the spatial words and the slow video stream, fuses the result with the fast video stream, and then calculates the mutual attention between the temporal words and the fast video stream to obtain the fusion features of the two modalities; The segment boundary positioning module first calculates the frame of the target segment endpoint, and then regresses the starting frame in the frame before the predicted endpoint frame; The multi-modal attention module is used for calculating the multi-modal attention between the slow video stream and the spatial words as the prior of the multi-modal attention between the fast video stream and the temporal words; First, the multi-head attention between the two modalities is calculated, and the process is represented as: MultiHead(X,Y) = [Attention 1…h (X,Y)]W O , # (4) where W Q , W K , W V and W O are learnable linear transformation parameters; h denotes the number of heads used by multi-head attention; X denotes the features of the video modality, and Y denotes the features of the text modality; d denotes the dimension of the features; and σ denotes the softmax function; the process of calculating the multi-modal attention of the slow video stream and the spatial word is represented as: wherein Q s represents a spatial word; represents the calculation result of the slow video stream in the l-th layer multimodal attention; MultiHead represents a multi-head attention mechanism; LN represents layer normalization; MLP represents a two-layer perceptron; after the multimodal attention calculation of the slow video stream and the spatial word, the slow video stream is more focused on each frame and the corresponding spatial semantics in the text; in order to use the multimodal attention calculation result of the slow video stream and the spatial word as prior knowledge for positioning in the time dimension, the attention map of the normalized slow video stream in the last layer multimodal attention is calculated by the following method: wherein N s represents the number of spatial words; A is the attention map of the slow video stream and the spatial words in the last layer of the multi-modal attention; A i represents the two-dimensional attention map in the i-th head of the multi-head attention, with the dimension of T slow ×N s ; represents the j-th column of the two-dimensional attention map in the i-th head of the multi-head attention; minmax represents the min-max normalization; the value corresponding to each frame in the normalized attention map a reflects the attention degree of the frame in the multi-modal attention with the spatial words, and the higher the value, the more the object, scene spatial semantics described in the text appears in the frame; the features of the slow video stream output by the last layer of the multi-modal attention are added to the fast video stream V fast , and the normalized attention map a is multiplied by the fast video stream V fast ; the process is represented as: where upsamplerepresents an up-sampling operation; represents element-wise multiplication; 1 T represents a vector with all elements being 1; since the frame number of the slow video stream and the fast video stream do not match, the slow video stream is up-sampled to T fast frames using linear interpolation method before fusion; after fusing the features and the attention map a of the slow video stream, the multi-modal attention of the fast video stream V fast,fused and the time word is calculated to obtain the final fused video and text spatio-temporal semantic feature V st with the dimension of Txd; The segment boundary positioning module first predicts the end point of the segment according to the modeled segment end point edge distribution P The process is represented as: where the endpoint edge distribution P is modeled using a two-layer perceptron acting independently on the fused features V st at each location, with a softmax normalization over the outputs at all locations to obtain the probability distribution P. The segment boundary location module also includes a regressor of conditional expectation of the modeling start point, which predicts the expectation E[start|end=i] of the normalized start point frame for the i-th frame the product of the output regression expectation and the end point; the process is represented as: where the regressor of conditional expectation E[start|end=i] is a two-layer perceptron that independently acts on the fused feature V st at each position and goes through a sigmoid activation function to output the normalized start frame.

2. The video clip positioning system of claim 1, wherein, The video encoder module is composed of a backbone network and a Transformer encoder; the text encoder module is composed of a backbone network, a Transformer encoder and a relational graph convolution network; wherein: The backbone network in the video encoder module adopts a C3D and I3D deep convolutional neural network to extract video features; the center frame image and its adjacent 31 frame images are extracted into a 4,096-dimensional or 1,024-dimensional vector; the Transformer encoder in the video encoder module contains two isomorphic calculation units, which act on the features extracted by the backbone network, encode the position information and context information of the video frame into the features, and the output video feature dimension is Txd; The backbone network in the text encoder module adopts a Glove word vector to extract text features; the Transformer encoder in the text encoder module has the same structure as the Transformer encoder in the video encoder module, but does not share parameters; the text features extracted by the backbone network are context coded by the Transformer encoder, and finally the relational graph convolution with residual connection is used on the syntax dependency tree structure of the text for relationship coding; the process is represented as: where, represents the feature of the ith word in the (l+1)th layer of the relational graph convolution; R represents the set of relations on the edges in the syntactic dependency tree structure; represents the neighbor of the ith word in the syntactic dependency tree structure with the relation r on the edge; is the learnable parameter in the lth layer of the relational graph convolution; since there are 55 kinds of relations on the edges in the syntactic dependency tree structure, the number of relations doubles after considering the direction of the edge, which will introduce too many parameter matrices The method of basis decomposition is used to reduce the number of parameters; the process is represented as: wherein, is the i-th basis parameter matrix in the l-th layer relational graph convolution; B is the number of basis parameter matrices; is a learnable linear combination coefficient.

3. The video clip positioning system of claim 2, wherein, The spatio-temporal semantic decomposition module respectively decomposes video and text into respective time semantics and space semantics; the spatio-temporal decomposition of the video takes the original frame rate video as a fast video stream, and performs sparse sampling of every 4 frames on the fast video stream with T frames to obtain a slow video stream with T slow frames; the spatio-temporal decomposition of the text firstly performs part-of-speech tagging on the text in the query, and takes nouns and pronouns in the query as space words and takes verbs and adverbs as time words.

4. The video clip positioning system of claim 1, wherein, An end-to-end training method is adopted to train the final output of the model, i.e. the edge distribution P of the endpoint and the conditional expectation E[start|end=i] of the normalized starting point; the IoU of the prediction interval with the fixed starting point and the real interval is used as the supervision information, and the calculation process of the label value of the i-th frame is represented as: Wherein, s is the real value of the starting point; e is the real value of the endpoint; according to formula (13), the label value of the endpoint frame e is 1; the label value of the frame after the starting frame is set as the IoU of the interval [s, i] and the real interval [s, e], and the label value becomes larger as i moves from left to right and gets closer to e, and becomes smaller when i is greater than e; the training target of the endpoint edge distribution P is to minimize the binary cross entropy function: where, is the end point edge distribution without softmax normalization; for the conditional expectation E[start|end=i] of the normalization start point, the training objective is to minimize the SmoothL1 loss function: Finally, the training target is the sum of the two loss functions: L = L end + L start , # (16).

5. The video segment positioning system according to one of claims 1 to 4, characterized in that The workflow is: (I) Firstly, according to the encoder module of the video and text, the pre-trained backbone network is used to extract the features of the video and text respectively, and then the position information and context information are fused using the Transformer encoder; for the text query, the relationship graph convolution network is used again to encode the relationship on the syntactic dependency tree structure; (II) The encoded video features and text features are taken as the input of the spatio-temporal semantic decomposition module, the text is divided into spatial words and temporal words according to the part of speech, and the video is sampled at different frame rates to obtain fast video stream and slow video stream; (III) The spatial words and temporal words, fast video stream and slow video stream are taken as the input of the multi-modal attention module, the multi-modal attention of the spatial words and slow video stream is calculated first, the fused features are added with the fast video stream, the attention map in the last layer of multi-modal attention is normalized and multiplied with the fast video stream, and then the multi-modal attention of the temporal words and fast video stream is calculated; (IV) The fused video features are taken as the input of the segment boundary positioning module, the end point distribution of the segment is modeled according to the greedy strategy to predict the end point of the segment, and then the start point frame is regressed in the frame before the predicted end point frame; (V) The IoU of the predicted interval and the real interval is taken as the supervision information, and the network is trained end to end to minimize the binary cross entropy loss function and the SmoothL1 loss function.

Images