Multi-modal video positioning method based on deep neural network

Abstract

The invention discloses a multi-modal video positioning method based on a deep neural network, and the method comprises the steps: firstly decomposing a sentence S inputted by a user, classifying each word in the sentence according to the part-of-speech, representing a classification result through one-hot coding, and initializing the sentence S to obtain a word feature matrix; secondly, obtaining a language query feature matrix according to the word feature matrix, and extracting entity information features and action information features according to one-hot codes; generating visual features based on a video input by a user, and obtaining visual features of an entity in combination with entity information features; and finally, based on the visual features of the entity, calculating the action correlation score of the video frame through a full connection layer, predicting the starting boundary and the ending boundary of a target action fragment through a double-branch network, obtaining a positioning result, and constructing a loss function for reverse training. The technical problem of low prediction precision in video positioning is solved, and the accuracy of motion boundary prediction in the video is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video positioning, and in particular relates to a multimodal video positioning method based on a deep neural network. Background Art

[0002] With the rapid development of the Internet and smart devices, especially the popularity of portable user devices such as smartphones, tablets, and laptops, the production and consumption of video content has become an indispensable part of daily life. New applications such as video recommendation systems, automatic video summarization, and video surveillance are constantly emerging and have attracted widespread attention. The operation of these applications requires the processing of large amounts of video data and requires strong understanding and retrieval capabilities. However, due to the characteristics of video data such as high dimensionality, multimodality, and time dependence, how to efficiently locate the time period corresponding to a specific natural language description in massive video data has become a challenge in current research.

[0003] As a key technology to solve this problem, video temporal localization is gradually becoming a research hotspot in academia and industry. The core goal of video temporal localization is to automatically identify and accurately locate the time period that matches a given natural language description from a video. To achieve this goal, the system needs to be able to understand the content of the video and accurately capture the semantic association between the video clip and the text description. However, the challenge of video temporal localization is that the complexity of video content and the diversity of natural language expressions often make it difficult to achieve accurate alignment and positioning.

[0004] The rise of multimodal learning provides a new research perspective and solution for video temporal localization. By combining visual and language information, researchers have proposed a variety of methods to improve the accuracy of video temporal localization. In addition, the introduction of pseudo-supervised learning further reduces the dependence on large-scale labeled data, thereby improving the generalization ability of the model.

[0005] Despite some progress, existing methods still have limitations in dealing with long videos, complex scenes, and fine-grained action recognition. For example, accurately locating a short and complex action segment in a long video remains a challenging task. With the increasing diversity and complexity of video content, the requirements for video temporal localization in terms of computational efficiency, accuracy, and robustness are becoming increasingly stringent.

[0006] In this context, how to further improve the accuracy and efficiency of video temporal localization through multimodal learning and optimization technology remains an important direction for future research. Researchers need to develop more efficient and robust models to cope with various challenges in practical applications, thereby promoting the development of video content understanding and retrieval technology. Summary of the invention

[0007] In order to solve the shortcomings of low precision and difficulty in optimization of traditional methods, the present invention proposes a multimodal video localization method based on deep neural network, and also proposes a computationally intensive loss function to solve the problem that the model is difficult to converge.

[0008] In order to achieve the above-mentioned purpose, the steps of the present invention are as follows:

[0009] 1. Decompose the sentence S input by the user, classify the words in the sentence according to their parts of speech, and use one-hot encoding to represent the classification results; use the GloVe model to initialize the sentence S and obtain the word feature matrix Q.

[0010] First, the sentence S input by the user is decomposed into entity information query and action information query. For each word T in the sentence i , classified according to part of speech, using one-hot encoding P i Indicates the classification result. If the part of speech is related to the entity (such as noun, adjective), then P i e =[1, 0, 0]; if it is related to action (such as verbs, adverbs), then P i m =[0, 1, 0]; otherwise P i o =[0, 0, 1].

[0011] Next, use the GloVe model to initialize the sentence S and obtain the word feature matrix where w i Represents the features of the i-th word, with a dimension of d w , N represents the number of words.

[0012] 2. Using Transformer q Module, based on the word feature matrix Q, learns the relationship between words, outputs the language query feature matrix, and extracts entity information features based on the unique hot encoding and action information features

[0013] Using Transformer qModule, which uses a standard Transformer to learn the relationship between words based on the word feature matrix Q and output the language query feature matrix:

[0014] F q =Transformer q FC1(Q),

[0015] in is the learned language query feature matrix, FC1 is a fully connected layer that transforms the word feature matrix from d w Project the d-dimensional image to the d-dimensional image.

[0016] Extract entity information features based on one-hot encoding and action information features The specific formula is:

[0017]

[0018]

[0019] Among them, P i e [0] is P i e One-hot encoding represents the first element in the matrix, P i o [2] is P i o One-hot encoding represents the third element in the matrix, P i m [1] is P i m One-hot encoding represents the third element in the matrix, Represents entity information characteristics, Represents action information features.

[0020] 3. Generate visual features based on user input video Combined with entity information features Computing visual features Entity Information Features Similarity matrix, obtain the visual features F of the entity ve .

[0021] Use the pre-trained 3D convolutional network to extract visual features of user input videos Then through Transformer e The module captures inter-frame contextual information. This module uses a Transformer without position encoding to generate updated visual features:

[0022]

[0023] The fully connected network FC2 is used to transform the visual features from d v The dimension is projected into d dimensions.

[0024] Calculate the similarity matrix between visual features and entity information features Decompose the similarity matrix S by row vector and calculate a similarity matrix expanded by row:

[0025]

[0026] Decompose the similarity matrix S by column vector and calculate a similarity matrix expanded by column:

[0027] S=[S c1 ... S cN ]

[0028] S c =[softmax(S c1 ) ... softmax(S cN )]

[0029] Where S ri S ci Represent row vectors and column vectors respectively, and obtain the similarity matrix S through softmax normalization. r and S c .

[0030] Using these similarity matrices, two attention weights are calculated:

[0031]

[0032] Next, the visual and query features are fused by the following formula to obtain the visual feature F of the entity: ve :

[0033]

[0034] Among them, the symbol ⊙ represents element-by-element multiplication, the symbol semicolon “;” represents concatenation, and the fully connected layer FC3 is used to project the concatenated features from the 4d dimension to the d dimension.

[0035] 4. Entity-based visual features F ve , the action-related score P of the video frame is calculated through the fully connected layer e ,The starting and ending boundaries of the target action segment are predicted through a dual-branch network, and the boundary probability distribution is calculated by softmax to obtain the positioning result, and the loss function is constructed for reverse training.

[0036] The action-related score P of the video frame is calculated through the fully connected layer e =sigmoid(FC5(ReLU(FC4(F ve )))), the higher the score, the greater the probability of being selected as the action area. The model uses LSTM Transformer to learn local and global motion changes of video features:

[0037]

[0038]

[0039] By and action related score P e The elements are combined by multiplication, and the video frames related to the action are obtained.

[0040] Calculate the similarity matrix Repeat step S3 to operate the similarity matrix to obtain the visual features of motion perception Predict the starting boundary of the target action segment through the fully connected network and get the starting score Then, the boundary probability distribution is calculated by softmax

[0041] Residual connection gets

[0042] Use Bi-LSTM Transformer to learn temporal motion changes of video features:

[0043]

[0044] Among them, the visual features are transformed from dimension d v Projection to dimension d; Transformer Bi-LSTM (·) represents Bi-LSTM Transformer.

[0045] Calculate the similarity matrix Repeat step S3 to operate the similarity matrix to obtain F vsq , predict the end boundary S through the fully connected network e =FC9(ReLU(FC8(F vsq ))), and then the boundary probability distribution is completed through softmax calculation

[0046] Compute the predicted start and end boundaries of the target action segment using the maximum joint probability:

[0047]

[0048] Define starting boundary probability and the end boundary probability Respectively represent a certain time t s is the starting probability of the action segment, and at a certain time t e is the ending probability of the action segment; by traversing all t s and t e , find the time point that maximizes the joint value of the start and end probabilities and

[0049] The training objective for action boundary prediction is expressed as:

[0050]

[0051] where f XE is the cross entropy loss function; where (τ s ,τ e ) is the actual boundary of the action.

[0052] The model also calculates the standard IoU loss

[0053] Dense loss L dense :

[0054] Use softmax with temperature to compute a smooth prediction boundary:

[0055]

[0056]

[0057] Set two scalars (α s ,α e ) to measure the degree of aggregation of boundary probabilities:

[0058]

[0059]

[0060] and Respectively with α s and α e By multiplying and summing each item, we can calculate whether the smooth prediction boundary is clustered near the true label, denoted as L dense :

[0061]

[0062] And through the final loss function L = λ1L boundary +λ2L IoU +λ3L dense Make optimizations.

[0063] 5. Loss Function

[0064] The loss function comprehensively considers the cross entropy loss L of boundary prediction boundary , IoU loss L IoU and the dense loss L dense , and finally the performance of the model is optimized through the weighted combination of hyperparameters λ1, λ2, and λ3.

[0065] Beneficial effects of the present invention:

[0066] The dual-stage network of the present invention predicts the start frame, and after obtaining the start frame, predicts the end frame through multimodal interaction. It expands the dimension of model learning, solves the technical problem of low prediction accuracy in video positioning, and is committed to improving the accuracy of action boundary prediction in videos through fine feature extraction, fusion and context learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is the model structure diagram. DETAILED DESCRIPTION

[0068] like Figure 1 A multimodal video positioning method based on deep neural network is shown, and the specific steps are as follows

[0069] 1 In the process of query decomposition of the input sentence S, we can divide it into two main query categories: entity information query and action information query. Specifically, for each word T in the sentence S i , we will classify it into one of three types of part-of-speech tags based on its part-of-speech features. These three categories are: entity class, action class, and other class. In order to more clearly distinguish each word T i For categories, we use one-hot encoding P i To represent the classification result of the word. If the part-of-speech tag of a word is related to an entity (for example, a noun or adjective is usually used to refer to an object, thing, or describe a feature), then we will classify it as entity information and represent its unique hot encoding as P i e = [1,0,0]. That is, the first position is 1, indicating that the word is related to the entity.

[0070] On the contrary, if the word's part-of-speech tag is action-related (for example, a verb or adverb that describes a behavior, process, state, or action), then we classify it as an action information query. In this case, the one-hot encoding will be different from the encoding method for entity information queries.

[0071] Through this one-hot encoding-based part-of-speech classification method, we can process each word in a sentence more systematically, thus laying a clear classification foundation for subsequent entity information queries and action information queries.

[0072] 2 When using the GloVe model to initialize the sentence S, each word in the sentence is first converted into a corresponding word vector representation. Through the GloVe model, a pre-trained word feature vector is assigned to each word in the sentence S, thereby obtaining the word feature matrix Q, which is expressed as:

[0073] where w i Represents the i-th word feature, with dimension d w , N represents the number of words in the language query. Through this initialization method, the sentence S is converted into an N×d w dimensional matrix Q, each row of which is w i The word vector corresponding to a word in the sentence. This representation method provides a compact and efficient word feature expression for subsequent language processing tasks.

[0074] 3. In order to learn the relationship between words in sentence S, a Transformer can be used q Module. This module uses a standard Transformer model. The Transformer model captures the dependencies between each word and other words in the input sequence through the self-attention mechanism, thereby learning the relationship between words. The following is the process of using the Transformer module to learn the relationship between words:

[0075] F q =Transformer q FFC1(Q)

[0076] in is the learned language query feature, and FFC1(·) is a fully connected layer used to transform word features from d w Dimension is projected to d dimension, Transformer q (·) is a standard Transformer module.

[0077] 4Use one-hot encoding to distinguish the parts of the sentence related to entity information and action information, and extract this information through a formula. The specific formula is defined as follows:

[0078]

[0079]

[0080] Among them, P ie [0] is P i e One-hot encoding represents the first element in the matrix, P i o [2] is P i o One-hot encoding represents the third element in the matrix, P i m [1] is P i m One-hot encoding represents the third element in the matrix, Represents entity information characteristics, Represents action information features.

[0081] 5 In the process of extracting visual features using a pre-trained 3D convolutional neural network (3D CNN), the input visual data (such as videos or 3D images) are processed through the network to obtain visual feature representations in the form of time series. where f v,i Represents the i-th visual feature, with dimension d v , T represents the number of features.

[0082] 6 Repeat step 3 using Transformer e The module uses a Transformer module without position encoding to focus on the contextual information between different frames.

[0083]

[0084] in is the updated visual feature, and FC2(·) is a fully connected layer that transforms the visual feature from dimension d v Projection to dimension d; Transformer e (·) represents a standard Transformer module without positional encoding.

[0085] 7 Given visual features and entity query features Calculate their similarity:

[0086]

[0087] The matrix is ​​normalized by row-by-row and column-by-column softmax operations to obtain two similarity matrices S r and S c , which is used to capture the similarity across rows and columns in a matrix. This operation applies the softmax function independently across rows and columns in a matrix, thereby calculating the similarity of each element relative to other elements.

[0088] Decompose S into row vectors and calculate a similarity matrix expanded by rows:

[0089]

[0090] Decompose S into column vectors and calculate a similarity matrix expanded by columns:

[0091] S=[S c1 ... S cN ]

[0092] S c =[softmax(S c1 ) ... softmax(S cN )]

[0093] Where S ri , S ci Represent row vectors and column vectors respectively, and obtain the similarity matrix S through softmax normalization. r and S c .

[0094] 8 Calculate two attention weights and Visual features of the entity F ve The two attention weight matrices are calculated by the following formula and To capture query features With visual features Then, these attention weights are combined with the original visual features and a fully connected layer FC3 is used to generate the final entity visual features.

[0095] in The symbol ⊙ represents element-wise multiplication; [] represents the concatenation of features; FC3(·) is a fully connected layer used to project the concatenated features from the 4d dimension to the d dimension.

[0096] 9 In order to predict the action position, two fully connected layers are used to gradually process the entity visual features F ve To calculate the action-related score P of the video frame e The process includes a ReLU activation function, two fully connected layers, and a final sigmoid activation function to map the output to a probability distribution.

[0097] P e =sigmoid(FC5(ReLU(FC4(F ve ))))

[0098] By using two fully connected layers FC 4 and FC5 and ReLU and sigmoid activation functions, the action-related score Pe of the video frame is successfully calculated. The higher the score, the greater the probability that it is an action area, which can be used for further action position prediction or editing action clips in the video. The output feature dimensions of FC4(·)FC5(·) are and 1. The higher the action-related score, the higher the probability that the corresponding frame is selected as an action region.

[0099] 10Use LSTM Transformer to learn local and global motion changes of video features to generate more representative motion features This process enables the model to not only focus on short-term frame-to-frame motion changes, but also capture long-term motion dependencies in the video.

[0100]

[0101] in, is the updated motion feature, which pays more attention to motion changes; FC2(·) is the fully connected layer in step 6, which transforms the visual features from dimension d v Projection to dimension d; Transformer LSTM (·) represents LSTM Transformer.

[0102] 11. P by action related score e To focus on action-related video frames, the formula is is the action relevance score of each frame. Element-wise multiplication enhances the features of action frames and weakens the features of frames that are not related to the action, so that the model can better capture the key action areas. This operation effectively improves the accuracy of action position prediction.

[0103] 12 Repeat step 8 to add motion query features Fusion to visual motion representation To obtain the motion-aware visual feature F vm :

[0104]

[0105] 13 is predicted by a dual-branch network with two fully connected layers:

[0106] S s =FC7(ReLU(FC6(F vm )))

[0107] The starting score of the target action segment The output feature dimensions of FC6(·) and FC7(·) are and 1.

[0108] 14 The probability distribution of the action start boundary is calculated by softmax:

[0109]

[0110] 15 In order to predict the end boundary of the action, first use the starting boundary probability Focus on the action-related video frames. The specific steps are as follows: Through the element-by-element multiplication operation ⊙, the starting boundary probability With sports characteristics Multiply and add back the original motion characteristics, the formula is:

[0111]

[0112] This operation strengthens the frame features related to the start boundary while retaining the original frame features for action end boundary prediction.

[0113] 16 Similar to step 10, use Bi-LSTM Transformer to learn the temporal motion changes of video features. The Bi-LSTM Transformer structure can better capture the long-term and short-term dependencies between frames. This method can focus on local and global motion changes at the same time and is suitable for processing video data with temporal dependencies.

[0114]

[0115] The input features are That is, through the action starting boundary probability Weighted motion feature matrix Contains motion feature information for each frame in the video, with a dimension of T×dv, where T is the number of video frames and dv is the feature dimension. is the updated motion feature, which pays more attention to motion changes; FC2(·) is the fully connected layer in step 6, which transforms the visual features from dimension d v Projection to dimension d; Transformer Bi-LSTM (·) represents Bi-LSTM Transformer.

[0116] 17 Repeat step 8 to add motion query features Fusion to focus on video information related to action and starting boundaries To obtain the visual feature F of the predicted end boundary vsq :

[0117]

[0118] 18 Repeat steps 13 and 14 to obtain the probability distribution of the end boundary:

[0119] S e =FC9(ReLU(FC8(F vsq )))

[0120]

[0121] The starting score of the target action segment Among them, the output feature dimensions of FC8(·) and FC9(·) are and 1.

[0122] 19 Define the starting boundary probability and the end boundary probability Respectively represent a certain time t s is the starting probability of the action segment, and at a certain time t e is the end probability of the action segment. To determine the best start and end time, we optimize by maximizing the joint probability:

[0123]

[0124] This step is done by traversing all t s and t e , find the time point τ that maximizes the joint value of the start and end probabilities s and τ e .

[0125] To further evaluate the accuracy of the prediction, the optimization score of the prediction boundary is calculated That is, the value of the joint probability at the optimal start and end time points:

[0126]

[0127] This optimization score Reflects the model's starting and ending boundaries for prediction The higher the score, the more confident the model is about the predicted boundary. By maximizing the joint probability of the start and end boundaries, we can predict the start time of the action segment and end time At the same time, the optimization score is calculated It is used to measure the reliability of the prediction results. This process helps to accurately locate the time range of the target action in the video.

[0128] 20 Given the predicted starting boundary probability distribution and the end boundary probability distribution The training objective of action boundary prediction can be expressed as a cross entropy loss function, which measures the difference between the boundary probability distribution predicted by the model and the actual boundary.

[0129]

[0130] Among them, f XE (·) is the cross entropy function, (τ s ,τ e ) is the true boundary of the action. By minimizing this loss, the model can better predict the start and end time points of the action clip, making the predicted boundary distribution closer to the true boundary.

[0131] 21 In order to measure the prediction boundary and the true boundary (τ s ,τ e ), we can use the IoU (Intersection over Union) loss, denoted as L IoU The IoU loss reflects the degree of overlap between the predicted action segment and the real action segment. The closer it is to 1, the closer the predicted and real boundaries are, and the smaller the loss. The ratio of the overlapping area of ​​the predicted action boundary to the real action boundary to the union area of ​​the two. The formula is:

[0132]

[0133] 22 In order to compute the smooth prediction boundary and evaluate whether it is clustered near the true label, a softmax function with temperature is used to smooth the prediction boundary distribution:

[0134]

[0135]

[0136] In order to measure whether the boundary probability distribution is concentrated on the true label τ s and τ e Nearby, define two scalar vectors α s and α e , which represents the distance between the predicted boundary and the true boundary:

[0137]

[0138]

[0139] and Respectively with α s and α e By multiplying and summing each item, we can calculate whether the smooth prediction boundary is clustered near the true label, denoted as Ldense :

[0140]

[0141] Where temp is the hyperparameter of softmax and || ||1 is the L1 norm L dense Measures whether the smooth prediction boundaries are clustered around the true label τ s and τ e If the predicted boundary is close to the true boundary, then α s and α e The value is small, the loss L dense is also smaller, indicating that the model is more accurate in predicting boundaries.

[0142] 23 The final training objective loss function L is to optimize the prediction ability of the model by combining multiple loss functions. These loss functions include action boundary loss L boundary , IoU loss L IoU and boundary aggregation loss L dense , each loss term is assigned different weights according to its importance. The formula is as follows:

[0143] L=λ1L boundary +λ2L IoU +λ3L dense

[0144] Where L boundary : Measuring prediction boundaries through cross entropy loss and and the true boundary τ s and τ e The difference L IoU : Based on IoU (intersection over union) loss, measure the overlap between the predicted boundary and the true boundary L dense : Calculate whether the predicted boundary is clustered near the true boundary by using softmax with temperature to ensure the smooth distribution of the boundary. λ1, λ2, and λ3 are hyperparameters used to adjust the weight of each loss item in the overall loss. By properly adjusting these hyperparameters, the model can pay more attention to specific loss items, such as boundary accuracy, IoU matching degree, or predicted smoothness.

[0145] The final loss function L optimizes the model training process by balancing multiple loss terms to ensure that the model achieves the best performance in boundary prediction, IoU matching, and boundary distribution. This combined loss can help the model locate the start and end boundaries of action clips more accurately.

[0146] Example:

[0147] The public dataset Charades is a dataset for action recognition, action classification, and temporal action localization. The main features of this dataset are: Data scale: It contains 9848 videos, of which 7985 videos are used for training and 1863 videos are used for testing.

[0148] The total length of the video is over 66 hours. Video content: The video content simulates common activities in a home environment (for example, cooking, tidying up the room). Each video can contain multiple actions, and even multiple actions can occur at the same time (i.e., overlap). Annotation information: Each video is annotated with one or more action segments (the start and end time of each action). Supports multi-label classification: A video may contain labels for multiple action categories. Provides timestamps for the start and end of the action, which makes it suitable for time segment video localization tasks.

[0149] The public dataset TACoS (Temporally-Annotated Cooking Scenes) is a video localization task dataset focusing on kitchen scenes. It is mainly used to evaluate the performance of natural language descriptions in time segment video localization. Data source: Based on the MPII Kitchen Activities Dataset (MPII Cooking Activities Dataset). The video content mainly involves cooking scenes, such as cutting vegetables, pouring water and other kitchen actions. Data scale: Contains 127 videos with a total length of about 19 hours. Annotation information: Each video is annotated with multiple natural language descriptions and their corresponding time segments (start and end time). The annotation information describes the semantic content of the action in the video, for example: "cut the tomatoes into slices."

[0150] Definition: IoU measures the overlap between the model’s predicted time period P = [tp1, tp2] and the true time period G = [tg1, tg2]. Value range: IoU values ​​are between [0, 1]. The closer the IoU is to 1, the higher the match between the predicted time period and the actual time period.

[0151] Table 1 Performance on the TACoS dataset

[0152]

[0153] Table 2 Performance on the Charades dataset

[0154]

[0155] IoU=1: The predicted time period is exactly the same as the actual time period.

[0156] IoU=0: There is no overlap between the predicted time period and the true time period.

[0157] IoU@n: indicates the proportion of prediction results IoU>=n.

[0158] The experimental results are shown in Table 1 and Table 2, from which it can be seen that the present invention is obviously superior to several other prior arts compared thereto.

Claims

1. A multimodal video positioning method based on deep neural network, characterized in that: The following steps are involved: S1. Decompose the sentence S input by the user, classify the words in the sentence according to their parts of speech, and use one-hot encoding to represent the classification results; Use the GloVe model to initialize the sentence S and obtain the word feature matrix Q; S2. Using Transformer q Module, based on the word feature matrix Q, learns the relationship between words, outputs the language query feature matrix, and extracts entity information features F based on the unique hot encoding e q and action information features S3. Generate visual features based on user input video Combined with entity information feature F e q , computing visual features and entity information feature F e q Similarity matrix, obtain the visual features F of the entity ve ; S4. Entity-based visual features F ve , the action-related score P of the video frame is calculated through the fully connected layer e ,The starting and ending boundaries of the target action segment are predicted through a dual-branch network, and the boundary probability distribution is calculated by softmax to obtain the positioning result, and the loss function is constructed for reverse training.

2. The multimodal video positioning method based on deep neural network according to claim 1 is characterized in that: The specific implementation process of step S1 is as follows: Decompose the sentence S input by the user into entity information query and action information query; for each word T in the sentence i , classified according to part of speech, using one-hot encoding P i Indicates its classification results; If the part of speech is related to the entity, then P i e =[1,0,0]; if it is related to the action, then P i m =[0,1,0]; otherwise P i o = [0,0,1]; Use the GloVe model to initialize the sentence S and obtain the word feature matrix where w i Represents the features of the i-th word, with a dimension of d w , N represents the number of words.

3. The multimodal video positioning method based on deep neural network according to claim 2 is characterized in that: The specific implementation process of step S2 is as follows: Using Transformer q Module, which uses Transformer to learn the relationship between words based on the word feature matrix Q and outputs the language query feature matrix: F q =Transformer q FC1(Q), is the learned language query feature matrix, FC1 is a fully connected layer that transforms the word feature matrix from d w Dimensional projection to d dimension; Extract entity information features F based on one-hot encoding e q and action information features The specific formula is: Among them, P i e [0] is P i e One-hot encoding represents the first element in the matrix, P i o [2] is P i o One-hot encoding represents the third element in the matrix, P i m [1] is P i m One-hot encoding represents the third element in the matrix, Represents entity information characteristics, Represents action information features.

4. The multimodal video positioning method based on deep neural network according to claim 3 is characterized in that: The specific implementation process of step S3 is as follows: Use the pre-trained 3D convolutional network to extract visual features of user input videos Then through Transformer e The module captures inter-frame contextual information. This module uses a Transformer without position encoding to generate updated visual features: The fully connected network FC2 is used to transform the visual features from d v Dimension is projected into d dimension; Calculate the similarity matrix between visual features and entity information features Decompose the similarity matrix S by row vector and calculate a similarity matrix expanded by row: Decompose the similarity matrix S into column vectors and calculate a similarity matrix expanded by columns S=[S c1 ...S cN ] S c =[softmax(Sc1)...softmax(S cN )] Where S ri , S ci Represent row vectors and column vectors respectively, and obtain the similarity matrix S through softmax normalization. r and S c ; Using the similarity matrix, two attention weights are calculated: The visual feature F of the entity is obtained by fusing the visual and query features through the following formula: ve : Among them, the symbol ⊙ represents element-by-element multiplication, the semicolon represents concatenation, and the fully connected layer FC3 is used to project the concatenated features from 4d dimension to d dimension.

5. The multimodal video positioning method based on deep neural network according to claim 4 is characterized in that: The specific implementation process of step S4 is as follows: The action-related score P of the video frame is calculated through the fully connected layer e =sigmoid(FC5(ReLU(FC4(f ve )))), using LSTM Transformer to learn local and global motion changes of video features: By and action related score P e Combine them by element-by-element multiplication to obtain the video frames related to the action; Calculate the similarity matrix Repeat step S3 to operate the similarity matrix to obtain the visual features of motion perception Predict the starting boundary of the target action segment through the fully connected network and get the starting score Then, the boundary probability distribution is calculated by softmax Residual connection gets Use Bi-LSTM Transformer to learn temporal motion changes of video features: Among them, the visual features are transformed from dimension d v Projection to dimension d; Transformer Bi-LSTM (·) represents Bi-LSTM Transformer; Calculate the similarity matrix Repeat step S3 to operate the similarity matrix to obtain F vsq , predict the end boundary through the fully connected network Then the softmax calculation ends the boundary probability distribution Compute the predicted start and end boundaries of the target action segment using the maximum joint probability: Define starting boundary probability and the end boundary probability Respectively represent a certain time t s is the starting probability of the action segment, and at a certain time t e is the ending probability of the action segment; by traversing all t s and t e , find the time point that maximizes the joint value of the start and end probabilities and The training objective for action boundary prediction is expressed as: where f XE is the cross entropy loss function; where (τ s ,τ e ) is the actual boundary of the action; The model also calculates the standard IoU loss Dense loss L dense : Use softmax with temperature to compute a smooth prediction boundary: Set two scalars (α s ,α e ) to measure the degree of aggregation of boundary probabilities: and Respectively with α s and α e By multiplying and summing each item, we can calculate whether the smooth prediction boundary is clustered near the true label, denoted as L dense And through the final loss function L = λ1L boundary +λ2L IoU +λ3L dense Make optimizations.

Images