A deep neural network-based multi-modal video localization method

Abstract

The application discloses a kind of multimodal video positioning methods based on deep neural network, the sentence S of first user input is decomposed, for each word in sentence, according to the part-of-speech classification, using one-hot encoding to express classification result, and the word feature matrix is obtained by initializing sentence S.Second, according to word feature matrix, obtain language query feature matrix, according to one-hot encoding extraction entity information feature and action information feature.Then based on user input video, generate visual feature, combine entity information feature, obtain the visual feature of entity.Finally, based on the visual feature of entity, the action related score of video frame is calculated through full connection layer, the starting boundary and end boundary of target action segment are predicted through double branch network, and the positioning result is obtained, and loss function is constructed for reverse training.The application solves the technical problem of low prediction accuracy in video positioning, improves the accuracy of action boundary prediction in video.

Description

Technical Field

[0001] This invention belongs to the field of video positioning technology, specifically relating to a multimodal video positioning method based on deep neural networks. Background Technology

[0002] With the rapid development of the internet and smart devices, especially the widespread use of portable user devices such as smartphones, tablets, and laptops, the production and consumption of video content has become an indispensable part of daily life. New applications such as video recommendation systems, automatic video summarization, and video surveillance are constantly emerging, attracting widespread attention. These applications require processing massive amounts of video data and possess strong understanding and retrieval capabilities. However, due to the high dimensionality, multimodality, and time dependence of video data, efficiently locating the time segment corresponding to a specific natural language description within massive amounts of video data has become a challenge in current research.

[0003] Video temporal localization, as a key technology for solving this problem, is gradually becoming a research hotspot in both academia and industry. The core objective of video temporal localization is to automatically identify and accurately locate the time segment in a video that matches a given natural language description. To achieve this, the system needs to understand the video content and accurately capture the semantic relationships between video segments and text descriptions. However, the challenge of video temporal localization lies in the fact that the complexity of video content and the diversity of natural language expressions often make accurate alignment and localization difficult.

[0004] The rise of multimodal learning has provided new research perspectives and solutions for video temporal localization. By combining visual and linguistic information, researchers have proposed various methods to improve the accuracy of video temporal localization. Furthermore, the introduction of pseudo-supervised learning has further reduced the dependence on large-scale labeled data, thereby enhancing the model's generalization ability.

[0005] Despite some progress, existing methods still have limitations in handling long videos, complex scenes, and fine-grained action recognition. For example, accurately locating a brief and complex action segment in a long video remains a challenging task. As the diversity and complexity of video content increase, the requirements for video temporal localization in terms of computational efficiency, accuracy, and robustness are also becoming increasingly stringent.

[0006] Against this backdrop, further improving the accuracy and efficiency of video temporal localization through multimodal learning and optimization techniques remains an important direction for future research. Researchers need to develop more efficient and robust models to address various challenges in practical applications, thereby promoting the development of video content understanding and retrieval technologies. Summary of the Invention

[0007] To address the shortcomings of traditional methods, such as low accuracy and difficulty in optimization, this invention proposes a multimodal video localization method based on deep neural networks. It also introduces a computationally intensive loss function to solve the problem of model convergence difficulties.

[0008] To achieve the above objectives, the steps of the present invention are as follows:

[0009] 1. Decompose the user-input sentence S, classify the words in the sentence according to their parts of speech, and use one-hot encoding to represent the classification results; use the GloVe model to initialize the sentence S and obtain the word feature matrix Q.

[0010] First, the user-input sentence S is decomposed into entity information query and action information query. For each word T in the sentence... i Classification is based on part of speech, using one-hot encoding P. i This indicates the classification result. If the part of speech is related to the entity (e.g., noun, adjective), then P... i e = [1, 0, 0]; if related to an action (such as a verb or adverb), then P i m = [0, 1, 0]; otherwise P i o = [0, 0, 1].

[0011] Next, the GloVe model is used to initialize sentence S, obtaining the word feature matrix. Where w i The feature of the i-th word is represented by dimension d. w N represents the number of words.

[0012] 2. Using Transformer q The module learns the relationships between words based on the word feature matrix Q, outputs a language query feature matrix, and extracts entity information features based on one-hot encoding. and action information features

[0013] Using Transformer qThis module uses a standard Transformer to learn the relationships between words based on the word feature matrix Q, and outputs a language query feature matrix.

[0014] F q =Transformer q FC1(Q),

[0015] in It is the learned language query feature matrix. FC1 is a fully connected layer that transforms the word feature matrix from d... w Projecting 1 dimension onto d dimension.

[0016] Extracting entity information features based on one-hot encoding and action information features The specific formula is as follows:

[0017]

[0018]

[0019] Among them, P i e [0] is P i e One-hot encoding represents the first element in the matrix, P i o [2] is P i o One-hot encoding represents the third element in the matrix, P i m [1] is P i m One-hot encoding represents the third element in the matrix. Represents entity information characteristics, It represents the characteristics of action information.

[0020] 3. Generate visual features based on user-input video. Combining entity information features Computational visual features With entity information features The similarity matrix is ​​used to obtain the visual features F of the entity. ve .

[0021] Extracting visual features from user-input videos using a pre-trained 3D convolutional network. Then through Transformer e The module captures inter-frame context information and uses a Transformer without position encoding to generate updated visual features:

[0022]

[0023] The fully connected network FC2 is used to transfer visual features from d v Project the dimension onto the d-dimensional plane.

[0024] Calculate the similarity matrix between visual features and entity information features. Decompose the similarity matrix S into row vectors and calculate a row-expanded similarity matrix:

[0025]

[0026] Decompose the similarity matrix S into column vectors and calculate a similarity matrix expanded by columns:

[0027] S = [S c1 ... S cN ]

[0028] S c =[softmax(S c1 ... softmax(S) cN )]

[0029] Where S ri S ci Let the row vectors and column vectors be represented respectively, and the similarity matrix S is obtained by softmax normalization. r and S c .

[0030] Using these similarity matrices, calculate two attention weights:

[0031]

[0032] Next, the visual features F of the entity are obtained by fusing visual and query features using the following formula. ve :

[0033]

[0034] In this context, the symbol ⊙ represents element-wise multiplication, the semicolon “;” represents concatenation, and the fully connected layer FC3 is used to project the concatenated features from 4d dimension to d dimension.

[0035] 4. Entity-based visual features F ve The motion-related score P of video frames is calculated through a fully connected layer. e The starting and ending boundaries of the target action segment are predicted by a dual-branch network, the boundary probability distribution is calculated by softmax, the localization result is obtained, and a loss function is constructed for reverse training.

[0036] The motion-related score P of the video frame is calculated using a fully connected layer. e =sigmoid(FC5(ReLU(FC4(F ve Frames with higher scores have a greater probability of being selected as action regions. The model uses an LSTM Transformer to learn the local and global motion changes of video features:

[0037]

[0038]

[0039] Through action-related scores P e By combining elements one by one, video frames related to the action are obtained.

[0040] Calculate the similarity matrix Repeat step S3 to operate on the similarity matrix to obtain motion-aware visual features. The initial boundary of the target action segment is predicted by a fully connected network, and the initial score is obtained. Then, the boundary probability distribution is calculated using softmax.

[0041] Residual connection obtained

[0042] Using Bi-LSTM Transformer to learn temporal motion variations of video features:

[0043]

[0044] Among them, visual features are divided from dimension d v Projected onto dimension d; Transformer Bi-LSTM (·) represents Bi-LSTMTransformer.

[0045] Calculate the similarity matrix Repeat step S3 to operate on the similarity matrix to obtain F. vsq Predict the termination boundary S using a fully connected network e =FC9(ReLU(FC8(F vsq Then, the boundary probability distribution is calculated using softmax.

[0046] Calculate the prediction start and end boundaries of the target action segment using the maximization of joint probability:

[0047]

[0048] Define the initial boundary probability and the probability of the termination boundary They represent a certain time t. s It is the initial probability of the action segment, and at a certain time t e It represents the probability of the action segment ending; by iterating through all t... s and t e Find the time point that maximizes the joint value of the start and end probabilities. and

[0049] The training objective for action boundary prediction is expressed as:

[0050]

[0051] Where f XE It is the cross-entropy loss function; where (τ) s ,τ e () is the true boundary of the action.

[0052] The model also calculates the standard IoU loss.

[0053] Dense loss L dense :

[0054] Calculate a smooth prediction boundary using temperature-controlled softmax:

[0055]

[0056]

[0057] Define two scalars (α) s ,α e To measure the degree of clustering of boundary probabilities:

[0058]

[0059]

[0060] and respectively with α s and α e Whether the smoothed prediction boundary clusters near the true label is calculated by multiplying and summing each term, denoted as L. dense :

[0061]

[0062] And through the final loss function L=λ1L boundary +λ2L IoU +λ3L dense Optimize.

[0063] 5. Loss Function

[0064] The loss function comprehensively considers the cross-entropy loss L of boundary prediction. boundary IoU loss L IoU and dense loss L dense Finally, the performance of the model is optimized by combining the weights of the hyperparameters λ1, λ2, and λ3.

[0065] Beneficial effects of this invention:

[0066] This invention employs a two-stage network to predict the starting frame, and then uses a multimodal interaction approach to predict the ending frame. This expands the dimensions of model learning, addresses the technical problem of low prediction accuracy in video localization, and aims to improve the accuracy of action boundary prediction in videos through refined feature extraction, fusion, and context learning. Attached Figure Description

[0067] Figure 1 This is a diagram of the model structure. Detailed Implementation

[0068] like Figure 1 This paper presents a multimodal video localization method based on deep neural networks. The specific steps are as follows:

[0069] 1. In the process of query decomposition of the input sentence S, we can divide it into two main query categories: entity information query and action information query. Specifically, for each word T in sentence S... i Based on its part-of-speech characteristics, we will categorize it into one of three part-of-speech tags: entity, action, and other. To more clearly distinguish each word T... i For the category, we use one-hot encoding P i This is used to represent the classification result of the word. If the part-of-speech tag of a word is related to an entity (e.g., a noun or adjective, which are usually used to refer to objects, things, or describe features), then we will classify it as entity information and represent it as P using one-hot encoding. i e = [1,0,0]. That is, the first position is 1, indicating that the word is related to the entity.

[0070] Conversely, if the word's part-of-speech tag is related to an action (e.g., a verb or adverb used to describe behavior, process, state, or action), then we will categorize it as an action information query. In this case, one-hot encoding will differ from the encoding method used for entity information queries.

[0071] By using this one-hot encoding-based part-of-speech classification method, we can process each word in a sentence more systematically, thus laying a clear classification foundation for subsequent entity information queries and action information queries.

[0072] 2. During the initialization of sentence S using the GloVe model, each word in the sentence is first converted into a corresponding word vector representation. The GloVe model assigns a pre-trained word feature vector to each word in sentence S, thus obtaining the word feature matrix Q, which is represented as follows:

[0073] Where w i The feature of the i-th word is represented by the dimension d. w N represents the number of words in the language query. Through this initialization method, sentence S is transformed into an N×d... w A matrix Q of dimension w, where each row... i This corresponds to the word vector of a specific word in a sentence. This representation provides a compact and efficient word feature representation for subsequent language processing tasks.

[0074] 3. To learn the relationships between the words in sentence S, a Transformer can be used. q This module uses a standard Transformer model. The Transformer model uses a self-attention mechanism to capture the dependencies between each word in the input sequence and other words, thus learning the relationships between words. The following is the process of how to learn the relationships between words using the Transformer module:

[0075] F q =Transformer q FFC1(Q)

[0076] in These are the learned language query features. FFC1(·) is a fully connected layer used to transfer word features from d... w Projecting dimension d onto dimension d, Transformer q (·) is a standard Transformer module.

[0077] 4. One-hot encoding is used to distinguish the parts of the sentence related to entity information and action information, and this information is extracted using a formula. The specific formula is defined as follows:

[0078]

[0079]

[0080] Among them, P ie [0] is P i e One-hot encoding represents the first element in the matrix, P i o [2] is P i o One-hot encoding represents the third element in the matrix, P i m [1] is P i m One-hot encoding represents the third element in the matrix. Represents entity information characteristics, It represents the characteristics of action information.

[0081] 5. In the process of extracting visual features using a pre-trained 3D convolutional neural network (3D CNN), the input visual data (such as video or 3D image) is processed through the network to obtain a time-series representation of visual features. Where f v,i Let d represent the i-th visual feature, with dimension d. v T represents the number of features.

[0082] 6. Repeat step 3 using Transformer e This module focuses on contextual information between different frames, and it uses a Transformer module without position encoding.

[0083]

[0084] in These are the updated visual features. FC2(·) is a fully connected layer that converts the visual features from dimension d. v Projected onto dimension d; Transformer e (·) represents a standard Transformer module without position encoding.

[0085] 7 Given visual features and entity query features Calculate their similarity:

[0086]

[0087] The matrix is ​​normalized by performing softmax operations row by row and column by column to obtain two similarity matrices S. r and S c This function is used to capture similarity in rows and columns of a matrix. It applies the softmax function independently to each row and column of the matrix, thus calculating the similarity of each element relative to the other elements.

[0088] Decompose S into row vectors and calculate a similarity matrix expanded by rows:

[0089]

[0090] Decompose S into column vectors and calculate a similarity matrix expanded by columns:

[0091] S = [S c1 ... S cN ]

[0092] S c =[softmax(S c1 ... softmax(S) cN )]

[0093] Where S ri S ci Let the row vectors and column vectors be represented respectively, and the similarity matrix S is obtained by softmax normalization. r and S c .

[0094] 8. Calculate the two attention weights and Visual features F of an entity ve The following formula is used to calculate the two attention weight matrices. and To capture query features With visual features The relationships between these attention weights are then considered. These attention weights are then combined with the original visual features and passed through a fully connected layer (FC3) to generate the final entity visual features.

[0095] in The symbol ⊙ indicates element-wise multiplication; [] indicates feature concatenation; FC3(·) is a fully connected layer used to project the concatenated features from the 4d dimension to the d dimension.

[0096] 9. To predict action position, two fully connected layers will be used to progressively process the entity visual features F. ve To calculate the motion-related score P of video frames e The process includes a ReLU activation function, two fully connected layers, and a final sigmoid activation function to map the output to a probability distribution.

[0097] P e =sigmoid(FC5(ReLU(FC4(F ve ))))

[0098] The motion-related score Pe of a video frame was successfully calculated using two fully connected layers, FC4 and FC5, and ReLU and sigmoid activation functions. A higher score indicates a greater probability that the frame is an action region, which can be used for further motion location prediction or editing motion segments in the video. The motion-related score... The output feature dimensions of FC4(·) and FC5(·) are respectively And 1. The higher the motion-related score, the higher the probability that the corresponding frame will be selected as the motion region.

[0099] 10. Using LSTM Transformer to learn local and global motion changes in video features to generate more representative motion features. This process enables the model to not only focus on short-term inter-frame motion changes, but also to capture long-term motion dependencies in the video.

[0100]

[0101] in, It is the updated motion feature, focusing more on motion changes; FC2(·) is the fully connected layer in step 6, which converts visual features from dimension d. v Projected onto dimension d; Transformer LSTM (·) represents LSTM Transformer.

[0102] 11. Based on the action-related score P e To focus on motion-related video frames, the formula is: This is the motion relevance score for each frame. Multiplying these scores element-wise enhances the features of action frames while weakening features of frames unrelated to the action, allowing the model to better capture key motion regions. This operation effectively improves the accuracy of motion location prediction.

[0103] 12. Repeat step 8 to query motion features. Integrating into visual motion representation In order to obtain visual features F for motion perception vm :

[0104]

[0105] 13 was predicted using a two-branch network containing two fully connected layers:

[0106] S s =FC7(ReLU(FC6(F vm )))

[0107] The initial score of the target action segment The output feature dimensions of FC6(·) and FC7(·) are respectively And 1.

[0108] The probability distribution of the action initiation boundary is obtained by softmax calculation:

[0109]

[0110] 15. To predict the action termination boundary, we first use the initial boundary probability. Focus on motion-related video frames. The specific steps are as follows: Perform element-wise multiplication (⊙) to calculate the initial boundary probabilities. With motion characteristics Multiplying and adding back the original motion characteristics, the formula is:

[0111]

[0112] This operation enhances frame features associated with the starting boundary while preserving the original frame features for action end boundary prediction.

[0113] Similar to step 10, step 16 uses a Bi-LSTM Transformer to learn the temporal motion variations of video features. The Bi-LSTM Transformer structure is used to better capture long-short-term dependencies between frames. This method can simultaneously focus on local and global motion variations, making it suitable for processing video data with temporal dependencies.

[0114]

[0115] Among them, the input features are That is, through the action initiation boundary probability Weighted motion feature matrix It contains motion feature information for each frame in the video, with a dimension of T×dv, where T is the number of video frames and dv is the feature dimension. It is the updated motion feature, focusing more on motion changes; FC2(·) is the fully connected layer in step 6, which converts visual features from dimension d. v Projected onto dimension d; Transformer Bi-LSTM (·) represents Bi-LSTMTransformer.

[0116] 17. Repeat step 8 to query motion features. Integrating into video information that focuses on motion-related and starting boundaries To obtain the visual features F of the predicted termination boundary. vsq :

[0117]

[0118] 18. Repeat steps 13 and 14 to obtain the probability distribution of the ending boundary:

[0119] S e =FC9(ReLU(FC8(F vsq )))

[0120]

[0121] Initial score of the target action segment Among them, the output feature dimensions of FC8(·) and FC9(·) are respectively And 1.

[0122] 19. Define the initial boundary probability and the probability of the termination boundary They represent a certain time t. s It is the initial probability of the action segment, and at a certain time t e This represents the probability of the action segment ending. To determine the optimal start and end times, optimization is performed by maximizing the joint probability:

[0123]

[0124] This step involves iterating through all t. s and t e Find the time point τ that maximizes the joint value of the start and end probabilities. s and τ e .

[0125] To further evaluate the accuracy of the predictions, the optimization score of the prediction boundary is calculated. That is, the joint probability at the optimal start and end time points:

[0126]

[0127] This optimization score This reflects the model's performance at the start and end boundaries of the prediction. The confidence score indicates the model is more confident in the predicted boundary. By maximizing the joint probability of the start and end boundaries, we can predict the start time of an action segment. and end time Meanwhile, the calculated optimization score This is used to measure the reliability of the prediction results. This process helps to accurately pinpoint the temporal range of target actions in a video.

[0128] 20. Given the initial boundary probability distribution for prediction and the probability distribution of the termination boundary The training objective for action boundary prediction can be represented by the cross-entropy loss function. This loss function measures the difference between the boundary probability distribution predicted by the model and the true boundary.

[0129]

[0130] Among them, f XE (·) is the cross-entropy function, (τ) s ,τ e The loss represents the true boundary of the action. By minimizing this loss, the model can better predict the start and end times of action segments, making the predicted boundary distribution closer to the true boundary.

[0131] 21. To measure the prediction boundary With the true boundary (τ) s ,τ e The difference between the two can be expressed using the IoU (Intersection over Union) loss, denoted as L. IoU The IoU loss reflects the degree of overlap between the predicted action segment and the actual action segment. A value closer to 1 indicates a closer proximity between the predicted and actual boundaries, resulting in a smaller loss. The IoU loss is the ratio of the overlapping area between the predicted and actual action boundaries to their union. The formula is:

[0132]

[0133] 22. To compute a smooth prediction boundary and assess whether it clusters near the true labels, a temperature-controlled softmax function is used to smooth the prediction boundary distribution:

[0134]

[0135]

[0136] To measure whether the boundary probability distribution clusters around the true label τ s and τ e Nearby, define two scalar vectors α s and α e , representing the distance between the predicted boundary and the true boundary:

[0137]

[0138]

[0139] and respectively with α s and α e Whether the smoothed prediction boundary clusters near the true label is calculated by multiplying and summing each term, denoted as L.dense :

[0140]

[0141] Where temp is a hyperparameter of softmax, and |||1 is the L1 norm L dense A measure of whether the smoothed prediction boundary clusters around the true label τ s and τ e Nearby. If the predicted boundary is close to the true boundary, then α s and α e The value is small, and the loss L dense The smaller value indicates that the model is more accurate in boundary prediction.

[0142] 23. The final training objective, loss function L, optimizes the model's predictive ability by combining multiple loss functions. These loss functions include the action boundary loss L. boundary IoU loss L IoU and boundary aggregation loss L dense Each loss term is assigned a different weight based on its importance. The formula is as follows:

[0143] L=λ1L boundary +λ2L IoU +λ3L dense

[0144] Where L boundary : Measure the prediction boundary using cross-entropy loss and With the true boundary τ s and τ e Difference L IoU Based on IoU (Intersection over Union) loss, it measures the degree of overlap between the predicted boundary and the true boundary. dense The predicted boundaries are clustered near the true boundaries using temperature-controlled softmax calculations to ensure a smooth boundary distribution. λ1, λ2, and λ3 are hyperparameters used to adjust the weight of each loss term in the overall loss. By appropriately adjusting these hyperparameters, the model can focus more on specific loss terms, such as boundary accuracy, IoU matching, or prediction smoothness.

[0145] The final loss function L optimizes the model's training process by balancing multiple loss terms, ensuring that the model achieves optimal performance in boundary prediction, IoU matching, and boundary distribution. This combined loss helps the model more accurately locate the start and end boundaries of action segments.

[0146] Example:

[0147] The Charades dataset is a public dataset used for action recognition, action classification, and temporal action localization. Its main characteristics are: Data size: It contains 9848 videos, of which 7985 are used for training and 1863 are used for testing.

[0148] The total video duration exceeds 66 hours. Video content: The video content simulates common activities in a home environment (e.g., cooking, tidying up). Each video segment can contain multiple actions, and multiple actions can even occur simultaneously (i.e., overlap). Annotation information: Each video is annotated with one or more action segments (start and end times of each action). Supports multi-tag classification: A video may contain tags for multiple action categories. Provides start and end timestamps for actions, making it suitable for time-segment video localization tasks.

[0149] The publicly available dataset TACoS (Temporally-Annotated Cooking Scenes) is a video localization task dataset focused on kitchen scenes. It primarily evaluates the performance of natural language descriptions in time-span video localization. Data Source: Based on the MPII Cooking Activities Dataset. The video content mainly involves cooking scenes, such as chopping vegetables and pouring water. Data Size: Contains 127 videos, with a total length of approximately 19 hours. Annotation Information: Each video is annotated with multiple natural language descriptions and their corresponding time periods (start and end times). The annotation information describes the semantic content of the actions in the video, such as "slicing tomatoes."

[0150] Definition: IoU measures the degree of overlap between the time period P = [tp1, tp2] predicted by the model and the actual time period G = [tg1, tg2]. Value range: IoU values ​​are between [0,1]. The closer the IoU is to 1, the higher the match between the predicted time period and the actual time period.

[0151] Table 1 shows the performance on the TACOS dataset.

[0152]

[0153] Table 2 shows the performance on the Charades dataset.

[0154]

[0155] IoU=1: The predicted time period is completely consistent with the actual time period.

[0156] IoU=0: The predicted time period does not overlap with the actual time period.

[0157] IoU@n: Represents the proportion of predicted results where IoU>=n.

[0158] The experimental results are shown in Tables 1 and 2, which show that the present invention is significantly superior to several existing technologies in other regions.

Claims

1. A multimodal video localization method based on deep neural networks, characterized in that, Includes the following steps: S1. Decompose the user-input sentence S, classify the words in the sentence according to their parts of speech, and use one-hot encoding to represent the classification results; The sentence S is initialized using the GloVe model to obtain the word feature matrix. ; S2. Use Module, based on word feature matrix It learns the relationships between words, outputs a language query feature matrix, and extracts entity information features based on one-hot encoding. and action information features ; S3. Generate visual features based on user-input video. Combining entity information features Computational visual features With entity information features The similarity matrix is ​​used to obtain the visual features of entities. ; S4. Entity-based visual features The motion-related scores of video frames are calculated through fully connected layers. The system predicts the start and end boundaries of target action segments using a dual-branch network, calculates the boundary probability distribution using softmax to obtain the localization result, and constructs a loss function for back-end training. The specific implementation process is as follows: Motion-related scores of video frames are calculated using fully connected layers. The LSTM Transformer is used to learn the local and global motion changes of video features: ； ； in, For users to input the visual features of the video, a fully connected network Used to transfer visual features from Dimension projection to Dimension; Based on action-related scores By combining elements one by one, video frames related to the action are obtained. Calculate the similarity matrix Repeat step S3 to operate on the similarity matrix to obtain motion-aware visual features. The initial boundary of the target action segment is predicted using a fully connected network to obtain the initial score. Then, the boundary probability distribution is calculated using softmax. ; Residual connection obtained ; Using Bi-LSTM Transformer to learn temporal motion variations of video features: ； in, Visual features from dimensions Projected to dimension ; Represents Bi-LSTMTransformer; Calculate the similarity matrix Repeat step S3 to operate on the similarity matrix to obtain Predicting the end boundary using a fully connected network Then, the boundary probability distribution is calculated using softmax. ; Calculate the prediction start and end boundaries of the target action segment using the maximization of joint probability: ； Define the initial boundary probability and the probability of the termination boundary , respectively representing a certain moment It is the probability of the action segment starting, and at a certain moment... It represents the probability of an action segment ending; this is achieved by iterating through all... and Find the time point that maximizes the joint value of the start and end probabilities. and ; The training objective for action boundary prediction is expressed as: ； in It is the cross-entropy loss function; where It is the true boundary of the action; The model also calculates the standard IoU loss. ; Dense loss : Calculate a smooth prediction boundary using temperature-controlled softmax: ； ； Define two scalars To measure the degree of clustering of boundary probabilities: ； ； in, Indicates the number of features; and respectively with and Whether the smoothed prediction boundary clusters near the true label is calculated by multiplying and summing each term, denoted as . ； And through the final loss function Optimize, , and This is a hyperparameter.

2. The multimodal video localization method based on deep neural networks according to claim 1, characterized in that, The specific implementation process of step S1 is as follows: The user-input sentence S is decomposed into entity information query and action information query; for each word in the sentence... Classify based on part of speech, using one-hot encoding. This indicates the classification result; If the part of speech is related to the entity, then If it is related to the action, then ; otherwise ; The sentence S is initialized using the GloVe model to obtain the word feature matrix. ,in Indicates the first The features of each word, with dimensions of . , Indicates the number of words.

3. The multimodal video localization method based on deep neural networks according to claim 2, characterized in that, The specific implementation process of step S2 is as follows: use This module uses Transformer to calculate the word feature matrix. Learn the relationships between words and output a language query feature matrix: ； It is the learned language query feature matrix. It is a fully connected layer that transfers the word feature matrix from Dimensional projection to Dimension; Extracting entity information features based on one-hot encoding and action information features The specific formula is as follows: ； ； in, for One-hot encoding represents the first element in the matrix. for One-hot encoding represents the third element in the matrix. for One-hot encoding represents the second element in the matrix. Represents entity information characteristics, It represents the characteristics of action information.

4. The multimodal video localization method based on deep neural networks according to claim 3, characterized in that, The specific implementation process of step S3 is as follows: Extracting visual features from user-input videos using a pre-trained 3D convolutional network. , Indicates the first A visual feature, and then through The module captures inter-frame context information and uses a Transformer without position encoding to generate updated visual features: ； Calculate the similarity matrix between visual features and entity information features. : Similarity matrix Decompose by row vectors and calculate a similarity matrix expanded by rows: ； Similarity matrix Calculate a similarity matrix by decomposing it into column vectors. ； ； in Let the row vectors and column vectors be represented respectively, and the similarity matrix is ​​obtained by softmax normalization. and ; Using the similarity matrix, calculate the two attention weights: ； The visual features of an entity are obtained by fusing visual and query features using the following formula. : ； In this context, the symbol ⊙ indicates element-wise multiplication, the semicolon indicates concatenation, and the fully connected layer... Used to transfer the spliced ​​features from Dimensional projection to Dimension.

Images