Video understanding method, device, equipment, storage medium and computer program product

Abstract

The present application relates to the field of video technologies, and provides a video understanding method, apparatus, computer device, storage medium, and computer program product. The present application can improve the efficiency and accuracy of video understanding. The method includes: obtaining a video to be understood, and respectively obtaining text feature information, dynamic feature information, and static feature information of the video to be understood by using a text feature acquisition network, a dynamic feature acquisition network, and a static feature acquisition network, and obtaining an understanding result of the video to be understood based on the text feature information, the dynamic feature information, and the static feature information.

Description

Technical Field

[0001] This application relates to the field of video technology, and in particular, to a video understanding method, apparatus, computer device, storage medium, and computer program product. Background Art

[0002] With the booming development of the fields of natural language processing and computer vision, video understanding has become a new hot topic after image understanding. Efficient video understanding of videos obtained in real time can greatly contribute to improving security issues.

[0003] Traditional technologies usually perform video understanding manually, but the efficiency of video understanding through this technology is relatively low. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a video understanding method, apparatus, computer device, computer-readable storage medium, and computer program product.

[0005] In a first aspect, this application provides a video understanding method. The method includes:

[0006] Obtain a video to be understood;

[0007] Use a text feature acquisition network, a dynamic feature acquisition network, and a static feature acquisition network to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood;

[0008] Based on the text feature information, dynamic feature information, and static feature information, obtain the understanding result of the video to be understood.

[0009] In one embodiment, obtaining the understanding result of the video to be understood based on the text feature information, dynamic feature information, and static feature information includes:

[0010] Perform guided attention fusion on the text feature information, dynamic feature information, and static feature information to obtain multiple modal representation information;

[0011] Use a graph attention network to obtain the aggregated feature information of the multiple modal representation information;

[0012] According to the aggregated feature information, obtain the understanding result of the video to be understood.

[0013] In one embodiment, obtaining the understanding result of the video to be understood according to the aggregated feature information includes:

[0014] Use a temporal memory update network to perform temporal memory update on the aggregated feature information;

[0015] According to the aggregated feature information after temporal memory update, obtain the understanding result of the video to be understood.

[0016] In one embodiment, the method further includes:

[0017] Obtaining a video sample carrying text information and an understanding result label corresponding to the video sample;

[0018] Training a text feature acquisition network, a dynamic feature acquisition network, a static feature acquisition network, a graph attention network, and a temporal memory update network by using the video sample and the understanding result label.

[0019] In one embodiment, obtaining text feature information of a video to be understood by using a text feature acquisition network includes:

[0020] Detecting whether the video to be understood carries corresponding text information;

[0021] If not, generating corresponding text information for the video to be understood by using a text information generation model;

[0022] Using the text feature acquisition network to obtain the text feature information of the video to be understood according to the corresponding text information.

[0023] In one embodiment, using a text feature acquisition network, a dynamic feature acquisition network, and a static feature acquisition network to respectively obtain text feature information, dynamic feature information, and static feature information of a video to be understood includes:

[0024] Using the text feature acquisition network, the dynamic feature acquisition network, and the static feature acquisition network to respectively obtain to-be-processed text feature information, to-be-processed dynamic feature information, and to-be-processed static feature information of the video to be understood;

[0025] Using a context information acquisition network to respectively perform context encoding on the to-be-processed text feature information, the to-be-processed dynamic feature information, and the to-be-processed static feature information to obtain text feature information, dynamic feature information, and static feature information.

[0026] In a second aspect, the present application further provides a video understanding device. The device includes:

[0027] A video to be understood acquisition module, configured to acquire a video to be understood;

[0028] A feature information acquisition module, configured to use a text feature acquisition network, a dynamic feature acquisition network, and a static feature acquisition network to respectively obtain text feature information, dynamic feature information, and static feature information of the video to be understood;

[0029] An understanding result acquisition module, configured to obtain an understanding result of the video to be understood based on the text feature information, the dynamic feature information, and the static feature information.

[0030] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] Obtain a video to be understood; use a text feature acquisition network, a dynamic feature acquisition network, and a static feature acquisition network to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood; based on the text feature information, dynamic feature information, and static feature information, obtain the understanding result of the video to be understood.

[0032] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0033] Obtain a video to be understood; use a text feature acquisition network, a dynamic feature acquisition network, and a static feature acquisition network to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood; based on the text feature information, dynamic feature information, and static feature information, obtain the understanding result of the video to be understood.

[0034] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0035] Obtain a video to be understood; use a text feature acquisition network, a dynamic feature acquisition network, and a static feature acquisition network to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood; based on the text feature information, dynamic feature information, and static feature information, obtain the understanding result of the video to be understood.

[0036] The above video understanding method, device, computer device, storage medium, and computer program product obtain a video to be understood, and use a text feature acquisition network, a dynamic feature acquisition network, and a static feature acquisition network to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood. Based on the text feature information, dynamic feature information, and static feature information, the understanding result of the video to be understood is obtained. In this solution, the video to be understood is obtained, input into a video understanding model, and the text feature acquisition network, dynamic feature acquisition network, and static feature acquisition network in the video understanding model are used to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood. Based on the text feature information, dynamic feature information, and static feature information, the understanding result of the video to be understood is obtained, thereby improving the efficiency of video understanding. Further, using the text feature acquisition network, dynamic feature acquisition network, and static feature acquisition network to jointly obtain feature information of the video to be understood can improve the accuracy of video understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic flowchart of a video understanding method in an embodiment;

[0038] Figure 2 is a schematic diagram of a video understanding model in an embodiment;

[0039] Figure 3 is a schematic flowchart of the construction process of a video understanding data set in an embodiment;

[0040] Figure 4 is a schematic flowchart of a video understanding method in another embodiment;

[0041] Figure 5 is a structural block diagram of a video understanding device in an embodiment;

[0042] Figure 6 is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] In one embodiment, as Figure 1 shown, a video understanding method is provided. In this embodiment, it is exemplified that the method is applied to a terminal, and includes the following steps:

[0045] Step S101, obtain a video to be understood.

[0046] In this step, the video to be understood can be a real-time video of a specific event.

[0047] Specifically, the terminal obtains the video to be understood.

[0048] Step S102: Use the text feature acquisition network, dynamic feature acquisition network, and static feature acquisition network to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood.

[0049] In this step, as Figure 2 shown, the text feature acquisition network (which can also be called the subtitle text feature extraction network) can be a network model for obtaining the text feature information of the video to be understood, such as the BERT network model; the text feature information can be the feature information obtained after the BERT network model performs word embedding on the subtitle text information of the video to be understood. Among them, as Figure 2 and 3 shown, the subtitle text information can be the subtitle description information summarized manually for the video to be understood (for example, each video segment is summarized by specific event staff to obtain a subtitle description information, such as "The specific content is: XXXX"), and the subtitle description information can be saved in the document corresponding to the video to be understood. The subtitle text information can also be the corresponding text information generated by the text information generation model according to the video to be understood, where the text information generation model can be the MDVC model (Multi-modal Dense Video Captioning); the dynamic feature acquisition network can be a network model for extracting the dynamic feature information of the video to be understood, such as the C3D network; the static feature acquisition network can be a network model for extracting the static feature information of the video to be understood, such as the VGG16 network model.

[0050] Specifically, as Figure 2 shown, the terminal uses the text feature acquisition network, dynamic feature acquisition network, and static feature acquisition network to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood.

[0051] Step S103: Based on the text feature information, dynamic feature information, and static feature information, obtain the understanding result of the video to be understood.

[0052] In this step, the understanding result can be which type the video to be understood belongs to among the preset types.

[0053] Specifically, the terminal obtains the understanding result of the video to be understood based on the text feature information, dynamic feature information, and static feature information.

[0054] In the above video understanding method, the video to be understood is obtained, and the text feature acquisition network, dynamic feature acquisition network, and static feature acquisition network are used to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood. Based on the text feature information, dynamic feature information, and static feature information, the understanding result of the video to be understood is obtained. In this solution, the video to be understood is obtained and input into the video understanding model. The text feature acquisition network, dynamic feature acquisition network, and static feature acquisition network in the video understanding model are used to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood. Based on the text feature information, dynamic feature information, and static feature information, the understanding result of the video to be understood is obtained, thereby improving the efficiency of video understanding. Further, the text feature acquisition network, dynamic feature acquisition network, and static feature acquisition network are used together to obtain the feature information of the video to be understood, which can improve the accuracy of video understanding.

[0055] In one embodiment, the use of the text feature acquisition network, dynamic feature acquisition network, and static feature acquisition network in the above step S102 to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood specifically includes: using the text feature acquisition network, dynamic feature acquisition network, and static feature acquisition network to respectively obtain the text feature information to be processed, dynamic feature information to be processed, and static feature information to be processed of the video to be understood; using the context information acquisition network to respectively perform context encoding on the text feature information to be processed, dynamic feature information to be processed, and static feature information to be processed to obtain the text feature information, dynamic feature information, and static feature information.

[0056] In this embodiment, the context information acquisition network can be a BiLSTM.

[0057] Specifically, the terminal uses the text feature acquisition network, dynamic feature acquisition network, and static feature acquisition network to respectively obtain the text feature information to be processed, dynamic feature information to be processed, and static feature information to be processed of the video to be understood, and uses the context information acquisition network to respectively perform context encoding on the text feature information to be processed, dynamic feature information to be processed, and static feature information to be processed to obtain the text feature information, dynamic feature information, and static feature information.

[0058] Exemplarily, such as Figure 2As shown in the figure, the terminal uses BERT (text feature acquisition network) to perform word embedding on the subtitle information in the video respectively, and uses BiLSTM (context information acquisition network) to perform context encoding and generate feature vectors, and uses 1D convolution and max pooling to form the final feature vectors (text feature information), forming the corresponding s vectors. Then, VGG16 (static feature acquisition network) and C3D (dynamic feature acquisition network) are used to extract the static feature information and dynamic feature information in the video (equivalent to the to-be-processed static feature information and to-be-processed dynamic feature information) respectively, and BiLSTM is used for feature encoding to form a vectors and m vectors (static feature information and dynamic feature information). Specifically, as Figure 2 shown, the mutual-guidance feature extraction module of the video understanding model includes a video dynamic feature extraction module, a video static feature extraction module, and a subtitle text feature extraction module. ① Video dynamic feature extraction module: The terminal extracts the dynamic features of the video through the C3D network. The actual input of the video is to obtain its dynamic features every 16 frames, that is, 16*N. At the same time, in order to align the context information of different modalities in the video, a sliding window sampling operation can be performed in the time dimension to ensure that each frame contains dynamic feature information. The dynamic feature m i is obtained from the last fully connected layer of it, where m i (i = 1, 2,..., N) is the i-th video dynamic feature, and the video dynamic feature U m =[m1, m2,..., m N ∈R 4096×N is obtained, where N is the number of frames of the video. In order to ensure the context information of the video dynamic features, a BiLSTM (context information acquisition network) of the same dimension is used to encode the video dynamic features h i m =BiLSTM(m i ), and the encoded video dynamic features (dynamic feature information) U m =[h1 m , h2 m ,..., h N m ∈R 4096×N is obtained, where h i m is the encoded feature vector of the i-th video dynamic feature, N is the number of frames of the video, and m represents the video dynamic features. ② Video static feature extraction module: The VGG16 network model (static feature acquisition network) is used to extract the static frame features of the video. In the video time domain, in order to synchronize the context relationship of different modalities in the video and accurately infer the evolution information of the static features of the video frames, the video static frame features are extracted at 1 FPS, and the static feature a i is obtained from the second-to-last fully connected layer of it, where a i(i = 1, 2, ……, N) is the i-th video static feature, obtaining the video static feature U a = [a1, a2, ……, a N ∈ R 4096×N , where N is the number of frames of the video, R represents the spatial dimension, 4096×N represents the size of the spatial dimension of R. To obtain the context information of the video static feature, use a BiLSTM of the same dimension to encode the video static feature h i a = BiLSTM(a i ), obtaining its encoded video dynamic feature (static feature information) U a = [h1 a , h2 a , ……, h N a ∈ R 4096×N , where h i a is the feature vector after encoding the i-th video static feature, N is the number of frames of the video, and a represents the video static feature. ③ Subtitle text feature extraction module: Use the BERT network model (text feature acquisition network) to extract subtitle text features, and select the 12-layer BERT network model to extract subtitle text features (equivalent to the text feature information to be processed), obtaining the subtitle text feature s i , where s i (i = 1, 2, ……, N) is the i-th subtitle text feature, obtaining the video subtitle text feature U s = [s1, s2, ……, s N ∈ R 768×N×L , where N is the number of frames of the video, L is the number of characters in the sentence. Since BERT belongs to word embedding, and at the same time to obtain the context information of the text, use a BiLSTM of the same dimension to encode the subtitle text feature h i s = BiLSTM(s i ), obtaining its encoded subtitle text feature U s = [h1 s , h2 s , ……, h N s ∈ R 768×N×L , where h i s is the feature vector after encoding the i-th subtitle text feature, N is the number of frames of the video, L is the number of sentences, s represents the video subtitle text feature, and finally use (conv1-ReLu-maxpool) to obtain its final text feature (text feature information) U s = [h1 s , h2s ,..., h N s ∈ R 1024×N .

[0059] In the technical solution of this embodiment, by using the context information acquisition network to perform context encoding on the text feature information to be processed, the dynamic feature information to be processed, and the static feature information to be processed respectively, the context information of the text feature information to be processed, the dynamic feature information to be processed, and the static feature information to be processed is obtained, which is conducive to obtaining more accurate text feature information, dynamic feature information, and static feature information, and improving the accuracy of video understanding.

[0060] In one embodiment, obtaining the understanding result of the video to be understood based on the text feature information, dynamic feature information, and static feature information in the above step S103 specifically includes: performing guided attention fusion on the text feature information, dynamic feature information, and static feature information to obtain multiple modal representation information; using a graph attention network to obtain the aggregated feature information of the multiple modal representation information; and obtaining the understanding result of the video to be understood according to the aggregated feature information.

[0061] In this embodiment, the guided attention fusion may be to perform attention weight analysis on the text feature information, dynamic feature information, and static feature information; the modal representation information may be the feature vectors most relevant to the problem selected after performing attention weight analysis; the graph attention network may be the graph attention network GAT; and the aggregated feature information may be the aggregated feature information of different modalities obtained through the training of the graph attention network.

[0062] Specifically, the terminal performs guided attention fusion on the text feature information, dynamic feature information, and static feature information to obtain multiple modal representation information, uses a graph attention network to obtain the aggregated feature information of the multiple modal representation information, and obtains the understanding result of the video to be understood according to the aggregated feature information.

[0063] Exemplarily, as Figure 2 shown, in order to obtain relevant modal features, three guided attention modules are designed. Since the soft attention mechanism has selectivity and differentiability, the soft attention mechanism is used to design the guided attention module, and at the same time, the correlation matrix is used for fusion, and the fused features are concatenated. ① The first guided attention module a2m-Attention, with N as the iteration condition, the terminal takes the static feature vector h1 a and the row vector h1 m in the video dynamic feature matrix as the input of the guided attention module a2m-Attention, where i represents the i-th understanding pair, and i represents the i-th row dynamic feature vector in the video dynamic feature matrix. The guided attention model is expressed as Soft_Attention(h ia , h i m ), and then learn the problem-guided attention weights for the video dynamic features, and generate features using the attention weights and and connect the generated features Then use the linear and tanh functions for dimensionality reduction, and its guided attention mechanism is as follows: T is the transpose of the vector, and they are jointly embedded in d ma ×d a dimensional space, h j e and m i e are the embedding representations of a i and h i m respectively. [,] is the concatenation operation. Attention is paid to the video dynamic features according to the probability after problem-guided attention to obtain the fused feature vector. To improve the attention weights of different modalities, attention operations are also performed on the static feature representations, and the subsequent feature vectors are concatenated. The differentiability of the guided attention mechanism makes it learnable. Finally, linear and non-linear transformations are performed to obtain the final feature vector, and attention is performed sequentially with N as the iteration condition to obtain the video dynamic feature matrix after attention (which can also be N×250), and the maximum pooling is used to reduce the dimensionality of each row of the video dynamic feature matrix to obtain its final feature matrix U m = [u1 m , u2 m , ……, u N m ∈ R 512×N , where N is the number of frames of the video. ② The second guided attention module also uses the same soft attention mechanism to calculate the attention for the subtitle text features. The second guided attention module s2a-Attention, with N as the iteration condition, takes the problem feature vector h i a and the row vector h i s in the video subtitle text feature matrix as the input of the guided attention module a2s-Attention. Here, i represents the i-th understanding pair, and i represents the i-th subtitle text feature vector in the subtitle text feature matrix. The guided attention model is expressed as Soft_Attention(h i a , h i s ), and then learn the problem-guided attention weights for the subtitle text features, and generate features using the attention weights and and connect the generated features Then use the linear and tanh functions for dimensionality reduction to obtain the final feature vector, and perform attention sequentially with N as the iteration condition to obtain the subtitle text feature matrix after attention. Then use max pooling to reduce the dimensionality of each row of the subtitle text feature matrix to obtain its final feature matrix U s =[u1 s , u2 s , ……, u N s ∈ R 512×N , where N is the number of frames of the video. ③ The third guided attention module also uses the same soft attention mechanism to calculate the attention for the static features of the video. The third guided attention module m2a-Attention, with N as the iteration condition, takes the problem feature vector h i m and the row vector h i a in the video static feature matrix as the input of the guided attention module m2a-Attention, where i represents the i-th understanding pair, and i represents the i-th row static feature vector in the static feature matrix. The guided attention model is expressed as Soft_Attention(h i m , h i a ). Then, learn the problem-guided attention weights for the video static features, generate features and and concatenate the generated features Then use the linear and tanh functions for dimensionality reduction to obtain the final feature vector, and perform attention sequentially with N as the iteration condition to obtain the video static feature matrix after attention. Then use max pooling to reduce the dimensionality of each row of the video static feature matrix to obtain its final feature matrix U a =[u1 a , u2 a , ……, u N a ∈ R 512×N , where N is the number of frames of the video. After passing through the three attention modules, use a 150-dimensional BiLSTM to encode the row vectors of the different modality feature matrices with N as the iteration condition, and obtain the different modality feature matrices U a =[u1 a , u2 a , ……, u N a ∈ R 150×N , U m =[u1 m , u2 m , ……, uN m ∈ R 150×N , U s = [u1 s , u2 s , ……, u N s ∈ R 150×N (equivalent to multiple modal representation information).

[0064] Next, as Figure 2 shown, the terminal inputs the multiple modal representation information obtained above into the feature enhancement memory module. Multimodality can improve the prior of video understanding, and the semantic relationships between different modalities can significantly improve the reasoning ability of the video understanding model. Through simple operations such as simple vector addition and multiplication, the semantic relationships between different modalities will be weakened. In order to accurately model the semantic relationships between different modalities, a graph attention network (GAT) is used to model the semantic relationships between different modalities, and the aggregated features of different modalities are obtained through the training of the graph attention network. At the same time, in order to model the context temporal relationship, a BiLSTM (temporal memory update network) is used to model the temporal relationship in the video. In order to establish the connection between modal information, first define an undirected fully connected graph G i = {V i , E attention}, where is the node set of the graph attention network at the i-th iteration, where type represents the corresponding node category, representing its video dynamic feature, video static feature, and video caption text feature respectively, node represents the number of nodes, and there are three vertices v i a , v i m , v i s in each iteration. The content of each node is the feature vector of different modalities, which are u i a , u i c , u i m , corresponding to the row vectors of the different modality feature matrices at the i-th iteration. E attention is the edge set between nodes in the graph attention network, represented as the attention cross-correlation coefficient between different nodes. By constructing a modal object graph and using the graph attention layer, the mutually enhanced features between different modalities can be obtained. A two-layer graph attention layer can be constructed, and the formula for each layer is where is the feature after the attention of the z-th layer, is the adjacency matrix. Since a fully connected undirected graph is constructed, the adjacency matrix is initialized as the corresponding fully connected undirected graph before each graph training. During the training process, the attention cross-correlation coefficient is used to update the adjacency matrix to form the attention coefficient matrix. The formula for updating its adjacency matrix is is the learnable weight matrix, α is the cross-correlation coefficient matrix, and its calculation process is as follows: First, pass the input feature vector through the self-attention mechanism sa: to obtain its shareable weight matrix: Then use the LeakyReLU activation function for non-linear calculation, and finally use softmax for regularization. Finally, calculate the cross-correlation coefficient matrix, and its calculation formula is where softmax is performed in each row, and the calculation formula (pointwise summation operation) for each node passing through self-attention and multi-head attention is

[0065]

[0066] where K is the number of multi-head attentions, which can be set to 3, σ represents the sigmod activation function, r represents the neighbor nodes of node h, and the graph attention layer of the second layer can be used as the output, defined as After the graph attention network performs self-attention and multi-head attention calculations on different modal features, the graph embedding representation is obtained Then the graph embedding is dimensionally reduced through 1D convolution, and then max pooling is used to obtain the most relevant feature o i ∈R 1×150 , and at the same time, in order to perform context relevance analysis and temporal memory on different modal features of the video, so that the model has the ability of multi-step reasoning, a bidirectional temporal memory network BiLSTM (temporal memory update network, also known as bidirectional recurrent neural network) is used as the memory update unit in the model. By updating the memory of multi-modal features at different times, the BiLSTM of the same dimension is used to encode oi, and the temporal relationship of each video can be iterated N times. Only the final output of BiLSTM is selected, that is, when t = N, the output h of BiLSTM is obtained again j = BiLSTM(o i ) t＝N , and finally, the final modal vector is obtained after passing through the fully connected layer where j is the number of each understanding pair, and finally each candidate answer is summarized to obtain the final modal feature Make a prediction. Then, the terminal can define the video understanding task as a generation understanding task. First, perform a concatenation operation on the output after fusion inference, then use a fully connected layer (FC) for linear and non-linear transformations, and finally use the cross-entropy loss function for training. Specifically, the terminal can be defined as a generation understanding task. First, use softmax to convert the features into prediction scores and use the cross-entropy loss function for optimization

[0067]

[0068] where y GT represents the GT ground truth, k represents the k-th prediction, and N represents the sample. Finally, the terminal uses the prediction module to select the maximum prediction score from the final prediction scores as the final result. The Softmax normalization layer can be used, and through the calculation formula: Pro = softmax(P q ), the prediction scores are obtained, and finally the one with the highest score is selected as the final prediction, using y = max(Pro). The terminal can also use this video understanding model to realize intelligent early warning and real-time detection of the current specific event system according to the understanding result of the video to be understood.

[0069] The technical solution of this embodiment obtains the aggregated feature information of multiple modal representation information obtained after guided attention fusion by using the graph attention network, thereby obtaining the understanding result of the video to be understood, which is beneficial to improving the accuracy of video understanding.

[0070] In one embodiment, the above method can also obtain the understanding result of the video to be understood through the following steps, specifically including: using the temporal memory update network to perform temporal memory update on the aggregated feature information; obtaining the understanding result of the video to be understood according to the aggregated feature information after temporal memory update.

[0071] In this embodiment, as Figure 2 shown, the temporal memory update network can be a bidirectional temporal memory network BiLSTM (also known as a bidirectional recurrent neural network).

[0072] Specifically, the terminal uses the temporal memory update network to perform temporal memory update on the aggregated feature information, and obtains the understanding result of the video to be understood according to the aggregated feature information after temporal memory update.

[0073] The technical solution of this embodiment is beneficial to improving the accuracy of video understanding by obtaining the understanding result of the video to be understood according to the aggregated feature information after temporal memory update.

[0074] In one embodiment, the above method can also train the text feature acquisition network, the dynamic feature acquisition network, the static feature acquisition network, the graph attention network, and the temporal memory update network through the following steps, specifically including: obtaining video samples carrying text information and the corresponding understanding result labels of the video samples; using the video samples and the understanding result labels to train the text feature acquisition network, the dynamic feature acquisition network, the static feature acquisition network, the graph attention network, and the temporal memory update network.

[0075] In this embodiment, as Figure 3 shown, the video samples carrying text information can be video samples carrying subtitle description information summarized by humans for the video to be understood; the understanding result labels corresponding to the video samples can be the true understanding result labels corresponding to the video samples, such as the labels of what specific event type the video sample belongs to in a preset manner.

[0076] Specifically, the terminal obtains video samples carrying text information and the corresponding understanding result labels of the video samples, and uses the video samples and the understanding result labels to train the text feature acquisition network, the dynamic feature acquisition network, the static feature acquisition network, the graph attention network, and the temporal memory update network.

[0077] Exemplarily, as Figure 3 and 4 shown, a large number of videos in a specific event system can be used to construct a dataset for related video understanding, and a specific event video understanding model with a mutually guided graph attention network can be designed to achieve the purpose of intelligent early warning and real-time detection of the current specific event system. Specifically, S1: The input is a large number of specific event videos in the specific event system, and the output is a video understanding dataset; among them, as Figure 3As shown in the figure, the terminal first uses a large number of existing specific event videos to construct a video understanding related dataset JQYJ-video data, selects 400 videos as the original video data, each video is 1 minute long, and each video is split into 10-second video segments (that is, each video is split into 6 video segments). Each video segment is summarized by specific event staff to obtain a subtitle description information (i.e., text information), resulting in 2400 (i.e., 400×6) specific event video segments and subtitle description files. The corresponding professionals classify the specific event videos into five preset specific event types (i.e., each video sample has a corresponding understanding result label). Until there are 2400 video samples for each of the preset specific event types, 12000 (i.e., 2400×5) video samples are obtained (each video sample already carries the corresponding text information). The 12000 video samples can be divided into a training set (which can contain 9000 video samples), a validation set (which can contain 1000 video samples), and a test set (which can contain 2000 video samples). Then the terminal obtains the above 12000 video samples carrying text information and the understanding result labels corresponding to the video samples. S2. The input is the video understanding dataset formed in S1, and the output is a pre-trained model (i.e., a video understanding model); among them, as Figure 2As shown in the figure, the pre-trained model may include three modules: a modality feature extraction module with mutual guidance attention (which can be simply referred to as a mutual guidance feature extraction module and can be divided into a modality feature extraction module and a mutual guidance attention module), a feature enhancement memory module (which can be simply referred to as a feature enhancement module), and a prediction module. The main function of the modality feature extraction module is to extract multi-modal features. That is, the trained BERT network model (text feature acquisition network) is used to perform word embedding on the caption text information, and BiLSTM (context information acquisition network) is used for context encoding and generating features. The C3D (dynamic feature acquisition network) is used to extract video dynamic features, and BiLSTM (context information acquisition network) is used for context encoding and generating features. The VGG16 network model (static feature acquisition network) is used to extract the static features of the video, and BiLSTM (context information acquisition network) is used for context encoding and generating features. The main function of the mutual guidance attention module is to obtain the most relevant feature information. The main function of the feature enhancement memory module is to use the graph attention network (GAT) to model the inference mechanism of the video understanding task, that is, to model the semantic relationship of different modality features using the graph attention network, and use BiLSTM (temporal memory update network) for temporal memory update. The main function of the prediction module is to perform answer prediction. The terminal inputs the video sample and the understanding result label obtained in S1 into the pre-trained model to train the pre-trained model, that is, to train the text feature acquisition network, dynamic feature acquisition network, static feature acquisition network, graph attention network, and temporal memory update network in the pre-trained model. S3, the input is a specific event video and the pre-trained model in S2, and the output is the detection data of the pre-trained model for video understanding. Among them, the terminal obtains the video to be understood (such as a specific event video) and inputs it into the pre-trained model, and outputs the understanding result of the video to be understood (such as the detection data of the pre-trained model for video understanding).

[0078] The technical solution of this embodiment is to train the text feature acquisition network, dynamic feature acquisition network, static feature acquisition network, graph attention network, and temporal memory update network by using video samples and understanding result labels, which is beneficial to improving the accuracy of each network model, and thus beneficial to improving the accuracy of video understanding.

[0079] In one embodiment, the specific steps of using the text feature acquisition network to obtain the text feature information of the video to be understood in step S102 specifically include: detecting whether the video to be understood carries corresponding text information; if not, using the text information generation model to generate corresponding text information according to the video to be understood; using the text feature acquisition network to obtain the text feature information of the video to be understood according to the corresponding text information.

[0080] In this embodiment, the text information generation model may be an MDVC model (Multi-modal Dense VideoCaptioning).

[0081] Specifically, the terminal detects whether the video to be understood carries corresponding text information. If not, the text information generation model is used to generate corresponding text information according to the video to be understood, and the text feature acquisition network is used to obtain the text feature information of the video to be understood according to the corresponding text information.

[0082] Exemplarily, the terminal detects whether the video to be understood carries corresponding text information. If not, the text information generation model (MDVC model) is used to generate the dense caption information of the video to be understood, and the text feature acquisition network is used to extract the text feature information of the dense caption information.

[0083] The technical solution of this embodiment is beneficial to obtaining the text feature information of the video to be understood even when the video to be understood does not carry corresponding text information, thereby improving the accuracy of video understanding by using the text information generation model to generate corresponding text information according to the video to be understood and using the text feature acquisition network to obtain the text feature information of the video to be understood according to the corresponding text information.

[0084] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are displayed in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps in other steps.

[0085] Based on the same inventive concept, an embodiment of the present application further provides a video understanding device for implementing the above-mentioned video understanding method. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the video understanding device provided below can be referred to the limitations on the video understanding method in the above text, and will not be repeated here.

[0086] In one embodiment, as Figure 5 shown, a video understanding device is provided. The device 500 may include:

[0087] The video to be understood acquisition module 501 is used to acquire the video to be understood;

[0088] The feature information acquisition module 502 is used to respectively acquire the text feature information, dynamic feature information, and static feature information of the video to be understood by using a text feature acquisition network, a dynamic feature acquisition network, and a static feature acquisition network;

[0089] The understanding result acquisition module 503 is used to acquire the understanding result of the video to be understood based on the text feature information, dynamic feature information, and static feature information.

[0090] In one embodiment, the understanding result acquisition module 503 is further used to perform guided attention fusion on the text feature information, dynamic feature information, and static feature information to obtain multiple modal representation information; use a graph attention network to obtain the aggregated feature information of the multiple modal representation information; and obtain the understanding result of the video to be understood according to the aggregated feature information.

[0091] In one embodiment, the understanding result acquisition module 503 is further used to perform temporal memory update on the aggregated feature information by using a temporal memory update network; and obtain the understanding result of the video to be understood according to the aggregated feature information after temporal memory update.

[0092] In one embodiment, the understanding result acquisition module 503 is further used to acquire a video sample carrying text information and the understanding result label corresponding to the video sample; and use the video sample and the understanding result label to train the text feature acquisition network, the dynamic feature acquisition network, the static feature acquisition network, the graph attention network, and the temporal memory update network.

[0093] In one embodiment, the video to be understood acquisition module 501 is further used to detect whether the video to be understood carries corresponding text information; if not, use a text information generation model to generate corresponding text information according to the video to be understood; and use the text feature acquisition network to acquire the text feature information of the video to be understood according to the corresponding text information.

[0094] In one embodiment, the video to be understood acquisition module 501 is further used to respectively acquire the text feature information to be processed, the dynamic feature information to be processed, and the static feature information to be processed of the video to be understood by using the text feature acquisition network, the dynamic feature acquisition network, and the static feature acquisition network; and use a context information acquisition network to respectively perform context encoding on the text feature information to be processed, the dynamic feature information to be processed, and the static feature information to be processed to obtain the text feature information, the dynamic feature information, and the static feature information.

[0095] Each module in the above video understanding device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in the processor of a computer device in hardware form or be independent of the processor, or can be stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0096] In one embodiment, a computer device is provided. The computer device can be a terminal, and its internal structure diagram can be as Figure 6 shown. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a video understanding method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0097] Those skilled in the art can understand that Figure 6 the structure shown in

[0098] is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0098] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented:

[0099] Obtain the video to be understood;

[0100] Utilize a text feature acquisition network, a dynamic feature acquisition network, and a static feature acquisition network to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood;

[0101] Based on the text feature information, dynamic feature information, and static feature information, obtain the understanding result of the video to be understood.

[0102] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0103] Perform guided attention fusion on the text feature information, dynamic feature information, and static feature information to obtain multiple modal representation information;

[0104] Use the graph attention network to obtain the aggregated feature information of the multiple modal representation information;

[0105] According to the aggregated feature information, obtain the understanding result of the video to be understood.

[0106] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0107] Use the temporal memory update network to perform temporal memory update on the aggregated feature information;

[0108] According to the aggregated feature information after temporal memory update, obtain the understanding result of the video to be understood.

[0109] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0110] Obtain a video sample carrying text information and the corresponding understanding result label of the video sample;

[0111] Use the video sample and the understanding result label to train the text feature acquisition network, dynamic feature acquisition network, static feature acquisition network, graph attention network, and temporal memory update network.

[0112] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0113] Detect whether the video to be understood carries the corresponding text information;

[0114] If not, use the text information generation model to generate the corresponding text information according to the video to be understood;

[0115] Use the text feature acquisition network to obtain the text feature information of the video to be understood according to the corresponding text information.

[0116] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0117] Use the text feature acquisition network, dynamic feature acquisition network, and static feature acquisition network to respectively obtain the text feature information to be processed, dynamic feature information to be processed, and static feature information to be processed of the video to be understood;

[0118] The context information acquisition network performs context encoding on the text feature information to be processed, the dynamic feature information to be processed, and the static feature information to be processed, and obtains the text feature information, the dynamic feature information, and the static feature information.

[0119] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0120] Obtain the video to be understood;

[0121] Using the text feature acquisition network, the dynamic feature acquisition network, and the static feature acquisition network, respectively obtain the text feature information, the dynamic feature information, and the static feature information of the video to be understood;

[0122] Based on the text feature information, the dynamic feature information, and the static feature information, obtain the understanding result of the video to be understood.

[0123] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0124] Perform guided attention fusion on the text feature information, the dynamic feature information, and the static feature information to obtain multiple modal representation information;

[0125] Use the graph attention network to obtain the aggregated feature information of the multiple modal representation information;

[0126] According to the aggregated feature information, obtain the understanding result of the video to be understood.

[0127] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0128] Use the temporal memory update network to perform temporal memory update on the aggregated feature information;

[0129] According to the aggregated feature information after temporal memory update, obtain the understanding result of the video to be understood.

[0130] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0131] Obtain the video sample carrying text information and the understanding result label corresponding to the video sample;

[0132] Use the video sample and the understanding result label to train the text feature acquisition network, the dynamic feature acquisition network, the static feature acquisition network, the graph attention network, and the temporal memory update network.

[0133] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0134] Detect whether the video to be understood carries corresponding text information;

[0135] If not, use the text information generation model to generate corresponding text information according to the video to be understood;

[0136] Use the text feature acquisition network to obtain the text feature information of the video to be understood according to the corresponding text information.

[0137] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0138] Use the text feature acquisition network, the dynamic feature acquisition network, and the static feature acquisition network to respectively obtain the text feature information to be processed, the dynamic feature information to be processed, and the static feature information to be processed of the video to be understood;

[0139] Use the context information acquisition network to perform context encoding on the text feature information to be processed, the dynamic feature information to be processed, and the static feature information to be processed respectively, and obtain the text feature information, the dynamic feature information, and the static feature information.

[0140] In one embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0141] Obtain the video to be understood;

[0142] Use the text feature acquisition network, the dynamic feature acquisition network, and the static feature acquisition network to respectively obtain the text feature information, the dynamic feature information, and the static feature information of the video to be understood;

[0143] Based on the text feature information, the dynamic feature information, and the static feature information, obtain the understanding result of the video to be understood.

[0144] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0145] Perform guided attention fusion on the text feature information, the dynamic feature information, and the static feature information to obtain multiple modal representation information;

[0146] Use the graph attention network to obtain the aggregated feature information of the multiple modal representation information;

[0147] According to the aggregated feature information, obtain the understanding result of the video to be understood.

[0148] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0149] Use the temporal memory update network to perform temporal memory update on the aggregated feature information;

[0150] Obtain the understanding result of the video to be understood according to the aggregated feature information updated by temporal memory.

[0151] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0152] Obtain video samples carrying text information and the understanding result labels corresponding to the video samples;

[0153] Use the video samples and understanding result labels to train the text feature acquisition network, dynamic feature acquisition network, static feature acquisition network, graph attention network, and temporal memory update network.

[0154] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0155] Detect whether the video to be understood carries corresponding text information;

[0156] If not, use the text information generation model to generate corresponding text information according to the video to be understood;

[0157] Use the text feature acquisition network to obtain the text feature information of the video to be understood according to the corresponding text information.

[0158] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0159] Use the text feature acquisition network, dynamic feature acquisition network, and static feature acquisition network to respectively obtain the text feature information to be processed, dynamic feature information to be processed, and static feature information to be processed of the video to be understood;

[0160] Use the context information acquisition network to perform context encoding on the text feature information to be processed, dynamic feature information to be processed, and static feature information to be processed respectively, to obtain text feature information, dynamic feature information, and static feature information.

[0161] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties.

[0162] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0163] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0164] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A video understanding method, characterized in that, The method includes: Obtain the video to be understood; Use a text feature acquisition network, a dynamic feature acquisition network, and a static feature acquisition network to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood; the text feature information is the feature information obtained after the text feature acquisition network performs word embedding on the subtitle text information of the video to be understood; the subtitle text information is the subtitle description information manually summarized from the video to be understood, or the corresponding text information generated by a text information generation model according to the video to be understood; Adopt a soft attention mechanism to perform guided attention fusion on the text feature information, the dynamic feature information, and the static feature information, and perform max-pooling dimensionality reduction processing on the feature information after guided attention fusion to obtain multiple modal representation information; the guided attention fusion is to perform attention weight analysis on the text feature information, the dynamic feature information, and the static feature information; the modal representation information is the feature vector most relevant to the question selected after the attention weight analysis; Use a graph attention network to obtain the aggregated feature information of the multiple modal representation information; Use a temporal memory update network to perform temporal memory update on the aggregated feature information; the temporal memory update network includes a bidirectional temporal memory network; According to the aggregated feature information after temporal memory update, obtain the understanding result of the video to be understood.

2. The method according to claim 1, wherein The video to be understood is a real-time specific event video.

3. The method according to claim 1, characterized in that, The text information generation model is an MDVC model.

4. The method according to claim 1, characterized in that, The method further includes: Obtain a video sample carrying text information and the understanding result label corresponding to the video sample; Use the video sample and the understanding result label to train the text feature acquisition network, the dynamic feature acquisition network, the static feature acquisition network, the graph attention network, and the temporal memory update network.

5. The method according to claim 1, wherein Using the text feature acquisition network to obtain the text feature information of the video to be understood includes: Detect whether the video to be understood carries corresponding text information; If not, use the text information generation model to generate corresponding text information according to the video to be understood; Use the text feature acquisition network to obtain the text feature information of the video to be understood according to the corresponding text information.

6. The method according to claim 1, wherein The step of using the text feature acquisition network, the dynamic feature acquisition network, and the static feature acquisition network to respectively obtain the text feature information, dynamic feature information, and static feature information of the video to be understood includes: Use the text feature acquisition network, the dynamic feature acquisition network, and the static feature acquisition network to respectively obtain the to-be-processed text feature information, to-be-processed dynamic feature information, and to-be-processed static feature information of the video to be understood; Use a context information acquisition network to respectively perform context encoding on the to-be-processed text feature information, to-be-processed dynamic feature information, and to-be-processed static feature information to obtain the text feature information, dynamic feature information, and static feature information.

7. A video understanding device, characterized in that, The device includes: A video to be understood acquisition module, configured to obtain a video to be understood; A feature information acquisition module, which is used to respectively acquire the text feature information, dynamic feature information, and static feature information of the video to be understood by using a text feature acquisition network, a dynamic feature acquisition network, and a static feature acquisition network; the text feature information is the feature information obtained after the text feature acquisition network performs word embedding on the subtitle text information of the video to be understood; the subtitle text information is the subtitle description information manually summarized from the video to be understood, or the corresponding text information generated by using a text information generation model according to the video to be understood. An understanding result acquisition module, which is used to perform guided attention fusion on the text feature information, the dynamic feature information, and the static feature information by adopting a soft attention mechanism, perform max-pooling dimensionality reduction processing on the feature information after the guided attention fusion to obtain multiple modal representation information; the guided attention fusion is to perform attention weight analysis on the text feature information, the dynamic feature information, and the static feature information; the modal representation information is the feature vector most relevant to the question selected after the attention weight analysis; use a graph attention network to obtain the aggregated feature information of the multiple modal representation information; use a temporal memory update network to perform temporal memory update on the aggregated feature information; the temporal memory update network includes a bidirectional temporal memory network; according to the aggregated feature information after the temporal memory update, obtain the understanding result of the video to be understood.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

Images