Video description method and system based on multimodal interaction and temporal semantic aggregation

Abstract

The present invention discloses a video description method and system based on multimodal interaction and temporal semantic aggregation. The method comprises: obtaining a video to be described; extracting multimodal features from the video to be described; the multimodal features comprise appearance features, motion features and object features; extracting semantic features from the multimodal features; inputting the extracted multimodal features and semantic features into a trained video description network, and outputting a natural language description result of the video; the trained video description network is a video description network based on multimodal information interaction and temporal semantic aggregation; the working principle of the trained video description network comprises: using a multimodal interaction module to integrate motion features and object features into appearance features respectively to form multimodal interaction features; using a hierarchical decoder with temporal semantic aggregation to process semantic features and multimodal interaction features to generate a video description language.

Description

Technical Field

[0001] The present invention relates to the field of video description technology, and in particular to a video description method and system based on multimodal interaction and temporal semantic aggregation. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] Video captioning, which involves automatically describing video content using natural language, has attracted increasing attention from computer vision researchers due to its wide range of applications. However, the complex scenes and object interactions in videos make this task challenging. These challenges include effectively representing video content using multimodal features and generating more detailed descriptions using existing visual and linguistic features. Video captioning methods are typically based on an encoder-decoder framework. The encoder encodes video content using convolutional neural networks, typically extracting appearance features using 2D CNNs, motion features using 3D CNNs, and object features using R-CNNs. These encoders capture diverse video content from different angles, forming multimodal input information. The more video content the input contains, the more accurate the generated sentences. In "Reconstruction network for video captioning," Wang et al. primarily used appearance features to extract salient regions from each frame. In "Controllable video captioning with pos sequence guidance based on gated fusion network," Wang et al. proposed a cross-gating module to interact with motion and appearance features. However, these methods only utilize one modality of information input or only interact with two of the information to form input information, and cannot include more video content.

[0004] The decoder, which includes a recurrent neural network (RNN) and a long short-term memory network (LSTM), decodes the input information to generate a sentence. Previous methods, on the one hand, only utilize the temporal information of visual features and ignore the temporal information of linguistic features. In fact, the generation of the current word is closely related to previously generated words. On the other hand, when using semantic information to guide feature aggregation, previous methods overly rely on the quality of the semantic features generated by the semantic generator. Summary of the Invention

[0005] In order to solve the problems of insufficient feature interaction and inaccurate focus on visual and language features in existing video description methods, the present invention provides a video description method and system based on multimodal interaction and temporal semantic aggregation; multimodal features are extracted from the video, semantic attributes are extracted using features, multimodal features are interactively fused, and finally a description sentence is generated through a hierarchical decoder with temporal semantic aggregation.

[0006] In a first aspect, the present invention provides a video description method based on multimodal interaction and temporal semantic aggregation;

[0007] Video description methods based on multimodal interaction and temporal semantic aggregation include:

[0008] Get the video to be described;

[0009] Extracting multimodal features from the video to be described; the multimodal features include: appearance features, motion features, and object features;

[0010] Extracting semantic features from multimodal features;

[0011] The extracted multimodal features and semantic features are input into a trained video description network to output a natural language description result of the video; the trained video description network is a video description network based on multimodal information interaction and temporal semantic aggregation;

[0012] Among them, the working principles of the trained video description network include: using a multimodal interaction module to integrate motion features and object features into appearance features respectively to form multimodal interaction features; using a hierarchical decoder with temporal semantic aggregation to process semantic features and multimodal interaction features to generate video description language.

[0013] In a second aspect, the present invention provides a video description system based on multimodal interaction and temporal semantic aggregation;

[0014] The video description system based on multimodal interaction and temporal semantic aggregation includes:

[0015] An acquisition module is configured to: acquire a video to be described;

[0016] A multimodal feature extraction module is configured to extract multimodal features from the video to be described; the multimodal features include appearance features, motion features, and object features;

[0017] A semantic feature extraction module, configured to: extract semantic features from multimodal features;

[0018] The video description module is configured to: input the extracted multimodal features and semantic features into a trained video description network, and output a natural language description result of the video; the trained video description network is a video description network based on multimodal information interaction and temporal semantic aggregation;

[0019] Among them, the working principles of the trained video description network include: using a multimodal interaction module to integrate motion features and object features into appearance features respectively to form multimodal interaction features; using a hierarchical decoder with temporal semantic aggregation to process semantic features and multimodal interaction features to generate video description language.

[0020] In a third aspect, the present invention further provides an electronic device, comprising:

[0021] a memory for non-transitory storage of computer-readable instructions; and

[0022] a processor for executing said computer-readable instructions,

[0023] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0024] In a fourth aspect, the present invention further provides a storage medium that non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0025] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention proposes a video description network MITSANet (Multimodal Interaction & Temporal Semantic Aggregation Networks for Video Captioning) based on multimodal information interaction and temporal semantic aggregation.

[0028] The present invention introduces an appearance-guided multimodal interaction module, which integrates object information and motion information into appearance features respectively to form a unified multimodal interaction feature.

[0029] This paper proposes a hierarchical decoder with temporal semantic aggregation, which dynamically aggregates visual and linguistic features when generating corresponding words. This network effectively improves the accuracy of generated sentences. Temporal semantic aggregation is a method of aggregating related visual and linguistic features. During decoding, it dynamically focuses on the most relevant temporal and semantic information in the visual and linguistic features, thereby generating more accurate sentences.

[0030] A multimodal feature interaction module is proposed to effectively fuse the relevant parts of different features so that the input features representing the video contain more video content.

[0031] The present invention proposes a temporal semantic aggregation module to effectively aggregate relevant visual features and language features, thereby improving the accuracy of word generation.

[0032] All the proposed methods constitute an effective video description approach, and experiments on two widely used datasets demonstrate the effectiveness of the proposed method. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0034] Figure 1 Flowchart of the video description method designed in embodiment 1 of the present invention;

[0035] FIG2( a ) shows the overall network structure of the MITSNet architecture according to the first embodiment of the present invention;

[0036] FIG2( b ) shows the multimodal interaction module structure of the MITSNet architecture according to the first embodiment of the present invention;

[0037] FIG2( c ) shows the semantic detection module structure of the MITSNet architecture according to the first embodiment of the present invention;

[0038] Figure 3(a) to Figure 3(d) This is a result diagram of the video description of Example 1 of the present invention. DETAILED DESCRIPTION

[0039] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0040] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0042] All data in this embodiment is obtained in compliance with laws and regulations and based on the consent of the user, and is used legally.

[0043] Example 1

[0044] This embodiment provides a video description method based on multimodal interaction and temporal semantic aggregation;

[0045] like Figure 1 As shown in FIG, the video description method based on multimodal interaction and temporal semantic aggregation includes:

[0046] S101: Obtain the video to be described;

[0047] S102: extracting multimodal features from the video to be described; the multimodal features include: appearance features, motion features, and object features;

[0048] S103: Extracting semantic features from multimodal features;

[0049] S104: Inputting the extracted multimodal features and semantic features into a trained video description network to output a natural language description result of the video; the trained video description network is a video description network based on multimodal information interaction and temporal semantic aggregation;

[0050] The working principle of the trained video description network includes the following:

[0051] A multimodal interaction module is used to integrate motion features and object features into appearance features to form multimodal interaction features.

[0052] A hierarchical decoder with temporal semantic aggregation is used to process semantic features and multimodal interaction features to generate video description language.

[0053] Furthermore, S102: extracting multimodal features from the video to be described, specifically including:

[0054] Through a two-dimensional convolutional neural network, the appearance features of the video to be described are extracted;

[0055] Extract motion features of the video to be described through a three-dimensional convolutional neural network;

[0056] The object features of the video to be described are extracted through regional convolutional neural networks.

[0057] For example, the video data is fed into pre-trained 2D convolutional neural networks (2D CNNs), 3D convolutional neural networks (3D CNNs), and regional convolutional neural networks (R-CNNs) to extract appearance features, motion features, and object features, respectively. The video is first pre-processed and uniformly sampled to 26 frames. InceptionResNetV2 is used as the 2D-CNNs to extract appearance features V a , I3D is used as 3D-CNNs to extract motion features V m , Fast-RCNN extracts object features V o , where 36 objects are extracted from each frame.

[0058] Furthermore, the step S103 of extracting semantic features from the multimodal features specifically includes:

[0059] The trained semantic detector is used to extract semantic features from multimodal features.

[0060] Furthermore, the trained semantic detector is implemented using a multi-layer perceptron.

[0061] Furthermore, as shown in FIG2(c), the trained semantic detector includes the following specific training processes:

[0062] Constructing a first training set and a first validation set; the first training set and the first validation set both contain video appearance features and motion features with known semantic labels;

[0063] The video appearance features and motion features corresponding to each semantic label are concatenated in series, and the concatenated results are input into the semantic detector for training. When the training reaches the set number of times, the training is stopped, and the detection accuracy of the semantic detector is verified using the first validation set. When the detection accuracy exceeds the set threshold, the training is stopped to obtain the trained semantic detector. Otherwise, the data of the first training set is replaced and the training is repeated.

[0064] For example, we first select k nouns, verbs, and adjectives with the highest frequency from the training set and the validation set as semantic labels, where k = 300. We then create a k-dimensional attribute tag s for each video.i =[s i1 ,s i2 ,...,s ik ]∈{0,1} k , if the video description contains the jth semantic tag, then s ij =1, otherwise s ij = 0. Then the appearance features and motion features of the training set are serially input into the semantic detector (multi-layer perceptron) for training, and the trained semantic detector is used to generate semantic features for the video test set.

[0065] Furthermore, as shown in FIG2( b ), the network structure of the trained video description network includes: interconnected multimodal interaction modules and a hierarchical decoder with temporal semantic aggregation.

[0066] Furthermore, the multimodal interaction module includes: a first LSTM network and a second LSTM network;

[0067] The input value of the first LSTM network is the appearance feature, and the output value of the first LSTM network is the high-level appearance feature;

[0068] The input value of the second LSTM network is the motion feature, and the output value of the second LSTM network is the high-level motion feature;

[0069] The high-level appearance features are sequentially processed through the fully connected layer and the mapping function layer to obtain a first mapping value;

[0070] The high-level motion features are sequentially processed through the fully connected layer and the mapping function layer to obtain a second mapping value;

[0071] Processing the first mapping value and the second mapping value through a multiplier to obtain a first correlation matrix;

[0072] Input the first correlation matrix into the softmax activation function layer to obtain the first normalized matrix;

[0073] Multiplying a first transposed result of the second mapping value by the first normalized matrix, transposing the multiplied result again to obtain a second transposed result, and summing the second transposed result with the high-level appearance feature to obtain an interactive feature of the appearance feature and the motion feature;

[0074] The object features are processed in sequence by the fully connected layer and the mapping function layer to obtain the third mapping value;

[0075] Processing the first mapping value and the third mapping value through a multiplier to obtain a second correlation matrix;

[0076] Input the second correlation matrix into the softmax activation function layer to obtain the second normalized matrix;

[0077] multiplying a third transposed result of the third mapping value by the second normalized matrix, transposing the multiplied result to obtain a fourth transposed result, and summing the fourth transposed result with the high-level appearance feature to obtain an interactive feature of the appearance feature and the object feature;

[0078] The interaction features between appearance features and motion features, and the interaction features between appearance features and object features are added together to obtain multimodal interaction features.

[0079] Furthermore, as shown in FIG2( a ), the network structure of the hierarchical decoder with temporal semantic aggregation includes:

[0080] The word embedding layer, attention LSTM network, visual time semantic aggregation layer, language LSTM network, language time semantic aggregation layer, and Softmax activation function layer are connected in sequence;

[0081] The input of the attention LSTM network is connected to the output of the average pooling layer. The input of the average pooling layer is used to input multimodal interaction features; the input of the visual time semantic aggregation layer is used to input multimodal interaction features and semantic features; the input of the language time semantic aggregation layer is used to input semantic features;

[0082] The output of the attention LSTM network is also connected to the input of the language LSTM network.

[0083] Furthermore, the word embedding layer maps words into multi-dimensional vectors;

[0084] Furthermore, the attention LSTM network is implemented by an LSTM network, and the attention LSTM network is used to generate a time-aggregated query value; the input value of the attention LSTM network is the average pooled value of the multimodal feature, the word embedding value and the output value of the language LSTM network at the previous moment.

[0085] Furthermore, the visual time semantic aggregation layer generates and aggregates the most relevant visual time features at the current moment and the most relevant semantic features; the visual time semantic aggregation layer is used to aggregate the multimodal interaction features that are most relevant to the currently generated word;

[0086] Furthermore, the language LSTM network is implemented by an LSTM network and is used to process the output value of the visual time semantic aggregation layer and the time aggregation query value to generate the language features of the current moment;

[0087] Furthermore, the language time semantic aggregation layer is used to process all language features generated at the current moment and before, so as to aggregate the language features most relevant to the currently generated word.

[0088] Furthermore, the Softmax activation function layer is used to output the word at the current moment.

[0089] To effectively utilize visual and linguistic information, this paper proposes a decoder with temporal semantic aggregation. The temporal semantic aggregation layer can focus on the most relevant information in visual and linguistic features when generating words, improving the accuracy of generated words.

[0090] Furthermore, the specific training process of the trained video description network includes:

[0091] Constructing a second training set and a second validation set; wherein the second training set and the second validation set both have video appearance features, motion features, object features, and semantic features of known video description sentences;

[0092] Input the second training set into the video description network and train the network. When the network loss function value no longer decreases, stop training to obtain the video description network after preliminary training.

[0093] The second validation set is input into the preliminarily trained video description network to verify the network. When the validation accuracy reaches the set threshold, the training is stopped and the current video description network is the trained video description network; otherwise, the second training set is replaced and training is continued.

[0094] Furthermore, the loss function adopts cross entropy. For the input video v, the model parameters are represented by θ, which is defined as follows:

[0095]

[0096] Where V represents the input video; θ represents the model parameters; y t represents the word distribution at the current time t; y 1:t-1, Represents the word distribution from the initial time to time t-1.

[0097] Use the backpropagation algorithm to update the network weights, obtain the optimal result and save it. Figure 3(a) to Figure 3(d) The final result of the present invention is shown.

[0098] Furthermore, the multimodal interaction module is used to integrate motion features and object features into appearance features to form multimodal interaction features, including:

[0099] Use a bidirectional LSTM network to capture the temporal information of appearance features and motion features:

[0100]

[0101]

[0102] Among them, V a and V m Indicates appearance features and motion features; h a and h m Represents the hidden layer state; and It is the high-level appearance features and motion features obtained by modeling temporal information through the LSTM network.

[0103] In order to model the interaction of different modal features in each frame image, a correlation matrix F is designed.

[0104] For the interaction between motion features and appearance features, the matrix F∈N*N represents the interaction between N-frame motion features and N-frame appearance features:

[0105]

[0106]

[0107]

[0108] Among them, tanh is the activation function; W m and W a is the learned embedding matrix; b m and b a is the deviation; α and β represent the mapping function; and are high-level appearance features and motion features; F is the correlation matrix; T represents the transpose. The mapping function aims to transform and Mapping to public space, computing and The similarities between the two.

[0109] Then the obtained correlation matrix F is normalized, and the motion information related to the appearance is integrated into the appearance features using the normalized matrix to obtain the interaction feature V between the appearance feature and the motion feature am :

[0110] F'=softmax(F,dim=1)

[0111]

[0112] Where F is the correlation matrix; F' is the normalized matrix; T represents the transpose; α represents the mapping function; and It is a high-end appearance feature and sports feature; V amis the interaction feature between appearance features and motion features. Similarly, for the interaction between appearance features and object features, first calculate the correlation matrix, then use the normalized matrix to integrate the object features into the appearance features, and obtain the interaction feature V between appearance features and object features. ao :

[0113]

[0114] Where, F' is the normalized matrix; γ represents the mapping function; It is a high-end appearance feature; V o is the object feature; T represents transposition; V ao It is the interactive feature of appearance features and object features.

[0115] Utilize element addition to fuse information and form a unified multimodal interaction feature:

[0116] V=V ao +V am

[0117] Among them, V ao is the interactive feature of appearance features and object features, V am It is the interactive feature of appearance features and motion features, V unified multimodal interactive features.

[0118] Furthermore, the hierarchical decoder with temporal semantic aggregation is used to process semantic features and multimodal interaction features to generate a video description language, including:

[0119] Attention LSTM network, the input value is the word embedding vector, video context and the output of the language LSTM network at the previous moment, and finally generates the time attention query value

[0120] Visual temporal semantic aggregation layer, using temporal attention query values and semantic features to fuse the most relevant visual features at the current moment;

[0121] First, use temporal attention to query the value Calculate the key time frame of the current moment, and then use the semantic features to obtain the visual time semantic aggregation features of the current moment:

[0122]

[0123]

[0124]

[0125] in, represents the learnable embedding matrix; is the temporal attention query value; V represents the multimodal interaction feature; att1 represents the attention distribution probability of each frame at the current moment; the temporal aggregation V' is a weighted combination based on att1 at each moment; s represents the semantic feature; Represents the learned weight of semantic information related to the current moment; V′ s Represents visual temporal semantic aggregation features;

[0126] Aggregated visual features and temporal attention query values Input language LSTM network to generate hidden state

[0127] Considering that the word generated at the current moment t is related to the word generated before, the normalized weights of all hidden states from the initial moment to the current moment are calculated:

[0128]

[0129]

[0130] in, represents the learnable embedding matrix; represents the temporal attention query value; h l is the hidden state from the initial moment to the current moment; att2 represents the attention distribution probability of the current moment to each moment; h' represents the hidden state after time aggregation.

[0131] Different from the direct fusion of visual semantic features, the most relevant semantic features are activated by activating the sigmoid gate, and then the relevant semantic features are fused with the language features to generate the language feature h′ that is currently being focused on. st , through the residual network, the relevant time features and semantic features are integrated to generate the final language features

[0132] s′=sigmoid(W g s⊙W s h′);

[0133]

[0134]

[0135] Among them, W g 、W s 、 represents the learnable embedding matrix; s represents the semantic feature; h' represents the hidden state after time aggregation; s' represents the semantic feature activated by the activation gate; h' st is the language feature h′ that is currently being focused on st ; Represents the generated language features.

[0136] Finally, the probability distribution at time t is obtained through a single-layer perceptron and softmax operation, where y t is the probability of a D-dimensional vocabulary:

[0137]

[0138] Among them, W y represents a learnable parameter; Indicates the generated language features; b y Indicates deviation; y t Represents the probability of word distribution in the D-dimensional vocabulary.

[0139] The effect of the present invention can be further illustrated by the following experiments:

[0140] The hardware conditions for all verification experiments of the present invention are: Ubuntu 16.04L, CPU: Intel Xeon 4114, RAM 250G, and three TITAN RTX graphics cards.

[0141] The software environment used in the experiment is: python 3.7.10, pytorch = 1.9.0, torchvision = 0.10.0

[0142] Experimental data 1 comes from the MSVD dataset, which consists of 1,970 short videos selected from YouTube. Each video shows an activity, including running, horseback riding, cycling, etc. The training set contains 1,200 videos, the validation set contains 100 videos, and 670 videos are used for testing.

[0143] Experimental Data 2 is the MSR-VTT dataset containing 10,000 video clips. These videos cover 20 categories, including movies, sports, music, etc., with 6,513 videos used for training, 497 videos for validation, and 2,990 videos for testing.

[0144] The present invention conducted video description experiments on both datasets. Table 1 compares the method designed by the present invention with previous algorithms. BLEU represents the n-garam accuracy between the generated sentence and the reference sentence. METEOR calculates the harmonic mean of the precision and recall between the generated sentence and the reference sentence. ROUGE-L calculates the recall of the longest common subsequence between the generated sentence and the reference sentence. CIDEr calculates the cosine similarity between the generated sentence and the reference sentence. The higher the indicators, the higher the quality of the generated sentence.

[0145] like Figure 3(a) to Figure 3(d)As shown in the figure, the video description model of the present invention can correctly describe the events in the video and include more detailed information about the objects, including long hair and blue shirt. This shows that the model of the present invention can better utilize the video content to generate more detailed sentences.

[0146] Table 1 Quantitative evaluation results of experimental data

[0147]

[0148] Video captioning is an important visual task that has been extensively studied in computer science. With the rapid growth of video data, the ability to "translate" videos into natural language is becoming increasingly important, accelerating video data query and search. Existing video captioning methods still face several challenges, including insufficient utilization of multimodal video input features, over-reliance on semantic detectors, and failure to consider the temporal correlation of linguistic features. To address this, we propose a video captioning network (MITSANet) based on multimodal information interaction and temporal semantic aggregation. We introduce an appearance-guided multimodal interaction module that integrates object and motion information into appearance features, forming a unified multimodal interaction feature. Secondly, we propose a hierarchical decoder with temporal semantic aggregation that dynamically aggregates visual and linguistic features when generating corresponding words. Finally, based on the proposed method, we design a video captioning network (MITSANet) based on multimodal interaction and temporal semantic aggregation to automatically generate text descriptions for input video clips.

[0149] Example 2

[0150] This embodiment provides a video description system based on multimodal interaction and temporal semantic aggregation;

[0151] The video description system based on multimodal interaction and temporal semantic aggregation includes:

[0152] An acquisition module is configured to: acquire a video to be described;

[0153] A multimodal feature extraction module is configured to extract multimodal features from the video to be described; the multimodal features include appearance features, motion features, and object features;

[0154] A semantic feature extraction module, configured to: extract semantic features from multimodal features;

[0155] The video description module is configured to: input the extracted multimodal features and semantic features into a trained video description network, and output a natural language description result of the video; the trained video description network is a video description network based on multimodal information interaction and temporal semantic aggregation;

[0156] Among them, the working principles of the trained video description network include: using a multimodal interaction module to integrate motion features and object features into appearance features respectively to form multimodal interaction features; using a hierarchical decoder with temporal semantic aggregation to process semantic features and multimodal interaction features to generate video description language.

[0157] It should be noted that the acquisition module, multimodal feature extraction module, semantic feature extraction module, and video description module described above correspond to steps S101 to S104 in Example 1. The examples and application scenarios implemented by these modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system, such as a set of computer-executable instructions.

[0158] The descriptions of the various embodiments in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0159] The proposed system can be implemented in other ways. For example, the system embodiment described above is merely illustrative. For example, the above module division is only a logical function division. In actual implementation, other division methods may be used. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not implemented.

[0160] Example 3

[0161] This embodiment also provides an electronic device, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method described in the above embodiment one.

[0162] It should be understood that in this embodiment, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0163] The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0164] During implementation, each step of the above method may be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software.

[0165] The method in Example 1 can be directly implemented as being executed by a hardware processor, or by a combination of hardware and software modules within the processor. The software module can be located in a storage medium well-established in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not given here.

[0166] Those skilled in the art will appreciate that the units and algorithm steps of the various examples described in conjunction with this embodiment can be implemented using electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0167] Example 4

[0168] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is performed.

[0169] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A video description method based on multimodal interaction and temporal semantic aggregation, characterized by: include: Get the video to be described; Extract multimodal features from the video to be described; The multimodal features include: appearance features, motion features and object features; Extracting semantic features from multimodal features; The extracted multimodal features and semantic features are input into a trained video description network to output a natural language description result of the video; the trained video description network is a video description network based on multimodal information interaction and temporal semantic aggregation; The trained video description network has a network structure including: Interconnected multimodal interaction modules and hierarchical decoders with temporal semantic aggregation; The multimodal interaction module adds the interaction features of the appearance features and the motion features and the interaction features of the appearance features and the object features to obtain the multimodal interaction features; The trained video description network works by: using a multimodal interaction module to integrate motion features and object features into appearance features to form multimodal interaction features; using a hierarchical decoder with temporal semantic aggregation to process semantic features and multimodal interaction features to generate a video description language; The network structure of the hierarchical decoder with temporal semantic aggregation includes: The word embedding layer, attention LSTM network, visual time semantic aggregation layer, language LSTM network, language time semantic aggregation layer, and Softmax activation function layer are connected in sequence; The input of the attention LSTM network is connected to the output of the average pooling layer. The input of the average pooling layer is used to input multimodal interaction features; the input of the visual time semantic aggregation layer is used to input multimodal interaction features and semantic features; the input of the language time semantic aggregation layer is used to input semantic features; The output of the attention LSTM network is also connected to the input of the language LSTM network; The visual time semantic aggregation layer generates and aggregates the most relevant visual time features at the current moment and the most relevant semantic features; the visual time semantic aggregation layer is used to aggregate the multimodal interaction features that are most relevant to the currently generated word; The language time semantic aggregation layer is used to process all language features generated at the current moment and before, so as to aggregate the language features most relevant to the currently generated word.

2. The video description method based on multimodal interaction and temporal semantic aggregation according to claim 1, characterized in that: Extract multimodal features from the video to be described, including: Through a two-dimensional convolutional neural network, the appearance features of the video to be described are extracted; Extract motion features of the video to be described through a three-dimensional convolutional neural network; The object features of the video to be described are extracted through regional convolutional neural networks.

3. The video description method based on multimodal interaction and temporal semantic aggregation according to claim 1, characterized in that: Extracting semantic features from multimodal features, specifically including: Use the trained semantic detector to extract semantic features from multimodal features; The semantic detector after training. The specific training process includes: Constructing a first training set and a first validation set; the first training set and the first validation set both contain video appearance features and motion features with known semantic labels; The video appearance features and motion features corresponding to each semantic label are concatenated in series, and the concatenated results are input into the semantic detector for training. When the training reaches the set number of times, the training is stopped, and the detection accuracy of the semantic detector is verified using the first validation set. When the detection accuracy exceeds the set threshold, the training is stopped to obtain the trained semantic detector. Otherwise, the data of the first training set is replaced and the training is repeated.

4. The video description method based on multimodal interaction and temporal semantic aggregation according to claim 3 is characterized in that: The interactive features of the appearance features and the motion features are obtained by: The input value of the first LSTM network is the appearance feature, and the output value of the first LSTM network is the high-level appearance feature; the input value of the second LSTM network is the motion feature, and the output value of the second LSTM network is the high-level motion feature; The high-level appearance features are sequentially processed through the fully connected layer and the mapping function layer to obtain a first mapping value; The high-level motion features are sequentially processed through the fully connected layer and the mapping function layer to obtain a second mapping value; Processing the first mapping value and the second mapping value through a multiplier to obtain a first correlation matrix; Input the first correlation matrix into the softmax activation function layer to obtain the first normalized matrix; The first transposed result of the second mapping value is multiplied by the first normalized matrix, the multiplied result is transposed again to obtain a second transposed result, and the second transposed result is summed with the high-level appearance feature to obtain an interactive feature of the appearance feature and the motion feature.

5. The video description method based on multimodal interaction and temporal semantic aggregation according to claim 4 is characterized in that: The interactive features of the appearance features and the object features are obtained by: The object features are processed in sequence by the fully connected layer and the mapping function layer to obtain the third mapping value; Processing the first mapping value and the third mapping value through a multiplier to obtain a second correlation matrix; Input the second correlation matrix into the softmax activation function layer to obtain the second normalized matrix; The third transposed result of the third mapping value is multiplied by the second normalized matrix, the multiplied result is further transposed to obtain a fourth transposed result, and the fourth transposed result is summed with the high-level appearance feature to obtain an interactive feature of the appearance feature and the object feature.

6. A video description system based on multimodal interaction and temporal semantic aggregation, characterized by: include: An acquisition module is configured to: acquire a video to be described; A multimodal feature extraction module is configured to: extract multimodal features from the video to be described; The multimodal features include: appearance features, motion features and object features; A semantic feature extraction module, configured to: extract semantic features from multimodal features; The video description module is configured to: input the extracted multimodal features and semantic features into a trained video description network, and output a natural language description result of the video; the trained video description network is a video description network based on multimodal information interaction and temporal semantic aggregation; The trained video description network has a network structure including: Interconnected multimodal interaction modules and hierarchical decoders with temporal semantic aggregation; The multimodal interaction module adds the interaction features of the appearance features and the motion features and the interaction features of the appearance features and the object features to obtain the multimodal interaction features; The trained video description network works by: using a multimodal interaction module to integrate motion features and object features into appearance features to form multimodal interaction features; using a hierarchical decoder with temporal semantic aggregation to process semantic features and multimodal interaction features to generate a video description language; The network structure of the hierarchical decoder with temporal semantic aggregation includes: The word embedding layer, attention LSTM network, visual time semantic aggregation layer, language LSTM network, language time semantic aggregation layer, and Softmax activation function layer are connected in sequence; The input of the attention LSTM network is connected to the output of the average pooling layer. The input of the average pooling layer is used to input multimodal interaction features; the input of the visual time semantic aggregation layer is used to input multimodal interaction features and semantic features; the input of the language time semantic aggregation layer is used to input semantic features; The output of the attention LSTM network is also connected to the input of the language LSTM network; The visual time semantic aggregation layer generates and aggregates the most relevant visual time features at the current moment and the most relevant semantic features; the visual time semantic aggregation layer is used to aggregate the multimodal interaction features that are most relevant to the currently generated word; The language time semantic aggregation layer is used to process all language features generated at the current moment and before, so as to aggregate the language features most relevant to the currently generated word.

7. An electronic device, comprising: a memory for non-transitory storage of computer-readable instructions; as well as a processor for executing said computer-readable instructions, When the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is executed.

8. A storage medium, characterized in that: Computer-readable instructions are non-transitory stored, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1 to 5 are executed.

Images