A video text description method integrating multi-granularity video semantic information

Abstract

The present invention proposes a video text description method that integrates multi-granularity video semantic information, which mainly solves the problems of incomplete video semantic information, insufficient semantic information fusion, and information redundancy when multiple semantic information exists in the prior art. Its implementation scheme is: 1) Establishing a data set, and using three neural network pre-training models to extract three-level spatial and temporal features of its video samples respectively; 2) Building a video text description network that integrates multi-granularity video semantic information, and constructing its loss function; 3) Training the video text description network that integrates multi-granularity video semantic information; 4) Inputting the video to be described into the trained network to generate a text description of the video. The present invention increases the types of extracted video features, fully integrates features, and selects appropriate fused features to generate text, which can not only effectively represent video information, but also reduce the degree of information redundancy, and can be used to generate accurate and fluent text describing video content.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and further relates to a video text description method, which can be used to generate a highly accurate and fluent text describing video content. Background Art

[0002] With the rapid development of scientific information technology and the rise of the short video trend, videos are becoming more and more common in people's lives. Videos are artificially created products with the purpose of disseminating information and entertainment, and they have diverse and multi-domain characteristics. Accurate understanding of video content is of great significance and wide application value in life. In the Internet field, video understanding technology is widely used in video search, video summarization, question-answering systems and other fields; in the security field, video understanding technology can be used for abnormal event identification, human and vehicle analysis, etc.; in terms of helping the disabled, video understanding technology can be used for navigation for the blind, converting movies or short videos into text, and then using text-to-speech technology to tell the video content to the blind in the form of voice broadcast, so using text to describe video content plays a very important role. Video text description is a hot research direction in the field of computer vision.

[0003] The purpose of video text description is to automatically generate a complete and natural sentence to describe the video content given a video. The higher the accuracy and fluency of the text description, the better. With the rapid development of deep learning in recent years, the use of deep neural network frameworks has become the mainstream way to handle this task, such as "encoder-decoder", but the existing video text description methods still have the following problems: the video content in natural scenes is complex, the number of entities is large, and the relationship between entities is complex. Most existing methods have a small number of extracted video semantic features and cannot fully represent video information; the insufficient fusion of multiple video semantic features leads to low accuracy of the generated video text description. In view of the above problems, how to extract video semantic features more comprehensively, fully integrate video semantic information, and generate logical video text descriptions is a key problem that needs to be solved in current video text descriptions.

[0004] Xi'an University of Technology disclosed a video text description encoding and decoding method in its patent document with application number: CN202010706821.2. The specific steps of this method are as follows: use a two-dimensional residual convolutional neural network RexNet and a three-dimensional convolutional neural network ECO to extract video features, obtain video semantic feature information, and then use S-LSTM network decoding to obtain a text description of the video, wherein more accurate semantic features are obtained by increasing the difference between words in the encoding stage. Although this method proposes to increase the difference between words to obtain more accurate semantic features, the types of extracted video features are relatively small, and it is impossible to represent the video more comprehensively. In addition, there are problems such as insufficient fusion of multiple video semantic information and insufficient utilization of text semantic information, resulting in low accuracy of the generated video text description.

[0005] Wenjie Pei, Jiyuan Zhang, Xiangrong Wang et al. proposed the memory-attended recurrent network MARN model in the paper "Memory-Attended Recurrent Network for Video Captioning" published in CVPR. The model mainly consists of three parts: an encoder, a recurrent encoder based on the attention mechanism, and an encoder with memory. It uses the designed memory to establish a mapping relationship between words in the vocabulary and related video content. The model uses ResNet-101 as a 2D convolutional neural network to extract 2D visual information from the video, and uses ResNeXt-101 as a 3D convolutional neural network to extract 3D visual information from the video. The experiments in the back of the article also discussed that as the types of video features increase, the running results of the model are getting better and better. However, the video features used by the model are not fully extracted due to the incomplete extraction of visual 2D information, and the extraction of visual 3D features contains insufficient 3D action semantic information, resulting in low accuracy in the generated video description text.

[0006] Qi Zheng, Chaoyue Wang, Dacheng Tao et al. published a paper titled "Syntax-AwareAction Targeting for Video Captioning" in CVPR. The paper proposed a semantic action-guided SAAT model, which uses the Inception-ResnetV2 neural network to extract video appearance features, the C3D three-dimensional neural network to extract video action features, and the Faster R-CNN network to extract regional target features in the middle frame of the video. The maximum number of regional targets is 10, and then the encoding part of the Transformer is used to fuse the regional target features, appearance features, and action features, respectively, and then the video text description is generated through the long short-term memory network LSTM decoder. Although this method uses a variety of video features, when extracting regional target features, only the middle frame of the video is used when extracting local features, resulting in fewer target areas and the inability to more comprehensively represent the video regional target features. In addition, the direct fusion of multiple features will result in information redundancy, resulting in low accuracy of the generated video text description.

[0007] In summary, the existing video text description methods have the problem of extracting fewer types of video features, resulting in incomplete video semantic information. Moreover, in the existing fusion methods of multiple video text descriptions, when multiple video features exist, the video semantic features are not fully and effectively fused, and there will be information redundancy problems, which will interfere with the generated video text description and cause low accuracy of the generated text. Summary of the invention

[0008] The purpose of the present invention is to propose a video text description method that integrates multi-granularity video semantic information to solve the problems proposed in the prior art that the types of video features are small, the video semantic information is incomplete, and when multiple video semantic features exist, the feature fusion is insufficient or the information is redundant, so as to improve the accuracy of the generated text.

[0009] The technical idea for implementing the present invention is to extract video key frame images by densely sampling the original video in an "equally spaced frame number" manner, and use three neural network pre-training models to extract the global features of these key frame images, local features with object types and spatial positions, and action features that reflect video timing information, so as to more comprehensively represent the video feature information, thereby realizing fine-grained modeling of spatial timing information of video modal data with integrity, timing and redundancy, so as to obtain a comprehensive unified video representation; the three video features are fully integrated through the self-attention mechanism method, and the three generated fusion features are interconnected with the three parts of speech information of the text to increase the accuracy of the generated text, and then the designed selection module is combined to select the fusion feature most relevant to the previous word to predict the next word, and the fusion features are selectively used to solve the information redundancy problem caused when there are multiple video semantic information.

[0010] According to the above ideas, the technical solutions of the present invention include the following:

[0011] (1) Establish a training set:

[0012] (1a) Select at least 1,200 videos to be described, and manually annotate at least 20 sentences in each video using natural language text annotations. The number of words in the natural language text annotation of each video sentence does not exceed 30, and a total of at least 42,000 video-natural language-text pairs are generated;

[0013] (1b) For each text annotation in each pair of samples, the POS part of speech of each text annotation is annotated using the part-of-speech tag tool provided by the SpaCy natural language toolkit. Each pair of samples is composed of "video-text-part-of-speech tag";

[0014] (1c) Count all the types of words that appear in the video text description and number them starting from 0 to form a dictionary. The dictionary format is: {number: word}. According to the dictionary, replace the text in the above sample pairs with the dictionary number to obtain the training set;

[0015] (2) Three neural network pre-training models are used to extract three-level spatial and temporal features of video samples:

[0016] (2a) In the training set, N frames of images are extracted from each video as key frame images of the video using the “equally spaced frame” sampling method;

[0017] (2b) Input the key frame image extracted from (2a) into the existing trained 2D convolutional neural network pre-trained model Inception-ResNetV2, and extract the 1536-dimensional features of each image as the global feature V a ∈R N×1536, the global feature dimension extracted from each video is N*1536;

[0018] (2c) Input the key frame image extracted from (2a) into the existing pre-trained neural network model Faster-RCNN, extract M local regions of each image, and the feature dimension of each region is 2048 dimensions as the local feature V o ∈R N×M×2048 , the dimension of local area features extracted from each video is N*M*2048;

[0019] (2d) Input the video in the training set into the existing pre-trained 3D convolutional neural network model I3D, take the time position of the key frame extracted in (2a) as N time points, and extract the 1024-dimensional features of each time point in the video as the action features V of each video along the time sequence. m ∈R N×1024 , the dimension of action features extracted from each video is N*1024.

[0020] (3) Building a video text description network that integrates multi-granular video semantic information:

[0021] (3a) constructing a video semantic feature embedding module composed of a global semantic information embedding submodule, an action semantic information embedding submodule and a local semantic information embedding submodule in parallel;

[0022] (3b) constructing a video semantic information fusion module consisting of a noun fusion module, a verb fusion module, and a logical connective fusion module in parallel;

[0023] (3c) constructing a video semantic fusion feature selector module;

[0024] (3d) Construct a decoder module consisting of a long short-term memory network neuron LSTMCell and a linear layer to output short-term memory and long-term memory

[0025] (3e) The video semantic feature embedding module, the semantic information fusion module, the fusion feature selector module and the decoder module are cascaded to form a video text description network with multi-granularity video semantic information.

[0026] (4) Define the video text description network loss function that integrates multi-granular video semantic information:

[0027]

[0028] in, is the cross entropy loss function, which represents the loss of text sentence information. is the KLD loss function, which indicates the loss of part-of-speech information, and λ is the set weight parameter;

[0029] (5) Training the video text description network:

[0030] (5a) Extract three-level features of the videos in the training set according to the steps in (2);

[0031] (5b) The features extracted in (5a) and the text descriptions and part-of-speech tags in the training set are randomly selected and input into the video text description network in batches, and the parameters in the current network are iteratively updated using the Adam optimization algorithm and the gradient descent method until the loss function converges or the training times are completed, thereby obtaining a trained video text description network;

[0032] (6) The user uses the trained video text description network to generate a video description sentence:

[0033] (6a) Using the same method as (2), three existing neural network pre-training models are used to extract global features, local features, and action features of user-submitted videos;

[0034] (6b) Input the three features extracted from (6a) into the trained video text description network and output the description sentence corresponding to the video.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] First, the present invention uses three different types of neural network pre-training models to extract global features, local features and action features from each video sample respectively, and extracts video image frames as key frames of the video by "equally spaced sampling". The spatial temporal information of the video modal data is fine-grainedly modeled to obtain a comprehensive unified representation of the video, which solves the problem that the existing video text description technology methods cannot fully represent the complex spatiotemporal semantic features of the video. From the "global-local" perspective, the temporal information and target features hidden in the video are fully mined as the main input features of the network model, thereby improving the accuracy of generating video text descriptions.

[0037] Second, the present invention adopts a video text description network that integrates multi-granularity video semantic information and combines it with a video semantic feature selector module. It adopts an attention mechanism-based method to mine the correlation between different video semantic information, overcomes the limitations of single video semantic information, and can correspond the global information, action information and local information of the video with the part-of-speech information such as nouns and verbs of the generated text.

[0038] Third, the present invention combines the fusion network module with the selector module and selectively uses relevant video semantic features, so that the generated words are more relevant to the video semantic information, which solves the information redundancy problem when the prior art uses multiple video semantic information, and makes the current word generated more closely related to the previous word, effectively improving the accuracy and fluency of the generated video text description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart for realizing the present invention. DETAILED DESCRIPTION

[0040] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0041] Reference Figure 1 , the implementation steps of this example are as follows:

[0042] Step 1: Create a training set.

[0043] (1.1) From the existing MSVD dataset, select at least 1,200 videos to be described and their corresponding description text data to form a sample set. Each video has at least 20 manually annotated natural language text annotations, and each natural language text has no more than 30 words, generating at least 42,000 pairs of samples;

[0044] (1.2) Extract the POS part-of-speech tags of the text in the sample pair using the part-of-speech tags provided in the SpaCy natural language processing package, where the "NN" tag represents a singular noun and the "JJ" tag represents an adjective with the number "0", the "VB" tag represents a verb with the number "1", and the tags of other parts of speech are represented by the number "2";

[0045] (1.3) Count all the types of words that appear in the text description and number them from 0 to form a dictionary. Use " <start>" tag, indicating the beginning of a sentence, and the end of the text description is used <end>" tag, indicating the end of a sentence, words that do not appear in the dictionary use " <unk>"Label, blank text uses " <pad>" tag, the number of word types in the text is 7351, then the total length of the dictionary is 7355 = 7351 + 4, and the content format of the dictionary is {number: word};

[0046] Assume that the description text of a video is X i , which consists of L words, indicating the length of the text description. The text description is numbered and annotated using the created dictionary. Then, a text description is represented as follows:

[0047] X i ={c i,1 ,c i,2 ,...,c i,t ,...,c i,L }

[0048] where c i,t Represented as the number of the t-th word in the ith text in the dictionary, i=1,2,...,7355.

[0049] Step 2: Use three neural network pre-training models to extract three-level spatial and temporal features of video samples respectively.

[0050] The three neural network pre-training models are:

[0051] The first is the 2D convolutional neural network pre-trained model Inception-ResNetV2;

[0052] The second is the 3D convolutional neural network pre-training model I3D;

[0053] The third is the target detection neural network pre-training model Faster-RCNN.

[0054] The specific implementation of this step is as follows:

[0055] (2.1) Select one video from the training set in turn using the "equally spaced frame sampling" method and calculate the specific moment position of each video:

[0056]

[0057] Among them, T represents the time of the video, V represents the frame rate of the video, N represents the number of intervals set, which is set to 26 in the experiment, and P i Indicates the specific moment position of the i-th video, i = 0, 1, 2...N;

[0058] (2.2) Using the existing OpenCV tools, extract N frames of the video from the calculated N specific time positions as key frames;

[0059] (2.3) The N key frame images extracted in (2.2) are respectively input into the three neural network pre-training models to extract the global features, local features and action features of the video:

[0060] (2.3.1) Input N key frame images into the 2D convolutional pre-trained neural network model Inception-ResNetV2, extract the 1536-dimensional features of each image, and obtain the global feature V of the video a ∈R N×1536 ;

[0061] (2.3.2) Input N key frame images into the object detection neural network pre-training model Faster-RCNN, extract M local regions of each image, and the local feature dimension of each region is 2048-dimensional features to obtain the local feature V of the video o ∈R N×M×2048 , in this example, M is set to 36;

[0062] (2.3.3) Input each video into the 3D convolutional neural network pre-training model I3D, extract the dynamic behavior features of each video along the time sequence, and obtain the action features V of the video m ∈R N×1024 ;

[0063] (2.4) Connect the three video features extracted from a video to obtain the overall feature representation of a video:

[0064]

[0065] in, represents the global features of the jth video, represents the local features of the jth video, represents the action feature of the jth video, V j Represents the overall features of the video, j represents the jth video in the dataset, j = 1, 2, ..., 1200.

[0066] Step 3: Build a video text description network that integrates multi-granularity video semantic information and process the above three video features.

[0067] (3.1) Construct a video semantic feature embedding module consisting of a global semantic information embedding submodule, an action semantic information embedding submodule, and a local semantic information embedding submodule in parallel. The structure, dimension, input and output settings of each submodule are as follows:

[0068] (3.1.1) A global semantic information embedding submodule is composed of a linear layer with a dimension of 1536×1000 and a bidirectional long short-term memory network Bi-LSTM with a dimension of 1024. The global feature V of (2.3.1) is input to this submodule. a ∈R N ×1536 , output global semantic feature V′ a ∈R N×1024 ;

[0069] (3.1.2) The action semantic information embedding submodule is composed of a linear layer with a dimension of 1024×1000 and a bidirectional long short-term memory network Bi-LSTM with a dimension of 1024. The action feature V in (2.3.3) is input to this submodule. m ∈R N ×1024 , output action semantic feature V′ m ∈R N×1024 ;

[0070] (3.1.3) A local semantic information embedding submodule is composed of a linear layer with a dimension of 2048×1000 and a cascade with a dimension of 1600×1000. The local feature V of (2.3.2) is input to this submodule. o ∈R N×M×2048 , output local semantic features V′ o ∈R N×M×1000 ;

[0071] (3.2) Construct a video semantic information fusion module consisting of a noun fusion submodule, a verb fusion submodule, and a logical connective fusion submodule in parallel. The three semantic features are processed by these three submodules to achieve the following:

[0072] (3.2.1) The noun fusion submodule consists of a 512×512 linear layer and a dot product attention network cascade to calculate the noun semantic feature V n :

[0073]

[0074] in is the short-term memory of the output of the previous time step, V′ o is the local semantic feature of (3.1.3), is V′ o Dimensions;

[0075] (3.2.2) The verb fusion submodule is composed of a 512×512 linear layer and a dot product attention network cascade to calculate the verb semantic feature V l :

[0076]

[0077] Where V′ m is the action semantic feature, is V′ m Dimensions;

[0078] (3.2.3) The logical connective fusion submodule is composed of a linear layer with a dimension of 512×512 and a dot product attention network, and the verb semantic feature V is calculated. f :

[0079]

[0080] in is the long-term memory of the previous time step, yes Dimension.

[0081] (3.3) Construct a video semantic fusion feature selector module to process the three semantic fusion features in (3.2):

[0082] (3.3.1) A linear layer with a dimension of 1024×512, a linear layer with a dimension of 512×512, an activation function tanh and a softmax function are cascaded to form a video semantic fusion feature selector module;

[0083] (3.3.2) Substitute the noun feature V in (3.2) n 、Verb Features V m and logical connective V f Input the first linear layer in the video semantic fusion feature selector module respectively to obtain noun embedding, verb embedding and logical connective embedding;

[0084] (3.3.3) The short-term memory output of the previous time step Input the second linear layer in the video semantic fusion feature selector module to obtain the short-term memory embedding;

[0085] (3.3.4) Add the three part-of-speech embeddings of (3.3.2) and the short-term memory embedding of (3.3.3) respectively, and then pass through the tanh activation function to generate noun score values, verb score values, and logical connective score values. Use the softmax function to select the part-of-speech feature with the largest score value among the three;

[0086] (3.4) Construct a decoder module consisting of a long short-term memory network neuron LSTMCell with a dimension of 1536, a linear layer with a dimension of 1536×7351, a linear layer with a dimension of 512×7351 and a softmax function cascade, input the part-of-speech feature with the largest score in (3.3) to the module, and output the word number with the highest probability in the dictionary;

[0087] (3.5) The video semantic feature embedding module, the semantic information fusion module, the semantic fusion feature selector module and the decoder module are cascaded to form a video text description network of multi-granularity video semantic information.

[0088] Step 4: Use the cross entropy loss function and KLD loss function to calculate the overall loss of the video text description network.

[0089] (4.1) Connect the most likely word numbers generated by each training sample input into the network to form the predicted video text description;

[0090] (4.2) Calculate the cross entropy loss of the text according to the following formula:

[0091]

[0092] Where Len is the length of the text, P t is the textual description of the network prediction in (5.1), is the correct text description in the training set;

[0093] (4.3) Calculate the KLD part-of-speech loss according to the following formula:

[0094]

[0095] in is the part-of-speech tag predicted by the network, is the correct part-of-speech tag in the training set

[0096] (4.4) Adding the above two losses together, we get the overall loss of the network:

[0097]

[0098] Where λ is the weight parameter set.

[0099] Step 5: Train a video text description network that integrates multi-granularity video semantic information.

[0100] (5.1) Extract three features of the videos in the training set using the same method as step 2;

[0101] (5.2) Assume that the initial value of the learning rate is 0.0001 and the number of training times is 20. When the network training times reaches 10 times, the learning rate is adjusted to 0.00001;

[0102] (5.3) The three features extracted in (5.1) and the correct text descriptions and part-of-speech tags in the training set are input into the video text description network constructed in step 3 in batches. The sample pair size of each batch is 32. The parameters in the current network are iteratively updated using the Adam optimization algorithm and gradient descent method.

[0103] (5.4) At the end of each training, calculate the overall network loss value according to step 4, and save the network parameters with the minimum value;

[0104] (5.5) Repeat (5.3) and (5.4) until the loss function converges or the training times are completed to obtain the trained video text description network.

[0105] Step 6: Generate a text description of the video.

[0106] The same method as step 2 is used for the video to be described. Three existing neural network pre-training models are used to extract the global features, local features and action features of the user-submitted video. These features are then input into the trained video text description network to output the description sentence corresponding to the video.

[0107] The effects of the present invention are further described below in conjunction with simulation experiments.

[0108] 1. Simulation experiment conditions:

[0109] The simulation experiment hardware platform of the present invention is: the processor is Intel Core Xeon 4210CPU, the main frequency is 2.2GHz, the memory is 16GB, and the graphics card is Nvidia GeForce RTX 3090.

[0110] The software platform of the simulation experiment of the present invention is: Linux 4.15 operating system and Python 3.7.

[0111] The data used in the simulation experiment of the present invention is: MSVD dataset, which contains 1970 sample pairs of "video-multiple description texts", and each sample pair has at least 30 text descriptions. 1200 sample pairs are selected from the MSVD dataset as the training set, and the remaining sample pairs are used as the test set to verify the accuracy of the network.

[0112] The present invention uses four existing evaluation indicators: bilingual evaluation accuracy BLUE@4, recall evaluation accuracy ROUGE_L, explicit sorting translation evaluation index METEOR and consistency-based image description evaluation index CIDER

[0113] 2. Simulation content and results analysis:

[0114] The simulation experiment of the present invention is to use the method of the present invention and four existing technologies (POS-CG, SAAT, MARN, Mixture) to construct video text description networks respectively, use the same training set data to train these five networks respectively, and obtain five trained video text description networks, and then use the same test set to input them into the five trained networks respectively to obtain corresponding video text descriptions, and use the existing four index methods to evaluate the output results, as shown in Table 1:

[0115] Table 1: Comparison of simulation results between the present invention and the prior art

[0116]

[0117] The four prior art sources in Table 1 are as follows:

[0118] POS-CG is a video text description network model proposed by Wang et al. in their paper "Controllable Video Captioning WithPOS Sequence Guidance Based on Gated Fusion Network" (IEEE / CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 2641-2650, doi: 10.1109 / ICCV.2019.00273.), abbreviated as POS-CG.

[0119] SAAT is a video text description network model proposed by Zheng et al. in their paper "Syntax-Aware Action Targeting for Video Captioning" (2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 13093-13102, doi: 10.1109 / CVPR42600.2020.01311), referred to as SAAT.

[0120] MARN is a video text description network model proposed by Pei et al. in their paper "Memory-Attended Recurrent Network for Video Captioning" (2019IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.), abbreviated as MARN.

[0121] Mixture is a video text description network model proposed by Hou et al. in their paper "Joint Syntax Representation Learning and Visual Cue Translation for Video Captioning" (IEEE / CVF International Conference on Computer Vision (ICCV). IEEE, 2020.), referred to as Mixture.

[0122] As can be seen from Table 1, the four evaluation indicators of the video text description of the method of the present invention are all higher than those of the four prior art methods, that is, the four evaluation indicators are improved by 8%, 3.7%, 0.4% and 3% respectively.

[0123] The above simulation experiments show that the present invention extracts three video features through three neural network pre-training models, and then outputs a description text by fusing a multi-granularity video semantic information network model, thereby solving the problem that the existing video text description method extracts fewer types of video features and cannot effectively represent video information. It further solves the problem that the existing method for fusing multiple video features does not effectively fuse video semantic features, and when multiple features exist, there will be information redundancy, which interferes with the generation of accurate video text. The above experiments verify that the present invention can obtain a more accurate and fluent video text description.< / pad> < / unk> < / end> < / start>

Claims

1. A video text description method integrating multi-granularity video semantic information, characterized in that it comprises the following steps: (1) Establish a training set: (1a) Select at least 1,200 videos to be described, and manually annotate at least 20 sentences in each video using natural language text annotations. The number of words in the natural language text annotation of each video sentence does not exceed 30, and a total of at least 42,000 video-natural language-text pairs are generated; (1b) For each text annotation in each pair of samples, the POS part of speech of each text annotation is annotated using the part-of-speech tag tool provided in the SpaCy natural language toolkit. Each pair of samples is composed of "video-text-part-of-speech tag"; (1c) Count all the types of words that appear in the video text description and number them starting from 0 to form a dictionary. The dictionary format is: {number: word}. According to the dictionary, replace the text in the above sample pairs with the dictionary number to obtain the training set; (2) Three neural network pre-training models are used to extract three-level spatial and temporal features of video samples: (2a) In the training set, N frames of images are extracted from each video as key frame images of the video using the "equally spaced frame" sampling method; (2b) Input the key frame image extracted from (2a) into the existing trained 2D convolutional neural network pre-trained model Inception-ResNetV2, and extract the 1536-dimensional features of each image as the global feature V a ∈R N×1536 , the global feature dimension extracted from each video is N*1536; (2c) Input the key frame image extracted from (2a) into the existing trained object detection neural network pre-training model Faster-RCNN, extract M local regions of each image, and the feature dimension of each region is 2048 dimensions as the local feature V o ∈R N×M×2048 , the dimension of local area features extracted from each video is N*M*2048; (2d) Input the video in the training set into the existing pre-trained 3D convolutional neural network model I3D, take the time position of the key frame extracted in (2a) as N time points, and extract the 1024-dimensional features of each time point in the video as the action features V of each video along the time sequence. m ∈R N×1024 ,The dimension of action features extracted from each video is N*1024; (3) Building a video text description network that integrates multi-granular video semantic information: (3a) constructing a video semantic feature embedding module composed of a global semantic information embedding submodule, an action semantic information embedding submodule and a local semantic information embedding submodule in parallel; the structure and parameter settings of each submodule in the video semantic feature embedding module are as follows: The global semantic information embedding submodule is composed of a linear layer with a dimension of 1536×1000 and a bidirectional long short-term memory network Bi-LSTM with a dimension of 1024, which is used to output the global semantic feature V a ′; The action semantic information embedding submodule is composed of a linear layer with a dimension of 1024×1000 and a bidirectional long short-term memory network Bi-LSTM with a dimension of 1024, which is used to output the action semantic feature V m ′; The local semantic information embedding submodule is composed of a linear layer with a dimension of 2048×1000 and a connection with a dimension of 1600×1000, which is used to output the local semantic feature V o ′; (3b) constructing a video semantic information fusion module composed of a noun fusion module, a verb fusion module and a logical connective fusion module in parallel; the structures and parameters of the submodules constituting the video semantic information fusion module are set as follows: The noun fusion submodule is composed of a linear layer with a dimension of 512×512 and a dot product attention network connection, which is used to output the noun semantic feature V n : in is the short-term memory of the output of the previous time step, V o ′ is a local semantic feature, Yes V o ' dimension; The verb fusion submodule is composed of a linear layer with a dimension of 512×512 and a dot product attention network connection, which is used to output the verb semantic feature V l : Where V m ′ is the action semantic feature, Yes V m ' dimension; The logical connective fusion submodule is composed of a linear layer with a dimension of 512×512 and a dot product attention network connection, which is used to output the verb semantic feature V f : in is the long-term memory of the previous time step, yes Dimensions; (3c) constructing a video semantic fusion feature selector module; (3d) Construct a decoder module consisting of a long short-term memory network neuron LSTMCell and a linear layer to output short-term memory and long-term memory (3e) cascading the video semantic feature embedding module, the semantic information fusion module and the decoder module to form a video text description network with multi-granular video semantic information; (4) Define the video text description network loss function that integrates multi-granular video semantic information: l=l1+λl2 Among them, l1 is the cross entropy loss function, which indicates the loss of text sentence information, l2 is the KLD loss function, which indicates the loss of part-of-speech information, and λ is the set weight parameter; (5) Training the video text description network: (5a) Extract three-level features of the videos in the training set according to the steps in (2); (5b) The features extracted in (5a) and the text descriptions and part-of-speech tags in the training set are randomly selected and input into the video text description network in batches, and the parameters in the current network are iteratively updated using the Adam optimization algorithm and the gradient descent method until the loss function converges or the training times are completed, thereby obtaining a trained video text description network; (6) The user uses the trained video text description network to generate a video description sentence: (6a) Using the same method as (2), three existing neural network pre-training models are used to extract global features, local features, and action features of user-submitted videos; (6b) Input the three features extracted from (6a) into the trained video text description network and output the description sentence corresponding to the video.

2. The method according to claim 1, characterized in that: Step (2a) extracts N frames of images from each video in the training set by sampling at "equal intervals", which is implemented as follows: (2a1) Select one video from the training set in turn and calculate the position of each key frame in the video at a specific moment: Where T represents the time of the video, V represents the frame rate of the video, N represents the number of intervals set, and P i represents the i-th sampling position, i=0,1,2...N; (2a2) Using the existing OpenCV tools, extract N frames of the video from the N specific time positions calculated in (2a1).

3. The method according to claim 1, characterized in that: The cross entropy loss function l1 and KLD loss function l2 in step (4) are expressed as follows: Where Len is the length of the text, P t is a textual description of the network’s predictions, is the correct text description in the training set, is the part-of-speech tag predicted by the network, is the correct part-of-speech tag in the training set.

4. The method according to claim 1, characterized in that: Step (5b) uses the Adam optimization algorithm and gradient descent method to iteratively update the parameters in the current network, which is implemented as follows: (5b1) Assume that the initial value of the learning rate is 0.0001 and the number of training times is 20. When the network training times reaches 10 times, the learning rate is adjusted to 0.00001; (5b2) In each training, sample pairs are selected from the training set in batches, and the size of each batch of sample pairs is set to 32. They are input into the network, the overall network loss is calculated, and the network parameters are updated iteratively through back-propagation. (5b3) At the end of each training, the network parameters with the minimum loss function are saved; (5b4) Repeat (5b2) and (5b3) until the loss function converges or the training times are completed to obtain the trained video text description network.

Images