Video description generation method and system integrating semantic feature selection and attention

Abstract

The present invention relates to the field of natural language processing and computer vision technology, and in particular to a video description generation method and system for semantic feature selection and attention fusion, mapping the image features of a video to be described to the text space of a description sentence, and obtaining the semantic object features in the video to be described; extracting the action features in the video to be described based on the semantic object features, and fusing different modal feature information in the video to be described based on an attention mechanism, wherein the different modal feature information includes at least semantic object features and action features; combining the previous word encoding embedding in the description sentence and using conditional probability distribution to gradually decode the fused feature information into a natural language description corresponding to the video to be described. The present invention can enhance the semantic relevance between the text description of a video entity and the video content, effectively alleviate the semantic deviation between the visual space and the semantic space, improve the output accuracy of the action encoder, and has a good application prospect in the field of automatic generation of video descriptions.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and computer vision technology, and in particular to a video description generation method and system oriented to semantic feature selection and attention fusion. Background Art

[0002] The video description task for video understanding requires extracting meaningful high-level semantic information such as objects, scenes, actions, etc. from the video, and generating logically clear and semantically coherent text content based on this information. Video description generation is an interdisciplinary problem involving computer vision and natural language processing. It can convert video content into text or semantic representation, which can help people retrieve and manage massive video resources more quickly and efficiently. Given the spatiotemporal characteristics of video data and the diversity and complexity of semantics, the main challenge is how to effectively combine the visual information of the video with the natural language description to generate an accurate and fluent description. How to understand the content of the video and describe its content has become a research hotspot in the current field of video understanding.

[0003] Video description generation methods can be roughly divided into three categories, mainly: methods based on manual feature extraction, video description methods based on deep learning, and generation methods based on reinforcement learning. Among them, the method based on manual feature extraction uses template technology to make the generated descriptive sentences more accurate in grammar, but the existence of fixed templates limits the descriptive sentences, making the generation of sentences extremely inflexible and not rich in content. In the method based on deep learning, the typical research framework is to describe the video using the "encoder-decoder" paradigm. In the method based on reinforcement learning, taking the GAN network as an example, by training the generator and the discriminator, the generator can generate new data similar to the real data. This method usually requires the use of additional data sets to train the GAN model. There are still some difficulties in the generation of video descriptions in these three methods: 1) There are a large number of static semantic objects in the video, and many works do not screen these semantic objects, but stay at the video frame level features to analyze the video content, lacking fine-grained semantic object selection. 2) Due to the motion characteristics of the video, some research works pay more attention to the capture of action semantics, breaking through the limitation that the original video action recognition task can only output limited action types according to the video content. However, it pays more attention to action semantics and ignores other components of descriptive sentences, which greatly restricts sentence structure and flexible expression. 3) In terms of feature fusion, most existing methods simply cascade features of different modalities, lack correlation mining between high-level semantics, and the feature fusion method is relatively simple and inefficient. Summary of the invention

[0004] To this end, the present invention provides a video description generation method and system for semantic feature selection and attention fusion, which solves the problems in the prior art that affect the application of video description due to the lack of fine-grained semantic object selection, limited action type restrictions and lack of correlation mining between high-level semantics.

[0005] According to the design scheme provided by the present invention, on the one hand, a video description generation method for semantic feature selection and attention fusion is provided, comprising:

[0006] Mapping the image features of the video to be described to the text space of the description sentence, and obtaining the semantic object features in the video to be described;

[0007] Extracting action features in the video to be described based on semantic object features, and fusing different modal feature information in the video to be described based on an attention mechanism, wherein the different modal feature information includes at least semantic object features and action features;

[0008] Combined with the previous word encoding embedding in the description sentence and using conditional probability distribution, the fused feature information is gradually decoded into the natural language description corresponding to the video to be described.

[0009] As a video description generation method for semantic feature selection and attention fusion of the present invention, further, the image features of the video to be described are mapped to the text space of the description sentence, including:

[0010] Firstly, several frames in the video to be described are selected as key frames, and the key frames and the video frames in the adjacent areas are used as the input of the pre-trained object detector model. The object detector model is used to obtain the object detection features in the video to be described, wherein the object detection features at least include initial object features, initial action features and video frame features;

[0011] Then, the encoder is used to encode and output the initial object features; and for the encoded output results, the video frame features and the initial action features are combined and the decoder is used to map the input to the description sentence text space to select and output the video semantic object features, wherein the decoder includes: an LSTM network that obtains the input data vector representation and a fully connected layer that maps the LSTM network output to the text space.

[0012] As a video description generation method for semantic feature selection and attention fusion of the present invention, further, in the fully connected layer, the LSTM network output is mapped to the text space, and the network is supervised and learned with the help of description sentences, and the learning process includes:

[0013] First, the loss function is constructed based on minimizing the distance between the entity in the description sentence and the entity in the video;

[0014] Then, the description sentences are used to perform supervised learning on the LSTM network based on the loss function to learn the vector representation of the video content in the text space.

[0015] As a video description generation method for semantic feature selection and attention fusion of the present invention, further, extracting action features in the video to be described based on semantic object features, comprising:

[0016] The pre-trained C3D three-dimensional convolutional neural network is used to extract the initial action in the sample data, and the action features in the video are obtained through cross-attention fusion in combination with the semantic object features. In the pre-training of the C3D three-dimensional convolutional neural network, the initial action and action-related features are spliced ​​and the spliced ​​features are mapped to the text space using the LSTM network and the fully connected layer. The learning optimization of the neural network is achieved by minimizing the distance between the relevant action features in the video description supervision signal text and the features after splicing and mapping to the text space.

[0017] As a video description generation method for semantic feature selection and attention fusion of the present invention, further, based on the attention mechanism, different modal feature information in the video to be described is fused, including:

[0018] First, the action features and semantic object features are fused using the additive attention mechanism;

[0019] Next, a multi-layer attention mechanism is used to extract deep hierarchical features from the fused features and obtain the similarity matrix;

[0020] Then, the softmax function is used to normalize the similarity matrix and obtain the attention weight. The attention weight and the deep level extracted features are weighted averaged to obtain the fusion features of different modal features.

[0021] As a video description generation method for semantic feature selection and attention fusion of the present invention, further, combining the encoding embedding of the previous word in the description sentence and using conditional probability distribution to gradually decode the fused feature information into a natural language description corresponding to the video to be described, it includes:

[0022] First, the fused features are converted into a sequence of decoder input vectors containing time steps;

[0023] Then, based on the visual features and the previous word encoding embedding in the description sentence, the pre-trained LSTM decoder is used to predict the current word. The natural language description corresponding to the video is obtained by step-by-step decoding until a complete description sentence is generated or the predefined maximum sentence length is reached.

[0024] As a video description generation method for semantic feature selection and attention fusion of the present invention, further, the LSTM decoder training process includes:

[0025] The training loss function is set using cross entropy loss, and the LSTM decoder is trained and optimized based on the given video and video description and the training loss function.

[0026] Furthermore, the present invention also provides a video description generation system for semantic feature selection and attention fusion, comprising: a preprocessing module, a fusion module and an output module, wherein:

[0027] A preprocessing module, used to map the image features of the video to be described to the text space of the description sentence, and obtain the semantic object features in the video to be described;

[0028] A fusion module is used to extract action features in the video to be described based on the semantic object features, and fuse different modal feature information in the video to be described based on the attention mechanism, wherein the different modal feature information at least includes semantic object features and action features;

[0029] The output module is used to gradually decode the fused feature information into a natural language description corresponding to the video to be described.

[0030] Beneficial effects of the present invention:

[0031] 1. There are a large number of static semantic objects in the video. Many works do not screen these semantic objects, but only analyze the video content based on the video frame level features. There is a lack of fine-grained semantic object selection. Through efficient semantic feature selection and using feature mapping to map image features to text space, the model is prompted to learn the semantic representation of semantic objects in text space from descriptive sentences, which can effectively alleviate the semantic deviation between visual space and semantic space.

[0032] 2. Regarding feature fusion, most existing methods simply concatenate features of different modalities, lack correlation mining between high-level semantics, and the feature fusion method is relatively simple and inefficient. By encoding action features through feature fusion and using semantic object features as queries to search for action features, the accuracy of the action encoder output can be improved.

[0033] 3. Comparative experiments on MSVD and MSR-VTT datasets show that the scheme in this case has achieved relatively advanced results. At the same time, a large number of ablation experiments can also verify the effectiveness of the scheme in this case, and it can be deployed in the automatic description scenario of multimedia videos, and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the video description generation process for semantic feature selection and attention fusion in the embodiment;

[0035] Figure 2 It is a schematic diagram of the overall model framework of the video description generation principle in the embodiment;

[0036] Figure 3 This is a schematic diagram of the performance effect of the model in the embodiment;

[0037] Figure 4 This is a schematic diagram of the changing trend of the BLEU4 indicator in the ablation experiment in the embodiment over the training rounds;

[0038] Figure 5 This is a visual analysis diagram in the embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and technical solutions.

[0040] In view of the problems that there is still semantic deviation between video content and generated description and low efficiency of feature fusion in the encoding-decoding process of existing video description generation, the embodiments of the present invention refer to Figure 1 As shown, a video description generation method for semantic feature selection and attention fusion is provided, which includes the following contents:

[0041] S101, mapping image features of a video to be described to a text space of a description sentence, and obtaining semantic object features in the video to be described.

[0042] Specifically, the image features of the video to be described are mapped to the text space of the description sentence, which can be designed to include the following content:

[0043] Firstly, several frames in the video to be described are selected as key frames, and the key frames and the video frames in the adjacent areas are used as the input of the pre-trained object detector model. The object detector model is used to obtain the object detection features in the video to be described, wherein the object detection features at least include initial object features, initial action features and video frame features;

[0044] Then, the encoder is used to encode and output the initial object features; and for the encoded output results, the video frame features and the initial action features are combined and the decoder is used to map the input to the description sentence text space to select and output the video semantic object features, wherein the decoder includes: an LSTM network that obtains the input data vector representation and a fully connected layer that maps the LSTM network output to the text space.

[0045] In the fully connected layer, the LSTM network output is mapped to the text space, and the network is supervised by the description sentence. The learning process may include:

[0046] First, the loss function is constructed based on minimizing the distance between the entity in the description sentence and the entity in the video;

[0047] Then, the description sentences are used to perform supervised learning on the LSTM network based on the loss function to learn the vector representation of the video content in the text space.

[0048] In the video description task, vision and language are two different but closely related data modalities. In order to handle this relationship, in this embodiment, see Figure 2 The framework of the video description generation algorithm (A method for generating video descriptions by fusing semantic selection and multi-attention stacking, SSMA) shown in the figure mainly includes four parts: semantic feature selection, action feature extraction, multimodal information fusion, and decoder. The video is represented as a series of visual features v. Each feature v i It is obtained through a visual feature extractor represented by a convolutional neural network. At the same time, each word described in natural language can be converted into an embedded vector form w through one-hot encoding. t In this case, the task of our solution is to use visual features and the previous word w t-1 To predict the output word w at the current moment t , that is, generating a natural language description sentence. In the specific implementation, this task can be completed by maximizing the conditional probability distribution of the given input sequence and output sequence, so as to select the word that can maximize the conditional probability at each moment as the predicted word at the current moment, and continue until a complete description sentence is generated or the predefined maximum sentence length is reached.

[0049] In order to make the model learn human judgment, in the embodiment of this case, semantic objects that are consistent with human intuition are first screened from the video, and its specific structure can be designed to include two parts: target detection and language model. Only those objects that are consistent with human intuition are selected. With the help of the DETR structure, the image features can be mapped to the text space through the pre-trained Transformer model, and then the description sentence is used as the label optimization. The description sentence can be directly applied as a supervisory signal to learn the vector representation of the video content in the text space.

[0050] For a single video, T frames can be selected as key frames and input into a pre-trained object detector along with the video frames around the key frames. Object regions are captured from each key frame and clustered based on the intersection between appearance and bounding boxes to obtain target detection features. M represents the number of objects. The initial object features will then be input into the encoder to obtain the hidden layer representation.

[0051]

[0052] The semantic selection decoder accepts three types of input

[0053]

[0054] XO h is the encoder output, θ is a randomly generated query parameter, X V It can be the video context vector from 2DCNN. As the output of the decoder, φ is mapped to the semantic space through a fully connected layer and LSTM, and supervised learning is performed with the help of video description.

[0055] When calculating the loss, in this embodiment, the synonym representation of each entity in the description sentence can be obtained from WordNet, and after removing the abstract nouns, SBERT (Sentence-BERT) is used as the text encoder to calculate the entity embedding feature. Obviously, the entities extracted from the description and the entities extracted from the video can form two sets. The optimization problem is actually an optimal allocation problem. The optimal allocation cost can be set as but

[0056]

[0057]

[0058] Dist means and The matching cost between them is finally minimized and The distance between them optimizes the above semantic selection step:

[0059]

[0060] S102. Extract action features in the video to be described based on semantic object features, and fuse different modal feature information in the video to be described based on an attention mechanism, wherein the different modal feature information includes at least semantic object features and action features.

[0061] Specifically, the action features in the video to be described are extracted based on the semantic object features, which can be designed to include the following contents:

[0062] The pre-trained C3D three-dimensional convolutional neural network is used to extract the initial action in the sample data, and the action features in the video are obtained through cross-attention fusion in combination with the semantic object features. In the pre-training of the C3D three-dimensional convolutional neural network, the initial action and action-related features are spliced ​​and the spliced ​​features are mapped to the text space using the LSTM network and the fully connected layer. The learning optimization of the neural network is achieved by minimizing the distance between the relevant action features in the video description supervision signal text and the features after splicing and mapping to the text space.

[0063] Among them, the different modal feature information in the video to be described is integrated based on the attention mechanism, which may include:

[0064] First, the action features and semantic object features are fused using the additive attention mechanism;

[0065] Next, a multi-layer attention mechanism is used to extract deep hierarchical features from the fused features and obtain the similarity matrix;

[0066] Then, the softmax function is used to normalize the similarity matrix and obtain the attention weight. The attention weight and the deep level extracted features are weighted averaged to obtain the fusion features of different modal features.

[0067] In action feature extraction, semantic object features can be used as queries to find action features, which can improve the accuracy of the action encoder output.

[0068]

[0069] The C3D network pre-trained on Kinetics-400 is used to extract action features in the video. The C3D network is a three-dimensional convolutional neural network that can effectively capture the spatiotemporal information in the video and extract the corresponding action features. These features can be used for subsequent feature fusion and text generation. The C3D network is used to extract T action initializations in each sample. Combine the output of the semantic selection step The action features are obtained by cross-attention fusion Similar to the semantic feature selection step, in order to use the video description text as a supervision signal, the initial action features and action-related features can be concatenated and input into the LSTM, and then mapped to the text space through the fully connected layer.

[0070]

[0071] By minimizing the action feature S from the text supervision signal a and The distance between them is used to optimize the above action feature extraction step.

[0072]

[0073] In order to associate action features with corresponding semantic objects, the SBERT encoder can still be used to extract verb embeddings from video descriptions as supervisory signals. In this way, subsequent multimodal information fusion can obtain richer feature representations and improve the accuracy and efficiency of video content understanding and analysis.

[0074] In order to obtain the semantic representation of the entire video in the text space, in multimodal feature fusion, the mechanism of additive attention and self-attention superposition can be used to fuse different modal information in the video.

[0075] First, the additive attention mechanism is used to fuse the video action features and the video semantic object features. The specific process can be expressed as:

[0076]

[0077] Among them, z i is the fused feature vector, W1, W2, W3, b are learnable parameters, and M is the number of semantic objects.

[0078] Next, a multi-layer self-attention mechanism is used to deeply extract the fused features. For each video feature fusion result z i ∈R (s×h) , where s represents the sequence length and h represents the hidden state dimension of each word.

[0079] Then, a three-layer self-attention mechanism is used for deep feature extraction, multiplying x by three learnable weight matrices W Q ∈R h×h , W K ∈R h×h and W V ∈R h×h , get the Query, Key, and Value matrices. Use the dot product method to calculate the similarity score matrix S between Query and Key.

[0080] S=QK T (10)

[0081] The similarity matrix is ​​normalized by the Softmax function to obtain the attention weight, and the attention weight is weighted averaged with the Value matrix to obtain the output sequence H.

[0082] H=Vsoftmax(S) (11)

[0083] S103, combining the previous word encoding embedding in the description sentence and using conditional probability distribution to gradually decode the fused feature information into a natural language description corresponding to the video to be described.

[0084] The specific design can include the following contents:

[0085] First, the fused features are converted into a sequence of decoder input vectors containing time steps;

[0086] Then, based on the visual features and the previous word encoding embedding in the description sentence, the pre-trained LSTM decoder is used to predict the current word. The natural language description corresponding to the video is obtained by step-by-step decoding until a complete description sentence is generated or the predefined maximum sentence length is reached.

[0087] When training the LSTM decoder, the training loss function can be set using the cross entropy loss, and the LSTM decoder is trained and optimized based on the training loss function using the given video and video description.

[0088] The decoder is responsible for extracting the vector fused with the previous text and decoding it into text. Its training goal is to minimize the cross entropy loss to optimize the performance of the description generator.

[0089] The output characteristics of the multimodal encoder are Where s is the length of the input sequence and h is the dimension of the feature vector. A single LSTM decoder can be used to map these features to the text space. The specific process is described as follows:

[0090] First, a linear layer is used to transform the output feature H of the self-attention model into a sequence of decoder input vectors for each time step. The output is Where h is the hidden state size of the LSTM decoder. It can be expressed as follows:

[0091] Z=HW+b (12)

[0092] in is the weight matrix of the fully connected layer. is the bias vector.

[0093] Then, the decoder is initialized to an all-zero hidden state vector and the embedding vector of a single token e is the dimension of the embedding vector, and the hidden state at each time step is calculated by the following steps

[0094] h t =LSTM(x t ,h t-1 ) (13)

[0095] Among them, LSTM represents the operation of the LSTM layer, using a fully connected layer to transform the hidden state h t Map to a Dimensional vector Where v is the vocabulary. It can be expressed by the following formula:

[0096] y t =h t V+c (14)

[0097] in, is the weight matrix of the fully connected layer, is the bias vector, and then the Softmax function is used to transform y t Convert to a probability distribution p t :

[0098] p t =softmax(y t ) (15)

[0099] Next, beam search is used to select k most likely sequence outputs to optimize the output sequence, where k=5.

[0100] When choosing the cross entropy loss to optimize the description generator, given a video and its video description [w1, ..., w L ], where L is the video length. ζ(w t ) is the word w t Representation in the vocabulary. Then the loss function can be expressed as:

[0101]

[0102] Furthermore, based on the above method, an embodiment of the present invention also provides a video description generation system for semantic feature selection and attention fusion, comprising: a preprocessing module, a fusion module and an output module, wherein:

[0103] A preprocessing module, used to map the image features of the video to be described to the text space of the description sentence, and obtain the semantic object features in the video to be described;

[0104] A fusion module is used to extract action features in the video to be described based on the semantic object features, and fuse different modal feature information in the video to be described based on the attention mechanism, wherein the different modal feature information at least includes semantic object features and action features;

[0105] The output module is used to gradually decode the fused feature information into a natural language description corresponding to the video to be described.

[0106] In order to verify the effectiveness of this solution, the following is a further explanation based on experimental data:

[0107] Experiments were conducted on the MSVD (Microsoft Video Description) dataset and the MSR-VTT (Microsoft Research Video to Text) dataset. The MSVD dataset is a small-scale dataset that contains more than 1,970 YouTube video clips. Each video clip is about 10 seconds long, and each video has an average of 40 words in the description. The MSR-VTT dataset is a large-scale dataset that contains 10,000 video clips. Each video clip is about 20 seconds long, and each video has an average of 20 words in the description.

[0108] Four main evaluation indicators are used for model evaluation: BLEU (Bilingual Evaluation Understudy, a similarity measure based on precision), ROUGE (Recall-Oriented Understudy for Gisting Evaluation, a similarity measure based on recall), CIDEr (Consensus-based Image Description Evaluation, a consensus-based image description evaluation), and METEOR (Metric for Evaluation of Translation with Explicit Ordering).

[0109] The model of this case adopts Figure 2 The architecture shown in the figure adopts the idea based on target detection in the process of semantic feature selection. First, the target detection method is used to analyze the video content to extract the semantic objects worth describing, such as people, cars, animals, etc.; in the action feature extraction based on the C3D network, these semantic objects are mapped to the text space by action feature mapping, so that they can be used as natural language labels, and these natural language description sentences are used as labels for optimization to further improve the performance of the model; and based on the multi-attention mechanism of multi-modal video information fusion, the multi-modal semantic fusion method is used to fuse the outputs from the two modules of visual image and language text, and the information of the two modalities of vision and language can be used to enrich the model, so as to more accurately describe the semantic information in the video. Based on the LSTM decoder, the decoder is used to decode the fused semantic information into a natural language description.

[0110] 1. Comparative test

[0111] To further compare the model performance, the experiments compared RestNet, SAAT, ORG-TRL, Uni-perceiver and other models on the MSVD and MSR-VTT datasets, and obtained good results on the BLEU4, ROUGE, METEOR and CIDEr indicators. Figure 3 As shown in Table 1 and Table 2. Figure 3 In the figure, red fonts represent the model generation results, and black fonts represent labels.

[0112] As can be seen from Table 1, the performance of ORG-TRL with the introduction of the target detector is significantly improved compared with SAAT of the same year, which illustrates the importance of target detection features in the video description task.

[0113] Table 1 Model performance on MSVD dataset

[0114]

[0115]

[0116] Table 2 shows the performance of the model on MSR-VTT. It can be seen that all indicators on the MSR-VTT dataset have declined. This is because the video scenes contained in MSR-VTT are more diverse and the number of actions is also greater, reflecting the shortcomings of the C3D network in modeling longer information. However, overall, the present invention still achieves good results.

[0117] Table 2 Model performance on the MSR-VTT dataset

[0118]

[0119] 2. Ablation experiment

[0120] In order to verify the effectiveness of the action encoder and feature fusion module, an ablation experiment is carried out based on the MSR-VTT dataset. The specific experimental data are shown in Table 3.

[0121] Table 3 Ablation experiment results

[0122]

[0123] In the experiment, this solution directly inputs the C3D features into the feature fusion step. As can be seen from the table, without feature selection, the model inputs a large number of features that are not helpful for semantic generation, and the quality of the generated text is greatly reduced.

[0124] In terms of multimodal fusion, after being completely replaced by DNN, all indicators have dropped significantly, and the BLEU indicator is only 25.3. This shows that multimodal fusion in this case is indispensable. Later, the additive attention mechanism was replaced by DNN in the solution of this case, and the self-attention mechanism was retained, but it did not achieve good results. However, when only the self-attention layer was removed and the additive attention was retained, the BLEU indicator was only 40.2, indicating that additive attention played a good role in feature fusion, and self-attention is also an indispensable part of the feature fusion module. Figure 4 The changing trend of BLEU4 index with each training round in the ablation experiment is shown.

[0125] In order to make a qualitative analysis of multimodal fusion, the Grad-CAM tool is used in the experiment to perform a visual analysis of multimodal fusion. The results are as follows: Figure 5 As shown in the figure, the darker areas in the picture represent the areas that the model focuses on. From the figure, we can see that after deep feature extraction, the model can focus on the key parts of the visual area very well. On this basis, the decoder can generate descriptions in a targeted manner.

[0126] The above experimental data show that this solution can enhance the semantic relevance between the text description of entities in the video and the video content, effectively alleviate the semantic deviation between the visual space and the semantic space, improve the accuracy of the action encoder output, and has good application prospects in the field of automatic video description generation.

[0127] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0128] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0129] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0130] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0131] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A video description generation method for semantic feature selection and attention fusion, characterized in that: Include: Mapping the image features of the video to be described to the text space of the description sentence, and obtaining the semantic object features in the video to be described; Extract action features in the video to be described based on semantic object features, and fuse different modal feature information in the video to be described based on an attention mechanism, wherein the different modal feature information at least includes semantic object features and action features, and fuse different modal feature information in the video to be described based on the attention mechanism, including: first, using an additive attention mechanism to fuse action features and semantic object features; then, using a multi-layer attention mechanism to perform deep hierarchical feature extraction on the fused features and obtain a similarity matrix; then, using a softmax function to normalize the similarity matrix and obtain an attention weight, and performing weighted averaging on the attention weight and the deep hierarchical extracted features to obtain fused features of different modal features; Combined with the previous word encoding embedding in the description sentence and using conditional probability distribution, the fused feature information is gradually decoded into the natural language description corresponding to the video to be described.

2. The video description generation method for semantic feature selection and attention fusion according to claim 1 is characterized in that: Map the image features of the video to be described to the text space of the description sentence, including: Firstly, several frames in the video to be described are selected as key frames, and the key frames and the video frames in the adjacent areas are used as the input of the pre-trained object detector model. The object detector model is used to obtain the object detection features in the video to be described, wherein the object detection features at least include initial object features, initial action features and video frame features; Then, the encoder is used to encode and output the initial object features; and for the encoded output results, the video frame features and the initial action features are combined and the decoder is used to map the input to the description sentence text space to select and output the video semantic object features, wherein the decoder includes: an LSTM network that obtains the input data vector representation and a fully connected layer that maps the LSTM network output to the text space.

3. The video description generation method for semantic feature selection and attention fusion according to claim 2 is characterized in that: In the fully connected layer, the LSTM network output is mapped to the text space, and the network is supervised by the description sentence. The learning process includes: First, the loss function is constructed based on minimizing the distance between the entity in the description sentence and the entity in the video; Then, the description sentences are used to perform supervised learning on the LSTM network based on the loss function to learn the vector representation of the video content in the text space.

4. The video description generation method for semantic feature selection and attention fusion according to claim 1 is characterized in that: Extract action features in the video to be described based on semantic object features, including: The pre-trained C3D three-dimensional convolutional neural network is used to extract the initial action in the sample data, and the action features in the video are obtained through cross-attention fusion in combination with the semantic object features. In the pre-training of the C3D three-dimensional convolutional neural network, the initial action and action-related features are spliced ​​and the spliced ​​features are mapped to the text space using the LSTM network and the fully connected layer. The learning optimization of the neural network is achieved by minimizing the distance between the relevant action features in the video description supervision signal text and the features after splicing and mapping to the text space.

5. The video description generation method for semantic feature selection and attention fusion according to claim 1 is characterized in that: Combined with the previous word encoding embedding in the description sentence and using conditional probability distribution, the fused feature information is gradually decoded into the natural language description corresponding to the video to be described, including: First, the fused features are converted into a sequence of decoder input vectors containing time steps; Then, based on the visual features and the previous word encoding embedding in the description sentence, the pre-trained LSTM decoder is used to predict the current word. The natural language description corresponding to the video is obtained by step-by-step decoding until a complete description sentence is generated or the predefined maximum sentence length is reached.

6. The video description generation method for semantic feature selection and attention fusion according to claim 1 is characterized in that: The LSTM decoder training process includes: The training loss function is set using the cross entropy loss, and the given video and video description are used to enable the LSTM decoder to learn the mapping relationship between the input video feature sequence and the video description text space based on the training loss function.

7. A video description generation system for semantic feature selection and attention fusion, characterized in that: Contains: preprocessing module, fusion module and output module, among which, A preprocessing module, used to map the image features of the video to be described to the text space of the description sentence, and obtain the semantic object features in the video to be described; A fusion module is used to extract action features in the video to be described based on semantic object features, and fuse different modal feature information in the video to be described based on an attention mechanism, wherein the different modal feature information at least includes semantic object features and action features. The different modal feature information in the video to be described is fused based on the attention mechanism, including: first, using an additive attention mechanism to fuse the action features and the semantic object features; then, using a multi-layer attention mechanism to perform deep hierarchical feature extraction on the fused features and obtain a similarity matrix; then, using a softmax function to normalize the similarity matrix and obtain an attention weight, and performing weighted averaging on the attention weight and the deep hierarchical extraction features to obtain fused features of different modal features; The output module is used to gradually decode the fused feature information into a natural language description corresponding to the video to be described.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: at least one processor, and a memory coupled to the at least one processor; The memory stores a computer program, and the computer program can be executed by the at least one processor to implement the method according to any one of claims 1 to 6.

Images