A video description generation method and medium based on fact enhancement and emotional arousal

By constructing a three-level video description generation model and adding a deep sequence module layer by layer, the problem of insufficient emotional information in existing video description generation models is solved, and the generated sentences achieve richer emotional expression and higher accuracy.

CN117765448BActive Publication Date: 2026-11-10JINGGANGSHAN UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311848700.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-29
Publication Date
2026-11-10
Estimated Expiration
2043-12-29

AI Technical Summary

Technical Problem

Existing video description generation models lack sufficient emotional information, flexibility, vividness, and semantic richness when generating descriptive sentences.

Method used

A video description generation method based on fact enhancement and emotional arousal is adopted. By constructing a three-level model and adding deep sequence modules layer by layer, the method utilizes fact heuristic module, fact enhancement module and emotional arousal module, combined with multi-level deep sequence fusion, to generate descriptive sentences containing rich emotional semantics.

Benefits of technology

It improves the expressive power of the video description generation model, resulting in more emotional and semantically richer, more accurate, and more semantically richer description sentences.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117765448B_ABST
    Figure CN117765448B_ABST
Patent Text Reader

Abstract

This invention relates to a method and medium for generating video descriptions based on fact enhancement and emotional arousal. The method includes the following steps: acquiring visual features of the video to be described; using the visual features as input to a trained video description generation model to obtain a descriptive statement; wherein the video description generation model includes a fact-inspired module, a fact-enhanced module, and an emotional arousal module; and obtaining the descriptive statement based on the fusion result of the outputs of the fact-inspired module, the fact-enhanced module, and the emotional arousal module. Compared with existing technologies, the descriptive statements generated by this invention for videos have advantages such as higher accuracy and richer emotional semantics.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of video description generation, and in particular to a method and medium for generating video descriptions based on fact enhancement and emotional arousal. Background Technology

[0002] Video description is a fundamental task in visual-language multimodal high-level understanding research. Given a video, it involves summarizing, generalizing, and expressing its content using accurate and appropriate natural language. This task has broad application prospects in areas such as automatic narration, autonomous navigation, and the development of smart wearable devices. Currently, researchers have developed various effective models and algorithms based on deep features, continuously improving the quality of the generated descriptive sentences. However, most current work focuses on describing the factual content of the video, paying insufficient attention to the emotional information, or simply assigning sentiment labels such as "positive," "negative," and "neural." The generated sentences lack flexibility and vividness, and their semantic richness needs further improvement. Summary of the Invention

[0003] The purpose of this invention is to address the problems of insufficient emotional richness and vivid expression in current video description tasks by providing a video description generation method and medium based on fact enhancement and emotional arousal, which can naturally embed emotional semantics into the generated description sentences, thereby improving the overall quality of the generated sentences.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] A video description generation method based on fact enhancement and emotional arousal includes the following steps:

[0006] Obtain the visual features of the video to be described;

[0007] The visual features are used as input to a trained video description generation model to obtain descriptive statements;

[0008] The video description generation model includes a fact-inspired module, a fact-enhanced module, and an emotion-aware module. The description statement is obtained based on the fusion result of the outputs of the fact-inspired module, the fact-enhanced module, and the emotion-aware module. The training process of the video description generation model includes:

[0009] Using a fact-heuristic module as a sequence model, the sequence model is trained using a first dataset, which contains fact descriptions.

[0010] Based on the sequence model, a fact augmentation module is added to form an augmented model. The augmented model is trained using a second dataset, which is a dataset containing fact descriptions and has a larger data size than the first dataset.

[0011] Based on the augmented model, an emotional arousal module is added to form a three-level model. The three-level model is trained using a third dataset, which is a dataset with added emotional information based on the first dataset.

[0012] Furthermore, the fact-inspired module is a two-layer LSTM sequence model constructed using a non-factorization approach, and the model is optimized based on the language features and video features extracted from the first dataset.

[0013] Furthermore, during training, a gradient backpropagation algorithm spanning multiple time steps is used to update and optimize the parameters in the fact-inspired module.

[0014] Furthermore, the fact augmentation module is a two-layer LSTM sequence model constructed using a non-factorization approach, and the model is optimized based on the language features and video features extracted from the second dataset.

[0015] Furthermore, the emotional arousal module is a single-layer LSTM sequence model, which is optimized based on the language features extracted from the third dataset.

[0016] Furthermore, the video features are extracted using a pre-trained CNN network and obtained using linear layer dimensionality reduction.

[0017] Furthermore, the language features are encoded using a one-hot encoding method and obtained using an embedding layer for dimensionality reduction.

[0018] Furthermore, when training the augmented model, the fact-inspired module updates parameters using a fine-tuning method, while the fact-enhancing module updates parameters using a full-training method.

[0019] Furthermore, when obtaining the descriptive statement, a multi-level deep sequence fusion method is used to perform element-wise post-fusion of the probability output vector of each module in the video description generation model at each time step, and the word corresponding to the maximum value of the final probability score is used as the prediction output at that time step.

[0020] The present invention also provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the video description generation method based on fact enhancement and emotional arousal as described above.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] This invention designs a deep incremental learning strategy, which builds a video description generation model with a deeper sequence network by adding deep sequence modules layer by layer. The training dataset is augmented and sentiment-description processed, and the model is jointly optimized by adding or expanding the sample space layer by layer. This effectively improves the model's expressive power and enables it to generate descriptive sentences for videos that contain more emotional semantics and richer expressions than other similar video description models, with higher accuracy. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the main steps of training the video description generation model of the present invention;

[0024] Figure 2 This is a schematic diagram of the structure of the video description generation model of the present invention;

[0025] Figure 3 Flowchart for VD-FEEA model training;

[0026] Figure 4 Flowchart for VD-FEEA test (sentence generation). Detailed Implementation

[0027] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0028] This embodiment provides a video description generation method based on fact enhancement and emotional arousal. The method includes the following steps: acquiring the visual features of the video to be described; using the visual features as input to a trained video description generation model to obtain a descriptive statement. In this method, the video description generation model includes a fact-inspired module, a fact-enhanced module, and an emotional arousal module. The fact-inspired module is used for initial perception of factual information, the fact-enhanced module is used to acquire more comprehensive factual information, and the emotional arousal module is used to enable the model to acquire emotional information based on the factual description. Based on the fusion result of the outputs of the fact-inspired module, the fact-enhanced module, and the emotional arousal module, a final descriptive statement containing rich emotional expression is obtained.

[0029] like Figure 1 As shown, in this embodiment, the training process of the video description generation model includes:

[0030] 1) Construct a fact-heuristic module, use the fact-heuristic module as a sequence model, and train the sequence model using a first dataset. The first dataset is a dataset containing fact descriptions. In this embodiment, it is specifically a subset of fact descriptions.

[0031] 2) Based on the sequence model, a new sequence network is added to deepen the network layers and construct a fact augmentation module to form an augmented model. The augmented model is trained using a second dataset of a larger scale. During training, the fact heuristic module updates parameters using a fine-tuning method, while the fact augmentation module updates parameters using a full training method. This helps the model avoid getting stuck in local optima and achieve stronger generalization ability. The second dataset is a dataset containing fact descriptions, and its data size is larger than that of the first dataset. In this embodiment, it is specifically a subset of the fact augmentation dataset.

[0032] 3) Based on the augmented model, the network model is further deepened by adding an emotion arousal module to form a three-level model. The three-level model is trained using a third dataset so that emotional information can be naturally integrated into the factual description. The third dataset is a dataset with added emotional information based on the first dataset. In this embodiment, it is specifically an emotion description dataset.

[0033] During the testing phase, the probability outputs of the three modules are weighted and fused using the idea of ​​deep sequence fusion to collaboratively decide the word output at each time step.

[0034] refer to Figure 2 and Figure 3 As shown, the specific training process in this embodiment includes:

[0035] In the first stage of training, S1, the model is trained using a fact-heuristic subset, which enables the model to acquire heuristic factual information.

[0036] To separate the modeling of language and vision, a two-layer LSTM sequence model is constructed using an improved factorization structure at this stage. For language features, each word, after feature embedding, is fed into the bottom-layer LSTM network to model the temporal structure of the language. Meanwhile, the top-layer LSTM receives the input... <pad>To maintain alignment; once all lexical features have entered the network, the decoding stage begins, and the underlying LSTM units are then input. <pad>To maintain alignment, video features are fed into a top-level LSTM to encode the video's dynamic features, while word prediction is performed at each time step until all video frames or time steps are exhausted. At each time step, the output vocabulary is compared with the input vocabulary, the fact-heuristic error E1 is calculated, and the parameters in the fact-heuristic module (including the sequence network, vocabulary feature embedding layer, and visual feature reduction layer) are updated and optimized using the Back Propagated Through Time (BPTT) algorithm.

[0037] The extraction of the aforementioned linguistic and visual features specifically includes:

[0038] 11) Use a CNN network pre-trained on the ImageNet network and the MSCOCO dataset to extract deep visual features from videos in the fact-inspired dataset, and use a linear layer to reduce the dimensionality of the visual features (visual feature reduction layer);

[0039] 12) Encode each word in the video description sentences in the fact-inspired dataset using the one-hot encoding method, and use a word embedding layer to reduce the dimensionality of the language features.

[0040] In the second stage, S2, the fact-inspired subset is expanded (e.g., by directly using the MSVD training set) to obtain the fact-enhanced subset. Simultaneously, the existing lexical feature embedding layer and visual feature reduction layer are retained, and a new two-layer LSTM network is constructed, adding a fact-enhanced sequence module and its corresponding objective function to the fact-inspired sequence module. To fully extract linguistic information, this module uses a non-factorization approach to construct the sequence model. Its bottom-layer LSTM only receives the hidden layer output from the fact-inspired module; visual features are not directly input into this network. For linguistic features, they are combined with the output of the bottom-layer LSTM and fed into the top-layer LSTM. In this stage, a joint optimization approach is used to train the two modules, resulting in more thorough optimization of the lower-layer fact-inspired module, while the higher-layer fact-enhanced module enables the entire model to acquire more abstract and expressive multimodal features.

[0041] In actual training operations, let the current iteration be the I-th iteration of the model, which includes two sub-iterations (let them be I-1 and I-2 respectively). tmp And I), its update formula can be expressed as:

[0042]

[0043] Where α is the learning rate set for the entire model during training, O af (·) and O ef (·) represent the objective functions of the fact-inspired module and the fact-enhancing module, respectively. λ af and λ ef Let λ be the harmonic factor of the two objective functions. Generally speaking, λ ef ≡1, while λ af <<λ ef Θ S1 The model parameters in the first stage S1 include the vocabulary embedding layer, the visual feature reduction layer, and the sequence model parameters, i.e., Θ. S1 =(Θ emb ,Θ re ,Θ seq1 ). (Θ S1 ,Θ seq2 ) indicates that it is in Θ S1 A new sequence module parameter, denoted as Θ, has been added to the existing parameters. S2 .

[0044] In the second sub-iteration, although O was used ef The (·) function updates the parameters, but still uses the harmonic factor λ from the fact-heuristic module. af Limit its update rate. During this phase, for parameter Θ′ in the fact enhancement module... S2 =Θ seq2 Then use the following formula to update:

[0045]

[0046] At this time, Θ S2 =(Θ emb ,Θ re ,Θ seq1 ,Θ seq2 ).

[0047] In the second stage, S3, the model is further deepened based on the fact-enhancing module by constructing an emotion-awareness module. In this stage, the sentiment description dataset uses the same video and fact-inspired subsets, but the factual descriptions are rewritten, incorporating emotional information to make the descriptions more flexible and engaging. The emotion-awareness module in this stage only requires a single LSTM layer to model the language, allowing the model to fully utilize factual information and naturally embed emotional information into the descriptive sentences. Joint optimization is used to update the model parameters in this stage.

[0048] In this phase, for the fact-heuristic module, each iteration includes three sub-iterations (let the three sub-iterations be...). And I), the parameters therein will also be updated three times, and its formula can be expressed as:

[0049]

[0050] Among them O ea (·) represents the objective function on the emotional arousal module.

[0051] For the fact-enhancing sequence module in this stage, each iteration contains two sub-iterations (let them be I and I). tmp And I), where the parameters will also be updated twice, and its formalization process can be expressed as:

[0052]

[0053] In the emotional arousal module, its parameter set Θ′ S3 The update formula for in one iteration is:

[0054]

[0055] Where Θ′ S3 =Θ seq3 .

[0056] refer to Figure 3 As shown, after three stages of expansion and training, the model is tested in the testing phase by using a multi-level deep sequence fusion method to perform element-wise late-stage fusion of the probability output vector of each module at each time step. The word corresponding to the maximum value of the final probability score is used as the prediction output at that time step.

[0057] To verify the performance of the above method, the following experiment was designed in this embodiment.

[0058] Six models were used as comparison models, as detailed below:

[0059] (1) Baseline(S): The baseline model trained on EmVidCap-S using the improved S2VT model.

[0060] (2) Baseline: The baseline model trained on EmVidCap using the improved S2VT model.

[0061] (3) VDFEEA2(S): The comparison model trained using EmVidCap-S in the second stage.

[0062] (4) VDFEEA2: Another pair of models trained using EmVidCap in the second stage.

[0063] (5) VDFEEA3(S): The comparison model trained using EmVidCap-S in the third stage.

[0064] (6) VDFEEA3: A comparative model trained using EmVidCap in the third stage.

[0065] The performance of each model on the EmVidCap-S and EmVidCap datasets using traditional metrics (BLEU-4 (B4), METEOR (M), ROUGE-L (R), and CIDEr (C)) is shown in Tables 1 and 2. The comparison results show that the VD-FEEA3 (S) and VD-FEEA3 models designed in this invention achieve superior performance on multiple metrics compared to the corresponding baseline models Baseline (S) / Baseline and the comparison models VD-FEEA2 (S) / VD-FEEA2. When the dataset is smaller, merging the fact-inspired stage and the fact-enhanced stage has a smaller impact on the overall model performance; the three-stage model construction approach and optimization strategy proposed in this invention can also effectively enhance the expressive power of multi-modal features and significantly improve the semantics of the generated sentences. On larger datasets, the model designed in this invention exhibits stronger generalization ability, indicating that the fact-inspired stage training plays a crucial role in improving the expressive power of subsequent multi-modal features, enhancing the model's generalization ability, and training the subsequent fact-enhanced and emotion-awakening stages.

[0066] Table 1. Performance of each model on the EmVidCap-S dataset (test set) (traditional metrics)

[0067] Model B4 M R C Baseline(S) 29.6 26.8 59.3 49.1 <![CDATA[VD-FEEA2(S)]]> 38.8 30.3 65.3 65.6 <![CDATA[VD-FEEA3(S)]]> 38.5 29.1 63.4 68.4

[0068] Table 2 shows the performance of each model on the EmVidCap dataset (test set) using traditional metrics.

[0069] Model B4 M R C Baseline 21.0 20.9 42.9 28.3 <![CDATA[VD-FEEA2]]> 22.7 20.1 43.8 33.2 <![CDATA[VD-FEEA3]]> 23.7 21.1 45.3 33.7

[0070] Each model was tested using the sentiment rating metric (ACC) on both the EmVidCap-S and EmVidCap datasets. sw ACC c The performance of the VD-FEEA3(S) model when evaluated using ACC and comprehensive evaluation metrics (BFS, CFS) is shown in Tables 3 and 4. The results show that on the EmVidCap-S dataset, the VD-FEEA3(S) model designed in this invention achieves high ACC. sw ACC c Both the accuracy of sentiment words and the CFS metric surpassed the Baseline(S) model, with significant improvements in the accuracy of sentiment words and semantics of generated sentences. Compared to VD-FEEA2(S), VD-FEEA3(S) also showed improvements in the accuracy of sentiment words and sentiment sentences. This indicates that although the training data is consistent, different training and fusion strategies have a significant impact on the model's sentiment mining and expression. The fact-inspired training strategy and three-level probabilistic fusion method designed in this invention can effectively improve the accuracy of sentiment in generated sentences. Regarding the CFS metric, the ACC of the VD-FEEA3(S) model is significantly lower than that of the Baseline(S) model. sw ACC c Both the CIDEr and CIDEr metrics of the VD-FEEA2(S) model are superior to those of the VD-FEEA2(S) model, thus its CFS performance improvement is also significant. On the EmVidCap dataset, the VD-FEEA3 model outperforms the VD-FEEA2(S) model in ACC. sw ACC c In terms of metrics such as BFS and CFS, its performance has also improved compared to other comparative models.

[0071] Table 3 shows the performance of each model on the EmVidCap-S dataset (test set) (sentiment rating and overall rating metrics).

[0072] Model <![CDATA[ACC sw ]]> <![CDATA[ACC c ]]> BFS CFS Baseline(S) 69.8 68.6 47.0 53.1 <![CDATA[VD-FEEA2(S)]]> 76.7 73.8 55.2 67.5 <![CDATA[VD-FEEA3(S)]]> 80.8 79.8 55.9 70.0

[0073] Table 4 shows the performance of each model on the EmVidCap dataset (test set) (sentiment rating and overall rating metrics).

[0074] Model <![CDATA[ACC sw ]]> <![CDATA[ACC c ]]> BFS CFS Baseline(S) 52.7 52.2 36.9 33.3 <![CDATA[VD-FEEA2(S)]]> 46.8 44.7 37.7 35.7 <![CDATA[VD-FEEA3(S)]]> 54.6 52.5 40.1 37.7

[0075] If the above methods are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.< / pad> < / pad>

Claims

1. A video description generation method based on fact enhancement and emotional arousal, characterized in that, The method includes the following steps: Obtain the visual features of the video to be described; The visual features are used as input to a trained video description generation model to obtain descriptive statements; The video description generation model includes a fact-inspired module, a fact-enhanced module, and an emotion-aware module. The description statement is obtained based on the fusion result of the outputs of the fact-inspired module, the fact-enhanced module, and the emotion-aware module. The training process of the video description generation model includes: Using a fact-heuristic module as a sequence model, the sequence model is trained using a first dataset, which contains fact descriptions. Based on the sequence model, a fact augmentation module is added to form an augmented model. The augmented model is trained using a second dataset, which is a dataset containing fact descriptions and has a larger data size than the first dataset. Based on the augmented model, an emotional arousal module is added to form a three-level model. The three-level model is trained using a third dataset, which is a dataset with added emotional information based on the first dataset. The fact-heuristic module is a two-layer LSTM sequence model constructed using a non-factorization approach, and the model is optimized based on the language features and video features extracted from the first dataset. The fact augmentation module is a two-layer LSTM sequence model constructed using a non-factorization approach, and the model is optimized based on the language features and video features extracted from the second dataset. The emotional arousal module is a single-layer LSTM sequence model, which is optimized based on the language features extracted from the third dataset. When obtaining the description statement, a multi-level deep sequence fusion method is used to perform element-wise post-fusion of the probability output vector of each module in the video description generation model at each time step, and the word corresponding to the maximum value of the final probability score is used as the prediction output at that time step.

2. The video description generation method based on fact enhancement and emotional arousal according to claim 1, characterized in that, During training, a gradient backpropagation algorithm spanning multiple time steps is used to update and optimize the parameters in the fact-inspired module.

3. The video description generation method based on fact enhancement and emotional arousal according to claim 1, characterized in that, The video features were extracted using a pre-trained CNN network and then obtained using linear layer dimensionality reduction.

4. The video description generation method based on fact enhancement and emotional arousal according to claim 1, characterized in that, The language features are encoded using a one-hot encoding method and obtained using an embedding layer for dimensionality reduction.

5. The video description generation method based on fact enhancement and emotional arousal according to claim 1, characterized in that, When training the augmented model, the fact-inspired module updates parameters using a fine-tuning method, while the fact-enhancing module updates parameters using a full training method.

6. A computer-readable storage medium, characterized in that, Includes one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the video description generation method based on fact enhancement and emotional arousal as described in any one of claims 1-5.