Dual-controllable video description method and system based on prompt question uncertainty distribution

Abstract

The invention belongs to the technical field of video description, and particularly relates to a double-controllable video description method and system based on prompt question sentence uncertainty allocation, and the method comprises the steps: firstly constructing and training a question-answer double-controllable video caption model; according to the model, firstly, a multi-level feature extractor is used for converting an input video sequence into multi-level semantic features, a plurality of natural language question prompt templates are designed and converted into marks, the marks and entity feature sets in the multi-level semantic features are combined and coded, and question feature representation is generated; selectively fusing the question feature representation with a predicate feature set and a global feature set in the multi-level semantic features; and inputting the new predicate feature set, the new global feature set and the entity feature set obtained by selective fusion into a subtitle decoder for fusion decoding to generate subtitles. The subtitle generation tendency of the self-adaptive adjustment model can be prompted according to the preset problem, and the subtitle generation accuracy is improved by deeply fusing the multi-granularity features.

Description

Technical Field

[0001] This invention belongs to the field of video description technology, specifically relating to a dual-controllable video description method and system based on the uncertainty allocation of prompt questions. Background Technology

[0002] Video captioning (VC) builds a visual-semantic cross-modal bridge by generating accurate text descriptions, supporting intelligent retrieval and interactive applications. With the massive emergence of short video data, traditional manual annotation systems are failing in both cost and efficiency. Developing efficient automated description algorithms has become a key breakthrough in overcoming technological bottlenecks and reconstructing the digital content ecosystem, possessing significant strategic value.

[0003] Video description tasks involve receiving a complete video, processing its features through an encoder, and then a decoder parsing the corresponding natural language subtitles from the feature space to summarize the video content. Existing methods can be categorized into LSTM-based, Transformer-based, and controllable signal-guided methods. These methods all extract appearance and motion features from the complete video and align the text with the video features to achieve video understanding. However, in controllable video description tasks, focusing solely on appearance and motion features is insufficient to generate customized subtitles that meet human expectations. Therefore, some methods use sentence length signals to control the length of generated subtitles. Furthermore, other researchers have set up bounding boxes to guide the model to describe selected regions in detail. Some researchers have even designed abstract semantic graphs containing objects, attributes, and relationships from a syntactic structure perspective. Given different numbers of abstract nodes in the model, descriptive sentences that strictly adhere to the abstract semantic structure can be generated.

[0004] Video captioning aims to generate natural language descriptions from video content through hierarchical architectures that collaboratively capture key visual elements. While entities, predicates, and syntactic structures are crucial for coherent descriptions, existing controllable methods often struggle to dynamically emphasize salient information due to their reliance on predefined entity lists and static fusion strategies. Existing controllable video captioning is implemented through multi-stage template-filled descriptions, which imposes rigid constraints on the controllable generation of video subtitles, limiting the controllability of the multi-stage framework. Furthermore, uncertainties in multimodal feature alignment (such as visual actions and textual semantics) degrade model performance. Summary of the Invention

[0005] The purpose of this invention is to address the aforementioned problems in the existing technology by providing a dual-controllable video description method and system based on the uncertainty allocation of prompt questions, which can adaptively adjust the model's tendency to generate subtitles according to preset prompts and can deeply integrate multi-granular features to improve the accuracy of generated subtitles.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] In a first aspect, the present invention provides a dual-controllable video description method based on the allocation of uncertainty in prompting questions, the method comprising:

[0008] S1. Construct and train a question-and-answer dual-controllable video subtitle model; including:

[0009] S11. Construct a multi-level feature extractor to convert the input video sequence into multi-level semantic features, which include a set of entity features. Predicate feature set Global feature set ;

[0010] S12. Design multiple natural language question prompt templates and convert them into tags. , mark The entity feature set obtained with S1 Combine encoding to generate problem feature representations The obtained problem features are represented The predicate feature sets obtained by S1 respectively Global feature set Selective fusion is performed to obtain a new set of predicate features. New global feature set ;

[0011] S13, Set entity features New predicate feature set New global feature set The input is fed into a subtitle decoder for fusion decoding to generate... Time Capsule;

[0012] S14. Design the model loss function and implement model training based on the model loss function;

[0013] S2. Use a trained question-and-answer dual-controllable video subtitle model to generate subtitles.

[0014] In S11, the multi-level feature extractor includes an entity feature extraction module, a predicate feature extraction module, and a global feature extraction module;

[0015] The entity feature extraction module detects all objects from the keyframes of the input video sequence as an initial object sequence, and inputs it into the Transformer encoder to obtain object representations. Next, the set of appearance features extracted from the keyframes will be used with a pre-trained 2D CNN. Compared to the set of motion features extracted from keyframes using a pre-trained 3D CNN After concatenation, the data is input into a bidirectional LSTM, and the embedded video content is calculated through max pooling. ; then, The same video content vector Random initialization Query parameters and object representation They are input together into the Transformer's decoder to decode the output. A set of entity features related to the video :

[0016] ;

[0017] In the above formula, For the decoder of Transformer; For entity feature set The first in Individual entity characteristics;

[0018] The predicate feature extraction module first uses a soft attention mechanism to calculate and combine the motion feature set extracted by the pre-trained 3D CNN. Related set of object-perceptual motion features ,in Indicates the first Object-aware motion features of keyframes The number of keyframes in the video sequence; then a BiLSTM network is used to set the object-aware motion features. With motion feature set Encode together as a set of predicate features :

[0019] ;

[0020] In the above formula, For the reason and The encoded result of the first Each predicate feature;

[0021] The global feature extraction module first uses a soft attention mechanism to calculate and combine the appearance feature set. Related set of object-aware appearance features and motion-aware appearance feature set ,in , The first Object-aware appearance features and action-aware appearance features are collected from each keyframe; then, a BiLSTM network is used to aggregate the appearance features. Object-aware appearance feature set With motion-aware appearance feature set Co-encode global feature set :

[0022] ;

[0023] In the above formula, global feature set The first in A global feature.

[0024] The set of object-perceived motion features is calculated using the following formula. The first in Object-aware motion features of keyframes :

[0025] ;

[0026] ;

[0027] ;

[0028] In the above formula, For the set of motion features The Middle Motion features of keyframes; Represents the set of entity features The first in Individual entity characteristics; It is a soft attention function; For entity feature set Size; , , , All are learnable parameters; It is the unnormalized raw attention score, used to measure and The strength of the association; It is an exponential function, used to... Mapped to positive numbers; It is the hyperbolic tangent activation function.

[0029] In S12, the text encoder of the BLIP model is used to process the tags. The entity feature set obtained with S1 Combine encoding to generate problem feature representations The text encoder consists of a bidirectional self-attention module, a cross-attention module, and a feedforward network layer, which processes the tags... First, a bidirectional self-attention module is used to encode information within the text modality. Then, a cross-attention module is used to capture the interactions between modalities. Finally, a feedforward network layer is used to refine the features and generate a problem feature representation. The expression for the text encoder is as follows:

[0030] ; ;

[0031] ; ;

[0032] ; ;

[0033] In the above formula, , , These represent the bidirectional self-attention module, the cross-attention module, and the feedforward network layer, respectively. , , These are the feature representations obtained after processing by the bidirectional self-attention module, the cross-attention module, and the feedforward network layer, respectively. , All are intermediate feature representations; It is a set of entity features.

[0034] In S12, choosing any of the following strategies will yield the problem feature representation. The predicate feature sets obtained by S1 respectively Global feature set Selective fusion can be performed using: element-wise additive fusion strategy, element-wise multiplicative fusion strategy, and weighted fusion strategy.

[0035] The weighted fusion strategy refers to introducing a Dirichlet distribution to calculate the problem feature representations separately. Predicate feature set Global feature set The dynamic weights are then used for weighted fusion according to the following formula:

[0036] ;

[0037] ;

[0038] In the above formula, , , Each is a set of predicate features. Global feature set Problem feature representation The dynamic weights and fusion weights are obtained by first mapping the corresponding features to a probability distribution through a linear projection layer to obtain the Dirichlet parameters, and then normalizing the Dirichlet parameters.

[0039] In S13, a language model is introduced, and the subtitle decoder is... The subtitles generated before the specified time are input into the language model, and the language model then processes them. Predicting timeline captions;

[0040] In S14, the model loss function includes the distance loss of the multi-level feature extractor, the cross-entropy loss of the subtitle decoder, and the KL divergence between the subtitle decoder and the language model. The distance loss of the multi-level feature extractor includes the distance loss of the entity feature extraction module, the predicate feature extraction module, and the global feature extraction module.

[0041] The expression for the model loss function is:

[0042] ;

[0043] ;

[0044] ;

[0045] In the above formula, The model loss function; , , These are the distance losses for the entity feature extraction module, the predicate feature extraction module, and the global feature extraction module, respectively. , These are the cross-entropy loss of the subtitle decoder and the KL divergence between the subtitle decoder and the language model, respectively. , , , , These are the weights corresponding to the loss; Indicates real subtitles The Middle one word One-hot encoding, This refers to the length of the subtitle; For subtitle decoders in Real-time generated subtitles; Resulting from language model predictions Time Capsule; This represents the KL divergence.

[0046] The formula for calculating the distance loss of the entity feature extraction module is as follows:

[0047] ;

[0048] ;

[0049] In the above formula, This represents the set of entity features. The Middle The vector representation generated by projecting entity features onto the linguistic semantic space This represents the first element in the object embedding set calculated from the actual subtitle object. Each object embedding, the collection of object embeddings uses an empty set. Fill to the set of entity features The sets of vectors generated by projecting onto the language semantic space are of equal size;

[0050] The formula for calculating the distance loss of the predicate feature extraction module is as follows:

[0051] ;

[0052] In the above formula, This represents the embedding vector obtained by encoding the predicates in the actual subtitles; Represents the set of features of the predicate The text semantic space is obtained by projecting the aggregated features through max pooling and then through a fully connected layer.

[0053] The formula for calculating the distance loss of the global feature extraction module is as follows:

[0054] ;

[0055] In the above formula, This represents the embedding vector obtained by encoding the entire real subtitle. Represents the global feature set After performing max pooling to aggregate features, the text semantic space is obtained by projecting through a fully connected layer.

[0056] Secondly, the present invention provides a dual-controllable video description system based on the allocation of uncertainty in prompting questions, the system comprising a model building module, a model training module, and a generation module;

[0057] The model building module is used to build a question-and-answer dual-controllable video subtitle model according to the following steps:

[0058] A multi-level feature extractor is constructed to convert the input video sequence into multi-level semantic features, which include a set of entity features. Predicate feature set Global feature set ;

[0059] Design multiple natural language question prompt templates and convert them into tags. , mark The entity feature set obtained with S1 Combine encoding to generate problem feature representations The obtained problem features are represented The predicate feature sets obtained by S1 respectively Global feature set Selective fusion is performed to obtain a new set of predicate features. New global feature set ;

[0060] Entity feature set New predicate feature set New global feature set The input is fed into a subtitle decoder for fusion decoding to generate... Time Capsule;

[0061] The model training module is used to design the model loss function and train the question-answering dual-controllable video subtitle model based on the model loss function;

[0062] The generation module is used to generate subtitles using a trained question-and-answer dual-controllable video subtitle model.

[0063] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0064] 1. In the dual-controllable video description method of the present invention, firstly, for fixed problems of different generated objects, after the video information is parsed into static features and motion features, the problem is mapped to the feature space by an encoder. During the text encoding process, visual features can be selectively introduced to further enhance the correlation between problem features and visual features. Subsequently, the problem features are selectively fused with global features or predicate features to deeply fuse multi-granular features, ultimately improving the accuracy and bias of the generated subtitles.

[0065] 2. In the dual controllable video description method of the present invention, the uncertainty in weight allocation is modeled based on the Dirichlet distribution, and dynamic weights are calculated to dynamically highlight features with high semantic relevance while suppressing irrelevant information, thereby improving the reliability of the model and its robustness against noise or degradation. Attached Figure Description

[0066] Figure 1 This is a flowchart of the dual controllable video description method described in this invention.

[0067] Figure 2This is a schematic diagram of the structure of the question-and-answer dual-controllable video subtitle model constructed in this invention.

[0068] Figure 3 This is a structural block diagram of the dual controllable video description system described in this invention. Detailed Implementation

[0069] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings.

[0070] Example 1:

[0071] See Figure 1 A dual-controllable video description method based on prompt question uncertainty allocation, the method comprising:

[0072] S1. Construct and train a question-and-answer dual-controllable video subtitle model; the structural diagram of the question-and-answer dual-controllable video subtitle model is shown below. Figure 2 As shown, the model building steps are as follows:

[0073] S11. Construct a multi-level feature extractor to convert the input video sequence into multi-level semantic features, which include a set of entity features. Predicate feature set Global feature set .

[0074] Specifically, the multi-level feature extractor includes an entity feature extraction module, a predicate feature extraction module, and a global feature extraction module;

[0075] The entity feature extraction module detects all objects from the keyframes of the input video sequence as an initial object sequence, and inputs it into the Transformer encoder to obtain object representations. Next, the set of appearance features extracted from the keyframes will be used with a pre-trained 2D CNN. Compared to the set of motion features extracted from keyframes using a pre-trained 3D CNN After concatenation, the data is input into a bidirectional LSTM, and the embedded video content is calculated through max pooling. ; then, The same video content vector Random initialization Query parameters and object representation They are input together into the Transformer's decoder to decode the output. A set of entity features related to the video :

[0076] ;

[0077] In the above formula, For the decoder of Transformer; For entity feature set The first in Individual entity characteristics;

[0078] The initial object sequence is obtained through a three-step process: keyframe sampling, object region extraction, and feature clustering and pooling. Specifically, all objects are first detected from the video keyframes, and then clustering and deduplication are performed using similarity and overlap. The resulting object feature sequence provides the basic input for subsequent accurate object selection (selecting the objects most likely to be mentioned in the subtitles).

[0079] The predicate feature extraction module first uses a soft attention mechanism to calculate and combine the motion feature set extracted by the pre-trained 3D CNN. Related set of object-perceptual motion features ,in Indicates the first Object-aware motion features of keyframes The number of keyframes in the video sequence; then a BiLSTM network is used to set the object-aware motion features. With motion feature set Encode together as a set of predicate features :

[0080] ;

[0081] In the above formula, For the reason and The encoded result of the first Each predicate feature;

[0082] The global feature extraction module first uses a soft attention mechanism to calculate and combine the appearance feature set. Related set of object-aware appearance features and motion-aware appearance feature set ,in , The first Object-aware appearance features and action-aware appearance features are collected from each keyframe; then, a BiLSTM network is used to aggregate the appearance features. Object-aware appearance feature set With motion-aware appearance feature set Co-encode global feature set :

[0083] ;

[0084] In the above formula, global feature set The first in A global feature.

[0085] The set of object-perceived motion features is calculated using the following formula. The first in Object-aware motion features of keyframes :

[0086] ;

[0087] ;

[0088] ;

[0089] In the above formula, For the set of motion features The Middle Motion features of keyframes; Represents the set of entity features The first in Individual entity characteristics; It is a soft attention function; For entity feature set Size; , , , All are learnable parameters; It is the unnormalized raw attention score, used to measure and The strength of the association; It is an exponential function, used to... Mapped to positive numbers; It is the hyperbolic tangent activation function.

[0090] S12. Design multiple natural language question prompt templates and use BLIP Tokenizer to convert them into tokens. , mark The entity feature set obtained with S1 Combine encoding to generate problem feature representations The obtained problem features are represented The predicate feature sets obtained by S1 respectively Global feature set Selective fusion is performed to obtain a new set of predicate features. New global feature set .

[0091] Natural language prompt templates are natural language forms of prompt questions, designed according to specific needs. For example, if the goal is to guide the model to understand the current task, the prompt question could be "What is this video currently describing?". If the goal is to guide the model to understand the main object, the prompt question could be "What is the main object and what is the information in the video?". If the goal is to generate captions highlighting actions, the prompt question could be "What is the obvious action in the video?".

[0092] Specifically, the text encoder of the BLIP model will process the tags. The entity feature set obtained with S1 Combine encoding to generate problem feature representations The text encoder consists of a bidirectional self-attention module, a cross-attention module, and a feedforward network layer, which processes the tags... First, a bidirectional self-attention module is used to encode information within the text modality. Then, a cross-attention module is used to capture the interactions between modalities. Finally, a feedforward network layer is used to refine the features and generate a problem feature representation. The expression for the text encoder is as follows:

[0093] ; ;

[0094] ; ;

[0095] ; ;

[0096] In the above formula, , , These represent the bidirectional self-attention module, the cross-attention module, and the feedforward network layer, respectively. , , These are the feature representations obtained after processing by the bidirectional self-attention module, the cross-attention module, and the feedforward network layer, respectively. , All are intermediate feature representations; It is a set of entity features.

[0097] Specifically, choosing any of the following strategies will yield the problem feature representation. The predicate feature sets obtained by S1 respectively Global feature set Selective fusion can be performed using: element-wise additive fusion strategy, element-wise multiplicative fusion strategy, and weighted fusion strategy.

[0098] Specifically, the calculation formula for the element-wise additive fusion strategy is as follows:

[0099] ;

[0100] ;

[0101] The calculation formula for the element-wise multiplication fusion strategy is as follows:

[0102] ;

[0103] ;

[0104] Specifically, the weighted fusion strategy refers to introducing a Dirichlet distribution to calculate the problem feature representation separately. Predicate feature set Global feature set The dynamic weights are then used for weighted fusion according to the following formula:

[0105] ;

[0106] ;

[0107] In the above formula, , , Each is a set of predicate features. Global feature set Problem feature representation The dynamic weights and fusion weights are obtained by first mapping the corresponding features to a probability distribution through a linear projection layer to obtain the Dirichlet parameters, and then normalizing the Dirichlet parameters.

[0108] S13, Set entity features New predicate feature set New global feature set The input is fed into a subtitle decoder for fusion decoding to generate... Real-time captions; simultaneously, a language model is introduced, integrating the caption decoder into... The subtitles generated before the specified time are input into the language model, and the language model then processes them. Predictive timings using captions.

[0109] Specifically, the subtitle decoder uses LSTM for fusion decoding, and the expression for this process is as follows:

[0110] ;

[0111] ;

[0112] in Embedding words from the previous moment; This is the hidden state from the previous moment; All were obtained through soft attention:

[0113] ;

[0114] Hidden state of LSTM output After a fully connected layer and softmax operation, we obtain Time Capsule;

[0115] S14. Design the model loss function and implement model training based on the model loss function.

[0116] Specifically, the model loss function includes the distance loss of the multi-level feature extractor, the cross-entropy loss of the subtitle decoder, and the KL divergence between the subtitle decoder and the language model. The distance loss of the multi-level feature extractor includes the distance losses of the entity feature extraction module, the predicate feature extraction module, and the global feature extraction module. Cross-entropy loss is used to transfer rich linguistic knowledge to the subtitle decoder, and the KL divergence between the subtitle decoder and the language model is used to address the long-tail word distribution problem.

[0117] The expression for the model loss function is:

[0118] ;

[0119] ;

[0120] ;

[0121] In the above formula, The model loss function; , , These are the distance losses for the entity feature extraction module, the predicate feature extraction module, and the global feature extraction module, respectively. , These are the cross-entropy loss of the subtitle decoder and the KL divergence between the subtitle decoder and the language model, respectively. , , , , These are the weights corresponding to the loss; Indicates real subtitles The Middle one word One-hot encoding, This refers to the length of the subtitle; For subtitle decoders in Real-time generated subtitles; Resulting from language model predictions Time Capsule; This represents the KL divergence.

[0122] The formula for calculating the distance loss of the entity feature extraction module is as follows:

[0123] ;

[0124] ;

[0125] In the above formula, This represents the set of entity features. The Middle The vector representation generated by projecting entity features onto the linguistic semantic space This represents the first element in the object embedding set calculated from the actual subtitle object. Each object embedding, the collection of object embeddings uses an empty set. Fill to the set of entity features The sets of vectors generated by projecting onto the language semantic space are of equal size;

[0126] The formula for calculating the distance loss of the predicate feature extraction module is as follows:

[0127] ;

[0128] In the above formula, This represents the embedding vector obtained by encoding the predicates in the actual subtitles; Represents the set of features of the predicate The text semantic space is obtained by projecting the aggregated features through max pooling and then through a fully connected layer.

[0129] The formula for calculating the distance loss of the global feature extraction module is as follows:

[0130] ;

[0131] In the above formula, This represents the embedding vector obtained by encoding the entire real subtitle. Represents the global feature set After performing max pooling to aggregate features, the text semantic space is obtained by projecting through a fully connected layer.

[0132] S2. Use a trained question-and-answer dual-controllable video subtitle model to generate subtitles.

[0133] Performance verification:

[0134] On the MSR-VTT video description benchmark dataset, BLEU-4 (B-4), METEOR (M), ROUGE (R), and CIDEr (C) were used as evaluation metrics to evaluate the performance of the question-answering dual-controllable video captioning model (QPDC) described in this invention compared with other existing models. MSR-VTT includes 6513 training clips, 497 validation clips, and 2990 test clips (each 15 seconds), each with 20 manually annotated captions covering 20 categories. BLEU is a classic text quality assessment tool that determines the similarity between generated content and standard text by analyzing the degree of overlap in N-gram syntax. METEOR is a text generation evaluation metric that simultaneously focuses on precision and recall. CIDEr is specifically designed for visual description evaluation and is the most important metric for video description tasks. Loss function weights. , , , , The values ​​were set to 0.6, 0.3, 1.0, 0.5, and 0.5 respectively. The Adam optimizer was used with a learning rate of 1×10⁻⁶. -4 The batch size was 64, and training lasted for 20 epochs. The decoder hidden size was fixed at 512. Beam search with a beam size of 5 was applied during inference. All experiments were performed on a PyTorch system equipped with an RTX-3090Ti (24GB). The results are shown in Table 1.

[0135] Table 1. Performance comparison of the proposed QPDC with other models on MSR-VTT.

[0136]

[0137] As shown in Table 1, the QPDC proposed in this invention achieves high scores on all metrics (BLEU-4, METEOR, ROUGE, and CIDEr). Particularly on CIDEr, the proposed QPDC improves upon O2NA by +1.4% and MSG by +1.2%. These results verify the effectiveness of the proposed QPDC in achieving dynamic multi-granularity content control through question-guided prompts and uncertainty modeling, thereby improving model performance in both standard caption generation and question-conditional caption generation scenarios.

[0138] The dual-controllable video description method of this invention employs a unified question-guided strategy to maintain consistency while enabling flexible attention shifts. Furthermore, it designs a question-prompt mechanism and a selective fusion module (FIM) for uncertainty modeling, forming a dual-controllable Question-Fusion mechanism that adaptively guides the focus of subtitles at multiple granular levels. The question-prompt mechanism dynamically guides the model's attention based on the question's intent, emphasizing salient content without being constrained by predefined limitations. The selective fusion module hierarchically integrates question guidance with local action features and global sentence context, achieving multi-granular control. Simultaneously, it utilizes Dirichlet distribution to model the weight distribution between text and visual features, thereby capturing multimodal uncertainty and dynamically adapting to different data conditions. This dual-controllable video description method can adaptively adjust the model's subtitle generation tendency based on preset question prompts, deeply fusing multi-granular features, thus constructing a novel paradigm for controllable video description methods.

[0139] Example 2:

[0140] See Figure 3 A dual-controllable video description system based on question-and-answer uncertainty allocation is disclosed. The system includes a model building module, a model training module, and a generation module. The model building module constructs a question-and-answer dual-controllable video caption model according to the following steps: constructing a multi-level feature extractor; using the multi-level feature extractor to convert the input video sequence into multi-level semantic features, wherein the multi-level semantic features include a set of entity features. Predicate feature set Global feature set Design multiple natural language question prompt templates and convert them into tags. , mark The entity feature set obtained with S1 Combine encoding to generate problem feature representations The obtained problem features are represented The predicate feature sets obtained by S1 respectively Global feature set Selective fusion is performed to obtain a new set of predicate features. New global feature set ; Set of entity features New predicate feature set New global feature set The input is fed into a subtitle decoder for fusion decoding to generate... The model building module is used to execute S11-S13 as described in Example 1, which will not be repeated here. The model training module is used to design the model loss function and train the question-and-answer dual-controllable video subtitle model based on the model loss function. Specifically, the model training module is used to execute S14 as described in Example 1, which will not be repeated here. The generation module is used to generate subtitles using the trained question-and-answer dual-controllable video subtitle model. Specifically, the model training module is used to execute S2 as described in Example 1, which will not be repeated here.

[0141] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program goods. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A dual-controllable video description method based on uncertain allocation of prompting questions, characterized in that: The method includes: S1. Construct and train a question-and-answer dual-controllable video subtitle model; including: S11. Construct a multi-level feature extractor to convert the input video sequence into multi-level semantic features, which include a set of entity features. Predicate feature set Global feature set ; S12. Design multiple natural language question prompt templates and convert them into tags. , mark The entity feature set obtained with S1 Combine encoding to generate problem feature representations The obtained problem features are represented The predicate feature sets obtained by S1 respectively Global feature set Selective fusion is performed to obtain a new set of predicate features. New global feature set ; S13, Set entity features New predicate feature set New global feature set The input is fed into a subtitle decoder for fusion decoding to generate... Time Capsule; S14. Design the model loss function and implement model training based on the model loss function; S2. Use a trained question-and-answer dual-controllable video subtitle model to generate subtitles.

2. The dual-controllable video description method based on uncertain allocation of prompting questions according to claim 1, characterized in that: In S11, the multi-level feature extractor includes an entity feature extraction module, a predicate feature extraction module, and a global feature extraction module; The entity feature extraction module detects all objects from the keyframes of the input video sequence as an initial object sequence, and inputs it into the Transformer encoder to obtain object representations. Next, the set of appearance features extracted from the keyframes will be used with a pre-trained 2D CNN. Compared to the set of motion features extracted from keyframes using a pre-trained 3D CNN After concatenation, the data is input into a bidirectional LSTM, and the embedded video content is calculated through max pooling. ; then, The same video content vector Random initialization Query parameters and object representation They are input together into the Transformer's decoder to decode the output. A set of entity features related to the video : ； In the above formula, For the decoder of Transformer; For entity feature set The first in Individual entity characteristics; The predicate feature extraction module first uses a soft attention mechanism to calculate and combine the motion feature set extracted by the pre-trained 3D CNN. Related set of object-perceptual motion features ,in Indicates the first Object-aware motion features of keyframes The number of keyframes in the video sequence; Then, a BiLSTM network is used to collect the object-aware motion features. With motion feature set Encode together as a set of predicate features : ； In the above formula, For the reason and The encoded result of the first Each predicate feature; The global feature extraction module first uses a soft attention mechanism to calculate and combine the appearance feature set. Related set of object-aware appearance features and motion-aware appearance feature set ,in , The first Object-aware appearance features and motion-aware appearance features of each keyframe; Then, a BiLSTM network is used to set the appearance features. Object-aware appearance feature set With motion-aware appearance feature set Co-encode global feature set : ； In the above formula, global feature set The first in A global feature.

3. The dual-controllable video description method based on uncertain allocation of prompting questions according to claim 2, characterized in that: The set of object-perceived motion features is calculated using the following formula. The first in Object-aware motion features of keyframes : ； ； ； In the above formula, For the set of motion features The Middle Motion features of keyframes; Represents the set of entity features The first in Individual entity characteristics; It is a soft attention function; For entity feature set Size; , , , All are learnable parameters; It is the unnormalized raw attention score, used to measure and The strength of the association; It is an exponential function, used to... Mapped to positive numbers; It is the hyperbolic tangent activation function.

4. The dual-controllable video description method based on prompt question uncertainty allocation according to any one of claims 1-3, characterized in that: In S12, the text encoder of the BLIP model is used to process the tags. The entity feature set obtained with S1 Combine encoding to generate problem feature representations The text encoder consists of a bidirectional self-attention module, a cross-attention module, and a feedforward network layer, which processes the tags... First, a bidirectional self-attention module is used to encode information within the text modality. Then, a cross-attention module is used to capture the interactions between modalities. Finally, a feedforward network layer is used to refine the features and generate a problem feature representation. The expression for the text encoder is as follows: ； ； ； ； ； ； In the above formula, , , These represent the bidirectional self-attention module, the cross-attention module, and the feedforward network layer, respectively. , , These are the feature representations obtained after processing by the bidirectional self-attention module, the cross-attention module, and the feedforward network layer, respectively. , All are intermediate feature representations; It is a set of entity features.

5. The dual-controllable video description method based on the uncertainty allocation of prompting questions according to any one of claims 1-3, characterized in that: In S12, choosing any of the following strategies will yield the problem feature representation. The predicate feature sets obtained by S1 respectively Global feature set Selective fusion can be performed using: element-wise additive fusion strategy, element-wise multiplicative fusion strategy, and weighted fusion strategy.

6. The dual-controllable video description method based on uncertain allocation of prompting questions according to claim 5, characterized in that: The weighted fusion strategy refers to introducing a Dirichlet distribution to calculate the problem feature representations separately. Predicate feature set Global feature set The dynamic weights are then used for weighted fusion according to the following formula: ； ； In the above formula, , , Each is a set of predicate features. Global feature set Problem feature representation The dynamic weights and fusion weights are obtained by first mapping the corresponding features to a probability distribution through a linear projection layer to obtain the Dirichlet parameters, and then normalizing the Dirichlet parameters.

7. The dual-controllable video description method based on prompt question uncertainty allocation according to any one of claims 1-3, characterized in that: In S13, a language model is introduced, and the subtitle decoder is... The subtitles generated before the specified time are input into the language model, and the language model then processes them. Predicting timeline captions; In S14, the model loss function includes the distance loss of the multi-level feature extractor, the cross-entropy loss of the subtitle decoder, and the KL divergence between the subtitle decoder and the language model. The distance loss of the multi-level feature extractor includes the distance loss of the entity feature extraction module, the predicate feature extraction module, and the global feature extraction module.

8. The dual-controllable video description method based on uncertain allocation of prompting questions according to claim 7, characterized in that: The expression for the model loss function is: ； ； ； In the above formula, The model loss function; , , These are the distance losses for the entity feature extraction module, the predicate feature extraction module, and the global feature extraction module, respectively. , These are the cross-entropy loss of the subtitle decoder and the KL divergence between the subtitle decoder and the language model, respectively. , , , , These are the weights corresponding to the loss; Indicates real subtitles The Middle one word One-hot encoding, This refers to the length of the subtitle; For subtitle decoders in Real-time generated subtitles; Resulting from language model predictions Time Capsule; This represents the KL divergence.

9. The dual-controllable video description method based on uncertain allocation of prompting questions according to claim 8, characterized in that: The formula for calculating the distance loss of the entity feature extraction module is as follows: ； ； In the above formula, This represents the set of entity features. The Middle The vector representation generated by projecting entity features onto the linguistic semantic space This represents the first element in the object embedding set calculated from the actual subtitle object. Each object embedding, the collection of object embeddings uses an empty set. Fill to the set of entity features The sets of vectors generated by projecting onto the language semantic space are of equal size; The formula for calculating the distance loss of the predicate feature extraction module is as follows: ； In the above formula, This represents the embedding vector obtained by encoding the predicates in the actual subtitles; Represents the set of features of the predicate The text semantic space is obtained by projecting the aggregated features through max pooling and then through a fully connected layer. The formula for calculating the distance loss of the global feature extraction module is as follows: ； In the above formula, This represents the embedding vector obtained by encoding the entire real subtitle. Represents the global feature set After performing max pooling to aggregate features, the text semantic space is obtained by projecting through a fully connected layer.

10. A dual-controllable video description system based on uncertain allocation of prompting questions, characterized in that: The system includes a model building module, a model training module, and a generation module; The model building module is used to build a question-and-answer dual-controllable video subtitle model according to the following steps: A multi-level feature extractor is constructed to convert the input video sequence into multi-level semantic features, which include a set of entity features. Predicate feature set Global feature set ; Design multiple natural language question prompt templates and convert them into tags. , mark The entity feature set obtained with S1 Combine encoding to generate problem feature representations The obtained problem features are represented The predicate feature sets obtained by S1 respectively Global feature set Selective fusion is performed to obtain a new set of predicate features. New global feature set ; Entity feature set New predicate feature set New global feature set The input is fed into a subtitle decoder for fusion decoding to generate... Time Capsule; The model training module is used to design the model loss function and train the question-answering dual-controllable video subtitle model based on the model loss function; The generation module is used to generate subtitles using a trained question-and-answer dual-controllable video subtitle model.

Images