A dense video description generation method, system, computer device and medium

Abstract

The application provides a dense video description generation method, system, computer device and medium, and belongs to the field of video processing, comprising: obtaining an uncropped video, frame sampling to obtain a video frame sequence of the uncropped video; extracting frame-level visual features from the video frame sequence; calculating cross-attention weights between the frame-level visual features and learnable event query vectors, and aggregating the frame-level visual features according to the cross-attention weights to obtain discrete event nodes; calculating event difference features between the discrete event nodes, determining causal probabilities between the corresponding discrete event nodes according to the event difference features, and constructing corresponding event-level causal structures according to the causal probabilities between the discrete event nodes; and generating corresponding natural language description sequences for multiple time segments in the uncropped video based on the frame-level visual features and the event-level causal structures, to obtain a dense video description result of the uncropped video. The method improves the accuracy of dense video description generation.

Description

Technical Field

[0001] This invention belongs to the field of video processing, and specifically relates to a method, system, computer device, and medium for generating dense video descriptions. Background Technology

[0002] Dense video description aims to generate corresponding natural language descriptions for multiple time segments in an uncropped video. Its goal is not only to accurately describe local segments, but also to maintain logical coherence and semantic consistency between different segments at the overall narrative level.

[0003] Existing technologies employ convolutional neural networks, Transformers, or video large language models to model uncropped video frame sequences. Specifically, they achieve causal modeling based on the correlation calculation of frame-level features or local region features, aiming to improve the coherence of narrative expression.

[0004] However, the correlation calculation of frame-level features or local region features is essentially the construction of a correlation matrix. Since it cannot distinguish whether the related events in the uncropped video frame sequence are related by time sequence or by cause, it cannot guarantee the coherence of the narrative expression, which in turn leads to poor accuracy in dense video description. Summary of the Invention

[0005] To address the problem of poor accuracy in existing dense video descriptions, this invention provides a dense video description generation method, system, computer device, and medium.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for generating dense video descriptions, comprising: The uncropped video to be processed is obtained, and the uncropped video is sampled frame by frame to obtain the video frame sequence of the uncropped video, wherein the uncropped video includes multiple time segments to be described; Extract frame-level visual features from video frame sequences; calculate the cross-attention weights between the frame-level visual features and multiple learnable event query vectors; aggregate the frame-level visual features based on the cross-attention weights to obtain discrete event nodes corresponding to each time segment to be described; calculate the event difference features between the discrete event nodes; determine the causal probability between the corresponding discrete event nodes based on the event difference features; and construct the corresponding event-level causal structure based on the causal probability between the discrete event nodes. Based on frame-level visual features and event-level causal structures, corresponding natural language description sequences are generated for multiple time segments in uncropped videos, resulting in dense video descriptions of the uncropped videos.

[0007] Optionally, the dense video description generation method provided by the present invention further includes: Initialize the learnable event query vector; The cross-attention mechanism is used to calculate the cross-attention weights between the learnable event query vector and the frame-level visual features. The frame-level visual features are then weighted and summed according to the cross-attention weights to obtain the event node features. The set of discrete event nodes is determined by the event node features corresponding to each learnable event query vector.

[0008] Optionally, the dense video description generation method provided by the present invention further includes: For any pair of event nodes consisting of two discrete event nodes, construct the event difference features between them; The event difference features are input into a pre-trained multilayer perceptron, which outputs the causal probability of event pairs. The causal probabilities of all event pairs are used to form a causal adjacency matrix, which serves as the event-level causal structure between discrete event nodes.

[0009] Optionally, the dense video description generation method provided by the present invention further includes: By enhancing the causal relationships of discrete event nodes based on the event-level causal structure, a causal-enhanced event representation is obtained; At each time step of the time segment, an event causal context vector is extracted from the causal enhanced event representation and the decoding state of the current time step based on an attention mechanism; and a visual context vector is extracted from the frame-level visual features and the decoding state of the current time step. The event causal context vector and the visual context vector are concatenated and mapped to obtain the descriptor probability distribution of the current time step; the descriptor of the current time step is determined by the descriptor probability distribution of the current time step, and the descriptor of the current time step is decoded to obtain the decoding result; a state update operation is performed based on the decoding result and the decoding state of the current time step to obtain the decoding state of the next time step. The natural language description sequence of time segments is obtained by combining the descriptive words at each time step, and the dense video description result is determined by the natural language description sequence of multiple time segments.

[0010] Optionally, the dense video description generation method provided by the present invention further includes: Aggregate discrete event nodes based on the event-level causal structure to obtain aggregated information; By injecting aggregated information into discrete event nodes, a causal-enhanced event representation is obtained.

[0011] Optionally, the dense video description generation method provided by the present invention further includes: Acquire training video data carrying corresponding descriptive annotations, extract training video data frame sequences from the training video data; input the training video data frame sequences into a deep neural network model based on the Transformer encoding and decoding framework, and output the corresponding training dense video description results; The cross-entropy loss function is calculated based on the training dense video description results and the corresponding description annotations to obtain the subtitle generation cross-entropy loss. The event diversity loss is determined based on the cosine similarity between the learnable event query vectors in the deep neural network model. The total loss is determined by the subtitle generation cross-entropy loss and the event diversity loss. The parameters of the dense video description generation model are updated according to the total loss to obtain the dense video description generation model used to generate dense video descriptions.

[0012] Optionally, the dense video description generation method provided by the present invention further includes: Calculate the average causal probability of the off-diagonal elements in the causal adjacency matrix corresponding to the training video data, and determine the causal sparsity loss by the difference between the average value and the preset target sparsity. The causal adjacency matrix corresponding to the training video data is multiplied element by element by its transpose matrix, and the causal directionality loss is determined by the sum of the absolute values ​​of each element after multiplication. The total loss is obtained by weighting the subtitle generation cross-entropy loss, event diversity loss, causal sparsity loss, and causal directionality loss.

[0013] The present invention also provides a dense video description generation system, comprising: The video sampling module is used to acquire the uncropped video to be processed, perform frame sampling on the uncropped video, and obtain the video frame sequence of the uncropped video, wherein the uncropped video includes multiple time segments to be described; The causal relationship construction module is used to extract frame-level visual features from video frame sequences; calculate the cross-attention weights between the frame-level visual features and multiple learnable event query vectors; aggregate the frame-level visual features according to the cross-attention weights to obtain discrete event nodes corresponding to each time segment to be described; calculate the event difference features between the discrete event nodes; determine the causal probability between the corresponding discrete event nodes based on the event difference features; and construct the corresponding event-level causal structure based on the causal probability between the discrete event nodes. The dense video description module is used to generate corresponding natural language description sequences for multiple time segments in an uncropped video based on frame-level visual features and event-level causal structures, thus obtaining the dense video description results of the uncropped video.

[0014] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement any of the steps of a dense video description generation method.

[0015] The present invention also provides a computer-readable storage medium storing a computer program that, when loaded by a processor, can execute any step of a dense video description generation method.

[0016] The dense video description generation method provided by this invention has the following beneficial effects: This invention, based on the state differences between event nodes, explicitly infers the existence, strength, and direction of causal dependencies between events. This allows for the differentiation between sequential or simultaneous events and their causes and consequences, avoiding the misinterpretation of accidental temporal co-occurrence as a necessary causal relationship. The resulting event-level causal structure reflects the evolutionary logic of events. Therefore, in dense video description generation, because it relies not on an association matrix that is difficult to distinguish between temporal and causal relationships, but on an event-level causal structure with event-level causal direction and dependencies, it reduces temporal confusion, causal attribution errors, and causal descriptions that do not conform to the video facts in multi-event scenarios. This makes the generated dense video descriptions more consistent with the true causal logic between events in the video, ultimately improving the accuracy of dense video description generation. Attached Figure Description

[0017] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a dense video description generation method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating the event-level causal structure modeling and narrative generation method for dense video description provided in this embodiment of the invention; Figure 3 A schematic diagram of the hierarchical module structure of the event-level causal structure modeling and narrative generation method provided in the embodiments of the present invention; Figure 4 This is a schematic diagram of an event-level causal structure provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a multi-event dense video and corresponding descriptive annotations provided in an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0020] Existing dense video description methods typically employ convolutional neural networks, Transformers, or large video language models to model video frame sequences. They then extract description-related information from frame-level features using attention mechanisms, and finally combine this with a language decoder to output descriptive text. However, these methods generally rely on frame-level temporal correlations or segment-level co-occurrence relationships for narrative organization, lacking a structured representation of events as high-level semantic units. When performing dense video descriptions in multi-event scenarios, this approach, which relies solely on temporal sequence or statistical co-occurrence for inference, easily misjudges simultaneous or sequential events as causal relationships. This leads to generated temporal confusion, causal attribution errors, and causal descriptions that do not conform to the facts of the video.

[0021] To address these shortcomings, some researchers have attempted to introduce strong causal reasoning capabilities into video narrative generation, such as by constructing causal attention matrices, adding temporal constraints, or using external language models for causal narrative generation. While these methods improve the coherence of narrative expression to some extent, their causal modeling often remains at the level of frame-level or local region feature correlation calculations, making it difficult to form event-level causal chain structures. Furthermore, relying on prior knowledge from language models for causal narrative may create causal illusions inconsistent with the video content, resulting in a lack of reliability in the generated results at the level of visual fact.

[0022] Furthermore, in multi-event videos, there are often cause-and-effect dependencies between events, such as an action triggering a subsequent behavior or a change in state causing a scene transition. Existing correlation-based modeling methods often do not explicitly distinguish between temporal order and causal direction, making them prone to statistical shortcuts during model training. They are more susceptible to interference when the background remains unchanged, actions are repetitive, or events are dense, making it difficult to identify the truly crucial cause-and-effect relationships. This limits the logical consistency and interpretability of dense video descriptions at the overall narrative level.

[0023] Therefore, there is an urgent need for a structured modeling method for dense video description generation that can automatically extract discrete semantic event nodes from continuous video frames and explicitly construct causal relationship structures at the event level. This method can enhance cross-segment narrative capabilities through event-level causal reasoning, thereby improving the coherence, accuracy, and reliability of dense video description generation in complex multi-event scenarios.

[0024] To address the aforementioned needs, this invention, based on the frame-level feature sequence of the input video, first discretizes continuous frame features into multiple semantic event nodes through an event extraction mechanism, thereby achieving an abstract transformation from frame-level representation to event-level representation. Subsequently, it constructs a causal structure representation between events at the event level, inferring causal directions and dependencies by modeling the differential features between event pairs, and optimizing the sparsity and directional consistency of the causal relationship graph using structured constraints, making the resulting causal structure more consistent with the evolution logic of real events. Based on this, it utilizes the constructed event-level causal structure to provide event causal context vectors for the description generation process, and integrates frame-level visual detail information with the event causal context vectors to generate dense video natural language description sequences corresponding to multiple time segments, thereby improving the logical consistency, causal coherence, and overall narrative quality of the description results in multi-event scenarios.

[0025] Example 1 This invention provides a method for generating dense video descriptions, specifically as follows: Figure 1 As shown, it includes the following steps: Step 11: Obtain the uncropped video to be processed, perform frame sampling on the uncropped video to obtain the video frame sequence of the uncropped video, wherein the uncropped video includes multiple time segments to be described.

[0026] Step 12: Extract frame-level visual features from the video frame sequence; calculate the cross-attention weights between the frame-level visual features and multiple learnable event query vectors; aggregate the frame-level visual features according to the cross-attention weights to obtain discrete event nodes corresponding to each time segment to be described; calculate the event difference features between the discrete event nodes; determine the causal probability between the corresponding discrete event nodes based on the event difference features; and construct the corresponding event-level causal structure based on the causal probability between the discrete event nodes.

[0027] Discrete event nodes can be determined through the following steps: Step 121: Initialize the learnable event query vector.

[0028] Step 122: Calculate the cross-attention weights between the learnable event query vector and the frame-level visual features based on the cross-attention mechanism. Then, sum the frame-level visual features according to the cross-attention weights to obtain the event node features.

[0029] Step 123: Determine the set of discrete event nodes based on the event node features corresponding to each learnable event query vector.

[0030] Specifically, such as Figure 2 As shown, the uncropped video to be processed is first acquired, and then the video is uniformly sampled at preset time intervals to obtain an image sequence containing T frames. Furthermore, for image sequence X, the differences in resolution and lighting conditions between different video sources can be reduced by methods such as frame size normalization, pixel standardization, and necessary frame cropping.

[0031] After frame sampling and preprocessing are completed, such as Figure 3 As shown, the video frame sequence is input into a pre-trained dense video description generation model. The video feature construction module in the dense video description generation model, such as a pre-trained visual encoder, is used to extract features to obtain frame-level visual features, as shown in formula (1): (1); in, For visual encoder feature extraction processing, For the t-th frame image, Let d represent the frame-level visual features corresponding to the t-th frame image, where d is the feature dimension. Then, these frame-level visual features are arranged in chronological order to obtain the frame-level feature sequence. .

[0032] Next, the event extraction module in the dense video description generation model extracts events from the frame-level feature sequence to obtain discrete event node features. For example, in the event extraction module, K learnable event query vectors are first initialized. Next, the event query vector is calculated. With frame-level features Cross attention weights As shown in formula (2): (2); in, Indicates the first A learnable event query vector, Indicates the first Frame-level visual features corresponding to the frame In the process of summation, the first... Frame-level visual features corresponding to the frame The summation index variable is used to iterate through all frames in the video frame sequence. Indicates the length of the video frame sequence. Representing vectors The transpose operation is used with frame-level features. or Perform inner product calculation.

[0033] Then based on the cross-attention weights The frame-level features are aggregated to obtain the event node features. As shown in formula (3): (3); in, Indicates the first Each event node characteristic Indicates the event query vector and the first Cross-attention weights between frame features Indicates the first Frame-level visual features corresponding to the frame Indicates the length of the video frame sequence.

[0034] And based on the characteristics of these event nodes Determine the feature set of discrete event nodes .

[0035] The causal adjacency matrix can be determined through the following steps: Step 124: For any two discrete event nodes forming an event pair, construct the event difference features between them.

[0036] Step 125: Input the event difference features into the pre-trained multilayer perceptron and output the causal probability of the event pair.

[0037] Step 126: Form a causal adjacency matrix from the causal probabilities of all event pairs, and use the causal adjacency matrix as the event-level causal structure between discrete event nodes.

[0038] Specifically, after the discrete event node features are constructed, the event causal relationship inference module in the dense video description generation model calculates the difference features between discrete event nodes, and then predicts the causal adjacency matrix between events to obtain the event-level causal structure.

[0039] For example, such as Figure 4 As shown, multiple event nodes are connected by directed edges, forming an event-level causal structure. For any two nodes in the discrete event nodes, an event pair... The event difference features are determined by the difference between the feature sets of their discrete event nodes. As shown in formula (4): (4); in, and They represent the first The characteristics of the first event node and the first Each event node characteristic This represents element-wise absolute value operation. Represents event nodes With event nodes Event difference characteristics between them.

[0040] Next, the event difference features Input a pre-trained multilayer perceptron to predict causal probabilities As shown in formula (5): (5); in, It is the Sigmoid activation function. This is a multilayer perceptron. Then, each event pair... causal probability Forming a causal adjacency matrix , as an event-level causal structure.

[0041] Step 13: Based on frame-level visual features and event-level causal structure, generate corresponding natural language description sequences for multiple time segments in the uncropped video to obtain dense video description results of the uncropped video.

[0042] Step 13 includes: Step 131: Enhance the causal relationships of discrete event nodes based on the event-level causal structure to obtain a causally enhanced event representation. For example, aggregate discrete event nodes based on the event-level causal structure to obtain aggregated information; inject the aggregated information into the discrete event nodes to obtain a causally enhanced event representation.

[0043] Step 132: In each time step of the time segment, extract the event causal context vector from the causal enhanced event representation and the decoding state of the current time step based on the attention mechanism, and extract the visual context vector from the frame-level visual features and the decoding state of the current time step.

[0044] Step 133: Concatenate and map the event causal context vector and visual context vector to obtain the descriptor probability distribution for this time step; determine the descriptor for this time step based on the descriptor probability distribution, and update the decoding state of the current time step to obtain the decoding state of the next time step. For example, input the descriptor as the output word of the current time step into the language decoder, and perform a state update operation based on the decoding state of the current time step to obtain the decoding state of the next time step. The decoding state represents the hidden state of the language decoder at the current time step, used to characterize the semantic context information of the currently generated descriptor sequence.

[0045] Step 134: Obtain the natural language description sequence of the time segment by combining the descriptive words of each time step, and determine the dense video description result by combining the natural language description sequences of multiple time segments.

[0046] Specifically, after determining the causal adjacency matrix between discrete event nodes, the narrative generation module in the dense video description generation model performs causal information propagation between events based on the causal adjacency matrix, generating causally enhanced event representations, i.e. causally enhanced event features. These causally enhanced event features are then fused with frame-level visual features to generate a logically coherent natural language description sequence across multiple time segments of the uncropped video, thereby outputting the dense video description result.

[0047] For example, when the causal adjacency matrix is ​​obtained Then, firstly based on the causal adjacency matrix Aggregate the feature set of discrete event nodes to obtain aggregated causal information. As shown in formula (6): (6); in, This is a discrete event node feature matrix, which consists of multiple event node features. composition; This is an event-level causal adjacency matrix, used to describe the causal dependencies between different event nodes.

[0048] Then, causal information will be aggregated. By injecting into discrete event nodes, a causal-enhanced event representation is obtained. As shown in formula (7): (7); in, This represents a nonlinear transformation function that aggregates causal information. When causality is reinforced, the event representation... After construction is complete, frame-level visual features will be... Features of causal reinforcement events As joint input, a dual-stream narrative generation process is constructed, consisting of frame-level visual features. Provides fine-grained visual content information, enhancing event features through causal relationships. Provides event-level structured causal context information. For example, at a time step t, based on the decoding state of the current time step. Frame-level visual features Perform attention reading to obtain the visual context vector. As shown in formula (8): (8); in, The attention function is calculated. Similarly, it is based on the decoding state at the current time step. Features of causal enhancement events Perform attention reading to obtain the event causal context vector. As shown in formula (9): (9); Next, the visual context vector With event causal context vector By merging, a probability distribution of descriptive words is generated. As shown in formula (10): (10); in, This represents a vector concatenation operation. For learnable mapping matrix, This represents the probability distribution of words generated at time step t, where Softmax is the activation function, and the probability distribution of the descriptive words is used as the basis for the activation. Select or sample to generate the descriptor for the current time step, and then decode the state. Updated to , and then perform subsequent visual context vectors With event causal context vector Attention is extracted until descriptive words for each time step within the time segment are obtained, generating a complete natural language description sequence. Finally, the dense video description result is determined from the natural language description sequences of multiple time segments.

[0049] In addition, the training phase also includes: Step 14: Obtain training video data with corresponding descriptive annotations, and extract training video data frame sequences from the training video data; input the training video data frame sequences into an untrained deep neural network model based on the Transformer encoder-decoder framework, and output the corresponding training dense video description results; calculate the cross-entropy loss function based on the training dense video description results and the corresponding descriptive annotations to obtain the subtitle generation cross-entropy loss, and determine the event diversity loss based on the cosine similarity between the learnable event query vectors in the deep neural network model; determine the total loss based on the subtitle generation cross-entropy loss and the event diversity loss, and update the parameters of the untrained deep neural network model based on the total loss to obtain the deep neural network model used to generate dense video descriptions. Specifically, the deep neural network model encodes temporal features through a Transformer encoder, predicts event time boundaries through a Transformer decoder, extracts discrete event nodes based on multi-head cross-attention between learnable event query vectors and frame-level visual features, predicts the event causal graph using a multilayer perceptron, and then generates the corresponding descriptive text using a recurrent neural network decoding module to obtain the training dense video description results.

[0050] The loss function also includes causal sparsity loss and causal directionality loss. Step 14 includes: Step 141: Calculate the average causal probability of the off-diagonal elements in the causal adjacency matrix corresponding to the training video data, and determine the causal sparsity loss by the difference between the average value and the preset target sparsity.

[0051] Step 142: Multiply the causal adjacency matrix corresponding to the training video data element by element with its transpose matrix, and determine the causal directionality loss by the sum of the absolute values ​​of each element after multiplication.

[0052] Step 143: Weight the subtitle generation cross-entropy loss, event diversity loss, causal sparsity loss and causal directionality loss to obtain the total loss.

[0053] Specifically, during the model training phase, training video data with corresponding descriptive annotations is first acquired. For example, a sample set containing multiple uncropped videos and their corresponding descriptive annotations is used. Frame-level visual features are extracted from the videos in the sample set, and a dense video description training dataset is constructed. Then, based on the above data processing, the training video data is input into an untrained deep neural network model, for example, by inputting frame-level feature sequences... With the corresponding descriptive annotation sequence Build as Figure 5 The training sample pairs shown ,in To describe the first in the sequence One word, This describes the sequence length. Next, the above training samples will be paired... The model parameters are iteratively updated by inputting an untrained deep neural network model. In each iteration cycle, the current iteration loss value is calculated using the total loss based on the predicted description and the real labeled description. The parameters of the dense video description generation model based on causal structure modeling are adjusted by the loss value through the backpropagation algorithm until the model converges or reaches the preset number of iterations, thus obtaining the trained dense video description generation model based on causal structure modeling.

[0054] The iterative loss value can be the total loss composed of subtitle generation cross-entropy loss, event diversity loss, causal sparsity loss, and causal directionality loss. As shown in formula (11): (11); in, Generate cross-entropy loss for the captions between the model's predictions and the descriptive annotation sequence. For the loss of event diversity, For causal sparsity loss, For causal directional loss, , and These are all weight coefficients for each loss function. The event diversity loss is shown in formula (12): (12); in, Learnable event query vectors and The cosine similarity between them. And the causal sparsity loss. and causal directionality loss The structured regularization constraints are applied to the causal adjacency matrix, as shown in Equations (13) and (14), respectively: (13); (14); Where ρ is the preset target sparsity of the target causal graph, which is used to control the proportion of non-zero connections in the causal adjacency matrix; The number of discrete event nodes; This is an event-level causal adjacency matrix, with matrix elements... For event nodes With event nodes The strength of the causal relationship between them; For matrix The transpose of the matrix; This is the element-wise multiplication of a matrix (Hadamard product). The sum of the absolute values ​​of the matrix elements Norm.

[0055] In summary, the dense video description generation method provided by this invention extracts discrete events from continuous video frames and explicitly models causal relationships at the event level, enabling the generated dense video description to reflect the cause-and-effect relationships between events, thereby improving the coherence and logic of the overall narrative. Furthermore, the modeling process distinguishes between temporal sequence and true causal relationships, effectively reducing erroneous causal inferences based solely on temporal co-occurrence and mitigating temporal confusion and causal misjudgments in multi-event scenarios. Moreover, the method utilizes both frame-level visual information and event-level causal structure information during description generation, ensuring that the description content conforms to the visual facts in the video and possesses a reasonable narrative structure, thus improving the accuracy and stability of dense video descriptions in complex scenarios.

[0056] To further illustrate the application process of the method of the present invention, the following is combined with... Figure 4 and Figure 5 Provide a specific example. For example... Figure 5As shown, in a video of making a bacon sandwich, the video contains multiple consecutive events, such as cutting tomatoes, adding oil to a pan, frying bacon, placing the bacon on the bread, and covering the bread with the sandwich. This invention first samples the video frames and extracts frame-level visual features. Then, using a learnable event query vector, it extracts discrete event nodes from the frame-level feature sequence, such as the "cutting tomato event," "frying bacon event," and "placing bacon event."

[0057] After obtaining discrete event nodes, this invention constructs event difference features and inputs them into a causal relationship prediction module to infer potential causal dependencies between events. For example, in this video, by adding oil to a pan and heating it, bacon can be fully cooked, and the cooked bacon is an important ingredient for making sandwiches. Therefore, the model can identify the causal dependencies between the events "adding oil to the pan and heating it," "frying bacon," and "placing bacon." By predicting causal relationships for multiple event pairs, the following can be obtained: Figure 4 The event-level causal structure is shown.

[0058] After constructing the event-level causal structure, the model utilizes both frame-level visual features and event-level causal structure information when generating video descriptions, thereby generating logically coherent dense video descriptions, such as "fry the bacon," "place the bacon on the sandwich," and "cover the top with bread," thus obtaining dense video description results that conform to the video facts and have causal coherence.

[0059] Example 2 The present invention also provides a dense video description generation system, comprising: The video sampling module is used to acquire the uncropped video to be processed, perform frame sampling on the uncropped video, and obtain the video frame sequence of the uncropped video. The uncropped video includes multiple time segments to be described.

[0060] The causal relationship construction module is used to extract frame-level visual features from video frame sequences; calculate the cross-attention weights between the frame-level visual features and multiple learnable event query vectors; aggregate the frame-level visual features according to the cross-attention weights to obtain discrete event nodes corresponding to each time segment to be described; calculate the event difference features between the discrete event nodes; determine the causal probability between the corresponding discrete event nodes based on the event difference features; and construct the corresponding event-level causal structure based on the causal probability between the discrete event nodes.

[0061] The dense video description module is used to generate corresponding natural language description sequences for multiple time segments in an uncropped video based on frame-level visual features and event-level causal structures, thus obtaining the dense video description results of the uncropped video.

[0062] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps in an embodiment of a dense video description generation method. Specific implementation methods can be found in the method embodiments, and will not be repeated here.

[0063] Furthermore, the present invention also provides a non-transitory computer-readable storage medium containing instructions on which a computer program is stored. For example, a memory containing instructions that can be executed by a processor of a computer device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the computer program is executed by the processor, it can implement the steps in an embodiment of a dense video description generation method. Specific implementation methods can be found in the method embodiments, which will not be repeated here.

[0064] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0068] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Any simple variations or equivalent substitutions of technical solutions that can be readily obtained by those skilled in the art within the scope of the technology disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for generating dense video descriptions, characterized in that, include: Obtain the uncropped video to be processed, perform frame sampling on the uncropped video to obtain the video frame sequence of the uncropped video, wherein the uncropped video includes multiple time segments to be described; Frame-level visual features are extracted from the video frame sequence; cross-attention weights are calculated between the frame-level visual features and multiple learnable event query vectors; the frame-level visual features are aggregated according to the cross-attention weights to obtain discrete event nodes corresponding to each time segment to be described; event difference features between the discrete event nodes are calculated; the causal probability between the corresponding discrete event nodes is determined by the event difference features; and the corresponding event-level causal structure is constructed by the causal probability between the discrete event nodes. Based on the frame-level visual features and event-level causal structure, corresponding natural language description sequences are generated for multiple time segments in the uncropped video, resulting in a dense video description of the uncropped video.

2. The dense video description generation method according to claim 1, characterized in that, Calculate the cross-attention weights between the frame-level visual features and multiple learnable event query vectors, and aggregate the frame-level visual features based on the cross-attention weights to obtain discrete event nodes corresponding to each time segment to be described, including: Initialize the learnable event query vector; The cross-attention weights between the learnable event query vector and the frame-level visual features are calculated based on the cross-attention mechanism. The frame-level visual features are then weighted and summed according to the cross-attention weights to obtain the event node features. The set of discrete event nodes is determined by the event node features corresponding to each learnable event query vector.

3. The dense video description generation method according to claim 1, characterized in that, Calculating the event difference features between the discrete event nodes, determining the causal probability between the corresponding discrete event nodes based on the event difference features, and constructing the corresponding event-level causal structure based on the causal probability between the discrete event nodes includes: For any pair of event nodes consisting of two discrete event nodes, construct the event difference features between them; The event difference features are input into a pre-trained multilayer perceptron, which outputs the causal probability of the event pair. The causal probabilities of all event pairs are used to form a causal adjacency matrix, which is then used as the event-level causal structure between discrete event nodes.

4. The dense video description generation method according to claim 1, characterized in that, Generate corresponding natural language description sequences for multiple time segments in the uncropped video to obtain a dense video description result for the uncropped video, including: By enhancing the causal relationships of discrete event nodes based on the event-level causal structure, a causal-enhanced event representation is obtained; In each time step of the time segment, an event causal context vector is extracted from the causal-enhanced event representation and the decoding state of the current time step based on an attention mechanism, and a visual context vector is extracted from the frame-level visual features and the decoding state of the current time step. The event causal context vector and the visual context vector are concatenated and mapped to obtain the descriptor probability distribution of the current time step; the descriptor of the current time step is determined by the descriptor probability distribution of the current time step, and the descriptor of the current time step is decoded to obtain the decoding result; a state update operation is performed based on the decoding result and the decoding state of the current time step to obtain the decoding state of the next time step. The natural language description sequence of the time segment is obtained by combining the descriptive words of each time step, and the dense video description result is determined by the natural language description sequences of multiple time segments.

5. The dense video description generation method according to claim 4, characterized in that, Based on the event-level causal structure, the causal relationships of discrete event nodes are enhanced, resulting in causally enhanced event representations including: Based on the event-level causal structure, discrete event nodes are aggregated to obtain aggregated information; The aggregated information is injected into discrete event nodes to obtain a causal-enhanced event representation.

6. The dense video description generation method according to claim 3, characterized in that, Before extracting frame-level visual features from the video frame sequence, the process also includes: Acquire training video data carrying corresponding descriptive annotations, extract training video data frame sequences from the training video data; input the training video data frame sequences into a deep neural network model based on the Transformer encoding and decoding framework, and output corresponding training dense video description results; The cross-entropy loss function is calculated based on the training dense video description results and the corresponding description annotations to obtain the subtitle generation cross-entropy loss. The event diversity loss is determined based on the cosine similarity between the learnable event query vectors in the deep neural network model. The total loss is determined by the subtitle generation cross-entropy loss and event diversity loss. The parameters of the untrained deep neural network model are updated based on the total loss to obtain a deep neural network model for generating dense video descriptions.

7. The dense video description generation method according to claim 6, characterized in that, The total loss, determined by the subtitle generation cross-entropy loss and event diversity loss, includes: Calculate the average value of the causal probabilities of the off-diagonal elements in the causal adjacency matrix corresponding to the training video data, and determine the causal sparsity loss by the difference between the average value and the preset target sparsity. The causal adjacency matrix corresponding to the training video data is multiplied element by element by its transpose matrix, and the causal directionality loss is determined by the sum of the absolute values ​​of each element after multiplication. The total loss is obtained by weighting the subtitle generation cross-entropy loss, event diversity loss, causal sparsity loss, and causal directionality loss.

8. A dense video description generation system, characterized in that, include: The video sampling module is used to acquire the uncropped video to be processed, perform frame sampling on the uncropped video, and obtain the video frame sequence of the uncropped video, wherein the uncropped video includes multiple time segments to be described; The causal relationship construction module is used to extract frame-level visual features from the video frame sequence; calculate the cross-attention weights between the frame-level visual features and multiple learnable event query vectors; aggregate the frame-level visual features according to the cross-attention weights to obtain discrete event nodes corresponding to each time segment to be described; calculate the event difference features between the discrete event nodes; determine the causal probability between the corresponding discrete event nodes based on the event difference features; and construct the corresponding event-level causal structure based on the causal probability between the discrete event nodes. The dense video description module is used to generate corresponding natural language description sequences for multiple time segments in the uncropped video based on the frame-level visual features and event-level causal structure, thereby obtaining the dense video description result of the uncropped video.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the dense video description generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is loaded by the processor, it is able to perform the steps of the dense video description generation method according to any one of claims 1 to 7.

Images