Dense Video Description Method Based on Multimodal Memory Knowledge

Abstract

This invention relates to the field of video description, specifically to a dense video description method based on multimodal memory knowledge. The method includes: extracting visual and audio features from an input video and performing cross-modal fusion to generate final audio and visual codes; determining event visual and event audio features of multiple candidate events from the input video based on the final audio and visual codes; for each candidate event, retrieving matching external knowledge from an external memory knowledge base based on its corresponding event visual and event audio features to generate its corresponding multimodal external memory knowledge; and gradually constructing a word embedding sequence through an autoregressive mechanism based on the multimodal external memory knowledge, event visual features, and event audio features of each candidate event to generate a description of the input video. This invention can learn the correspondence between events and descriptions from more comprehensive information, significantly improving the accuracy and richness of the generated descriptions.

Description

Technical Field

[0001] This invention relates to the field of video description, and more specifically to a dense video description method based on multimodal memory knowledge. Background Technology

[0002] Video description refers to the accurate summary of visual information, plot development, and key elements in a video through text, transforming it into readable text. It can be applied to video platforms, streaming services, and educational scenarios to improve the accessibility and dissemination efficiency of video content.

[0003] Most existing dense video description methods are based on an encoder-decoder framework, where the encoder extracts multimodal features from the video, and the decoder generates corresponding event descriptions. However, the descriptions generated by these methods often lack accuracy and richness. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a dense video description method based on multimodal memory knowledge.

[0005] The method includes:

[0006] Extract visual and audio features from the input video;

[0007] Cross-modal fusion of visual and audio features is performed to generate the final audio code and the final visual code;

[0008] Based on the final audio encoding and final visual encoding, the visual and audio features of multiple candidate events are determined from the input video.

[0009] For each candidate event, based on its corresponding visual and audio features, the matching external knowledge is retrieved from the pre-built external memory knowledge base to generate its corresponding multimodal external memory knowledge.

[0010] Based on the multimodal external memory knowledge, visual features, and audio features of each candidate event, a word embedding sequence is gradually constructed through an autoregressive mechanism to generate a description of the input video.

[0011] Furthermore, the steps for cross-modal fusion of visual and audio features to generate the final audio and visual codes include:

[0012] Multi-layer coding fusion of visual and audio features is performed using multiple coding layers. In each coding fusion layer:

[0013] Self-attention calculations are performed on the visual features of the previous coding layer and the audio features of the previous coding layer to obtain the visual self-attention features and the audio self-attention features of the coding layer.

[0014] After cross-modal fusion of the visual self-attention features and the audio self-attention features of the coding layer, visual features and audio features of the coding layer are generated.

[0015] The audio and visual features of the last coding layer are processed by a feedforward network to obtain the final audio and visual codes.

[0016] Furthermore, the steps for generating the visual features and audio features of the coding layer by cross-modal fusion of the visual self-attention features and the audio self-attention features of the coding layer include:

[0017] Using the visual self-attention features of the coding layer as the query and the audio self-attention features of the coding layer as the key, the coding layer visual cross-modal attention is generated.

[0018] Using the audio self-attention features of the coding layer as the query and the visual self-attention features of the coding layer as the key, the audio cross-modal attention of the coding layer is generated.

[0019] Feature fusion is performed on the visual cross-modal attention and audio cross-modal attention of the coding layer to generate the visual features and audio features of the coding layer.

[0020] Furthermore, the steps of fusing features of the visual cross-modal attention and audio cross-modal attention of the coding layer to generate the visual features and audio features of the coding layer include:

[0021] The first Layer coding layer vision cross-modal attention and the Layer coding layer audio cross-modal attention Perform feature concatenation to obtain the first... Layer coding layer mode invariant representation ;

[0022] The following formula is used to generate the first... Layer encoding layer visual features and the Layer coding layer audio features :

[0023] ;

[0024] ;

[0025] in, This indicates multi-head attention calculation. Indicates the first Layered encoding layer audio cross-modal attention, Indicates the first Layered coding layer visual cross-modal attention.

[0026] Furthermore, based on the final audio encoding and final visual encoding, the steps for determining the visual and audio features of multiple candidate events from the input video include:

[0027] The final audio and visual codes are input into the proposal generator to calculate the segment center, segment length, and confidence score of the event segment;

[0028] From all event fragments, filter them according to confidence scores to obtain a first number of event fragments as candidate events;

[0029] Based on the start and end times of each candidate event, extract the visual features of that candidate event. and event audio features :

[0030] ;

[0031] ;

[0032] in, For final visual encoding The features corresponding to the start time of the candidate events. For final visual encoding The features corresponding to the end times of candidate events. For final audio encoding The features corresponding to the start time of the candidate events. For final audio encoding The features corresponding to the end times of candidate events. This indicates a range truncation operation.

[0033] Furthermore, the steps for calculating the fragment center, fragment length, and confidence score of an event fragment include:

[0034] The following formulas are used to calculate the fragment center, fragment length, and confidence score of an event fragment:

[0035] ;

[0036] ;

[0037] ;

[0038] in, The center of the event segment. Indicates the length of an event segment. This represents the confidence score of an event segment. This represents the initial center position offset. This represents the preset baseline event length. Represents the sigmoid function. This represents the offset predicted by the convolutional network. This represents the logarithm of the length predicted by the convolutional network. Represents an exponential function. This represents the confidence score of the convolutional network's prediction.

[0039] Furthermore, the construction steps of the external memory knowledge base include:

[0040] Extract audio and visual features from videos in a pre-prepared target dataset;

[0041] Clustering algorithms are used to cluster the audio features and visual features of the dataset respectively, resulting in multiple visual cluster centers and multiple audio cluster centers. Each visual cluster center is used as an external visual knowledge base, and each audio cluster center is used as an external audio knowledge base.

[0042] A text encoder is used to encode the human-written descriptive text corresponding to the videos in the target dataset, resulting in multiple external text knowledge bases;

[0043] By combining external visual knowledge base, external audio knowledge base, and external text knowledge base, an external memory knowledge base is obtained.

[0044] Furthermore, for each candidate event, based on its corresponding visual and audio features, the steps of retrieving matching external knowledge from a pre-built external memory knowledge base to generate its corresponding multimodal external memory knowledge include:

[0045] Visual features of each candidate event and event audio features The visual compressed features of the candidate event are obtained by performing average pooling operations. and event audio compression features ;

[0046] Calculate the audio compression features of the event With each external audio knowledge base The cosine similarity is calculated, and external audio memories are obtained by filtering from all external audio knowledge bases based on the calculation results;

[0047] Calculate visual compression features of events With each external visual knowledge base The cosine similarity is calculated, and external visual memories are obtained by filtering from all external visual knowledge bases based on the calculation results.

[0048] Calculate visual compression features of events With an external text knowledge base The cosine similarity is calculated, and external text memories are obtained by filtering from all external text knowledge bases based on the calculation results.

[0049] The selected external audio memories, external visual memories, and external text memories are each subjected to self-attention computation to obtain memory representations in their respective modalities, thus yielding multimodal external memory knowledge. ,in Represents audio memory representation. This represents visual memory representation. This represents the textual memory representation. This indicates a splicing operation.

[0050] Furthermore, based on the multimodal external memory knowledge, visual features, and audio features of each candidate event, the steps to generate a description of the input video by progressively constructing a word embedding sequence through an autoregressive mechanism include:

[0051] Using gating networks to access multimodal external memory knowledge Filtering is performed to obtain matching and fused memory features :

[0052] ;

[0053] in, This indicates the processing of linear layers. Represents the sigmoid function;

[0054] Using a preset feature vector as the starting marker of the word embedding sequence, an autoregressive iteration is performed in multiple steps. In each iteration, the next word embedding is generated based on the multimodal external memory knowledge, visual features of the event, and audio features of the event for each candidate event and then added to the word embedding sequence; until a preset ending marker is generated.

[0055] All word embeddings except for the start and end markers in the word embedding sequence are converted into corresponding natural language words through a pre-trained word embedding mapping table. The natural language words are then concatenated sequentially according to the order in the word embedding sequence to obtain the description of the input video.

[0056] Furthermore, the step of generating the next word embedding based on the multimodal external memory knowledge, visual features of the event, and audio features of the event for each candidate event includes:

[0057] The word embedding sequence is processed through a masked self-attention layer to obtain the self-attention embedding sequence. ;

[0058] The self-attention embedding sequence is combined with event visual features through a cross-attention layer. Event audio characteristics Matching and fusing memory features Interact to obtain interactive visual features Interactive audio features Interactive memory features :

[0059] ;

[0060] ;

[0061] ;

[0062] The next word embedding is then generated through a fully connected layer and an activation function. :

[0063] ;

[0064] in, For activation function, This represents the processing of the fully connected layer.

[0065] This invention overcomes the shortcomings of existing methods that rely solely on internal video information by integrating internal video features with external memory knowledge. This allows the model to learn the correspondence between events and descriptions from more comprehensive information. By employing a memory-enhanced decoder, it achieves deep interaction between multimodal video features, multimodal memory knowledge, and descriptive text, thereby significantly improving the accuracy and richness of the generated descriptions. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a flowchart illustrating a dense video description method based on multimodal memory knowledge disclosed in an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of the structure of the event location module disclosed in an embodiment of the present invention;

[0069] Figure 3 This is a schematic diagram of the process for generating candidate event descriptions based on multimodal external memory knowledge, as disclosed in an embodiment of the present invention. Detailed Implementation

[0070] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, or product comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or units.

[0072] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0073] This invention proposes a dense video description method based on multimodal memory knowledge. It extracts visual and audio features from the input video and locates potential event segments by analyzing the dynamic correlation between these features. Cross-modal interaction technology is used to identify the temporal boundaries of event segments, and the confidence score of each event segment is calculated. The segment with the highest confidence score is then selected as candidate events. For each candidate event, related information is retrieved from a pre-constructed external memory knowledge base. This knowledge base forms cross-scene visual / audio topic centers by clustering the visual and audio features of massive amounts of video data and constructs a text semantic base by combining semantic embedding of manually described text. During retrieval, the similarity between the features corresponding to the candidate event and the visual and audio cluster centers in the knowledge base is calculated. Simultaneously, descriptive keywords related to the event content in the text semantic base are matched to select the most relevant multimodal memory knowledge. Subsequently, a dynamic gating mechanism is used to adjust the weights of the retrieved external memories, suppressing redundant information and retaining visual cues, audio cues, and text semantics highly relevant to the candidate events. Finally, the multimodal features of candidate events are fused with the filtered external memory knowledge, and combined with temporal context information to generate a detailed natural language description that matches the event content. Figure 1 This is a flowchart of a dense video description method based on multimodal memory knowledge provided in an embodiment of the present invention. The steps of the present invention will be described in detail below with reference to a specific embodiment.

[0074] 1. Generate candidate events

[0075] To extract potential event segments from input videos, this invention first performs feature extraction on the input videos. Specifically, a pre-trained visual feature extraction network is used to extract visual features from the input videos, resulting in visual features that contain semantic and spatial information of the video frames in the input videos. Meanwhile, in order to capture audio information from the input video, a pre-trained audio feature extraction network is used to extract audio features from the input video, obtaining audio features that characterize the spectral and temporal properties of the audio signal in the input video. Pre-trained visual feature extraction networks, such as CLIP (Contrastive Language-Image Pre-training), I3D (3D Convolutional Network), and SlowFast (Dual-Path Spatiotemporal Network), are not limited in this embodiment of the invention; similarly, pre-trained audio feature extraction networks, such as VGGish and OpenL3 (Audio-Image Alignment Model), are not limited in this embodiment of the invention. Through these pre-trained feature extraction networks, representative visual and audio features can be efficiently extracted from the input video.

[0076] 1.1 Multimodal Feature Interaction

[0077] In obtaining visual features and audio features Next, it is necessary to further explore the relationships between them to enhance the model's ability to perceive event segments in the video. Visual features and audio features Each module contains information from different modalities. Through self-attention and cross-attention mechanisms, contextual association and information complementarity within and between modalities can be achieved, thereby enhancing the model's ability to perceive event segments in videos and capturing potential cross-modal associations.

[0078] This invention achieves intermodal information exchange through a multimodal encoder. The multimodal encoder consists of multiple coding layers, each including two bimodal encoders and one shared modal encoder. The bimodal encoder consists of two self-attention layers and two cross-attention layers. The shared modal encoder is composed of two bimodal encoders.

[0079] In the In the dual-mode encoder of the coding layer, for the first... Layer encoding layer visual features and the Layer coding layer audio features Perform self-attention calculations separately to capture the contextual relationships within each modality, and obtain the first... Layer-encoded layer visual self-attention features With the Layer-encoded layer audio self-attention features Among them, the first Layer encoding layer visual features Visual features , No. Layer coding layer audio features Audio features The layer count for the coding layer starts from 1.

[0080] With the first Layer-encoded layer audio self-attention features As the key matrix and value matrix of the multi-head attention mechanism, with the first... Layer-encoded layer visual self-attention features Using the query matrix as the multi-head attention mechanism, multi-head attention calculation is performed to obtain the first... Layer coding layer vision cross-modal attention With the first Layer-encoded layer visual self-attention features As the key matrix and value matrix of the multi-head attention mechanism, with the first... Layer-encoded layer audio self-attention features Using the query matrix as the multi-head attention mechanism, multi-head attention calculation is performed to obtain the first... Layer coding layer audio cross-modal attention .

[0081] The first Layer coding layer vision cross-modal attention and the Layer coding layer audio cross-modal attention Perform feature concatenation to obtain the first... Layer coding layer mode invariant representation The first Layer coding layer vision cross-modal attention , No. Layer coding layer audio cross-modal attention , No. Layer coding layer mode invariant representation Enter the first A shared modality encoder in the layer coding layer is used to further enhance multimodal feature fusion, resulting in the 1st... Layer coding layer audio features and the Layer encoding layer visual features :

[0082] ;

[0083] ;

[0084] in, This indicates multi-head attention calculation.

[0085] The audio and visual features of the last coding layer are processed by a feedforward network to obtain the final audio code. and final visual encoding Thus, the multimodal encoder has achieved full interaction and fusion of visual and audio features, resulting in a fused multimodal feature representation.

[0086] 1.2 Generating Candidate Events

[0087] Based on the aforementioned fused multimodal features, this invention further utilizes a proposal generator to identify event segments in the input video. The final audio is then encoded. and final visual encoding The input video is fed into a proposal generator consisting of a series of proposal headers. Each proposal header is composed of a three-layer fully convolutional network. The kernel size of the first convolutional layer is a preset constant k, while the kernel size of the second and third convolutional layers is 1. The proposal generator predicts each timestamp in the input video by calculating the segment center of a possible event segment. Fragment length and confidence score It identifies event segments of different lengths in the input video, and the calculation formula is as follows:

[0088] ;

[0089] ;

[0090] ;

[0091] in, This represents the initial center position offset. This represents the preset baseline event length. This represents the sigmoid function, with normalized output to the [0,1] interval. This represents the offset predicted by the convolutional network. This represents the logarithm of the length predicted by the convolutional network. Represents an exponential function to ensure fragment length The value is positive. This represents the confidence score of the convolutional network's prediction.

[0092] In the public pool, the first set of event segments with the highest confidence scores are selected to obtain candidate events. For each candidate event, the midpoint between its start and end times in the input audio is the segment center. Its duration is the same as the segment length. .

[0093] Final audio encoding and final visual encoding All events are sequential in time. Based on the start and end times of each candidate event, the visual features of that candidate event are extracted. and event audio features :

[0094] ;

[0095] ;

[0096] in, For final visual encoding The features corresponding to the start time of the candidate events. For final visual encoding The features corresponding to the end times of candidate events. For final audio encoding The features corresponding to the start time of the candidate events. For final audio encoding The features corresponding to the end times of candidate events. This indicates a range truncation operation.

[0097] This invention uses an event location module to complete this step, as shown in the example below. Figure 2 As shown, Figure 2 In this context, Cross_Attention represents a cross-attention layer, Self_Attention represents a self-attention layer, FFN represents a feedforward network, 1-DConv represents a one-dimensional convolutional layer, and Kernel represents the kernel size of the convolutional layer. This represents a modality-invariant representation in a coding layer. Figure 2 Other variables refer only to features of a module before / after processing, such as This represents a feature before processing by the dual-modal encoder. express Features processed by a dual-modal encoder.

[0098] Through the above steps, this invention extracts a series of potential candidate events from the input video, laying the foundation for subsequent external memory knowledge fusion and event description generation. These candidate events not only contain key visual and audio information from the video but also capture the dynamic relationships between them, providing ample information support for generating accurate and rich event descriptions.

[0099] 2. External Memory Knowledge Matching and Retrieval

[0100] After generating candidate events, in order to further enrich the content of the event description, this invention introduces an external memory knowledge base and selects the most relevant external memory knowledge for each candidate event through matching and retrieval.

[0101] 2.1 Pre-built external memory knowledge base

[0102] This invention pre-constructs an external memory knowledge base integrating visual, audio, and text elements based on a target dataset. The target dataset consists of a large number of video files, with a data size that meets preset conditions to extract rich visual and audio thematic cues through clustering. Furthermore, each video in the target dataset is accompanied by corresponding manually described text, used to construct a text-based external memory knowledge base containing rich semantic information.

[0103] Specifically, following the same steps as the feature extraction method described in Section 1.1 Multimodal Feature Interaction, audio and visual features are extracted from the videos in the target dataset. The K-means algorithm is then used to cluster these visual and audio features, resulting in multiple visual cluster centers and multiple audio cluster centers. Each visual cluster center serves as an external visual knowledge base, and each audio cluster center serves as an external audio knowledge base. A text encoder is then used to encode the manually described text, resulting in multiple external text knowledge bases.

[0104] 2.2 External Memory Knowledge Matching

[0105] After constructing the external memory knowledge base, this invention further matches and retrieves the most relevant external memory knowledge from the external memory knowledge base for each candidate event. This involves analyzing the visual features of each candidate event. and event audio features Each feature is compressed into a fixed-length global feature vector through average pooling, thus obtaining the compressed visual features of the candidate event. and event audio compression features .

[0106] By calculating the visual compression features of events Event audio compression features The cosine similarity with external visual knowledge base, external audio knowledge base, and external text knowledge base is used to filter the external memory knowledge with the highest matching degree.

[0107] Event audio compression features With an external audio knowledge base The cosine similarity is:

[0108] ;

[0109] Event visual compression features With an external visual knowledge base The cosine similarity is:

[0110] ;

[0111] Event visual compression features With an external text knowledge base The cosine similarity is:

[0112] ;

[0113] in, Represents the dot product operator. This represents the L2 norm.

[0114] For each candidate event, the corresponding event audio compression features are selected from all external audio knowledge bases. The N external audio knowledge bases with the highest cosine similarity are used to obtain the N external audio memories corresponding to the candidate event; the visual compression features of the event corresponding to it are then selected from all external visual knowledge bases. The N external visual knowledge bases with the highest cosine similarity are used to obtain the N external visual memories corresponding to the candidate event; the event visual compression features corresponding to it are then selected from all external text knowledge bases. The N external text knowledge bases with the highest cosine similarity are used to obtain the N external text memories corresponding to the candidate event.

[0115] By performing self-attention computation on N external audio memories, N external visual memories, and N external text memories respectively, memory representations in their respective modalities are obtained, thus yielding multimodal external memory knowledge. ,in Represents audio memory representation. This represents visual memory representation. This represents the textual memory representation. This indicates a splicing operation.

[0116] Through the above matching and retrieval process, this invention selects the most relevant external memory knowledge for each candidate event, including audio memory, visual memory, and text memory. This external memory knowledge contains rich semantic information, which can effectively supplement the information of the candidate event itself and provide important support for generating more detailed and accurate event descriptions in the future.

[0117] 3. Gated network dynamic filtering redundancy memory

[0118] In intensive video description tasks, models need to retrieve visual, audio, and textual cues related to candidate events from massive external memory databases. However, not all retrieved external memory knowledge is relevant to the candidate events—some external memory knowledge may contain cross-scene distracting information or repetitive content. Therefore, such as... Figure 3 As shown, this invention introduces a gating network, which dynamically adjusts the importance of each piece of knowledge memory through an adaptive "soft selection" mechanism. The gating network generates a weight vector using linear transformation and a sigmoid function, quantifying the importance of each piece of knowledge memory. Based on the weights, the memory is scaled element-wise. Through the adaptive "soft selection" mechanism, memories highly relevant to the event are reinforced, while irrelevant or redundant information is suppressed. This dynamic adjustment improves the accuracy and richness of the generated descriptions and prevents irrelevant memories from interfering with model decisions, thus achieving efficient knowledge fusion in complex multimodal environments.

[0119] Gated networks for multimodal external memory knowledge After processing, matching and fusion memory features are obtained. :

[0120] ;

[0121] in, This indicates the processing of linear layers. Represents the sigmoid function

[0122] The introduction of gating networks enables the model to focus on the external memory knowledge most relevant to candidate events and appropriately suppress interfering information, providing a high-quality knowledge base for subsequent event description generation.

[0123] 4. Memory Enhancement Decoder Generates Description

[0124] The memory-enhanced decoder progressively constructs word embedding sequences through an autoregressive mechanism to generate descriptions. This invention embodiment uses " <start>"Serves as the starting marker for the word embedding sequence."

[0125] The process involves multiple steps of autoregressive iteration. In each iteration, the word embedding sequence is passed through a masked self-attention layer for self-attention computation to obtain a self-attention embedding sequence. Then, the self-attention embedding sequence is combined with the visual features of the event through a cross-attention layer. Event audio characteristics Matching and fusing memory features Interact to obtain interactive visual features Interactive audio features Interactive memory features :

[0126] ;

[0127] ;

[0128] ;

[0129] The next word embedding is then generated through a fully connected layer and an activation function. :

[0130] ;

[0131] in, For activation function, This represents the processing of the fully connected layer.

[0132] Embed words After incorporating the word embedding sequence, the iteration continues until an end marker is generated; in this embodiment of the invention, this is technically marked as "". <end>".

[0133] When the memory-enhanced decoder generates an end marker, it signifies that a complete event description has been generated. All word embeddings in the word embedding sequence, excluding the start and end markers, are converted into corresponding natural language words using a pre-trained word embedding map. These words are then concatenated sequentially according to the order in the embedding sequence to obtain the detailed description text for each candidate event. This process is repeated until all candidate events have generated corresponding descriptions, ultimately resulting in a dense description of the entire input video.

[0134] This memory-enhanced decoding mechanism can fully utilize the multimodal information of events and external memory knowledge to generate rich and accurate event descriptions. At the same time, the autoregressive iterative process makes the generated descriptive text more grammatically and semantically fluent and natural.

[0135] In summary, the dense video description method based on multimodal memory knowledge disclosed in this invention overcomes the shortcomings of existing methods that rely solely on internal video information by integrating internal video features with external memory knowledge. This allows the model to learn the correspondence between events and descriptions from more comprehensive information. Furthermore, the use of a memory-enhanced decoder enables deep interaction between multimodal video features, multimodal memory knowledge, and descriptive text, significantly improving the accuracy and richness of the generated descriptions. Therefore, this invention effectively overcomes the various shortcomings of existing technologies and possesses high industrial application value.

[0136] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.< / end> < / start>

Claims

1. A dense video description method based on multimodal memory knowledge, characterized in that, The method includes: Extract visual and audio features from the input video; Cross-modal fusion of visual and audio features is performed to generate the final audio code and the final visual code; Based on the final audio encoding and final visual encoding, the visual and audio features of multiple candidate events are determined from the input video. For each candidate event, based on its corresponding visual and audio features, matching external knowledge is retrieved from a pre-built external memory knowledge base to generate its corresponding multimodal external memory knowledge. The construction steps of the external memory knowledge base include: Extract audio and visual features from videos in a pre-prepared target dataset; Clustering algorithms are used to cluster the audio features and visual features of the dataset respectively, resulting in multiple visual cluster centers and multiple audio cluster centers. Each visual cluster center is used as an external visual knowledge base, and each audio cluster center is used as an external audio knowledge base. A text encoder is used to encode the human-written descriptive text corresponding to the videos in the target dataset, resulting in multiple external text knowledge bases; By combining external visual knowledge base, external audio knowledge base, and external text knowledge base, an external memory knowledge base is obtained. Based on the multimodal external memory knowledge, visual features, and audio features of each candidate event, a word embedding sequence is gradually constructed through an autoregressive mechanism to generate a description of the input video.

2. The dense video description method based on multimodal memory knowledge according to claim 1, characterized in that, The steps for cross-modal fusion of visual and audio features to generate the final audio and visual codes include: Multi-layer coding fusion of visual and audio features is performed using multiple coding layers. In each coding fusion layer: Self-attention calculations are performed on the visual features of the previous coding layer and the audio features of the previous coding layer to obtain the visual self-attention features and the audio self-attention features of the coding layer. After cross-modal fusion of the visual self-attention features and the audio self-attention features of the coding layer, visual features and audio features of the coding layer are generated. The audio and visual features of the last coding layer are processed by a feedforward network to obtain the final audio and visual codes.

3. The dense video description method based on multimodal memory knowledge according to claim 2, characterized in that, The steps for generating the visual and audio features of the coding layer by cross-modal fusion of the visual self-attention features and the audio self-attention features of the coding layer include: Using the visual self-attention features of the coding layer as the query and the audio self-attention features of the coding layer as the key, the coding layer visual cross-modal attention is generated. Using the audio self-attention features of the coding layer as the query and the visual self-attention features of the coding layer as the key, the audio cross-modal attention of the coding layer is generated. Feature fusion is performed on the visual cross-modal attention and audio cross-modal attention of the coding layer to generate the visual features and audio features of the coding layer.

4. The dense video description method based on multimodal memory knowledge according to claim 3, characterized in that, The steps for fusing features of the visual cross-modal attention and audio cross-modal attention of the coding layer to generate visual and audio features of the coding layer include: The first Layer coding layer vision cross-modal attention and the Layer coding layer audio cross-modal attention Perform feature concatenation to obtain the first... Layer coding layer mode invariant representation ; The following formula generates the first... Layer encoding layer visual features and the Layer coding layer audio features : ； ； in, This indicates multi-head attention computation. Indicates the first Layered encoding layer audio cross-modal attention, Indicates the first Layered coding layer visual cross-modal attention.

5. A dense video description method based on multimodal memory knowledge according to any one of claims 1-4, characterized in that, Based on the final audio encoding and final visual encoding, the steps for determining the visual and audio features of multiple candidate events from the input video include: The final audio and visual codes are input into the proposal generator to calculate the segment center, segment length, and confidence score of the event segment; From all event fragments, filter them according to confidence scores to obtain a first number of event fragments as candidate events; Based on the start and end times of each candidate event, extract the visual features of that candidate event. and event audio features : ； ； in, For final visual encoding The features corresponding to the start time of the candidate events. For final visual encoding The features corresponding to the end times of candidate events. For final audio encoding The features corresponding to the start time of the candidate events. For final audio encoding The features corresponding to the end times of candidate events. This indicates a range truncation operation.

6. The dense video description method based on multimodal memory knowledge according to claim 5, characterized in that, The steps for calculating the fragment center, fragment length, and confidence score of an event fragment include: The following formulas are used to calculate the fragment center, fragment length, and confidence score of an event fragment: ； ； ； in, The center of the event segment. Indicates the length of an event segment. This represents the confidence score of an event segment. This represents the initial center position offset. This represents the preset baseline event length. Represents the sigmoid function. This represents the offset predicted by the convolutional network. This represents the logarithm of the length predicted by the convolutional network. Represents an exponential function. This represents the confidence score of the convolutional network's prediction.

7. The dense video description method based on multimodal memory knowledge according to claim 1, characterized in that, For each candidate event, the steps of retrieving matching external knowledge from a pre-built external memory knowledge base and generating its corresponding multimodal external memory knowledge, based on its corresponding visual and audio features, include: Visual features of each candidate event and event audio features The visual compressed features of the candidate event are obtained by performing average pooling operations. and event audio compression features ; Calculate the audio compression features of the event With each external audio knowledge base The cosine similarity is calculated, and external audio memories are obtained by filtering from all external audio knowledge bases based on the calculation results; Calculate visual compression features of events With each external visual knowledge base The cosine similarity is calculated, and external visual memories are obtained by filtering from all external visual knowledge bases based on the calculation results. Calculate visual compression features of events With an external text knowledge base The cosine similarity is calculated, and external text memories are obtained by filtering from all external text knowledge bases based on the calculation results. The selected external audio memories, external visual memories, and external text memories are each subjected to self-attention computation to obtain memory representations in their respective modalities, thus yielding multimodal external memory knowledge. ,in Represents audio memory representation. This represents visual memory representation. This represents the textual memory representation. This indicates a splicing operation.

8. The dense video description method based on multimodal memory knowledge according to claim 1, characterized in that, Based on the multimodal external memory knowledge, visual features, and audio features of each candidate event, the steps to generate a description of the input video by progressively constructing a word embedding sequence through an autoregressive mechanism include: Using gating networks to access multimodal external memory knowledge Filtering is performed to obtain matching and fused memory features : ； in, This indicates the processing of linear layers. Represents the sigmoid function; Using a preset feature vector as the starting marker of the word embedding sequence, an autoregressive iteration is performed in multiple steps. In each iteration, the next word embedding is generated based on the multimodal external memory knowledge, visual features of the event, and audio features of the event for each candidate event and then added to the word embedding sequence; until a preset ending marker is generated. All word embeddings except for the start and end markers in the word embedding sequence are converted into corresponding natural language words through a pre-trained word embedding mapping table. The natural language words are then concatenated sequentially according to the order in the word embedding sequence to obtain the description of the input video.

9. The dense video description method based on multimodal memory knowledge according to claim 8, characterized in that, The steps for generating the next word embedding based on multimodal external memory knowledge, visual features of the event, and audio features of the event for each candidate event include: The word embedding sequence is processed through a masked self-attention layer to obtain the self-attention embedding sequence. ; The self-attention embedding sequence is combined with event visual features through a cross-attention layer. Event audio characteristics Matching and fusing memory features Interact to obtain interactive visual features Interactive audio features Interactive memory features : ； ； ； The next word embedding is then generated through a fully connected layer and an activation function. : ； in, For activation function, This represents the processing of the fully connected layer.

Images