Dense audio and video description method based on fine-grained feature extraction and multi-time scale fusion

Abstract

The invention provides a dense audio and video description method based on fine-grained feature extraction and multi-time scale fusion. The method comprises the following steps: preprocessing an original video into a standardized video frame and a transliteration text fragment set with a timestamp; generating an audio feature vector sequence fusing a time sequence and global semantic information by using the transliteration text fragment with the timestamp; generating a visual feature vector sequence with a fine-grained feature and a multi-time scale feature from the visual frame sequence; fusing the audio feature vector sequence and the visual feature vector sequence to generate a combined multi-modal feature sequence; and performing single-stage decoding on the combined multi-modal feature sequence to generate an audio and video description text. Compared with the prior art, a large number of similar actions and long-time continuous actions in a complex real scene video can be accurately understood, an accurate natural language description text can be generated, and the method is widely applied to scenes such as video abstraction, retrieval, recommendation, content auditing and teaching video understanding and has good popularization value.

Description

Technical Field

[0001] This invention relates to the field of video description processing, specifically to a dense audio and video description method based on fine-grained feature extraction and multi-temporal scale fusion, belonging to the interdisciplinary field of computer vision and natural language processing. Background Technology

[0002] Dense audio-visual description is a key technology for automatically detecting multiple event segments in a video and generating accurate natural language descriptions for each event. It has significant application value in fields such as video content understanding, retrieval, and human-computer interaction.

[0003] Complex real-world video scenes often contain multiple modalities of audio and video information, and the presence of numerous similar actions and long-duration continuous actions poses a challenge to the accuracy of dense audio and video descriptions. Existing dense video description methods typically follow an encoder-decoder framework, where a video encoder extracts features, and then a decoder completes the event localization and description generation. The C3D network proposed by Tran et al. extracts spatiotemporal features of the video through a 3D convolutional network. However, because it mainly relies on a single visual modality, it is prone to confusing behaviors with opposite intentions but similar appearances (such as "push" and "pull") when processing videos with similar actions, resulting in insufficient accuracy in the generated descriptions. The Temporal Segmentation Network (TSN) proposed by Wang et al. uses a sparse temporal sampling strategy to cover the global information of the video and adopts simple average pooling or max pooling to aggregate the features of each segment. This processing method is prone to the loss or confusion of key temporal information, making it difficult to accurately locate and describe the complete process of long-duration actions. Chen Guowen et al., in their patent "One The patent titled "A Multimodal Video Dense Event Description Algorithm with Interactive Transformer" integrates visual and audio multimodal features for encoding to generate video descriptions. This patent decomposes the dense video description task into two independent stages: segment proposal and description generation. It is a two-stage method and does not involve an end-to-end architecture. In the patent "A Video Dense Description Generation Method Based on Deep Local Self-Attention Network," Yu Zhou et al. proposed to jointly model event localization and description as a single sequence generation task, realizing an end-to-end generation framework. This method aims to simplify the model architecture but does not involve the fusion of multimodal information such as audio and video, which limits its ability to understand complex scenes. Summary of the Invention

[0004] The purpose of this invention is to provide a dense audio-visual description method based on fine-grained feature extraction and multi-temporal scale fusion. For complex videos containing numerous similar or long-duration actions, an end-to-end multimodal encoder-decoder architecture is designed. Through a motion feature activation module based on the differential method and a hierarchical cascaded multi-temporal scale feature aggregation module, high-quality visual features are automatically generated and enhanced. Combined with high-precision transcribed audio features, a single-stage decoder achieves high-precision and high-smoothness description of complex video content. The specific process includes: 1) Preprocessing the original video into a set of standardized video frames and timestamped transcribed text segments; 2) Generating an audio feature vector sequence that integrates temporal and global semantic information based on the timestamped transcribed text segments; 3) Generating a visual feature vector sequence with fine-grained and multi-temporal scale features based on the visual frame sequence; 4) Fusing the audio and visual feature vector sequences to generate a joint multimodal feature sequence. 5) Based on joint multimodal feature sequences Single-stage decoding is performed to generate audio and video description text. Specifically, the method of the present invention includes the following steps:

[0005] A. The original video is preprocessed into a set of standardized video frames and timestamped transcribed text segments. The specific steps are as follows:

[0006] A.1 Visually sample the input raw video and normalize it into a visual frame sequence. The specific steps are as follows:

[0007] A.1.1 Extract frame images from the original video at a fixed FPS to obtain the original frame sequence;

[0008] A.1.2 The extracted frame sequence is downsampled or padded to fix the number of frames to F frames, where F represents the total number of frames selected.

[0009] A.1.3 Perform resolution standardization and normalization on the extracted frame images to obtain a visual frame sequence with temporal order;

[0010] A.2 The speech of the input raw video is processed into a set of timestamped text segments. The specific steps are as follows:

[0011] A.2.1 The audio content in the video is transcribed using automatic speech recognition, and automatic punctuation and sentence segmentation are performed to divide the continuous audio into several sentence-level segments, each of which contains corresponding text content;

[0012] A.2.2 For each transcribed segment, extract its physical start time. With physical end time Forming a triple ( , );

[0013] A.2.3 Summarize the triples of all transcribed fragments in video order to obtain a set of transcribed text fragments with timestamps;

[0014] A.3 The processed visual frame sequence and the transcribed text fragments with timestamp information are used as input to the subsequent encoder to construct multimodal input data;

[0015] B. Based on the timestamped transcribed text fragments, generate an audio feature vector sequence that integrates temporal and global semantic information. The specific steps are as follows:

[0016] B.1 Transcribe the timestamped text fragments obtained in A.2 according to their physical start time. Sort the fragments from smallest to largest to obtain a sequence of transcribed fragments with temporal relationships;

[0017] B.2 Quantize the video timeline and construct a time-based lexicon system. The specific steps are as follows:

[0018] B.2.1 Obtain the total duration of the input video ;

[0019] B.2.2 Preset number of time quantization intervals and time interval Divided into equal parts A time interval;

[0020] B.2.3 Introducing vocabulary into the text segmenter Time-related words to ;

[0021] B.2.4 The physical start time of each transcribed segment Mapped to discrete-time index ,in ;

[0022] B.2.5 The physical end time of each transcribed segment Mapped to discrete-time index ;

[0023] B.3 Perform text segmentation on the transcribed text fragments and construct a text-time hybrid word sequence. The specific steps are as follows:

[0024] B.3.1 The text content of each transcribed segment is segmented using a word segmenter to obtain a text word sequence;

[0025] B.3.2 Concatenate the lexical units of each transcribed segment in the order of "start time lexical unit + end time lexical unit + text lexical unit sequence" to obtain a segment-level mixed lexical unit sequence;

[0026] B.3.3 Concatenate the fragment-level mixed word sequences of all transcribed fragments sequentially in time order to obtain the encoder input word sequence. ;

[0027] B.4 Embedding and mapping the text-time hybrid word sequence and encoding it into an audio feature vector are performed as follows:

[0028] B.4.1 The input word sequence obtained in B.3 The input is fed into the lexical embedding layer, resulting in a vector sequence containing positional information for each lexical unit. ;

[0029] B.4.2 Transform the vector sequence The input is encoded by a Transformer encoder to obtain a sequence of output feature vectors. , as an audio feature vector that integrates temporal and global semantic information;

[0030] C. Based on the visual frame sequence, generate a visual feature vector sequence with fine-grained features and multi-temporal scale features. The specific steps are as follows:

[0031] C.1 Extract frame-level features from the visual frame sequence and construct the original visual temporal feature tensor. The specific steps are as follows:

[0032] C.1.1 The visual frame sequence with temporal order obtained in step A.1 (number of frames) The input is a visual feature extraction network, which performs feature extraction independently on each frame to obtain frame-level features arranged in time.

[0033] C.1.2 Stack the features of each frame in temporal order along the time dimension to construct the original visual temporal feature tensor. , in This indicates the total number of frames selected from the input video. The number of channels representing frame-level features. and These represent the height and width of the frame-level feature in the spatial dimension, respectively;

[0034] C.2 Convert the original visual feature tensor Motion feature enhancement based on the difference method is performed to obtain enhanced visual features. The specific steps are as follows:

[0035] C.2.1 On the original visual feature tensor Perform channel compression / transformation to obtain compression characteristics. ,in This refers to the compression ratio;

[0036] C.2.2 Perform feature transformation processing on the compressed features of adjacent time steps in the time dimension, and calculate the difference between the compressed features of the current time step and the transformed compressed features of adjacent time steps to obtain the inter-frame motion representation. ,in ,when The process involves copying the first frame.

[0037] C.2.3 Representing inter-frame motion at each time point By splicing the sequences chronologically along the time dimension, a motion sequence can be obtained. ;

[0038] C.2.4 Motion Sequences Perform pooling operations in the spatial dimension to obtain motion summary features. ;

[0039] C.2.5 Characteristics of motion summary A averaging operation is performed in the time dimension to extract the global motion feature vector. The vector is then subjected to channel modeling and nonlinear transformation to generate channel excitation weights. ;

[0040] C.2.6 Utilizing Channel Excitation Weights Original visual features Recalibration is performed, and residual connections are used for fusion to obtain enhanced visual features after motion excitation. ;

[0041] C.3 Enhancement of visual features Perform hierarchical, cascaded, multi-timescale feature aggregation to obtain a sequence of visual feature vectors. The specific steps are as follows:

[0042] C.3.1 will enhance visual features Divided along the channel into A subset of features, wherein ;

[0043] C.3.2 The first feature subset is directly output as the short-time baseline feature;

[0044] C.3.3 Regarding the first A subset of features ( ) and the The output features of the first level are fused, and the fusion result is subjected to temporal modeling to form the second level. Level output characteristics;

[0045] C.3.4 Concatenate the output features at each level along the channel dimension to obtain a visual feature vector sequence with fine-grained motion features and multi-timescale features. ;

[0046] D. Fuse audio feature vector sequences and visual feature vector sequences to generate a joint multimodal feature sequence. The specific steps are as follows:

[0047] D.1 The audio feature vector sequence obtained in step B Perform a linear mapping to map it to the preset feature dimension. The aligned audio feature sequence is obtained. ;

[0048] D.2 Visual features obtained in step C Perform pooling to obtain a sequence of visual feature vectors arranged in time. ;

[0049] D.3 Visual feature vector sequences Perform a linear mapping, mapping it to a preset feature dimension. The aligned visual feature sequence is obtained. ;

[0050] D.4 Aligned audio feature sequences Aligned visual feature sequence By concatenating along the sequence dimension, a joint multimodal feature sequence is obtained. Here, alignment refers to feature dimension alignment (dimensions are unified as follows). The sequence is spliced ​​along the sequence dimension to construct a joint sequence input for subsequent decoder processing, rather than aligning audio features and visual features one-to-one in time steps;

[0051] E. Based on joint multimodal feature sequences Perform single-stage decoding to generate audio and video description text. The specific steps are as follows:

[0052] E.1 Starting word <bos>Write the first and second characters of the output word sequence to obtain the initial output word sequence. ;

[0053] E.2 The current output word sequence The decoder performs mask self-attention calculation to obtain the current decoding hidden state;

[0054] E.3 Combine the decoded hidden state with the conditional feature sequence obtained in step D. Perform cross-attention calculation to obtain the hidden state that incorporates multimodal conditional information;

[0055] E.4 Perform output layer mapping on the fused hidden state to obtain the probability distribution of the next word and generate the next word. and will Append to the output word sequence;

[0056] E.5 Repeat steps E.2 to E.4 until the end word is generated. <eos>And terminate the generation;

[0057] E.6 Parse the final output word sequence, and restore the time words and text words into event fragments and their descriptive text in the format of "start time word + end time word + text word sequence".

[0058] Compared with existing technologies, this invention has the following advantages and functions: This invention provides a dense audio-video description method based on fine-grained feature extraction and multi-temporal scale fusion. Through an end-to-end multimodal encoder-decoder architecture, it achieves the generation of descriptions for complex video content. In audio processing, by introducing a temporal lexical system, speech semantics and discretized temporal indices are uniformly encoded, and a Transformer encoder is used to jointly model semantic expression and temporal boundary information, thereby improving temporal expression capabilities and semantic-temporal boundary alignment capabilities. In visual feature extraction, motion feature excitation based on inter-frame difference and hierarchical cascaded multi-temporal scale feature aggregation are adopted to enhance the representation ability of fine-grained dynamic changes and long-term evolution processes, helping to alleviate the difficulty in distinguishing similar actions and the description bias caused by long-term information decay. In the decoding and generation stages, a single-stage decoding method is used to generate an output sequence containing temporal boundary information, and it is parsed and restored into event segments with start and end times and their corresponding text descriptions, achieving effective fusion and structured output of multimodal audio-video information. The structured results generated by this method can be applied to scenarios such as video summarization, retrieval and recommendation, content review, and comprehension of teaching videos, and have good engineering application and promotion value. Attached Figure Description

[0059] Figure 1 Flowchart of a dense audio and video description method based on fine-grained feature extraction and multi-temporal scale fusion Detailed Implementation

[0060] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0061] This invention provides a dense audio-visual description method based on fine-grained feature extraction and multi-temporal-scale fusion. The method includes: 1) preprocessing the original video into a set of standardized video frames and timestamped transcribed text segments; 2) generating an audio feature vector that integrates temporal and global semantic information based on the timestamped transcribed text segments; 3) generating a visual feature vector with fine-grained and multi-temporal-scale features based on a visual frame sequence; and 4) fusing the audio feature sequence and the visual feature vector sequence to generate a joint multimodal feature sequence. 5) Based on joint multimodal feature sequences Perform single-stage decoding to generate audio and video description text. Taking the "mer7ECSKflk" video from the YOUCOOK2 dense video description dataset as an example, the steps are explained in detail below:

[0062] 1. Preprocess the original video into a set of standardized video frames and timestamped transcribed text segments. The specific steps are as follows:

[0063] 1.1 Visually sample the input raw video and normalize it into a visual frame sequence. The specific steps are as follows:

[0064] 1.1.1 Setting the sampling rate For a duration of The video is extracted frame by frame, second by second, to obtain the original frame sequence. A total of 255 frames;

[0065] 1.1.2 Setting the system standard input frame number Due to the original number of frames Perform uniform downsampling to obtain The downsampling index is defined as:

[0066]

[0067] And order Therefore, the first frame corresponds to The 100th frame corresponds to ;

[0068] 1.1.3 Resize the selected 100 frames to... The image frame sequence tensor is then normalized according to the ImageNet mean and variance to obtain the image frame sequence tensor. ;

[0069] 1.2 The speech of the input original video is processed into a set of timestamped text fragments. The specific steps are as follows:

[0070] 1.2.1 WhisperX is used to transcribe the audio content in the video, and automatic punctuation and sentence segmentation are performed. Continuous audio is divided into several sentence-level segments, each containing corresponding text content. Example of transcription result:

[0071] Seg_1: "I called these Carne Asada but the skirt steak is cookedArrachera style."

[0072] Seg_2: "Marinade: 1 / 2 Cup Beer."

[0073] Seg_3: "Juice of 1 Lime."

[0074] Seg_4: "1 1 / 2 Tablespoon Worcestershire Sauce."

[0075] Seg_5: "2 Cloves of Garlic Sliced."

[0076] Seg_6: "Marinate meat for 2 or 3 hours in the refrigerator."

[0077] Seg_7: "Cook on grill until desired doneness is achieved."

[0078] Seg_8: "Dice into cubes, for the final cut cut across the grain.";

[0079] 1.2.2 For each transcribed segment, extract the physical start time and physical end time to form a triple. For example, for the segment "Marinade 1 / 2 Cup Beer.", we get: ;

[0080] 1.2.3 Summarize the triples of all transcribed fragments in video order to obtain a set of timestamped transcribed text fragments. ;

[0081] 1.3 Transforming the visual frame sequence tensor With a collection of transcribed text fragments with timestamps As input to the subsequent encoder, multimodal input data is constructed;

[0082] 2. Based on the timestamped transcribed text fragments, generate an audio feature vector sequence that integrates temporal and global semantic information. The specific steps are as follows:

[0083] 2.1 Sorting of transcribed fragments by The speech segments are sorted to obtain a sequence of transcribed segments with temporal relationships;

[0084] 2.2 The video timeline is quantized and a time-based lexicon system is constructed. The specific steps are as follows:

[0085] 2.2.1 Obtain the total video duration ;

[0086] 2.2.2 Setting the number of quantization intervals ,Will Divided into 100 equal time intervals;

[0087] 2.2.3 Introduce 100 time-related terms into the vocabulary of the text segmenter. to ;

[0088] 2.2.4 Map the physical start time of each transcribed fragment to a discrete-time index, using the fragment "Marinade 1 / 2 Cup Beer" as an example. For example:

[0089]

[0090] 2.2.5 Map the physical end time of each transcribed segment to a discrete-time index. For the same segment:

[0091]

[0092] 2.3 Perform text segmentation on the transcribed text fragments and construct a text-time hybrid word sequence. The specific steps are as follows:

[0093] 2.3.1 Perform Subword segmentation on the text content of each transcribed segment. Taking "Marinade 1 / 2 CupBeer" as an example, the text word sequence is: ["Marinade", "1 / 2", "Cup", "Beer"];

[0094] 2.3.2 Concatenate the lexical units of each transcribed segment in the order of "start time lexical unit + end time lexical unit + text lexical unit sequence" to obtain a segment-level mixed lexical unit sequence: [ , ,"Marinade","1 / 2","Cup","Beer"];

[0095] 2.3.3 Concatenate all fragment-level mixed word sequences sequentially according to time order to obtain the encoder input word sequence. ,when If the length exceeds the maximum length of 512, it is truncated; if it is insufficient, it is padded to make the final length 512.

[0096] 2.4 Embedding and mapping the text-time hybrid word sequence and encoding it into an audio feature vector are performed as follows:

[0097] 2.4.1 Input sequence Through lexical embedding layer and position embedding layer Mapped to a vector sequence containing location information :

[0098]

[0099] in, To be related to the input sequence The corresponding position index sequence is used to extract the position embedding information of each word;

[0100] 2.4.2 Will The input is to a 6-layer, 12-head Transformer encoder ( The audio feature vector sequence, which integrates temporal and global semantic information, is encoded in the following manner:

[0101]

[0102] 3. Based on the visual frame sequence, generate a visual feature vector sequence with fine-grained features and multi-temporal scale features. The specific steps are as follows:

[0103] 3.1 Extract frame-level features from the visual frame sequence and construct the original visual temporal feature tensor. The specific steps are as follows:

[0104] 3.1.1 Input the 100 frames of images obtained in step 1.1 into ResNet-50, extract the last convolutional features for each frame, and obtain the feature size of each frame as follows: ;

[0105] 3.1.2 Stack the features of each frame along the time dimension to construct the original visual temporal feature tensor: ;

[0106] 3.2 Convert the original visual feature tensor Motion feature enhancement based on the difference method is performed to obtain enhanced visual features. The specific steps are as follows:

[0107] 3.2.1 Setting the compression ratio ,pass Convolution compresses the number of channels from 2048 to 2048 / 16 = 128, resulting in: ;

[0108] 3.2.2 Compression features of adjacent moments in the time dimension Take the identity mapping, that is The difference between the current compressed feature and the transformed compressed feature of the adjacent time step is calculated and differentially analyzed to obtain the inter-frame motion representation:

[0109]

[0110] 3.2.3 At each time point By splicing the sequences chronologically along the time dimension, a motion sequence can be obtained. :

[0111]

[0112] 3.2.4 Motion Sequences In spatial dimension Perform global average pooling to obtain motion summary features. :

[0113]

[0114] 3.2.5 Characteristics of Motion Summary Channel modeling is performed and processed through a nonlinear function to generate channel excitation weights α. Motion summarization features are then analyzed first. Averaging is performed over time to obtain the global motion feature vector. :

[0115]

[0116] Then, the global motion feature vector is processed through two fully connected layers. The dimension is mapped from 128 to the original number of channels 2048, and the channel activation weights are obtained through the Sigmoid function. :

[0117]

[0118] in

[0119] 3.2.6 Utilizing Channel Excitation Weights Original visual features Recalibration is performed, and residual connections are used for fusion to obtain enhanced visual features:

[0120]

[0121] in This indicates element-wise channel multiplication;

[0122] 3.3 Enhancement of visual features Perform hierarchical, cascaded, multi-timescale feature aggregation to obtain a sequence of visual feature vectors. The specific steps are as follows:

[0123] 3.3.1 Will Divided along the channel into There are feature subsets, each subset having channels. :

[0124]

[0125] 3.3.2 Directly output the first feature subset as the short-time baseline feature:

[0126]

[0127] 3.3.3 Regarding the first A subset of features ( ) and the The output features of the first level are fused, and the fusion result is subjected to temporal modeling to form the second level. Level output characteristics:

[0128]

[0129]

[0130] in, This represents a temporal convolution operation performed in the time dimension;

[0131] 3.3.4 The output features at each level are concatenated along the channel dimension to obtain a sequence of visual feature vectors. :

[0132]

[0133] 4. Fuse audio feature vector sequences and visual feature vector sequences to generate a joint multimodal feature sequence. The specific steps are as follows:

[0134] 4.1 The audio features obtained in step 2 are then processed... Perform linear mapping to a preset feature dimension The aligned audio feature sequence is obtained. :

[0135]

[0136] in, The learnable weight matrix is ​​a linear mapping. For bias terms;

[0137] 4.2 Visual features obtained in step 3 In space dimension Perform global average pooling to obtain a sequence of visual feature vectors arranged in time:

[0138] 4.3 Visual Feature Vector Sequences Perform linear mapping to a preset feature dimension The aligned visual feature sequence is obtained. :

[0139]

[0140] in, The learnable weight matrix is ​​a linear mapping. For the corresponding bias term;

[0141] 4.4 and By concatenating along the sequence dimension, a joint multimodal feature sequence is obtained. :

[0142]

[0143] Here, alignment refers to feature dimension alignment (unified as) The sequence is spliced ​​along the sequence dimension to construct a joint sequence input for subsequent decoder processing, rather than aligning audio and visual features one-to-one at time steps;

[0144] 5. Based on joint multimodal feature sequences Perform single-stage decoding to generate audio and video description text. The specific steps are as follows:

[0145] 5.1 Starting word Write the first and second parts of the output word sequence to obtain the initial output word sequence. 5.2 Output the current word sequence Input the decoder and perform mask self-attention computation to obtain the current decoding hidden state. :

[0146]

[0147] in, This represents the embedding vector of the generated word sequence; This represents a masked multi-head self-attention mechanism used to ensure the first... Step decoding can only observe the previous step. Information of each word element;

[0148] 5.3 Decode the hidden state The conditional feature sequence obtained in step 4 Perform cross-attention computation to obtain the hidden state that incorporates multimodal conditional information. :

[0149]

[0150] in, It is a learnable linear mapping matrix;

[0151] 5.4 Hidden states after fusion Perform output layer mapping to obtain the probability distribution of the next word and generate the next word. And append it to the output word sequence;

[0152] 5.5 Repeat steps 5.2 to 5.4 until the end word is generated. And terminate the generation;

[0153] 5.6 Parse the final output lexical sequence: Reconstruct the event fragment and its descriptive text according to the format "start time lexical + end time lexical + text lexical sequence"; and reverse map the time lexical index to physical time, using the following formula:

[0154]

[0155] In this embodiment The model output fragment is:

[0156]

[0157] The corresponding physical time is:

[0158]

[0159] Get event fragments Its text description is "Grill the steak on the grill".

[0160] This embodiment implements a complete method for dense audio and video description of long videos: taking the 255-second video mer7ECSKflk.mp4 as an example, it adopts... Frame extraction and uniform downsampling are used to fix the visual frame count. Simultaneously, ASR transcription is performed on the speech, and start and end time triples of sentence-level segments are extracted; the total video duration is calculated on the audio side. Quantified as Each time interval is divided into time segments, and time terms are introduced. A hybrid sequence is constructed for each segment using "start time term + end time term + text term sequence". This sequence is then encoded using Transformer to obtain audio features that fuse temporal and semantic information. On the visual side, convolutional features are extracted from 100 frames, and a compression ratio is applied... The adjacent time-difference motion excitation, and with The grouping and cascading method is used to aggregate features at multiple time scales to obtain enhanced visual features that take into account both short-term actions and long-term processes; in the fusion and generation stage, audio and visual features are mapped to a unified dimension. The sequence is then concatenated along the sequence dimension to form a joint conditional sequence. A single-stage autoregressive decoding process generates an output sequence containing both temporal and textual terms. The start and end times of the event segments and their corresponding textual descriptions are then parsed and restored according to a preset format, resulting in a structured, dense description with temporal boundaries. This embodiment collaboratively implements temporal term constraints, motion excitation enhancement, and multi-timescale aggregation within an end-to-end framework. It can automatically locate and textualize key events in long videos, and can be used in scenarios such as video summarization, retrieval and recommendation, content moderation, and instructional video comprehension, demonstrating significant engineering application and promotional value.

[0161] Finally, it should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the claims.< / eos> < / bos>

Claims

1. A dense audio-visual description method based on fine-grained feature extraction and multi-temporal scale fusion, comprising the following steps: The original video is preprocessed into a set of standardized video frames and timestamped transcribed text segments. The specific steps are as follows: A.1 Visually sample the input raw video and normalize it into a visual frame sequence. The specific steps are as follows: A.1.1 Extract frame images from the original video at a fixed FPS to obtain the original frame sequence; A.1.2 The extracted frame sequence is downsampled or padded to fix the number of frames to F frames, where F represents the total number of frames selected. A.1.3 Perform resolution standardization and normalization on the extracted frame images to obtain a visual frame sequence with temporal order; A.2 The speech of the input raw video is processed into a set of timestamped text segments. The specific steps are as follows: A.2.1 The audio content in the video is transcribed using automatic speech recognition, and automatic punctuation and sentence segmentation are performed to divide the continuous audio into several sentence-level segments, each of which contains corresponding text content; A.2.2 For each transcribed segment, extract its physical start time. With physical end time Forming a triple ( , ); A.2.3 Summarize the triples of all transcribed fragments in video order to obtain a set of transcribed text fragments with timestamps; A.3 The processed visual frame sequence and the transcribed text fragments with timestamp information are used as input to the subsequent encoder to construct multimodal input data; Based on the timestamped transcribed text fragments, an audio feature vector sequence that integrates temporal and global semantic information is generated. The specific steps are as follows: B.1 Transcribe the timestamped text fragments obtained in A.2 according to their physical start time. Sort the fragments from smallest to largest to obtain a sequence of transcribed fragments with temporal relationships; B.2 Quantize the video timeline and construct a time-based lexicon system. The specific steps are as follows: B.2.1 Obtain the total duration of the input video ; B.2.2 Preset number of time quantization intervals and time interval Divided into equal parts A time interval; B.2.3 Introducing vocabulary into the text segmenter Time-related words to ; B.2.4 The physical start time of each transcribed segment Mapped to discrete-time index ,in ; B.2.5 The physical end time of each transcribed segment Mapped to discrete-time index ; B.3 Perform text segmentation on the transcribed text fragments and construct a text-time hybrid word sequence. The specific steps are as follows: B.3.1 The text content of each transcribed segment is segmented using a word segmenter to obtain a text word sequence; B.3.2 Concatenate the lexical units of each transcribed segment in the order of "start time lexical unit + end time lexical unit + text lexical unit sequence" to obtain a segment-level mixed lexical unit sequence; B.3.3 Concatenate the fragment-level mixed word sequences of all transcribed fragments sequentially in time order to obtain the encoder input word sequence. ; B.4 Embed and map the text-time hybrid word sequence, and encode it into an audio feature vector; Based on the visual frame sequence, a visual feature vector sequence with fine-grained features and multi-temporal scale features is generated. The specific steps are as follows: C.1 Extract frame-level features from the visual frame sequence and construct the original visual temporal feature tensor. ; C.2 Convert the original visual feature tensor Motion feature enhancement based on the difference method is performed to obtain enhanced visual features. ; C.3 Enhancement of visual features Perform hierarchical, cascaded, multi-timescale feature aggregation to obtain a sequence of visual feature vectors. ; By fusing audio feature vector sequences and visual feature vector sequences, a joint multimodal feature sequence is generated. The specific steps are as follows: D.1 The audio feature vector sequence obtained in step B Perform a linear mapping to map it to the preset feature dimension. The aligned audio feature sequence is obtained. ; D.2 Visual features obtained in step C Perform pooling to obtain a sequence of visual feature vectors arranged in time. ; D.3 Visual feature vector sequences Perform a linear mapping, mapping it to a preset feature dimension. The aligned visual feature sequence is obtained. ; D.4 Aligned audio feature sequences Aligned visual feature sequence By concatenating along the sequence dimension, a joint multimodal feature sequence is obtained. Here, alignment refers to feature dimension alignment (dimensions are unified as follows). The sequence is spliced ​​along the sequence dimension to construct a joint sequence input for subsequent decoder processing, rather than aligning audio features and visual features one-to-one in time steps; Based on joint multimodal feature sequences Perform single-stage decoding to generate audio and video description text. The specific steps are as follows: E.1 Starting word <bos>Write the first and second characters of the output word sequence to obtain the initial output word sequence. ;< / bos> E.2 The current output word sequence The decoder performs mask self-attention calculation to obtain the current decoding hidden state; E.3 Combine the decoded hidden state with the conditional feature sequence obtained in step D. Perform cross-attention calculation to obtain the hidden state that incorporates multimodal conditional information; E.4 Perform output layer mapping on the fused hidden state to obtain the probability distribution of the next word and generate the next word. and will Append to the output word sequence; E.5 Repeat steps E.2 to E.4 until the end word is generated. <eos> And terminate the generation;< / eos> E.6 Parse the final output word sequence, and restore the time words and text words into event fragments and their descriptive text in the format of "start time word + end time word + text word sequence".

2. The dense audio-video description method based on fine-grained feature extraction and multi-temporal scale fusion as described in claim 1, which embeds and maps text-temporal mixed word sequences and encodes them into audio feature vectors, includes the following specific steps: B.4.1 The input word sequence obtained in B.3 The input is fed into the lexical embedding layer, resulting in a vector sequence containing positional information for each lexical unit. ; B.4.2 Transform the vector sequence The input is encoded by a Transformer encoder to obtain a sequence of output feature vectors. , as an audio feature vector that integrates temporal and global semantic information.

3. The dense audio-visual description method based on fine-grained feature extraction and multi-temporal-scale fusion as described in claim 1 extracts frame-level features of the visual frame sequence and constructs the original visual temporal feature tensor. The specific steps are as follows: C.1.1 The visual frame sequence with temporal order obtained in step A.1 (number of frames) The input is a visual feature extraction network, which performs feature extraction independently on each frame to obtain frame-level features arranged in time. C.1.2 Stack the features of each frame in temporal order along the time dimension to construct the original visual temporal feature tensor. , in This indicates the total number of frames selected from the input video. The number of channels representing frame-level features. and These represent the height and width of the frame-level feature in the spatial dimension, respectively.

4. The dense audio-visual description method based on fine-grained feature extraction and multi-temporal-scale fusion as described in claim 1, wherein the original visual feature tensor... Motion feature enhancement based on the difference method is performed to obtain enhanced visual features. The specific steps are as follows: C.2.1 On the original visual feature tensor Perform channel compression / transformation to obtain compression characteristics. ,in This refers to the compression ratio; C.2.2 Perform feature transformation processing on the compressed features of adjacent time steps in the time dimension, and calculate the difference between the compressed features of the current time step and the transformed compressed features of adjacent time steps to obtain the inter-frame motion representation. ,in ,when The process involves copying the first frame. C.2.3 Representing inter-frame motion at each time point By splicing the sequences chronologically along the time dimension, a motion sequence can be obtained. ; C.2.4 Motion Sequences Perform pooling operations in the spatial dimension to obtain motion summary features. ; C.2.5 Characteristics of motion summary A averaging operation is performed in the time dimension to extract the global motion feature vector. The vector is then subjected to channel modeling and nonlinear transformation to generate channel excitation weights. ; C.2.6 Utilizing Channel Excitation Weights Original visual features Recalibration is performed, and residual connection is used for x-fusion to obtain enhanced visual features after motion excitation. .

5. The dense audio-visual description method based on fine-grained feature extraction and multi-temporal scale fusion as described in claim 1, enhances visual features. Perform hierarchical, cascaded, multi-timescale feature aggregation to obtain a sequence of visual feature vectors. The specific steps are as follows: C.3.1 will enhance visual features Divided along the channel into A subset of features, wherein ; C.3.2 The first feature subset is directly output as the short-time baseline feature; C.3.3 Regarding the first A subset of features ( ) and the The output features of the first level are fused, and the fusion result is subjected to temporal modeling to form the second level. Level output characteristics; C.3.4 Concatenate the output features at each level along the channel dimension to obtain a visual feature vector sequence with fine-grained motion features and multi-timescale features. .

Images