Time dynamic perception model system (TDPM) for audiovisual questioning and answering and control method thereof

Abstract

The invention discloses a time dynamic perception model system (TDPM) for audiovisual questions and answers and a control method thereof, which are specially designed for audiovisual questions and answers tasks and aim at mining time dynamic information in videos and carrying out comprehensive spatio-temporal reasoning to accurately answer questions. Comprising the following steps: respectively extracting text features, visual features and audio features from an input question text, a video frame and an audio clip, and initializing the text features, the visual features and the audio features into feature vectors; through a visual time aligner and an audio time aligner, capturing time dynamic dependence between a video frame and an audio clip by using an autoregressive task; meanwhile, a text semantic aligner is constructed, question-answer pairs are converted into descriptive sentences by constructing a prompt template, the semantic gap between the question and an original text of a pre-training model is reduced, the ability of the model to understand text description is enhanced, and cross-modal interaction is promoted; and fusing the processed visual, audio and text features to carry out answer prediction. Experimental results show that the TDPM model provided by the invention shows excellent space-time reasoning ability in a plurality of audiovisual question and answer tasks, which proves that the method has huge potential in dynamic audiovisual scene understanding and has robustness to noise information.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and multimodal information processing technology, and specifically to a Temporal Dynamic Perception Model (TDPM) system for audiovisual question answering and its control method. Background Technology

[0002] With the rapid growth of multimedia data, audiovisual question answering (AVQA) has gradually become an important research direction in the field of artificial intelligence. This task requires models to infer and provide accurate answers based on understanding the complex spatiotemporal information in videos and combining natural language questions. However, existing methods still have significant shortcomings in modeling the temporal dynamics of videos and cross-modal semantic alignment: on the one hand, most models only focus on local visual or audio features, making it difficult to effectively capture the dynamic evolution patterns in long-term sequences; on the other hand, there are semantic differences between the question text and the original corpus of the pre-trained model, limiting the model's understanding ability. Therefore, there is an urgent need for a new method that can integrate language guidance information, accurately model multimodal temporal dependencies, and achieve cross-modal semantic alignment to improve the spatiotemporal reasoning performance in AVQA tasks. Summary of the Invention

[0003] To address the aforementioned issues, this invention proposes a Temporal Dynamic Perception Model (TDPM) system for audiovisual question answering and temporal dynamic dependencies in audio scenarios, and combines language modalities for complex spatial-temporal reasoning, including: A Time Dynamic Perception Model (TDPM) system and its control method for audiovisual question answering, the model and method comprising the following steps: Step 1: Extract text features, visual features, and audio features from the input question text, video frames, and audio clips, respectively, and initialize them as feature vectors; Step 2: Capture the temporal dynamic dependencies between video frames and audio clips using an autoregressive task through a visual time aligner and an audio time aligner; Step 3: Construct a text semantic aligner. By constructing prompt templates, question-answer pairs are transformed into descriptive sentences, which narrows the semantic gap between the question and the original text of the pre-trained model, enhances the model's ability to understand text descriptions, and promotes cross-modal interaction. Step 4: Fuse visual, audio, and text features; Step 5: Predict the answer.

[0004] According to a preferred embodiment, step 1 further includes: Feature Extraction: For each 1-second visual segment, a pre-trained CLIP (ViT-B / 32) visual encoder (parameters frozen) is used to extract a fixed number of frames as visual features. Specifically, a fixed number of image frames are sampled from the video per second, and then the CLIP visual encoder is used to extract a 512-dimensional feature vector for each frame. ; Audio feature extraction: For each audio segment, a pre-trained VGGish model is used to extract 128-dimensional audio features from the 16kHz sampled audio clip, and these features are then projected onto a 512-dimensional feature space through a linear layer. ; Text Feature Extraction: Question-answer pairs are converted into descriptive sentences using prompt templates. For example, the question "Where is the first sounding instrument?" and the answer "right" are transformed into "The first sounding instrument is on the right side of the video." Then, CLIP's text encoder is used to extract 512-dimensional text features. .

[0005] According to a preferred embodiment, step 2 further includes: Visual Temporal Aligner: This function uses a Transformer encoder to learn the dynamic temporal dependencies between video frames. First, the video frames are input into the Transformer encoder to generate updated visual feature vectors, as shown in the formula: ,in Indicates Transformer encoder, These are learnable parameters. Next, based on these feature vectors, an autoregressive model is constructed. This model generates future information based on historical information and language guidance, as shown in the formula: ,in This represents the previous visual frame. and These are visual and textual features, respectively. The mean squared error loss (MSE Loss) between the generated result and the actual frame is calculated to optimize the model. ; Audio Temporal Aligner: Similarly, a Transformer encoder is used to learn the dynamic temporal dependencies between audio segments. The audio segments are fed into the Transformer encoder to generate updated audio feature vectors. Then, based on these feature vectors, an autoregressive model is constructed. The model predicts the current audio segment based on historical audio segments and calculates the mean squared error loss (MSE Loss) between the generated result and the actual audio segment to optimize the model's ability to capture the temporal dynamic dependencies between audio segments. .

[0006] According to a preferred embodiment, step 3 further includes: Hint template construction: Construct hint templates to transform question-answer pairs into descriptive sentences, making the questions more consistent with the original text of the pre-trained model. For example, "Where is the first sounding instrument?" and the answer "right" are transformed into the descriptive sentence "The first sounding instrument is on the right side of the video."; Cross-modal interaction: Combining descriptive sentences with audiovisual information, and using a Transformer decoder for cross-modal interaction. ,in This refers to the audio-visual stream formed by alternating splicing of the visual and audio streams. and These are learnable parameters. Specifically, the visual and audio streams, after being alternately spliced, are integrated over time to form an audio-visual stream. This audio-visual stream is then used as a key input to the Transformer decoder, with residual links added to prevent overfitting, generating updated text features. Text feature update: The updated text features are used in subsequent cross-modal interaction and reasoning processes to improve the model's ability to understand complex scenarios.

[0007] According to a preferred embodiment, step 4 further includes: Feature fusion: Visual and audio features are alternately spliced ​​together over time to form an integrated audiovisual feature set. Specifically, visual and audio features are arranged in chronological order to create a sequential feature representation. .

[0008] According to a preferred embodiment, step 5 further includes: Model Optimization: The visual, audio, and text features processed in steps 2 and 3 are fused. Hinge loss combined with visual and audio autoregressive losses is used to optimize the overall model performance. Specifically, a Hinge loss function is defined to combine visual and audio autoregressive losses, ensuring that the model not only accurately predicts answers but also effectively captures and understands the spatiotemporal dynamics in the video. ,in , It is a comprehensive representation obtained by average pooling the integrated audiovisual features. It is the true label of the answer. It is a balancing factor; Answer score calculation: The cosine similarity is used to calculate the score function of the candidate answer; an integrated representation is obtained by performing average pooling on the integrated audiovisual features, and then compared with the text features to obtain the probability distribution of each candidate answer; finally, the answer with the highest score is selected as the prediction result. The present invention also provides an electronic device, comprising: Processor: Used to execute instructions to implement all the functions in steps 1 to 5 above. Specifically, the processor is responsible for extracting features from the input question text, video frames, and audio clips; capturing temporal dynamic dependencies through visual and audio time aligners; constructing a text semantic aligner for cross-modal interaction; and finally fusing features to predict the answer. Memory: Used to store program code and data, including pre-trained model parameters, prompt templates, intermediate calculation results, and final answer prediction results; Graphics Processing Unit (GPU): Used to accelerate the training and inference process of deep learning models. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating the Time Dynamic Perception Model System (TDPM) and its control method for audiovisual question answering according to the present invention. Figure 2 This is a schematic diagram of the overall architecture of the Time Dynamic Perception Model (TDPM) for audiovisual question answering in this invention; Figure 3 This is a table showing the experimental comparison results of the Temporal Dynamic Perception Model (TDPM) of this invention for audiovisual question answering with existing models on the MUSIC-AVQA dataset.

Claims

1. A Time Dynamic Awareness Model (TDPM) system for audiovisual question answering and its control method, characterized in that, include: Step 1: Extract text features, visual features, and audio features from the input question text, video frames, and audio clips, respectively, and initialize them as feature vectors; Step 2: Capture the temporal dynamic dependencies between video frames and audio clips using an autoregressive task through a visual time aligner and an audio time aligner; Step 3: Construct a text semantic aligner. By constructing prompt templates, question-answer pairs are transformed into descriptive sentences, which narrows the semantic gap between the question and the original text of the pre-trained model, enhances the model's ability to understand text descriptions, and promotes cross-modal interaction. Step 4: Fuse visual, audio, and text features; Step 5: Predict the answer.

2. The Time Dynamic Perception Model System (TDPM) and its control method for audiovisual question answering as described in claim 1, characterized in that, Step 1 includes: Visual Feature Extraction: For each 1-second visual segment, a pre-trained CLIP (ViT-B / 32) visual encoder (parameters frozen) is used to extract a fixed number of frames as visual features. Specifically, a fixed number of image frames are sampled from the video per second, and then the CLIP visual encoder is used to extract a 512-dimensional feature vector for each frame. ; Audio feature extraction: For each audio segment, a pre-trained VGGish model is used to extract 128-dimensional audio features from the 16kHz sampled audio clip, and these features are then projected onto a 512-dimensional feature space through a linear layer. ; Text Feature Extraction: Question-answer pairs are converted into descriptive sentences using prompt templates. For example, the question "Where is the first sounding instrument?" and the answer "right" are transformed into "The first sounding instrument is on the right side of the video." Then, CLIP's text encoder is used to extract 512-dimensional text features. .

3. The Time Dynamic Perception Model System (TDPM) and its control method for audiovisual question answering as described in claim 1, characterized in that, Step 2 includes: Visual Temporal Aligner: This function uses a Transformer encoder to learn the dynamic temporal dependencies between video frames. First, the video frames are input into the Transformer encoder to generate updated visual feature vectors, as shown in the formula: ,in Indicates Transformer encoder, These are learnable parameters; then, based on these feature vectors, an autoregressive model is constructed, which generates future information based on historical information and language guidance, as shown in the formula: ,in This represents the previous visual frame. and These are visual and textual features; the mean squared error loss (MSE Loss) between the generated result and the actual frame is calculated to optimize the model: ; Inter-segment alignment: Similarly, a Transformer encoder is used to learn the temporal dynamic dependencies between audio segments. The audio segments are input into the Transformer encoder to generate updated audio feature vectors. Then, based on these characteristics, an autoregressive model is constructed. The model calculates the mean squared error loss (MSE Loss) between the generated result and the actual audio segment based on historical audio segment predictions, in order to optimize the model's ability to capture the temporal dynamic dependencies between audio segments. .

4. The Time Dynamic Perception Model System (TDPM) and its control method for audiovisual question answering as described in claim 1, characterized in that, Step 3 includes: Hint template construction: Construct hint templates to transform question-answer pairs into descriptive sentences, making the questions more consistent with the original text of the pre-trained model; for example, "Where is the first sounding instrument?" and the answer "right" are transformed into the descriptive sentence "The first sounding instrument is on the right side of the video."; Cross-modal interaction: Combining descriptive sentences with audiovisual information, and using a Transformer decoder for cross-modal interaction. ,in This refers to the audio-visual stream formed by alternating splicing of the visual and audio streams. and These are learnable parameters; specifically, the visual and audio streams, after being alternately spliced, are integrated in the time dimension to form an audio-visual stream; then, this audio-visual stream is used as a key input to the Transformer decoder, while residual links are added to avoid overfitting, generating updated text features; Text feature update: The updated text features are used in subsequent cross-modal interaction and reasoning processes to improve the model's ability to understand complex scenarios.

5. The Time Dynamic Perception Model System (TDPM) and its control method for audiovisual question answering as described in claim 1, characterized in that, Step 4 includes: Feature fusion: Visual and audio features are alternately spliced ​​together over time to form an integrated audiovisual feature set; specifically, visual and audio features are arranged in chronological order to form a sequential feature representation. .

6. The Time Dynamic Perception Model System (TDPM) and its control method for audiovisual question answering as described in claim 1, characterized in that, Step 5 includes: Model optimization: Visual, audio, and text features processed in steps 2 and 3 are fused; Hinge loss combined with visual and audio autoregressive losses is used to optimize the overall model performance; specifically, a Hinge loss function is defined to combine visual and audio autoregressive losses to ensure that the model can not only accurately predict answers but also effectively capture and understand spatiotemporal dynamic changes in the video. ;in, , It is a comprehensive representation obtained by average pooling the integrated audiovisual features. It is the true label of the answer. It is a balancing factor; Answer score calculation: The cosine similarity is used to calculate the score function of the candidate answer; an integrated representation is obtained by performing average pooling on the integrated audiovisual features, and then compared with the text features to obtain the probability distribution of each candidate answer; finally, the answer with the highest score is selected as the prediction result.

7. The electronic device of the present invention includes: Processor: Used to execute instructions to implement all the functions in steps 1 to 5 above; specifically, the processor is responsible for extracting features from the input question text, video frames and audio clips, capturing temporal dynamic dependencies through visual time aligners and audio time aligners, constructing a text semantic aligner for cross-modal interaction, and finally fusing features to predict the answer. Memory: Used to store program code and data, including pre-trained model parameters, prompt templates, intermediate calculation results, and final answer prediction results; Graphics Processing Unit (GPU): Used to accelerate the training and inference process of deep learning models, especially for processing large-scale visual and audio features.

Images