Audio and video question answering method, device and equipment based on multi-modal scene graph, medium and product

Abstract

The embodiment of the invention provides an audio and video question answering method and device based on a multi-modal scene graph, equipment, a medium and a product. The method comprises the following steps: obtaining a pre-trained audio and video question and answer model, wherein the pre-trained audio and video question and answer model comprises a multi-modal feature fusion network, a pre-trained multi-modal scene graph generation network and an edge learning expert time sequence integration network; inputting the multi-modal initial data into a pre-training encoder to obtain multiple modal features; inputting the multiple modal features into a multi-modal feature fusion network to obtain audio and video features and modal fusion features; inputting the audio and video features into a pre-training multi-modal scene graph generation network to obtain scene graph triple features; inputting the modal fusion features into an edge learning expert time sequence integration network to obtain time sequence integration features; performing feature aggregation processing on the scene graph triple feature and the time sequence integration feature to obtain a target feature vector; and inputting the target feature vector into a pre-training classifier to output a target question and answer result. The effect of improving the accuracy of the question and answer result is achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an audio and video question answering method, apparatus, device, medium and product based on multimodal scene graphs. Background Technology

[0002] With the continuous development of multimodal artificial intelligence, audio-visual question answering (AVQA) is a technology that simulates the comprehensive perception and reasoning ability of humans. It requires the model to receive visual, auditory and natural language input at the same time, and to comprehensively process temporal information, cross-modal relationships and semantic requirements to obtain an answer that matches the intent of the question.

[0003] In existing technologies, video content is generally the core element. The video is first encoded at the frame or segment level to obtain the primary visual representation. Audio is often simplified into global or coarse-grained auxiliary features, and text questions, after semantic encoding, are mainly used to guide attention in allocating weights among visual segments. The fusion stage typically maps visual, audio, and text to a unified representation space, and then uses a general prediction head for classification or generative output. However, existing technologies suffer from insufficient analysis of video-related information when dealing with multimodal data, resulting in low accuracy of question-answering outputs. Summary of the Invention

[0004] This application provides an audio-visual question-answering method, apparatus, device, medium, and product based on a multimodal scene graph, which aims to improve the accuracy of question-answering output results.

[0005] In a first aspect, embodiments of this application provide an audio / video question-answering method based on a multimodal scene graph, including:

[0006] Obtain a pre-trained audio / video question answering model and multimodal initial data, wherein the pre-trained audio / video question answering model includes a multimodal feature fusion network, a pre-trained multimodal scene graph generation network, an edge learning expert temporal integration network, and a pre-trained classifier;

[0007] Feature extraction is performed based on the multimodal initial data and the pre-trained encoder to obtain multiple modal features;

[0008] The various modal features are input into the multimodal feature fusion network to obtain audio-visual features and modal fusion features;

[0009] The audio and video features are input into a pre-trained multimodal scene graph generation network to obtain the scene. Figure 3 Tuple characteristics;

[0010] The modality fusion features are input into the temporal integration network of the edge learning expert to obtain the temporal integration features.

[0011] The scenario Figure 3 The tuple features and the aforementioned temporal integration features are subjected to feature aggregation processing to obtain a target feature vector that aggregates multimodal temporal information;

[0012] The target feature vector is input into a pre-trained classifier, and the target question answering result is output based on multiple predefined candidate answers.

[0013] In one possible implementation, the multimodal initial data includes audio data, video data, and text question data;

[0014] Then, feature extraction is performed based on the multimodal initial data and the pre-trained encoder to obtain multiple modal features, including:

[0015] Audio features are obtained by extracting audio data from a pre-trained audio encoder.

[0016] Based on the video data, visual extraction is performed using a pre-trained visual encoder to obtain frame-level features and image patch-level features;

[0017] Based on the text question data, text extraction is performed using a pre-trained text encoder to obtain sentence-level features and word-level features;

[0018] The audio features, frame-level features, image patch-level features, sentence-level features, and word-level features constitute multiple modal features.

[0019] In one possible implementation, the classifier is composed of fully connected layers, which calculate the probability distribution through a normalized exponential function;

[0020] The step of inputting the target feature vector into a pre-trained classifier and outputting the target question-answering result based on multiple predefined candidate answers includes:

[0021] The target feature vector is input into the pre-trained classifier, and the fully connected layer calculates the probability value corresponding to each predefined candidate answer through a normalized exponential function;

[0022] Based on the probability values ​​corresponding to each predefined candidate answer, a target predefined candidate answer is determined from multiple predefined candidate answers as the target question answer result output, wherein the probability value corresponding to the target predefined candidate answer is greater than the probability value corresponding to any one of the multiple predefined candidate answers.

[0023] In one possible implementation, obtaining the pre-trained audio / video question-answering model includes:

[0024] Obtain a multimodal dataset and an initial audio / video question answering model, wherein the initial audio / video question answering model includes a multimodal feature fusion network, an initial multimodal scene graph generation network, an edge learning expert temporal integration network, and an initial classifier;

[0025] The annotation set of the multimodal scene graph is obtained by annotating the multimodal dataset and the pre-trained multimodal large language model.

[0026] The multimodal dataset is input into the pre-trained encoder for feature extraction to obtain multimodal sample features;

[0027] The multimodal sample features are input into the multimodal feature fusion network to obtain sample audio and video features and sample modality fusion features;

[0028] The sample audio and video features are input into the initial multimodal scene graph generation network to obtain the predicted scene triplet features;

[0029] The differences in triplet features are determined based on the preset training loss function, the annotation set of the multimodal scene graph, and the triplet features of the predicted scene.

[0030] When the detected difference in the triplet features exceeds a preset difference threshold, the initial multimodal scene graph generation network is adjusted to obtain an adjusted multimodal scene graph generation network. The process then returns to the step of inputting the sample audio and video features into the initial multimodal scene graph generation network to obtain predicted scene triplet features, until the difference in the triplet features is less than or equal to the preset difference threshold. The last adjusted multimodal scene graph generation network is then used as a pre-trained multimodal scene graph generation network, and the last predicted scene triplet features are used as the target sample scene triplet features.

[0031] The sample modality fusion features are input into the edge learning expert temporal integration network to obtain sample temporal integration features;

[0032] The sample time-series integrated features and the target sample scene triplet features are subjected to feature aggregation processing to obtain the sample feature vector;

[0033] The sample feature vector is input into the initial classifier and question-answering result supervision processing is performed according to multiple predefined candidate answers and a preset task classification loss function to train the initial classifier and obtain a pre-trained classifier.

[0034] The multimodal feature fusion network, the pre-trained multimodal scene graph generation network, the edge learning expert temporal integration network, and the pre-trained classifier are used to construct a pre-trained audio and video question answering model.

[0035] In one possible implementation, the multimodal dataset includes each video and the corresponding audio.

[0036] The annotation process, which involves labeling a multimodal dataset and a pre-trained multimodal large language model to obtain an annotation set for the multimodal scene graph, includes:

[0037] Based on each video and the corresponding audio, and a pre-trained multimodal large language model, the pre-trained multimodal large language model is used to annotate each time segment of each video using a preset prompt template, resulting in audio-video information relationship triples as the annotation set for the multimodal scene graph.

[0038] In one possible implementation, it also includes:

[0039] Obtain a multimodal question-answering test set and preset standard answers;

[0040] The multimodal question answering test set is input into the pre-trained audio and video question answering model to obtain the test prediction question answering results;

[0041] The test prediction results are compared with the preset standard answers to fine-tune the pre-trained audio and video question answering model and obtain the target audio and video question answering model.

[0042] Secondly, embodiments of this application provide an audio / video question-answering device based on a multimodal scene graph, comprising:

[0043] The acquisition module is used to acquire a pre-trained audio and video question answering model and multimodal initial data, wherein the pre-trained audio and video question answering model includes a multimodal feature fusion network, a pre-trained multimodal scene graph generation network, an edge learning expert temporal integration network, and a pre-trained classifier;

[0044] The model question answering module is used to perform feature extraction processing based on the multimodal initial data and the pre-trained encoder to obtain multimodal features;

[0045] The model question answering module is also used to input the multiple modal features into the multimodal feature fusion network to obtain audio and video features and modal fusion features;

[0046] The model question-answering module is also used to input the audio and video features into a pre-trained multimodal scene graph generation network to obtain the scene. Figure 3 Tuple characteristics;

[0047] The model question answering module is also used to input the modality fusion features into the edge learning expert temporal integration network to obtain temporal integration features;

[0048] The model question-answering module is also used to interpret the scenario. Figure 3The tuple features and the aforementioned temporal integration features are subjected to feature aggregation processing to obtain a target feature vector that aggregates multimodal temporal information;

[0049] The model question answering module is also used to input the target feature vector into a pre-trained classifier and output the target question answering result based on multiple predefined candidate answers.

[0050] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0051] The memory stores computer-executed instructions;

[0052] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0053] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0054] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0055] This application provides an audio / video question-answering method, apparatus, device, medium, and product based on a multimodal scene graph. By constructing a pre-trained audio / video question-answering model comprising a multimodal feature fusion network, a multimodal scene graph generation network, an edge learning expert temporal integration network, and a pre-trained classifier, it achieves the synergy of semantic integration and contextual understanding of initial multimodal data. The multimodal feature fusion network fuses features from different modalities, improving the correlation between audio, video, and text information. The pre-trained multimodal scene graph generation network integrates audio-related relationships and visual spatial relationships into a unified graph structure, resulting in a multimodal scene graph, enhancing the model's ability to abstractly express the semantics of audio / video scenes. The edge learning expert temporal integration network further strengthens the contextual linkage across time segments, enabling the model to capture causal relationships and behavioral trajectories across time sequences. Finally, through feature aggregation, scene semantics and cross-modal temporal semantics are jointly encoded into a target feature vector, which is input into the pre-trained classifier to output accurate question-answering results, ensuring the matching degree between the question-answering results and the user's context. This application addresses the problems of insufficient understanding of multimodal data, inadequate contextual analysis capabilities, and low accuracy of prediction results in existing audio and video question-answering models, and achieves the effect of deep fusion of semantic information and high-confidence answer output in complex question-answering scenarios. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0057] Figure 1 A flowchart illustrating the audio / video question-answering method based on multimodal scene graphs provided in this application;

[0058] Figure 2 This is a schematic flowchart of the method for fine-tuning the model provided in the embodiments of this application;

[0059] Figure 3 A schematic diagram of the architecture of the audio / video question-answering model provided in the embodiments of this application;

[0060] Figure 4 A schematic diagram of the structure of the audio and video question-answering device based on a multimodal scene graph provided in this application;

[0061] Figure 5 A schematic diagram of the electronic structure provided in this application.

[0062] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0063] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0064] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0065] First, let me explain the terms used in this application:

[0066] Audio-Visual Question Answering (AVQA): A multimodal reasoning task that extracts image and audio information from video clips and answers questions about the scene.

[0067] Scene Graph: A structured graph representation used to express objects in a scene and the relationships between them.

[0068] Edge-Learnable Network: A neural network structure based on the Kolmogorov–Arnold network, capable of interpretable activation function modeling, suitable for modeling complex temporal relationships.

[0069] Mixture of Experts (MoE): A neural network model based on ensemble learning. It consists of multiple expert subnetworks and gating networks.

[0070] Existing technologies typically employ a video-centric processing flow: frame-level or segment-level encoding of the input video is used to obtain the main visual representations, while only global or coarse-grained embeddings are extracted from the audio. Text questions, after semantic encoding, are mainly used to assign attention weights to the visual sequence. Cross-modal fusion often maps multi-source features to a unified representation space through attention or weighted convergence, and then outputs the answer using a general prediction mechanism. This visual-centric paradigm, coupled with weak modeling of audio cues, is prone to problems such as answer bias and unclear basis in situations where the image is blurry, severely occluded, or where key information is mainly embodied in sound (e.g., speaker switching, sound source appearance and disappearance, off-screen cues).

[0071] Based on the aforementioned problems and needs, the inventive concept of this application is to address the issue of insufficient information utilization and low accuracy of output results in existing audio and video question answering methods for handling multimodal question answering tasks. This is achieved by proposing an audio and video question answering method based on multimodal scene graphs to fully utilize complex multimodal initial data, thereby improving the accuracy of question answering output. Specifically, an audio and video question answering model is constructed, comprising a multimodal feature fusion network, a pre-trained multimodal scene graph generation network, an edge learning expert temporal integration network, and a pre-trained classifier. By acquiring the pre-trained audio and video question answering model and multimodal initial data, and extracting multiple modal features based on the pre-trained encoder, audio and video features and modal fusion features representing static semantics and dynamic structures can be obtained, respectively. Subsequently, on the one hand, the audio and video features are input into the scene graph generation network to extract scene graph semantics with triple structures, enhancing the model's ability to analyze question answering background and entity relationships; on the other hand, the modal fusion features are input into the temporal integration network to dynamically acquire features with temporal correlations, achieving accurate analysis of event evolution patterns. Finally, by using the scene... Figure 3 Tuple features and temporal integration features are aggregated to form a target feature vector containing multimodal temporal and structural information, which is then input into a classifier to output the optimal question-answering result. This improves accuracy, robustness, and interpretability in multimodal question-answering scenarios.

[0072] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0073] Figure 1 The flowchart illustrating the audio / video question-answering method based on a multimodal scene graph provided in this application is as follows: Figure 1 As shown, this audio / video question-and-answer method includes:

[0074] S11, Obtain the pre-trained audio and video question answering model and multimodal initial data, wherein the pre-trained audio and video question answering model includes a multimodal feature fusion network, a pre-trained multimodal scene graph generation network, an edge learning expert temporal integration network, and a pre-trained classifier.

[0075] In this embodiment, the pre-trained audio / video question-answering model is a model trained on multimodal data for handling question-answering tasks involving audio, video, and text. The model includes a multimodal feature fusion network, a pre-trained multimodal scene graph generation network, an edge learning expert temporal integration network, and a pre-trained classifier. The multimodal feature fusion network receives and fuses input features from different modalities, achieving alignment and collaborative expression of semantic information between modalities. The pre-trained multimodal scene graph generation network, as the structured semantic modeling part of the model, has the ability to extract structural relationship features from the fused features. The edge learning expert temporal integration network is responsible for analyzing the fused features in the time dimension to capture temporal signals. The pre-trained classifier is used to make the final prediction of the question-answering result based on the integrated features.

[0076] Simultaneously, it also includes acquiring multimodal initial data. For example, multimodal initial data refers to an input data set that simultaneously includes audio data, video data, and text question data while maintaining temporal consistency. Audio data can be represented as the original audio stream or its spectral transformation form, video data is typically temporally continuous image frames or video segments, and text question data is a query statement expressed in natural language. The aforementioned multimodal data can originate from manual collection, pre-set datasets, or real-time user input. This embodiment, through the acquisition of a pre-trained audio / video question-answering model and the preparation of initial data, ensures that subsequent feature extraction, fusion analysis, and question-answering result generation processes can be performed with standardized input and model support, providing a fundamental guarantee for overall performance and response stability.

[0077] S12, based on the multimodal initial data and the pre-trained encoder, performs feature extraction processing to obtain multimodal features.

[0078] In this embodiment, multimodal initial data refers to multiple modal information collected synchronously within the same time window, which can reflect complex semantics and contextual understanding. This typically includes, but is not limited to, video frame sequences, audio streams, and possible subtitle information and user dialogue text. A pre-trained encoder is used to process the multimodal initial data. The encoder extracts features from the video modality, audio modality, and text modality respectively, obtaining multimodal features with temporal structure and semantic expressive capabilities. Through the pre-trained encoder, multimodal feature extraction is achieved, laying the foundation for subsequent question-answering prediction. This improves the semantic expressive power of the features and enhances the accuracy and robustness of subsequent analysis.

[0079] S13 inputs multiple modal features into a multimodal feature fusion network to obtain audio-visual features and modal fusion features.

[0080] In this embodiment, the multimodal feature fusion network is a deep neural network module used to achieve collaborative understanding between different modalities at the feature level. Its goal is to guide the interaction of information from different modalities through semantic channels while maintaining modality specificity, thereby obtaining fused features with stronger joint expressive power. For example, a self-attention mechanism is executed for each feature within a modality to capture the contextual dependencies and temporal patterns within the modality. For instance, audio features, after self-attention, can obtain a more prominent representation within the audio's own context; similarly, video features (frame-level features) also obtain a visual sequence representation with stronger temporal consistency through self-attention. Next, the multimodal feature fusion network establishes semantic alignment relationships between modalities through a cross-attention mechanism. For example, using video features as query vectors and audio features as keys and values ​​for cross-attention can enhance the responsiveness of video representation to sound-related cues, and vice versa. Through bidirectional or multidirectional cross-operations, deep fusion of audio and video can be achieved, forming audio-video features under a joint temporal structure. This feature maintains consistency with the original modality in the temporal dimension, while encoding integrated information from the two perceptual signals in the representation dimension, providing a complementary semantic foundation for subsequent tasks.

[0081] Building upon this foundation, the multimodal feature fusion network further integrates the aforementioned audio-visual features with sentence-level and word-level features from the textual question data. Textual features serve as guiding signals, enhancing the model's ability to focus on question-related information. By using audio-visual features as keys and values, and textual features as query execution cross-attention, semantic-level question-driven enhancement is achieved, making the fusion results more closely aligned with the user's query intent. The final modal fusion feature output by the fusion network is a composite expression integrating visual, auditory, and linguistic semantic cues, possessing cross-modal semantic consistency and question-related guidance capabilities. This embodiment, based on the extracted multimodal features, establishes semantic associations and alignments between modalities through structured fusion, thereby obtaining audio-visual features representing audio and video fusion information, as well as modal fusion features further integrating textual semantics. This maximizes the information integration effect and improves the accuracy and robustness of question-answering result prediction.

[0082] S14, input the audio and video features into the pre-trained multimodal scene graph generation network to obtain the scene. Figure 3 Tuple characteristics.

[0083] In this embodiment, the pre-trained multimodal scene graph generation network generates a set of structured triples based on the target entities (such as people, objects, scenes, etc.) identified in the audio and video features and their behaviors or interaction relationships. For example, for a scene in a video where "a person is opening a door," the generated triples would be (person, opening, door). These triples not only include static relationships between visual targets but can also include dynamic behavioral information captured through the audio modality, such as speaking, tapping, or other action features. Thus, this embodiment achieves a transition from modal features to semantic structures, facilitating subsequent more complex multimodal understanding tasks such as event recognition, behavior analysis, or interaction modeling. It realizes a structured representation of cross-modal semantic information, providing a foundation for subsequent analysis and response.

[0084] S15, input the modality fusion features into the temporal integration network of the edge learning expert to obtain the temporal integration features.

[0085] In this embodiment, the modality fusion features are input into an edge learning expert temporal integration network for processing. This network is used to extract and integrate the evolutionary relationships of the initial multimodal data over time. Specifically, it performs unified modeling and semantic completion processing on cross-modal temporal information to obtain temporal integrated features. The edge learning expert temporal integration network is used to achieve contextual modeling of modality fusion features over time, integration of cross-frame semantic relationships, and capture of dynamic behavioral evolution trends. Specifically, the edge learning expert temporal integration network can divide the modality fusion features into temporally ordered sequence inputs at the time segment level, and use a set of expert sub-modules to perform differentiated processing on different semantic structures or scene types.

[0086] The expert submodule implements time-series modeling through a kernelized attention network (KAN) structure. KAN utilizes a kernel function mechanism to capture the dynamic changes of modality fusion features over time in a non-linear manner. A sparse gating mechanism is used to dynamically allocate weights among the expert submodules, assigning different weights to each expert in the kernelized attention network. The outputs of each expert are then aggregated according to these weights to obtain temporal integrated features focusing on key segments. The output temporal integrated features not only capture the semantic extensibility between consecutive segments in audio and video but also retain the dynamic contextual information of the behavioral evolution path, providing a structured and dynamically consistent input representation for subsequent tasks such as behavior recognition and event understanding. This embodiment achieves the integration of modality fusion features over time through the multi-expert dynamic adaptation mechanism of the kernelized attention network, improving the expressive power of multimodal scene graphs in terms of temporal continuity, behavioral interpretability, and modal collaboration, ensuring the semantic integrity and recognition accuracy of audio and video data in complex event modeling.

[0087] S16, the scene Figure 3 Tuple features and temporal integrated features are subjected to feature aggregation processing to obtain a target feature vector that aggregates multimodal temporal information.

[0088] In this embodiment, the scene Figure 3 Tuple features and temporally integrated features are subjected to feature aggregation processing. This aggregation can be implemented using weighted attention mechanisms, gating unit mechanisms, or graph structure matching mechanisms to align the semantic spaces of features from different sources and dynamically adjust the feature fusion ratio. To reduce information redundancy or semantic conflicts, the input features can be linearly transformed and normalized before fusion to ensure that the fusion operation is performed within a unified feature space. The fused target feature vector fully preserves scene structure information and temporal evolution features, and improves representation consistency across modalities.

[0089] S17: Input the target feature vector into the pre-trained classifier and output the target question answering result based on multiple predefined candidate answers.

[0090] In this embodiment, the target feature vector is input into a pre-trained classifier. This classifier has undergone model parameter optimization based on a large amount of labeled data to enable semantic discrimination and result inference for multimodal question-answering behavior. The classifier first compresses and reconstructs the input feature dimension through a fully connected layer, then extracts high-order discriminative features through several activation functions and normalization operations, and finally outputs the probability distribution among several predefined candidate answers to determine the final target question-answering result. This embodiment achieves structured output of complex multimodal question-answering semantics by inputting the aggregated target feature vector into a pre-trained classifier and outputting the target question-answering result based on multiple predefined candidate answers, ensuring generalization ability and accuracy of output results under multiple scenarios and tasks.

[0091] In one specific embodiment, in a multimodal human-computer interaction scenario, user input is typically not limited to single-modal information but includes information sources from multiple modalities. Generally, the initial multimodal data includes audio data, video data, and text question data, used to comprehensively reflect the multimodal interaction behavior between the user and the system. Audio data is usually the audio track content carried in the video, from which sound events, speaker intonation, and audio event tags can be extracted. Video data corresponds to visual information, including visual dynamic information such as character actions, scene background, and object movement. Text question data is the question text posed by the user, which mainly carries the query intent and semantic context, and is usually expressed in natural language sentences.

[0092] Next, multiple modal features are obtained from step S12 above, and a specific implementation method is provided here. Based on the above embodiment, it includes:

[0093] S121, Audio features are obtained by extracting audio data using a pre-trained audio encoder.

[0094] Specifically, the VGGish model was used for feature encoding. VGGish is a pre-trained audio feature extraction model based on the VGG architecture, possessing good general audio perception capabilities and effectively processing various types of audio signals such as ambient sound, speech, background vocals, and musical rhythm. First, the audio track is segmented into t time segments of fixed length to ensure temporal structure. Each time segment is converted into a spectrogram using Short-Time Fourier Transform (STFT) or log-mel frequency-cephalic coefficients (log-mel spectrogram) and then input into the VGGish model for forward propagation. The VGGish model progressively extracts local temporal frequency patterns through multiple convolutional and pooling operations, outputting a 128-dimensional embedded feature vector in the last layer, representing the audio semantics of the current time segment. The above operations are repeated for each time segment to obtain the audio feature 'a' of the first time segment. 1 The audio features of the second time segment a 2 ...; ultimately forming audio features , where a i This represents the audio feature vector corresponding to the i-th time segment. This embodiment employs the VGGish model to encode audio segments, reducing the limitations of speech recognition or manual feature construction. It achieves robust representation of speech context and background noise, improves the expressive quality and temporal consistency of audio modalities, and lays the foundation for cross-modal problem understanding and target feature vector construction. Audio feature F a This includes audio features across multiple time segments, typically in the form of a sequence or set. It should be noted that, for ease of description, for F... a Describing audio features does not change its sequential or set form. Similarly, describing other feature sequences does not change their inherent sequential or set form.

[0095] S122, based on the video data, visual extraction is performed using a pre-trained visual encoder to obtain frame-level features and image patch-level features.

[0096] In this embodiment, a pre-trained visual encoder is used to simultaneously extract the overall visual content of static frame images and fine-grained image information of local regions. Video data is uniformly sampled at time intervals to extract keyframe images and form a frame sequence; then each frame image is divided into image patches, and the visual encoder extracts frame-level features (reflecting the semantics of the whole image) and image patch-level features (reflecting changes in local regions) respectively.

[0097] Specifically, the video stream is divided into several frames along the timeline so that subsequent feature extraction processes can capture video information at different points in time. To comprehensively extract semantic information and spatial details from the video data, the CLIP visual encoder is used to embed and represent each frame. The CLIP visual encoder is an image semantic representation model based on the Transformer structure. Its training method involves comparative learning of multimodal pairs of images and text, enabling it to generate visual feature representations closely corresponding to semantics. For each frame, the CLIP visual encoder generates two types of feature vectors: one is frame-level features, represented as a feature sequence. The first is used to characterize the global visual semantic information of the video within t time segments; the second is image patch-level features, represented as feature sequences. This dual-feature representation method is used to capture detailed information in local spatial regions. It enhances the ability to identify key local details while maintaining overall semantic understanding. This embodiment achieves hierarchical encoding of information at different levels within the video modality through a frame-level and block-level dual feature extraction mechanism, effectively enhancing the expressive power of video modalities in multimodal data. Simultaneously, it ensures complete coverage of video features in both temporal and spatial dimensions, providing a semantically rich and structurally clear input foundation for subsequent cross-modal semantic alignment with textual problems.

[0098] S123, based on the text question data, text is extracted using a pre-trained text encoder to obtain sentence-level features and word-level features.

[0099] Among them, audio features, frame-level features, image patch-level features, sentence-level features, and word-level features constitute multiple modal features.

[0100] In this embodiment, for the processing of text question data, a pre-trained text encoder (such as BERT) is used to perform semantic analysis on the natural language questions input by the user. First, the data undergoes word segmentation and is mapped to word vectors, which are then input into the encoder. Through multi-layer attention mechanisms and context modeling, sentence-level global feature representations and word-level feature representations that retain word-by-word semantic information are generated. Sentence-level features reflect the overall semantics of the entire question; word-level features are used to capture the fine-grained expression of keywords in the question.

[0101] Specifically, the input question is represented as a standard natural language text sequence, and this text sequence undergoes word segmentation and preprocessing to remove invalid symbols, standardize formatting, and adjust segmentation granularity, thereby improving the accuracy of feature extraction. In the text feature extraction stage, the CLIP text encoder is used to encode the text. Based on a Transformer structure, the CLIP text encoder can effectively model contextual dependencies and semantic relationships between words, thus generating multi-granular text embedding representations. The CLIP text encoder is then used to extract sentence-level features q representing the overall semantics. s And word-level features q of each word w By combining sentence-level and word-level features, multi-scale semantic representation from global to local is achieved. This enables the model to take into account both the overall problem intent and specific details in subsequent cross-modal alignment, attention guidance, and feature fusion processes, thereby improving the accuracy of response generation.

[0102] In summary, audio features, frame-level features, image patch-level features, sentence-level features, and word-level features are uniformly organized into multi-modal features, forming a feature set aligned by time index and semantic hierarchy. Audio features and frame-level features are paired with the same time segment index, ensuring a one-to-one correspondence between auditory cues and visual images across t time segments. Image patch-level features generate local detail descriptions within each time segment by aggregating several spatial blocks from the same frame, sharing the time index with frame-level features but retaining a finer-grained spatial resolution. Sentence-level features serve as global semantic anchors, used to express the overall semantic meaning of the question's intent. Word-level features constitute a semantic sequence arranged in word order, used to establish fine-grained correspondences with local visual / auditory details within the same time segment. To ensure cross-modal fusion, a unified representation of multiple granularities (global / local, sentence / word) and multiple channels (audio / video) is achieved within the same semantic coordinate system, improving the stability of cross-modal alignment and the discriminative power of feature representation.

[0103] In one specific embodiment, the classifier consists of fully connected layers, which calculate the probability distribution using a normalized exponential function. A fully connected layer is a typical neural network structure where each neuron is connected to all neurons in the previous layer, mapping upstream features to the final prediction result. To convert the final output into a class probability distribution, the vector output from the last fully connected layer is normalized using a normalized exponential function, typically the Softmax function. The Softmax function maps any real-valued vector to a probability distribution; each dimension of its output vector represents the model's predicted probability for that class, with all output values ​​mapped to probabilities between 0 and 1. This normalization process ensures the interpretability of the classification results, facilitating final decision-making in multi-class tasks.

[0104] Furthermore, the acquisition of the target question-and-answer result in step S17 above will be further explained here. Based on the above embodiment, it includes:

[0105] S171, Input the target feature vector into the pre-trained classifier, and let the fully connected layer calculate the probability value corresponding to each predefined candidate answer through a normalized exponential function;

[0106] S172, based on the probability values ​​corresponding to each predefined candidate answer, determine the target predefined candidate answer as the target question answer result output from multiple predefined candidate answers, wherein the probability value corresponding to the target predefined candidate answer is greater than the probability value corresponding to any one of the multiple predefined candidate answers.

[0107] In this embodiment, a fully connected layer is used to map the input high-dimensional feature vector to a fixed-dimensional candidate answer space. The output of the fully connected layer is processed by a normalized exponential function (such as the Softmax function) to exponentially map the scores of all candidate answer outputs and normalize them into a probability distribution, such that the probability value corresponding to each candidate answer is between 0 and 1, and the sum of the probabilities of all candidate answers is 1. After obtaining the probability value of each candidate answer, a maximum value selection operation is performed on the probability distribution to determine the candidate answer with the highest probability value as the current target question answer result. Specifically, iterate through the probability values ​​corresponding to all predefined candidate answers and determine the candidate answer with the largest probability value, which is the final output target predefined candidate answer. This ensures the optimal matching between the output result and the current input features, thereby improving the accuracy and robustness of the response to user multimodal input. Furthermore, the number of fully connected layers, the dimension of hidden units, and the form of the normalization function (such as whether to introduce a temperature parameter to adjust the distribution smoothness) in this classifier structure can all be flexibly configured to adapt to the number of answer categories and feature distribution characteristics under different interaction scenarios, further enhancing its applicability in different multimodal tasks.

[0108] In one embodiment, the acquisition of the pre-trained audio / video question-answering model in step S11 above is provided as an implementation method. Based on the above embodiment, it includes:

[0109] S1101, Obtain the multimodal dataset and the initial audio / video question answering model. The initial audio / video question answering model includes a multimodal feature fusion network, an initial multimodal scene graph generation network, an edge learning expert temporal integration network, and an initial classifier.

[0110] Specifically, a multimodal feature fusion network is constructed, consisting of multiple layers of self-attention and cross-attention modules to achieve alignment and interaction of features across different modalities. The self-attention module models the temporal structure and semantic relationships within each modality of data, such as phoneme variation patterns in audio sequences, spatial relationships between image patch-level features, or contextual dependencies between words in text. The cross-attention module models the correlations between different modalities, such as the alignment between local regions in video images and audio segments, or the semantic orientation between textual questions and targets in image frames. This feature fusion network, through parallel stacking of multiple attention layers, achieves unified representation and complementary enhancement of features from different modalities in a high-dimensional representation space.

[0111] Next, a multimodal scene graph generation network is constructed, mainly including a decoder module for decoding relations from fused features and a relation triplet selection module for filtering triples based on the question. The decoder module is a three-layer Transformer decoder that uses a multi-head attention mechanism and a feedforward network to decode the fused features. In each layer, the final fused features are used as the keys and values ​​for cross-attention, updating the decoded query and outputting candidate relation representations, forming a candidate relation triplet feature set. The relation triplet selection module consists of a scaled dot product attention mechanism and a top-k selection operator. The scaled dot product attention mechanism uses sentence-level features of the question as the query, calculates and normalizes relevance weights on the candidate relation triplet set for the current time segment; then, the top-k selection operator selects the triples most relevant to the current question, constructing a structured scene semantic graph.

[0112] Furthermore, a temporal integration module for edge-learnable experts is constructed, primarily comprising a Gaussian generator and a hybrid expert (MoE) framework with edge-learnable networks as the core experts. The Gaussian generator consists of a cross-attention module and a fully connected layer, designed to generate Gaussian distribution parameters based on fused features, predicting key temporal segments or regional locations within each modality. The generated Gaussian distribution guides the expert networks to focus on semantically dense regions, improving the model's efficiency in utilizing key details. The hybrid expert framework consists of multiple parallel edge-learnable expert networks and a gated routing network. The gated routing network assigns weight scores to each expert network, allocating higher attention weights to expert outputs with greater representational power. This allows for a greater focus on key segments and crucial information, improving the accuracy and stability of the results. The complementary and balanced capabilities of different experts enhance generalization ability in complex scenarios.

[0113] S1102, based on the multimodal dataset and the pre-trained multimodal large language model, the annotation set of the multimodal scene graph is obtained.

[0114] In this embodiment, the multimodal dataset is annotated and generated in one go. For each video sample in the dataset, its corresponding audio data is fed into a multimodal large language model (MLLM) with audio-visual semantic understanding capabilities, such as mini CPM-o. The video is analyzed and processed at each time segment, outputting relational triples in the form of <subject, predicate, object>. These relational triples include, but are not limited to, visual spatial relations (e.g., <piano, to the left of, flute>, representing the relative positions between objects), audio event relations (e.g., <piano, sounding at, flute>, representing the sequence or response of sounds), and other cross-modal relations (e.g., <person, speaking, subtitle>, representing the mapping between speech and subtitles). In this way, a labeled set of a multimodal scene graph can be formed without human intervention.

[0115] S1103, input the multimodal dataset into the pre-trained encoder for feature extraction processing to obtain multimodal sample features.

[0116] The audio, video, and text data in the multimodal dataset are respectively input into a pre-trained encoder for feature extraction processing, and the output represents the audio features, video features, and question text features of each sample. The specific implementation method has been described in previous embodiments and will not be repeated here.

[0117] S1104: Input the multimodal sample features into the multimodal feature fusion network to obtain sample audio and video features and sample modal fusion features.

[0118] Specifically, for frame-level feature F v and audio features F a The system enhances intramodal information interaction through self-attention (SA) modules, while achieving intermodal information fusion using cross-attention (CA) modules, as shown in the following formula:

[0119]

[0120]

[0121] In this context, SA represents the self-attention module, and CA represents the cross-attention module.

[0122] The enhanced frame-level features are first self-attention (intra-frame context enhancement) on the video channel, then cross-attention (aligned with audio) is performed using audio features as keys / values, and finally compared with the original F.v The result after adding the residuals.

[0123] The enhanced audio features are first self-attention (intra-segment context enhancement) on the audio channels, then cross-attention (alignment with the video) is performed using video features as keys / values, and finally compared with the original F. a The result after adding the residuals.

[0124] Next, using the modal features as the query vector, the word-level features q corresponding to the question are input. w The key and value are constructed using a cross-attention module, and residual connections are made with the audio and video features. The specific formula is as follows:

[0125]

[0126]

[0127] Among them, v q This indicates enhanced features for video queries, represented by F. v For querying, using word-level features q w Perform cross-attention on key / value pairs (question-guided semantic alignment), then... The sum of the residuals represents the video channel representation that has been incorporated into the semantics of the problem.

[0128] a q This indicates an enhanced feature for audio queries, represented by F. a For querying, using word-level features q w Perform cross-attention on key / value pairs (question-guided semantic alignment), then... The sum of the residuals represents the audio channel representation that has been incorporated into the semantics of the problem.

[0129] Similarly, image patch-level features F p Also in a similar way with v q and a q To align, use the following formula:

[0130]

[0131] in, To enhance image patch-level features, self-attention (intra-patch / inter-patch context enhancement) is performed on the patch-level visual channels and then connected with F... p The result after adding the residuals.

[0132] p v Block-level visual response features aligned with video queries, in v q For the query, retrieve and aggregate the local visual information most relevant to the video and the question from the enhanced image patch-level features.

[0133] p a Block-level visual response features aligned with audio queries, in a q For the query, retrieve and aggregate the local visual information most relevant to the audio and the question from the enhanced image patch-level features.

[0134] S1105, input the sample audio and video features into the initial multimodal scene graph generation network to obtain the predicted scene triplet features.

[0135] Specifically, enhanced frame-level features and enhanced audio features are concatenated along the feature dimension to form a joint representation that simultaneously incorporates visual global semantics and auditory temporal cues. To inject local visual details into this joint representation, enhanced image patch-level features are introduced as keys / values ​​using a cross-attention module. The joint representation focuses on fine-grained regions relevant to the problem, resulting in the final fused feature F. av The specific formula is as follows:

[0136]

[0137] Here, || represents the concatenation operation. Then, the fused features F... av As the core context, it is input into a three-layer Transformer decoder. The formula is:

[0138]

[0139] Among them, F av Used as the key and value in the cross-attention module of each layer of the decoder, the decoder uses the fused features as the external context to decode the query state Q. （l） Perform layer-by-layer updates and output Q （l+1） The final decoding state is obtained after three layers of stacking. It represents the set of candidate relation triplet features within the current time segment, including various semantic combinations of subject-predicate-object.

[0140] Since the decoder outputs a large number of triples, not all of which are directly relevant to the current problem, a problem-based filtering process is needed. This process utilizes a scaled dot product attention mechanism. First, the problem feature q is calculated. s Features of all triples in the current time segment t The dot product between the points measures the semantic relevance or alignment between the question and each triple. This raw score, after being normalized by a scaling factor and processed by the Softmax function, generates a set of attention weights W. rel This set of weights can be understood as the importance distribution of each triple in answering the current question. The formula is as follows:

[0141]

[0142] Where T represents the matrix transpose operation, and D represents... Feature dimension, q s Representing sentence-level features, The feature matrix of relational triplets representing time segment t.

[0143] Finally, a top-k selection operator is used. Based on this set of weights W rel From the characteristics of triples Select the top with the highest weight. k One. Received It contains the structured information most closely related to the problem in the current time segment t, namely the predicted scene triplet features. The specific formula is as follows:

[0144]

[0145] The final fused feature F is obtained through cross-attention with block-level details as keys / values. av This enables the injection of fine-grained spatial information into the joint audio and video representation; and the final fusion feature F av As a decoder, it crosses the attention context to complete the structured decoding from fused representations to relation triple features; then, using the question semantics q... s Perform scaling dot product scoring and By performing sparse selection, redundancy and noise are reduced, and the most valuable structural clues are retained, thereby improving the accuracy and interpretability of subsequent aggregation and classification.

[0146] S1106. Based on the preset training loss function, the labeled set of the multimodal scene graph, and the predicted scene triplet features, determine the triplet feature differences.

[0147] S1107, when the detected triplet feature difference exceeds the preset difference threshold, the initial multimodal scene graph generation network is adjusted to obtain the adjusted multimodal scene graph generation network, and the process returns to the step of inputting the sample audio and video features into the initial multimodal scene graph generation network to obtain the predicted scene triplet features, until the triplet feature difference is less than or equal to the preset difference threshold, and the last adjusted multimodal scene graph generation network is used as the pre-trained multimodal scene graph generation network, and the last obtained predicted scene triplet features are used as the target sample scene triplet features.

[0148] In this embodiment, to supervise the training of the scene graph generation network, the predicted scene triplet features are compared with the real scene obtained through the annotation set. Figure 3The tuples are compared, and their feature differences are calculated according to a preset training loss function (such as triplet matching loss). When the detected triplet feature difference exceeds a preset difference threshold, it indicates that the current scene graph generation effect has not yet reached the expected accuracy. At this time, adjustments are performed on the initial multimodal scene graph generation network, such as updating model parameters or introducing a new attention path, to generate a new scene graph generation network. This network is then used again to process the sample audio and video features, forming a closed-loop optimization, until the difference between the generated triplet features and the labeled set meets the preset threshold. Finally, the latest scene graph generation network that meets the error condition is used as the pre-trained multimodal scene graph generation network, and the triplet features generated in the last time are retained as the triplet features of the target sample scene.

[0149] Specifically, Hungarian Matching Loss (HML) is employed to enumerate all possible matching methods at each time segment. The matching cost function is used to calculate the cost between each pair of predicted triples and ground truth triples. The optimal bijective match with the minimum total cost is then found using the Hungarian algorithm. After obtaining a match, the matched predicted triples are used as constraints for supervised learning of classification / discrimination probabilities. Unmatched predictions (or ground truth triples) are treated as empty categories.

[0150] The calculation process includes:

[0151]

[0152] in, This represents all possible matching methods. This indicates that the Hungarian algorithm is used in The optimal matching permutation is obtained from the given information, where t is the t-th time segment, T is the total number of time segments, and i is the index of the triplet. The i-th truth relation triple within time segment t (from the annotation set of the multimodal scene graph). This indicates the matched predicted relation triples (from the predicted scenario triple features). L represents the predicted index corresponding to the truth index i under the matching permutation σ. match This represents the matching cost function.

[0153] L rel The loss represents relational learning loss, which provides probabilistic supervision to the matched prediction-true pairs and sums the cross-entropy or negative log-likelihood penalty terms on the matched terms. This represents the truth value (class indicator) of the i-th triple at time t. Let represent the prediction probability of the i-th predicted triplet for the true class at time t.

[0154] This embodiment employs Hungarian matching loss to perform optimal bijective matching between the predicted scene triplet features and the labeled set of the multimodal scene graph. After globally minimizing the matching cost at the time slice level, a relation learning loss is applied, reducing permutation uncertainty and duplicate matching issues in set prediction. Thus, the supervision signal shifts from the point level to the triplet structure level, strengthening the constraint on the overall consistency of the subject-predicate-object relationship. This training mechanism of matching first and then supervision improves training convergence speed and stability, suppresses loss propagation caused by redundancy and false detections, enhances the accuracy and robustness of triple extraction, and provides a higher-quality, more interpretable intermediate representation for subsequent feature aggregation and answer prediction based on structured relations.

[0155] S1108 inputs the sample modality fusion features into the edge learning expert temporal integration network to obtain the sample temporal integration features.

[0156] In this embodiment, the sample modality fusion features are input into the edge learning expert temporal integration network. This network integrates multiple specific modality expert modules and performs joint modeling of modality temporal information through an edge distribution modeling strategy to obtain the temporal integration features of the samples, which can express the intrinsic structure and change law of the modality over time.

[0157] Specifically, the formula is as follows:

[0158]

[0159] First, the question-visual query is encoded using a Gaussian generator to obtain sentence-level features q. s For queries, video query enhancement features v q It is a key / value pair, obtained through the cross-attention module. Subsequently, a fully connected layer was used to... Mapped to a set of Gaussian distribution parameters Based on this, Gaussian priors are generated on the timeline. The Gaussian prior is used to map different experts to different segments of the time axis, so that each expert is primarily responsible for their corresponding key time range, thereby forming a position-sensitive and interpretable time-weighted aggregation in subsequent aggregation.

[0160] Based on this, for any input feature stream, the hybrid expert framework utilizes these Gaussian priors and combines them with the characteristics of the data itself to perform the final temporal information integration. Each marginal learnable expert is an independent function fitter, which constructs a highly flexible, learnable activation function by learning the internal B-spline basis functions, as shown in the formula:

[0161]

[0162] in, This represents a learnable function fitter based on B-splines, where d represents the number of dimensions of the input vector. This indicates that the action is applied to the input value x. i The k-th B-spline basis function on the dimension, where K represents the number of B-spline basis functions used for each dimension; This represents the learnable coefficient.

[0163] The outputs of all experts are fused into the final temporal integrated feature through a weighted aggregation mechanism:

[0164]

[0165] Specifically, the output of the Hybrid Expert Framework (MoE) is aggregated from two types of weights: one is the temporal positioning weights derived from the Gaussian prior. Another type is the expert selection weights assigned by routing / gating. For each target channel (Corresponding to audio, block-level visual response features aligned with audio queries, and block-level visual response features aligned with video queries, respectively), the outputs of each expert are weighted and summed to obtain the temporal integrated features. .

[0166] This process, while ensuring lightweight efficiency, aligns the problem semantics, temporal location, and expert capabilities to output... Three types of interpretable MoE temporal aggregation features provide a temporally well-defined and noise-suppressed representation for subsequent feature aggregation and answer prediction.

[0167] S1109, perform feature aggregation processing on the sample time series integrated features and the target sample scene triplet features to obtain the sample feature vector.

[0168] Specifically, the temporal integration features of the samples are combined with the scene triple features of the target samples for feature aggregation, and the final sample feature vector is constructed by feature concatenation. This feature vector combines modal temporal semantics and structured scene semantics, providing multi-dimensional information support for question answering.

[0169] S1110: Input the sample feature vector into the initial classifier and perform question-answering result supervision processing based on multiple predefined candidate answers and a preset task classification loss function to train the initial classifier and obtain a pre-trained classifier.

[0170] The sample feature vector is input into the initial classifier, and supervised training is performed based on multiple predefined candidate answers and corresponding preset task classification loss functions (such as cross-entropy loss) to optimize the classifier's output probability distribution, gradually approximating the user's actual question-and-answer results. The final trained pre-trained classifier is obtained. The formula is:

[0171]

[0172] Among them, L task Let M be the cross-entropy loss function, and p be the number of questions. c This represents the probability of the correct answer.

[0173] S1111 constructs a pre-trained audio and video question answering model by combining a multimodal feature fusion network, a pre-trained multimodal scene graph generation network, an edge learning expert temporal integration network, and a pre-trained classifier.

[0174] After completing the training of each module, the trained multimodal feature fusion network, pre-trained multimodal scene graph generation network, edge learning expert temporal integration network, and pre-trained classifier are integrated to construct a complete pre-trained audio and video question answering model, which improves the ability to understand complex multimodal inputs and the accuracy of question answering.

[0175] In one specific embodiment, the multimodal dataset includes each video and its corresponding audio. The video data is a continuous sequence of image frames, while the audio data is the synchronously acquired sound signal corresponding to the video. The aforementioned audio and video data are typically paired according to timeline alignment to ensure that each image frame and its corresponding audio segment are consistent in time, thereby facilitating subsequent multimodal fusion and feature extraction.

[0176] Furthermore, obtain the annotation set of the multimodal scene graph, including:

[0177] S11021, based on each video and the corresponding audio, and the pre-trained multimodal large language model, through a preset prompt template, the pre-trained multimodal large language model performs annotation processing on each time segment of each video, and obtains audio-video information relationship triples as the annotation set of the multimodal scene graph.

[0178] In this embodiment, the Multimodal Large Language Model (MLLM) refers to a deep learning model capable of simultaneously processing multiple modal inputs such as images, text, and audio, and outputting structured language information. It possesses cross-modal feature fusion and semantic abstraction capabilities. To facilitate the model in generating structured annotation information for specific tasks, a set of preset prompt templates is constructed. These prompt templates guide the model to perform specific relation extraction tasks in natural language. For example, the prompt templates could be set as: "Please describe the correspondence between the people and their speech content in this time segment," or "Based on the image and sound, extract the main behavior and context in this segment."

[0179] In the specific implementation, each video segment is divided into multiple time segments. Each time segment can be of fixed length (e.g., 1 second or 2 seconds) or automatically divided based on video semantics or shot changes. Within each time segment, the corresponding video frame information and synchronized audio segments are extracted and fed as input into a pre-trained multimodal large language model. This model processes the provided modal data and prompt templates, outputting semantic triples for that time segment, i.e., audio-video information relation triples. The triple structure is <subject, relation, object>, where the subject can be a person, object, or scene, the relation typically describes an action, state, or semantic connection, and the object is supplementary information, such as the content of a person's speech, the action performed, or the current interactive object. In summary, this embodiment can generate a set of audio-video information relation triples covering each time segment, constituting a multimodal scene icon annotation set for the video. This annotation set provides a structured semantic foundation for subsequent tasks such as scene understanding, behavior recognition, and event extraction. By setting preset prompt templates, the consistency and controllability of model output are improved, while ensuring the semantic accuracy and cross-modal consistency of annotation results, thus enhancing the expressiveness and usability of multimodal scene graphs.

[0180] Figure 2 The flowchart of the method for fine-tuning the model provided in this application embodiment is based on the above embodiment and includes:

[0181] S21, Obtain the multimodal question-answering test set and preset standard answers;

[0182] S22, input the multimodal question-and-answer test set into the pre-trained audio and video question-and-answer model to obtain the test prediction question-and-answer results;

[0183] S23. The test prediction results are compared with the preset standard answers to fine-tune the pre-trained audio and video question answering model and obtain the target audio and video question answering model.

[0184] In this embodiment, a multimodal question-and-answer test set containing real-world question-and-answer content is constructed or selected. This test set includes multiple video clips and their corresponding audio content. For each set of audio and video data, one or more corresponding standard question-and-answer pairs are set as preset standard answers to verify the model's understanding and answering capabilities. The pre-trained audio and video question-and-answer model receives video frame sequences, audio waveform data, and question statements from the test set to obtain test predictions. The predicted question-and-answer results are compared with the preset standard answers in the multimodal question-and-answer test set. For example, based on metrics such as semantic matching scores, the accuracy of the model's output is calculated to determine the model's current capabilities. If the model's performance does not reach a preset performance threshold, the model parameters are updated using a backpropagation algorithm with the standard answers as supervision signals. This allows for fine-tuning of the model, enabling it to more accurately learn the complex correspondence between audio and video information and language. Finally, the fine-tuned model becomes the target audio and video question-and-answer model, possessing stronger semantic understanding and multimodal processing capabilities, and able to provide more accurate responses when facing audio and video question-and-answer tasks in real-world scenarios.

[0185] In the application phase, a pre-trained audio / video question-answering model is used for question answering. First, multimodal initial data (video and its corresponding audio, and user-submitted text questions) is received. This data is processed by a pre-trained encoder to obtain audio features, frame-level features, image patch-level features, sentence-level features, and word-level features. Then, these multimodal features are input into a multimodal feature fusion network. Intramodal enhancement and cross-modal alignment are achieved through self-attention and cross-attention. Guided by question word-level features, audio / video features and modal fusion features incorporating question semantics are obtained. Based on this, the audio / video features and enhanced image patch-level features are processed through cross-attention to obtain the final fused features. These are then input into the Transformer decoder of a pre-trained multimodal scene graph generation network to decode candidate relation triplet features. Sentence-level features are then used for scaling, dot product attention scoring, and top-k filtering to retain the triplets most relevant to the current question. In parallel, the modal fusion features are input into an edge learning expert temporal integration network. Based on a question-guided Gaussian prior and expert routing, interpretable temporal aggregation of key time segments is performed to obtain temporal integrated features. Finally, the filtered scene... Figure 3 Tuple features and temporal integrated features are aggregated to form a target feature vector, which is then input into a pre-trained classifier. The softmax probability distribution is calculated on multiple predefined candidate answers, and the target question answering result is output according to the maximum probability principle.

[0186] Figure 3 This is a schematic diagram of the architecture of the audio / video question-answering model provided in the embodiments of this application.

[0187] Based on the above embodiments, such as Figure 3As shown, the audio / video question answering model consists of data preprocessing, modality fusion, structured relationship modeling, temporal integration, feature aggregation, and discriminative output. The input is a video and its corresponding audio and text questions. A pre-trained encoder yields audio features, frame-level features, image patch-level features, sentence-level features, and word-level features. The multimodal feature fusion network comprises self-attention and cross-attention. First, intramodal enhancement is performed on both audio and video. Then, semantics are injected into both channels using the word-level features of the question as keys and values, forming enhanced frame-level features and enhanced audio features. These are then aligned with the enhanced image patch-level features to obtain block-level responses corresponding to the video and audio queries.

[0188] The structured relationship modeling adopts a multimodal scene graph generation network: the enhanced audio and video joint representation and block-level details are fused with cross-attention as context, and the three-layer Transformer decoder is input to decode the candidate relationship triple features; then the sentence-level features are scaled dot product attention scoring and the top-k selection module is used to filter out the triples most relevant to the question.

[0189] In parallel, temporal integration is accomplished by an edge-learning expert temporal integration network: a Gaussian generator produces temporal Gaussian priors corresponding to key time periods based on question and query representations; a hybrid expert (MoE) uses built-in learnable spline experts to perform expert-weighted aggregation on multiple inputs (audio channels, block-level channels aligned with video queries, and block-level channels aligned with audio queries), outputting temporal integration features. Through feature aggregation, the scene after question filtering is... Figure 3 Tuple features and temporal integrated features are unified in the feature dimension to form the target feature vector, which is input into a pre-trained classifier (fully connected layer and Softmax) to give the probability distribution on multiple predefined candidate answers and output the target question answering result.

[0190] It should be noted here that... Figure 3 In this process, video is input into a VLM (Vision-Language Model) to obtain the scene. Figure 3 Tuples represent the processing flow during the model training phase. Specifically, during training, the scene obtained from the VLM model can be... Figure 3 Tuples are used to train a multimodal scene graph generation network.

[0191] Figure 4 The schematic diagram of the audio and video question answering device based on a multimodal scene graph provided in this application is as follows: Figure 4 As shown, the audio / video question-and-answer device 40 provided in this embodiment includes:

[0192] The acquisition module 401 is used to acquire the pre-trained audio and video question answering model and multimodal initial data. The pre-trained audio and video question answering model includes a multimodal feature fusion network, a pre-trained multimodal scene graph generation network, an edge learning expert temporal integration network, and a pre-trained classifier.

[0193] The model question answering module 402 is used to perform feature extraction processing based on multimodal initial data and pre-trained encoder to obtain multimodal features;

[0194] The model question answering module 402 is also used to input multiple modal features into a multimodal feature fusion network to obtain audio and video features and modal fusion features;

[0195] The model question answering module 402 is also used to input audio and video features into a pre-trained multimodal scene graph generation network to obtain scene graphs. Figure 3 Tuple characteristics;

[0196] The model question answering module 402 is also used to input modality fusion features into the temporal integration network of the edge learning expert to obtain temporal integration features;

[0197] The model question-answering module 402 is also used to connect scenes Figure 3 Tuple features and temporal integrated features are subjected to feature aggregation processing to obtain a target feature vector that aggregates multimodal temporal information;

[0198] The model question answering module 402 is also used to input the target feature vector into the pre-trained classifier and output the target question answering result based on multiple predefined candidate answers.

[0199] The audio and video question-and-answer device 40 provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0200] Figure 5 A schematic diagram of the electronic structure provided in this application. (See attached diagram.) Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0201] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0202] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0203] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0204] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0205] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0206] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0207] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0208] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0209] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0210] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0211] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0212] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0213] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0214] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An audio-visual question-answering method based on multimodal scene graphs, characterized in that, include: Obtain a pre-trained audio / video question answering model and multimodal initial data, wherein the pre-trained audio / video question answering model includes a multimodal feature fusion network, a pre-trained multimodal scene graph generation network, an edge learning expert temporal integration network, and a pre-trained classifier; Feature extraction is performed based on the multimodal initial data and the pre-trained encoder to obtain multiple modal features; The various modal features are input into the multimodal feature fusion network to obtain audio-visual features and modal fusion features; The audio and video features are input into a pre-trained multimodal scene graph generation network to obtain a multimodal scene graph; wherein, the multimodal scene graph includes scene graph triplet features; The modality fusion features are input into the temporal integration network of the edge learning expert to obtain the temporal integration features. The scene graph triplet features and the temporal integration features are subjected to feature aggregation processing to obtain a target feature vector that aggregates multimodal temporal information; The target feature vector is input into a pre-trained classifier, and the target question answering result is output based on multiple predefined candidate answers.

2. The audio / video question-and-answer method according to claim 1, characterized in that, The multimodal initial data includes audio data, video data, and text question data; Then, feature extraction is performed based on the multimodal initial data and the pre-trained encoder to obtain multiple modal features, including: Audio features are obtained by extracting audio data from a pre-trained audio encoder. Based on the video data, visual extraction is performed using a pre-trained visual encoder to obtain frame-level features and image patch-level features; Based on the text question data, text extraction is performed using a pre-trained text encoder to obtain sentence-level features and word-level features; The audio features, frame-level features, image patch-level features, sentence-level features, and word-level features constitute multiple modal features.

3. The audio / video question-and-answer method according to claim 1, characterized in that, The classifier is composed of fully connected layers, and the probability distribution is calculated by a normalized exponential function. The step of inputting the target feature vector into a pre-trained classifier and outputting the target question-answering result based on multiple predefined candidate answers includes: The target feature vector is input into the pre-trained classifier, and the fully connected layer calculates the probability value corresponding to each predefined candidate answer through a normalized exponential function; Based on the probability values ​​corresponding to each predefined candidate answer, a target predefined candidate answer is determined from multiple predefined candidate answers as the target question answer result output, wherein the probability value corresponding to the target predefined candidate answer is greater than the probability value corresponding to any one of the multiple predefined candidate answers.

4. The audio / video question-and-answer method according to claim 1, characterized in that, The process of obtaining the pre-trained audio / video question-answering model includes: Obtain a multimodal dataset and an initial audio / video question answering model, wherein the initial audio / video question answering model includes a multimodal feature fusion network, an initial multimodal scene graph generation network, an edge learning expert temporal integration network, and an initial classifier; The annotation set of the multimodal scene graph is obtained by annotating the multimodal dataset and the pre-trained multimodal large language model. The multimodal dataset is input into the pre-trained encoder for feature extraction to obtain multimodal sample features; The multimodal sample features are input into the multimodal feature fusion network to obtain sample audio and video features and sample modality fusion features; The sample audio and video features are input into the initial multimodal scene graph generation network to obtain the predicted scene triplet features; The differences in triplet features are determined based on the preset training loss function, the annotation set of the multimodal scene graph, and the triplet features of the predicted scene. When the detected difference in the triplet features exceeds a preset difference threshold, the initial multimodal scene graph generation network is adjusted to obtain an adjusted multimodal scene graph generation network. The process then returns to the step of inputting the sample audio and video features into the initial multimodal scene graph generation network to obtain predicted scene triplet features, until the difference in the triplet features is less than or equal to the preset difference threshold. The last adjusted multimodal scene graph generation network is then used as a pre-trained multimodal scene graph generation network, and the last predicted scene triplet features are used as the target sample scene triplet features. The sample modality fusion features are input into the edge learning expert temporal integration network to obtain sample temporal integration features; The sample time-series integrated features and the target sample scene triplet features are subjected to feature aggregation processing to obtain the sample feature vector; The sample feature vector is input into the initial classifier and question-answering result supervision processing is performed according to multiple predefined candidate answers and a preset task classification loss function to train the initial classifier and obtain a pre-trained classifier. The multimodal feature fusion network, the pre-trained multimodal scene graph generation network, the edge learning expert temporal integration network, and the pre-trained classifier are used to construct a pre-trained audio and video question answering model.

5. The audio / video question-and-answer method according to claim 4, characterized in that, The multimodal dataset includes each video and the corresponding audio. The annotation process, which involves labeling a multimodal dataset and a pre-trained multimodal large language model to obtain an annotation set for the multimodal scene graph, includes: Based on each video and the corresponding audio, and a pre-trained multimodal large language model, the pre-trained multimodal large language model is used to annotate each time segment of each video using a preset prompt template, resulting in audio-video information relationship triples as the annotation set for the multimodal scene graph.

6. The audio / video question-and-answer method according to claim 4, characterized in that, Also includes: Obtain a multimodal question-answering test set and preset standard answers; The multimodal question answering test set is input into the pre-trained audio and video question answering model to obtain the test prediction question answering results; The test prediction results are compared with the preset standard answers to fine-tune the pre-trained audio and video question answering model and obtain the target audio and video question answering model.

7. An audio-visual question-answering device based on a multimodal scene graph, characterized in that, include: The acquisition module is used to acquire a pre-trained audio and video question answering model and multimodal initial data, wherein the pre-trained audio and video question answering model includes a multimodal feature fusion network, a pre-trained multimodal scene graph generation network, an edge learning expert temporal integration network, and a pre-trained classifier; The model question answering module is used to perform feature extraction processing based on the multimodal initial data and the pre-trained encoder to obtain multimodal features; The model question answering module is also used to input the multiple modal features into the multimodal feature fusion network to obtain audio and video features and modal fusion features; The model question answering module is further configured to input the audio and video features into a pre-trained multimodal scene graph generation network to obtain a multimodal scene graph; wherein, the multimodal scene graph includes scene graph triplet features; The model question answering module is also used to input the modality fusion features into the edge learning expert temporal integration network to obtain temporal integration features; The model question answering module is also used to perform feature aggregation processing on the scene graph triplet features and the temporal integration features to obtain a target feature vector that aggregates multimodal temporal information; The model question answering module is also used to input the target feature vector into a pre-trained classifier and output the target question answering result based on multiple predefined candidate answers.

8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the audio / video question-and-answer method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the audio / video question-and-answer method according to any one of claims 1 to 6.

Images