Multi-modal large model training method, audio-visual question and answer method, medium and equipment

Abstract

The invention discloses a multi-modal large model training method, an audiovisual question and answer method, a medium and equipment, and the multi-modal large model training method comprises the steps: obtaining a training sample set; inputting the question sample, the dynamic video sample and the corresponding synchronous audio sample in the training sample set into a multi-modal large model to obtain an answer token; according to the answer token, the answer label and features generated in the processing process of the first branch, the second branch and the third branch, constructing a total loss function of the multi-modal large model; and training the multi-modal large model based on the total loss function to obtain a trained multi-modal large model. The multi-modal large model trained by the training method adopts a single-stage question-guided audio-visual question-answering framework, and the multi-modal large model obtained by training takes question semantics as a core to realize efficient audio-visual question-answering.

Description

Technical Field

[0001] This invention relates to the field of audiovisual question-answering task technology, and in particular to a training method for a multimodal large model, an audiovisual question-answering method, and media and equipment. Background Technology

[0002] Audio-Visual Question Answering (AVQA), as an important direction in the field of multimodal artificial intelligence, requires models to deeply integrate auditory and visual modal information to achieve advanced semantic reasoning on complex multimodal inputs. In real-world scenarios, videos contain acoustic cues (such as speech and background music) and dynamic visual content. An effective AVQA system needs to possess cross-modal content association and deep understanding capabilities. However, large-scale AVQA models decouple representation alignment from task objectives when processing audio-visual question answering tasks, resulting in a lack of task-specificity in cross-modal representations and complex pre-training processes that prevent end-to-end joint learning. Summary of the Invention

[0003] This invention aims to at least partially address one of the technical problems in related technologies. To this end, one objective of this invention is to propose a training method for a multimodal large model. The trained multimodal large model employs a single-stage question-guided audiovisual question-answering framework. The trained multimodal large model, with question semantics as its core, achieves efficient audiovisual question answering.

[0004] The second objective of this invention is to propose an audiovisual question-and-answer method.

[0005] A third objective of this invention is to provide a computer-readable storage medium.

[0006] The fourth objective of this invention is to provide an electronic device.

[0007] To achieve the above objectives, a first aspect of the present invention proposes a training method for a multimodal large model. The multimodal large model includes a first branch, a second branch, a third branch, a fusion module, and a large language module. The input terminal of the first branch is used to input question-and-answer questions, the input terminal of the second branch is used to input dynamic video, and the input terminal of the third branch is used to input synchronized audio corresponding to the dynamic video. The output terminals of the second branch and the third branch are connected to the input terminal of the fusion module, and the output terminals of the first branch and the fusion module are connected to the input terminal of the large language module. The large language module is used to output answer tokens. The training method includes: acquiring a training sample set, wherein the training sample set includes question samples, dynamic video samples and their corresponding synchronized audio samples, and answer tags corresponding to the question samples; inputting the question samples, dynamic video samples, and their corresponding synchronized audio samples from the training sample set into the multimodal large model. A multimodal large model is constructed to obtain an answer token. Specifically, the first branch processes the question sample to obtain a question embedding vector, a text token, and a global semantic vector; the second branch processes the dynamic video sample while generating video compression features based on the question embedding vector and the global semantic vector; the third branch processes the synchronized audio sample while generating audio compression features based on the question embedding vector and the global semantic vector; a fusion module fuses the video compression features and the audio compression features to obtain an audio-video token; and a large language module processes the text token and the audio-video token to obtain an answer token. A total loss function for the multimodal large model is constructed based on the answer token, the answer label, and the features generated during the processing of the first, second, and third branches. The multimodal large model is then trained based on the total loss function to obtain a trained multimodal large model.

[0008] According to the training method of the multimodal large model of the present invention, the trained multimodal large model adopts a single-stage question-guided audiovisual question answering framework, and the trained multimodal large model takes question semantics as the core to achieve efficient audiovisual question answering.

[0009] In addition, the training method for multimodal large models proposed in the above embodiments of the present invention may also have the following additional technical features: According to an embodiment of the present invention, the first branch includes a text decoding unit and a text encoding unit. The first branch processes the question sample by: decoding the question sample using the text decoding unit to obtain a question embedding vector output by the embedding layer of the decoding unit and a text token output by the output layer of the decoding unit; and encoding the question sample using the text encoding unit to obtain a global semantic vector.

[0010] According to one embodiment of the present invention, the second branch includes a visual encoding unit and a visual compression unit. While processing the dynamic video sample, the second branch generates video compression features based on the question embedding vector and the global semantic vector. This includes: using the visual encoding unit to extract spatiotemporal features from the dynamic video sample to obtain visual features; flattening the visual features to obtain visual flattened features; and using the visual compression unit to calculate the visual weights corresponding to the visual flattened features based on the visual flattened features and the global semantic vector, and performing attention pooling on the enhanced visual features and visual weights corresponding to the visual flattened features to obtain the video compression features. The enhanced visual features corresponding to the visual flattened features are obtained by a multilayer perceptron performing question enhancement processing on the visual flattened features and the question embedding vector.

[0011] According to one embodiment of the present invention, the third branch includes an auditory coding unit and an auditory compression unit. The third branch processes the synchronized audio sample while generating audio compression features based on the question embedding vector and the global semantic vector. This includes: using the auditory coding unit to extract time-frequency features from the synchronized audio sample to obtain auditory features; flattening the auditory features to obtain auditory flattened features; and using the auditory compression unit to calculate the auditory weights corresponding to the auditory flattened features based on the auditory flattened features and the global semantic vector, and performing attention pooling on the enhanced auditory features and auditory weights corresponding to the auditory flattened features to obtain the audio compression features. The enhanced auditory features corresponding to the auditory flattened features are obtained by a multilayer perceptron performing question enhancement processing on the auditory flattened features and the question embedding vector.

[0012] According to one embodiment of the present invention, the fusion module includes a gating unit and a normalization unit. The fusion module performs fusion processing on the video compression features and the audio compression features to obtain an audio-video token, including: dynamically selecting an expert group based on the video compression features and the audio compression features, and performing dynamic weight fusion on the output of the expert group to obtain fused features.

[0013] According to an embodiment of the present invention, constructing the total loss function of the multimodal large model based on the answer token, the answer label, and the features generated during the processing of the first branch, the second branch, and the third branch includes: Construct a large language loss function based on the answer label and the answer token; Construct a visual loss function for the problem based on the problem embedding vector and the visual flattening features; Construct a question auditory loss function based on the question embedding vector and the auditory flattening features; Construct an audio-video loss function based on the video compression features and the audio compression features; The total loss function of the multimodal large model is constructed based on the large language loss function, the problem visual loss function, the problem auditory loss function, and the audio-visual loss function.

[0014] According to an embodiment of the present invention, the total loss function of the multimodal large model is:

[0015] in, Represents the total loss function, Represents a large language loss function. This indicates the auditory alignment loss weights in the problem area. This represents the auditory loss function of the problem. , The representation distribution of the problem embedding vector. The representational distribution of auditory flattening features, This indicates the visual alignment loss weights for the problem. The visual loss function represents the problem. , The representation distribution of the problem embedding vector. The representation distribution of visual flattening features. Indicates the alignment loss weights. Represents the audio / video loss function. , The representation distribution of audio compression features. This represents the distribution of characteristics of video compression.

[0016] To achieve the above objectives, a second aspect of the present invention provides an audiovisual question-and-answer method, comprising: Obtain the question to be answered and the dynamic video and its corresponding synchronized audio; input the question to be answered, the dynamic video and its corresponding synchronized audio into a pre-trained multimodal large model to obtain the answer result, wherein the pre-trained multimodal large model is trained using the multimodal large model training method described above.

[0017] To achieve the above objectives, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the multimodal large model training method described above, or the audiovisual question answering method described above.

[0018] To achieve the above objectives, a fourth aspect of the present invention provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements the multimodal large model training method or the audiovisual question answering method described above.

[0019] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] Figure 1 This is an architecture diagram of a multimodal large model according to an embodiment of the present invention; Figure 2 This is a flowchart of a training method for a multimodal large model according to an embodiment of the present invention; Figure 3 This is an architecture diagram of a multimodal large model according to a specific embodiment of the present invention; Figure 4 This is a flowchart of an audiovisual question-and-answer method according to an embodiment of the present invention; Figure 5 This is a structural block diagram of the controller according to an embodiment of the present invention. Detailed Implementation

[0021] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0022] As research progresses, AVQA benchmark datasets continue to evolve: the early AVQA and Pano-AVQA datasets focused on object-level semantic understanding and panoramic environment question answering in static scenes; the subsequent MUSIC-AVQA series datasets, on the other hand, are designed for dynamic music performance scenarios, introducing fine-grained spatiotemporal annotations and covering complex problems such as comparative, temporal, and counting types, providing a standardized platform for evaluating AVQA systems.

[0023] Among them, the most representative technologies in the AVQA field are mainly divided into two categories: I. Cross-modal models with a two-stage training paradigm. This type of method first performs cross-modal pre-alignment using self-supervised methods such as contrastive learning and masked target reconstruction, and then fine-tunes it for downstream tasks. For example, the VALOR model strengthens the semantic association between modalities through multimodal grouping alignment and description generation tasks; the InternVideo series of models optimizes spatiotemporal consistency through a progressive training strategy, and performs outstandingly in long video understanding.

[0024] II. Question-guided audiovisual feature processing models. These methods optimize feature alignment and extraction through question semantics. For example, the TWM model uses a query-guided attention mechanism to focus on information-rich segments; the QA-TIGER model uses Gaussian modeling to highlight question-related keyframes, improving the accuracy of temporal dynamic modeling.

[0025] It should be noted that the two-stage training paradigm decouples representation alignment from the task objective, resulting in a lack of task-specificity in cross-modal representations. Furthermore, the complex pre-training process prevents end-to-end joint learning, limiting the potential for cross-modal interaction. In long video processing, related methods struggle to adaptively adjust the granularity of information extraction based on problem requirements (global / local understanding), failing to support cross-modal reasoning at different levels of abstraction.

[0026] To address the aforementioned problems, embodiments of the present invention provide a training method, an audiovisual question-answering method, and a medium and device for a multimodal large model. The training method, audiovisual question-answering method, and medium and device of the present invention will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0027] The multimodal large model in this embodiment of the invention may include a first branch, a second branch, a third branch, a fusion module, and a large language module. For example... Figure 1 As shown, the input terminal of the first branch is used to input the question and answer, the input terminal of the second branch is used to input the dynamic video, the input terminal of the third branch is used to input the synchronized audio corresponding to the dynamic video, the output terminals of the second branch and the third branch are connected to the input terminal of the fusion module, and the output terminal of the first branch and the output terminal of the fusion module are connected to the input terminal of the large language module, which is used to output the answer token.

[0028] The multimodal large model in this embodiment of the invention adopts a single-stage question-guided audiovisual question answering framework. With question semantics as the core, it achieves efficient question answering through an end-to-end process of "feature extraction - question-guided compression - cross-modal alignment - dynamic fusion - inference output".

[0029] Figure 2 This is a flowchart of a training method for a multimodal large model according to an embodiment of the present invention. Figure 2 As shown, training methods for multimodal large models may include: S101, Obtain the training sample set, which includes question samples, dynamic video samples and their corresponding synchronized audio samples, and answer labels corresponding to the question samples.

[0030] Specifically, a large number of dynamic videos (visual) with synchronized audio (auditory) are acquired. One or more questions (text) and corresponding answers can be set for each dynamic video with synchronized audio. The dynamic video with synchronized audio and the answer corresponding to the question are used as a training sample. The questions, dynamic videos, synchronized audio, and answers in the training sample are respectively used as question samples, dynamic video samples, synchronized audio samples, and answer labels.

[0031] The problems in the embodiments of this invention are presented in natural language, such as Chinese and English.

[0032] S102, the question samples, dynamic video samples, and their corresponding synchronized audio samples from the training sample set are input into the multimodal large model to obtain the answer token. The first branch processes the question samples to obtain the question embedding vector, text token, and global semantic vector. The second branch processes the dynamic video samples and generates video compression features based on the question embedding vector and global semantic vector. The third branch processes the synchronized audio samples and generates audio compression features based on the question embedding vector and global semantic vector. The fusion module fuses the video compression features and audio compression features to obtain the audio-video token. The large language module processes the text token and audio-video token to obtain the answer token.

[0033] Specifically, question samples, dynamic video samples, and their corresponding synchronized audio samples from the training sample set are input into the multimodal large model. The first branch processes the question samples to obtain question embedding vectors, text tokens, and global semantic vectors. The second branch extracts features from dynamic video samples and uses the extracted visual features... Question embedding vector and global semantic vector Generate video compression features The third branch extracts features from synchronized audio samples and uses the extracted auditory features... Question embedding vector and global semantic vector Generate audio compression features The fusion module incorporates video compression features. and audio compression features The data is then fused to obtain an audio / video token. The large language module processes the text token and the audio / video token to obtain a response token.

[0034] S103, construct the total loss function of the multimodal large model based on the answer token, answer label, and features generated during the processing of the first, second, and third branches.

[0035] Specifically, the total loss function of the multimodal large model is constructed based on the answer token, answer label, and features generated during the processing of the first, second, and third branches.

[0036] S104, The multimodal large model is trained based on the total loss function to obtain the trained multimodal large model.

[0037] Specifically, the multimodal large model is trained based on the total loss function, and the trained multimodal large model is obtained when the total loss function converges to the minimum value.

[0038] The multimodal large model training method in this embodiment of the invention uses a single-stage question-guided audiovisual question answering framework. The trained multimodal large model has cross-modal content association and depth capabilities, and with question semantics as the core, it can achieve efficient audiovisual question answering.

[0039] In one embodiment of the present invention, such as Figure 3 As shown, the first branch may include a text decoding unit and a text encoding unit. The first branch processes the problem sample and may include: The text decoding unit is used to decode the question sample to obtain the question embedding vector output by the embedding layer of the decoding unit and the text token output by the output layer of the decoding unit; The problem samples are encoded using text encoding units to obtain a global semantic vector.

[0040] In embodiments of the present invention, the text decoding unit may include a text decoder, such as an LLaMA decoder.

[0041] In practice, the question sample is decoded using an LLaMA decoder equipped with a SentencePiece tokenizer to obtain the question embedding vector output by the embedding layer of the LLaMA decoder and the text token output by the output layer of the LLaMA decoder.

[0042] In embodiments of the present invention, the text encoding unit may include a text encoder, such as a BERT encoder.

[0043] In practice, the input question sample is encoded using the BERT encoder to obtain the global semantic vector output by the BERT encoder output layer. .in, .

[0044] Global semantic vector in the embodiments of the present invention , serving as a guiding signal for the extraction of visual features in the second branch and auditory features in the third branch.

[0045] As a specific embodiment, see Figure 3A sample question could be "Are there electric bass and guzheng instruments in the video?". Inputting "Are there electric bass and guzheng instruments in the video?" into the first branch, the LLaMA decoder and BERT encoder can process the question separately, obtaining the question embedding vector, text token, and global semantic vector. .

[0046] In one embodiment of the present invention, such as Figure 3 As shown, the second branch may include a visual coding unit and a visual compression unit. While processing dynamic video samples, the second branch generates video compression features based on the question embedding vector and the global semantic vector, and may include: Visual features are obtained by extracting spatiotemporal features from dynamic video samples using visual coding units. Visual features are flattened to obtain visual flattened features; Using a visual compression unit, visual weights corresponding to visual flattening features are calculated based on visual flattening features and global semantic vectors. Attention pooling is then applied to the enhanced visual features and visual weights corresponding to the visual flattening features to obtain video compression features. The enhanced visual features corresponding to the visual flattening features are obtained by a multilayer perceptron performing question enhancement processing on the visual flattening features and question embedding vectors.

[0047] In embodiments of the present invention, the visual encoding unit may include a visual encoder, such as an I3D encoder.

[0048] In practice, dynamic video samples are input into the I3D encoder, which extracts the spatiotemporal features of the input dynamic video samples and outputs the visual features. ,in, , Indicates the temporal length of dynamic video samples. Indicates the height of pixels in the dynamic video sample space. The width of the dynamic video sample space pixels is represented by , and D represents the hidden dimension.

[0049] Visual features output by the I3D encoder Perform feature reshaping, that is, visual feature Flattening, obtaining visual flattening features .in, , .

[0050] Visual compression units based on visual flattening features and global semantic vector Computational vision flattening features Corresponding visual weights Specifically, according to the first Visual flattening features and global semantic vector The inner product calculation of the first Visual flattening features Corresponding visual weights , ,in, This indicates a hidden dimension.

[0051] In computational vision flattening features Corresponding visual weights Simultaneously, multilayer perceptrons are used to analyze visual flattening features. Visual flattening features are obtained by performing question augmentation processing on the question embedding vector. Corresponding enhanced visual features In practice, a modality-specific MLP (Multi-Layer Perceptron) can be used to align the question embedding vector to the hidden dimension of the video. By dynamically selecting visually relevant features through question semantics, "relevant enhancement and irrelevant suppression" of visual information can be achieved.

[0052] The attention pooling layer in the visual compression unit enhances the visual features corresponding to the visual flattening features. and visual weight Attention pooling is performed, and weighted aggregation is used to obtain video compression features. , .

[0053] In one embodiment of the present invention, such as Figure 3 As shown, the third branch may include an auditory coding unit and an auditory compression unit. While processing synchronized audio samples, the third branch generates audio compression features based on the question embedding vector and the global semantic vector, and may include: Auditory features are obtained by extracting time-frequency features from synchronized audio samples using auditory coding units. The auditory features are flattened to obtain the flattened auditory features; Using an auditory compression unit, auditory weights corresponding to auditory flattening features are calculated based on auditory flattening features and global semantic vectors. Attention pooling is then applied to the enhanced auditory features and auditory weights corresponding to the auditory flattening features to obtain audio compression features. The enhanced auditory features corresponding to the auditory flattening features are obtained by question enhancement of the auditory flattening features and question embedding vectors by a multilayer perceptron.

[0054] In embodiments of the present invention, the auditory encoding unit may include an audio encoder and a speech encoder. The audio encoder is used to process external ambient sounds, and the speech encoder is used to process human voices. The audio encoder may be a Mel-CNN encoder.

[0055] In practice, (when there are no voices in the synchronized audio) the Mel-CNN encoder can be used to extract the time-frequency features of the input synchronized audio samples, and the Mel-CNN encoder outputs auditory features. ,in, , Indicates the timing length of the synchronized audio sample. This represents the frequency domain dimension of the synchronized audio samples. This indicates a hidden dimension.

[0056] Auditory features of the Mel-CNN encoder output Perform feature reconstruction, that is, auditory features Flattening, obtaining auditory flattening characteristics .in, , .

[0057] The auditory compression unit is based on auditory flattening characteristics and global semantic vector Computational vision flattening features Corresponding visual weights Specifically, according to the first Auditory flattening features and global semantic vector The inner product calculation of the first Individual auditory flattening features Corresponding auditory weight , , This indicates a hidden dimension.

[0058] Computational vision flattening features Corresponding visual weights Simultaneously, multilayer perceptrons were used to study auditory flattening features. Visual flattening features are obtained by performing question augmentation processing on the question embedding vector. Corresponding enhanced auditory features In practice, a modality-specific MLP can be used to align the question embedding vector to the audio hidden layer dimension. By dynamically selecting auditory-relevant features through question semantics, auditory "relevant enhancement and irrelevant suppression" can be achieved. Attention pooling layers in auditory compression units, for visual flattening features Corresponding enhanced auditory features and auditory weight Attention pooling is performed, and weighted aggregation is obtained. , .

[0059] In one embodiment of the present invention, the fusion module performs fusion processing on video compression features and audio compression features to obtain an audio-video token, which may include: Based on video compression characteristics and audio compression features A dynamic expert group is selected, and the outputs of the expert group are dynamically weighted and fused to obtain the fused features. .

[0060] The fusion module of this invention supports switching between multiple fusion strategies based on the dynamic integration features of a hybrid expert architecture.

[0061] The embodiments of the present invention address video compression features. and audio compression features When performing fusion processing, early fusion strategies, mid-term fusion strategies, and late fusion strategies are designed according to the specific fusion processing method.

[0062] Among them, early fusion strategies can improve visual flattening features. and auditory flattening features The splicing is performed to obtain the splicing features. Sharing the same route and expert group to process stitched features yields fused features. Early fusion strategies are suitable for problems requiring correlation of original features, such as "instrument counting." Mid-term fusion strategies utilize video compression features. and audio compression features Each component has its own routing, which leads to the same group of experts for fusion, resulting in fused features. Mid-stage fusion strategies can adapt to timing issues such as changes occurring after the drumbeat stops. Late-stage fusion strategies utilize video compression features. and audio compression features Each module has its own routing and expert group, which concatenates the features fused within each module to obtain the fused feature. The late-stage fusion strategy can be adapted to comprehensive judgment problems such as "rhythm comparison".

[0063] Dynamic routing is achieved by a gated network based on video compression features. and audio compression features The gating weight g is calculated token by token, which determines which expert group (expert 1, expert 2, ..., expert n) the user enters, and the final output is the fused feature. .

[0064] The fusion module in this embodiment of the invention can adopt a flexible expert fusion mechanism (Flex Fuse MoE module), which dynamically activates corresponding experts according to the problem type (e.g., time-series problems rely on mid-level time-series interactions, and comparison problems rely on late-stage decision comparisons), supporting multi-level abstraction reasoning. Based on a hybrid expert architecture, it includes early, mid-level, and late-stage fusion-related experts, and adaptively selects expert combinations through a gating network.

[0065] In one embodiment of the present invention, the total loss function of the multimodal large model is constructed based on the answer token, the answer label, and the features generated during the processing of the first branch, the second branch, and the third branch, and may include: Construct a large language loss function based on answer labels and response tokens; Construct a visual loss function for the question based on the question embedding vector and visual flattening features; Construct a question-based auditory loss function based on the question embedding vector and auditory flattening features; Construct an audio-video loss function based on video compression features and audio compression features; The total loss function of the multimodal large model is constructed based on the large language loss function, the problem visual loss function, the problem auditory loss function, and the audio-visual loss function.

[0066] In constructing the total loss function of a multimodal large model, this invention implements an incremental optimization of "alignment-compression-realignment" based on optimal transport theory.

[0067] Specifically, problem-visual modality alignment is performed before compression of the second branch. Specifically, the original features (visual flattening features) are calculated. The Wasserstein distance between the problem sample and the problem embedding vector is obtained. , The representation distribution of the problem embedding vector. Indicates visual flattening features The representation distribution, This indicates that the video and text are aligned.

[0068] For the third branch, before compression, a problem-auditory modality correspondence is performed. Specifically, the original features (auditory flattening features) are calculated. The Wasserstein distance between the problem sample and the problem embedding vector is obtained. , The representation distribution of the problem embedding vector. Indicating auditory flattening characteristics The representation distribution, This indicates that the auditory information is aligned with the text.

[0069] After compression processing in the second and third branches, this embodiment of the invention performs cross-modal alignment. Specifically, the audio-video Wasserstein distance of the compressed features is calculated to obtain... , Indicates audio compression features The representation distribution, Represents video compression features The distribution of representation.

[0070] The aforementioned cross-modal alignment assigns a high transmission cost to padding tokens (invalid features) to resist interference. This means that while left-aligning, because the number of audio and video padding tokens is different (due to the different lengths of the audio and video sequences), a high transmission distance loss is artificially assigned to the positions of these padding tokens to prevent the learning of shortcuts. The loss cannot represent the alignment of the effective semantic positions of the audio and video.

[0071] It should be noted that the Wasserstein distance calculation is performed by iteratively optimizing the transport plan matrix T (based on cosine similarity), and the converged distance is output as the alignment loss. The transport plan only retains high-similarity feature pairs (sparseness improves robustness).

[0072] This invention employs an interleaved optimal transport loss function (OTLoss). Through triple alignment losses (audio-text, video-text, and audio-video), it forces modal features to accurately match under the guidance of question semantics, avoiding cross-modal representation shifts. Furthermore, it uses the Wasserstein distance metric to measure modal distribution differences, performing "question-modal" and "cross-modal" alignment before and after feature compression, forming an "alignment-compression-re-alignment" process.

[0073] The text token output by the text encoding unit and the fusion feature output by the fusion module are combined. Input a large language module (such as an LLaMA3 module), and the large language module outputs an answer token. Construct a large language loss function for each answer token and its corresponding answer label. .

[0074] Large Language Loss Function in Embodiments of the Invention The cross-entropy loss function is used; therefore, the large language loss function Possible forms: Where N represents the number of training samples and C represents the classification categories. This represents the one-hot encoding of the answer label corresponding to the i-th question sample. This represents the probability that the i-th problem sample is predicted to belong to class C.

[0075] In one embodiment of the present invention, the total loss function of the multimodal large model is:

[0076] in, Represents the total loss function. Represents a large language loss function. This indicates the auditory alignment loss weights in the problem area. The auditory loss function represents the problem. , This represents the distribution of the global semantic vector. The representational distribution of auditory flattening features, This indicates that the auditory and text are aligned. This indicates the visual alignment loss weights for the problem. The visual loss function represents the problem. , This represents the distribution of the global semantic vector. The representation distribution of visual flattening features. This indicates that the video and text are aligned. Indicates the alignment loss weights. Represents the audio / video loss function. , The representation distribution of audio compression features. This represents the distribution of characteristics of video compression.

[0077] In this embodiment of the invention, when using the question-guided pooling module, a deformable convolutional pooling module can also be employed. That is, the deformation parameters of the convolutional kernel are predicted through question semantics to focus on relevant regions and achieve feature selection. The attention mechanism is more efficient than the deformable convolutional pooling mechanism for handling long time series.

[0078] Ablation experiments conducted in this embodiment of the invention (removing relevant OT Loss at different stages) show that the complete OT Loss configuration (86.17%) improves accuracy by 2.99% compared to the baseline without OT Loss (83.18%), demonstrating the crucial role of alignment.

[0079] When the pooling layer configuration in the compression unit is optimized in this embodiment of the invention, the model accuracy reaches 86.36% when the audio kernel stride is [6,8] and the visual kernel stride is [6,6,8], verifying the rationality of the feature compression parameters in this embodiment of the invention.

[0080] This invention employs a single-stage end-to-end training framework, eliminating the separate pre-training and fine-tuning processes. It directly focuses on the question-answering task objective, simultaneously optimizing feature alignment and inference to avoid "task-irrelevant" representations and reduce engineering complexity. Cross-modal alignment, feature compression, and fusion inference are unified within a single-stage architecture and jointly optimized end-to-end, eliminating the need for a pre-training stage.

[0081] This invention employs a question-guided pooling design, encoding the question into a semantic query vector. It calculates and weights the cross-modal similarity between the original features and the question, strengthening key features (e.g., focusing on the visual instrument region for the "counting instruments" question) while weakening irrelevant information, dynamically adapting to the granularity requirements of the question. Based on a dynamic attention mechanism of question semantics, it adaptively selects audiovisual features related to the question, achieving weighted feature compression.

[0082] Compared with the two-stage model, the multimodal large model trained in this embodiment of the invention eliminates the pre-training stage. End-to-end training makes the representation more consistent with the question-answering task, and there is no need to design auxiliary loss and specific pre-training components, making engineering implementation simpler.

[0083] The multimodal large model trained in this embodiment of the invention is compared with the problem-guided model. Through the process of "alignment-compression-realignment", the feature alignment is more accurate (experiments show that the accuracy is improved by 2.99% after introducing OTLouss); FlexFuseMoE (audio and video multimodal fusion module) supports dynamic switching of multiple fusion strategies and adapts to more problem types (the overall accuracy on the MUSIC-AVQA dataset reaches 86.17%, which is better than the baseline model).

[0084] The multimodal large model trained in this embodiment of the invention has stronger adaptability. Through dynamic pooling and expert fusion, it can adaptively adjust the processing logic according to the needs of the problem (global / local, temporal / counting) and perform more stably in complex problem scenarios.

[0085] This invention provides an audiovisual question-and-answer method.

[0086] Figure 4 This is a flowchart of an audiovisual question-and-answer method according to an embodiment of the present invention. Figure 4 As shown, audiovisual question-and-answer methods may include: S201, Obtain the questions to be answered and the dynamic video and its corresponding synchronized audio; S202, the question to be answered, the dynamic video and its corresponding synchronized audio are input into the pre-trained multimodal large model to obtain the answer result. The pre-trained multimodal large model is trained using the multimodal large model training method described above.

[0087] Specifically, when asking and answering questions about dynamic videos with synchronized audio, the question to be answered and the dynamic video with synchronized audio can be input into a pre-trained multimodal large model, and the pre-trained multimodal large model can output the answer to the question to be answered.

[0088] Using the pre-trained multimodal large model in this embodiment of the invention for audiovisual question answering has the advantages of high efficiency and accurate question answering.

[0089] This invention provides a computer-readable storage medium.

[0090] In one embodiment, a computer program is stored on a computer-readable storage medium, which, when executed by a processor, implements the training method for a multimodal large model as described above.

[0091] In one embodiment, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, it implements the audiovisual question-and-answer method as described above.

[0092] This invention provides an electronic device.

[0093] In one embodiment, the electronic device may include a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the training method for the multimodal large model as described above.

[0094] In one embodiment, the electronic device may include a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the audiovisual question-and-answer method described above.

[0095] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of the present invention.

[0096] like Figure 5 As shown, the electronic device 500 includes a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, for example, via a bus 502. Optionally, the electronic device 500 may also include a transceiver 504. It should be noted that in practical applications, the transceiver 504 is not limited to one type, and the structure of this electronic device 500 does not constitute a limitation on the embodiments of the present invention.

[0097] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 501 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0098] Bus 502 may include a pathway for transmitting information between the aforementioned components. Bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 502 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0099] The memory 503 is used to store computer programs corresponding to the multimodal large model training method or audiovisual question-answering method of the above embodiments of the present invention. The computer programs are controlled and executed by the processor 501. The processor 501 is used to execute the computer programs stored in the memory 503 to implement the content shown in the foregoing method embodiments.

[0100] Among them, electronic devices 500 include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device 500 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0101] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0102] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0103] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0104] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0105] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0106] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0107] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0108] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A training method for a multimodal large model, characterized in that, The multimodal large model includes a first branch, a second branch, a third branch, a fusion module, and a large language module. The input of the first branch is used to input question-and-answer questions, the input of the second branch is used to input dynamic video, and the input of the third branch is used to input synchronized audio corresponding to the dynamic video. The outputs of the second and third branches are connected to the input of the fusion module, and the outputs of the first branch and the fusion module are connected to the input of the large language module. The large language module is used to output answer tokens. The training method includes: Obtain a training sample set, wherein the training sample set includes question samples, dynamic video samples and their corresponding synchronized audio samples, and answer labels corresponding to the question samples; The question samples, dynamic video samples, and their corresponding synchronized audio samples from the training sample set are input into the multimodal large model to obtain an answer token. Specifically, the first branch processes the question samples to obtain a question embedding vector, a text token, and a global semantic vector; the second branch processes the dynamic video samples while generating video compression features based on the question embedding vector and the global semantic vector; the third branch processes the synchronized audio samples while generating audio compression features based on the question embedding vector and the global semantic vector; the fusion module fuses the video compression features and the audio compression features to obtain an audio-video token; and the large language module processes the text token and the audio-video token to obtain an answer token. The total loss function of the multimodal large model is constructed based on the answer token, the answer label, and the features generated during the processing of the first branch, the second branch, and the third branch. The multimodal large model is trained based on the total loss function to obtain a trained multimodal large model.

2. The training method for a multimodal large model according to claim 1, characterized in that, The first branch includes a text decoding unit and a text encoding unit. The first branch processes the problem sample, including: The text decoding unit is used to decode the question sample to obtain the question embedding vector output by the embedding layer of the decoding unit and the text token output by the output layer of the decoding unit; The problem sample is encoded using the text encoding unit to obtain a global semantic vector.

3. The training method for a multimodal large model according to claim 2, characterized in that, The second branch includes a visual coding unit and a visual compression unit. While processing the dynamic video samples, the second branch generates video compression features based on the question embedding vector and the global semantic vector, including: Using the visual coding unit, spatiotemporal features are extracted from the dynamic video sample to obtain visual features; The visual features are flattened to obtain the visual flattened features; Using the visual compression unit, the visual weights corresponding to the visual flattening features are calculated based on the visual flattening features and the global semantic vector. Attention pooling is then performed on the enhanced visual features and visual weights corresponding to the visual flattening features to obtain the video compression features. The enhanced visual features corresponding to the visual flattening features are obtained by performing question enhancement processing on the visual flattening features and the question embedding vector using a multilayer perceptron.

4. The training method for a multimodal large model according to claim 3, characterized in that, The third branch includes an auditory coding unit and an auditory compression unit. While processing the synchronized audio samples, the third branch generates audio compression features based on the question embedding vector and the global semantic vector, including: Using the auditory coding unit, time-frequency features are extracted from the synchronized audio samples to obtain auditory features; The auditory features are flattened to obtain the flattened auditory features; Using the auditory compression unit, the auditory weights corresponding to the auditory flattening features are calculated based on the auditory flattening features and the global semantic vector. Attention pooling is then applied to the enhanced auditory features and auditory weights corresponding to the auditory flattening features to obtain the audio compression features. The enhanced auditory features corresponding to the auditory flattening features are obtained by performing question enhancement processing on the auditory flattening features and the question embedding vector using a multilayer perceptron.

5. The training method for a multimodal large model according to claim 4, characterized in that, The fusion module performs fusion processing on the video compression features and the audio compression features to obtain an audio / video token, including: Based on the video compression features and the audio compression features, an expert group is dynamically selected, and the outputs of the expert group are dynamically weighted and fused to obtain the fused features.

6. The training method for a multimodal large model according to claim 5, characterized in that, The step of constructing the total loss function of the multimodal large model based on the answer token, the answer label, and the features generated during the processing of the first branch, the second branch, and the third branch includes: Construct a large language loss function based on the answer label and the answer token; Construct a visual loss function for the problem based on the problem embedding vector and the visual flattening features; Construct a question auditory loss function based on the question embedding vector and the auditory flattening features; Construct an audio-video loss function based on the video compression features and the audio compression features; The total loss function of the multimodal large model is constructed based on the large language loss function, the problem visual loss function, the problem auditory loss function, and the audio-visual loss function.

7. The training method for a multimodal large model according to claim 6, characterized in that, The total loss function of the multimodal large model is: in, Represents the total loss function, Represents a large language loss function. This indicates the auditory alignment loss weights in the problem area. This represents the auditory loss function of the problem. , The representation distribution of the problem embedding vector. The representational distribution of auditory flattening features, This indicates the visual alignment loss weights for the problem. The visual loss function represents the problem. , The representation distribution of the problem embedding vector. The representation distribution of visual flattening features. Indicates the alignment loss weights. Represents the audio / video loss function. , The representation distribution of audio compression features. This represents the distribution of characteristics of video compression.

8. An audiovisual question-and-answer method, characterized in that, include: Retrieve unanswered questions and dynamic videos along with their corresponding synchronized audio; The question to be answered, the dynamic video, and the corresponding synchronized audio are input into a pre-trained multimodal large model to obtain the answer result. The pre-trained multimodal large model is trained using the training method of the multimodal large model as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method for the multimodal large model as described in any one of claims 1-7, or the audiovisual question answering method as described in claim 8.

10. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method for a multimodal large model as described in any one of claims 1-7, or the audiovisual question answering method as described in claim 8.

Images