An asymmetric end-cloud collaborative-based multi-modal large language model audio-video question answering method

Abstract

The application discloses a multimodal large language model audio and video question and answer method, system, device, medium and program based on asymmetric end cloud cooperation, which comprises the following steps: receiving an audio and video question and answer request, reconstructing a single modal input according to the modal type supported by an edge device, performing single modal inference on the edge device side and generating an end side initial answer, constructing routing features based on intermediate information and performing routing decision, forwarding the features of the single modal input and a text question to the cloud if the routing label is 0, performing memory bank retrieval, modal enhancement and full modal inference on the cloud side to generate a cloud side answer. The application realizes the cooperative work of end side lightweight inference, automatic identification of difficult samples, cloud memory bank modal supplement and full modal enhancement inference, reduces the end side computing pressure, improves the reliability of routing decision, reduces the transmission overhead and cloud calling cost.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and multimodal data processing technology, specifically to a method, system, device, medium, and program for audio and video question answering based on asymmetric end-cloud collaborative multimodal large language model. Background Technology

[0002] Audio-Visual Question Answering (AVQA) is gradually becoming an important foundational task for multimodal reasoning in mobile systems, with applications including smart homes, intelligent transportation systems, and in-vehicle intelligent perception in the Internet of Vehicles (IoV). In these scenarios, cameras and microphones jointly provide complementary perceptual evidence to answer questions relevant to the current activity. Although Multimodal Large Language Models (MLLMs) have significantly improved the performance of AVQA tasks, direct deployment on edge devices remains infeasible. Recent research shows that large language models with billions of parameters often exceed the memory budget of edge devices; for example, the LLaMA-2-7B requires more than 8GB of space just to store FP16 weights, while most typical edge platforms can only support models with 80 million to 3 billion parameters. Therefore, most AVQA solutions based on multimodal large language models rely on cloud computing, which can lead to longer inference latency, higher bandwidth costs, and privacy concerns. To enable large language models to run at the edge, potential solutions include model compression techniques such as model pruning, knowledge distillation, and low-rank decomposition. While these methods effectively reduce model size, compressed language models typically suffer from performance degradation. Another approach involves dividing large language models into multiple parts and distributing them across other devices or cloud servers for joint inference. Although this method guarantees accuracy, it often introduces significant inference latency due to the time required for transmitting intermediate data or results. Therefore, researchers have proposed a gaining attention paradigm: small-model-large-model collaboration. In this paradigm, smaller-scale large language models, such as those obtained through model compression, collaborate with regular large-scale language models, achieving a synergistic balance between inference efficiency and accuracy.

[0003] In existing small-model-large-model collaboration methods, a core objective is to improve the collaboration efficiency between the small and large models by reducing the amount of interaction data at the token, query, or time levels. For example, in the methods proposed by Hao et al. and Wang et al., a lightweight large language model on the client side first generates initial tokens, which are then further refined and improved by the cloud model. DiSCo dynamically evaluates the importance of tokens on the small model and only transmits key tokens to the large model. Hybrid-LLM and RouteLLM directly evaluate query complexity and only route difficult queries to the large model. LSCI comprehensively considers inference accuracy and resource constraints, making adaptive query offloading decisions between the client and cloud. Ding et al. used a lightweight model to generate initial answers locally, and then retrieved supplementary information from a periodically updated cloud knowledge base to enhance inference quality. However, existing technologies still have the following shortcomings and deficiencies when facing audio and video question answering tasks: (1) Existing small-model-large-model collaboration schemes typically require complete modal data for inference. The input of audio and video question answering tasks includes multiple modalities such as text, audio, and video. Complete modal data not only increases the computational burden on the edge and the transmission burden between the edge and the cloud, but also makes it difficult for existing methods to handle stably when some modalities are missing or inconvenient to upload in practical applications. Therefore, there is an urgent need for a lightweight audio and video question answering processing method that can tolerate modal missing data.

[0004] (2) Existing small-model-large-model collaboration schemes typically require both collaborators to have the same modality processing capabilities. This approach is more suitable for single-modal large language models, but due to inference scalability issues, it is difficult to extend to complex audio and video question-answering tasks. Specifically, under the same parameter budget and the same application task, multimodal large language models usually exhibit weaker modality-specific representation capabilities compared to single-modal models. This is because multimodal large language models must simultaneously consider the specific processing of multiple modalities, additional cross-modal alignment, and modality fusion. While these components are necessary for complete modality understanding, they also consume parameters and computational resources that could be used to enhance single-modality capabilities, thereby increasing the burden on the edge model and affecting edge inference performance.

[0005] (3) Existing edge-cloud collaborative inference methods typically rely on single uncertainty indicators such as confidence, entropy, or perplexity for routing decisions, making it difficult to accurately determine whether the edge-side answer is reliable. In audio and video question answering tasks, edge-side models may exhibit high confidence but incorrect answers, or low confidence but correct answers. If the routing decision is inaccurate, difficult sample errors that the edge cannot answer correctly may be retained locally, leading to a decrease in overall question answering accuracy; simple samples that the edge can handle correctly may be forwarded to the cloud, causing unnecessary cloud calls, transmission overhead, and inference latency. Summary of the Invention

[0006] This application discloses a method, system, device, medium, and program for audio and video question answering based on asymmetric end-to-cloud collaboration of a multimodal large language model. It aims to solve the problems that existing small-model-large-model collaboration schemes require complete modal data for inference, the end-to-end and cloud-to-end sides usually require the same modal processing capabilities, and existing routing mechanisms have difficulty in accurately judging the reliability of end-to-end answers.

[0007] This application discloses an asymmetric end-to-end collaborative multimodal large language model audio and video question answering method, including the following steps: Receive an audio / video question-and-answer request, wherein the audio / video question-and-answer request includes video data V, audio data A, and text question Q; Based on the modality types supported by the edge device, the audio and video question-and-answer request is reconstructed into a single-modal input (U, Q), where U in the single-modal input includes one modality of video data V and audio data A; On the edge device side, a small-scale single-modal multimodal large language model is invoked to perform local reasoning on the single-modal input, generating an initial answer on the edge, and recording intermediate information during the edge reasoning process; routing features are constructed based on the intermediate information, and routing decisions are made based on the routing features to obtain a routing label; if the routing label is 1, the initial answer on the edge is returned directly; if the routing label is 0, the features U of the single-modal input and the text question Q are forwarded to the cloud; On the cloud side, a hash key value is calculated based on the feature U of the unimodal input, and historical samples with similar features are retrieved from the cloud memory based on the hash key value. Another modal feature complementary to the feature U of the unimodal input is extracted. The feature U of the unimodal input is combined with the extracted complementary modal feature to construct an enhanced full-modal input E. A large-scale full-modal multimodal language model is invoked to perform joint reasoning based on the enhanced full-modal input E and the text question Q to generate a cloud answer, and the cloud answer is returned.

[0008] Optionally, constructing routing features based on the intermediate information includes: during inference using a small-scale multimodal large language model on the client side, overcoding the text question Q to obtain the lexical hidden state of the text question Q, and performing average pooling on the lexical hidden state to obtain question semantic features; calculating the answer uncertainty features of the initial answer on the client side, the answer uncertainty features including answer sequence confidence, answer sequence perplexity, answer sequence log likelihood value, and answer average entropy; and concatenating the question semantic features with the answer uncertainty features to obtain the routing features.

[0009] Optionally, the step of making routing decisions based on the routing features includes: calculating the average token confidence of the initial answer on the client side, and obtaining a confidence label based on whether the average token confidence is greater than a preset threshold; inputting the routing features into a lightweight multilayer perceptron classifier to predict the correctness label of the initial answer on the client side; if both the confidence label and the correctness label are present, then the initial answer on the client side is determined to be reliable, and the routing label is set to 1; otherwise, the routing label is set to 0.

[0010] Optionally, the step of retrieving historical samples with similar features from the cloud memory includes: if the feature U of the single-modal input is an audio modality, then using audio fingerprint matching and time offset histogram voting, the historical audio sample most similar to the feature U of the single-modal input is retrieved from the audio memory; if the feature U of the single-modal input is a video modality, then first using perceptual hash features for coarse-grained screening, and then using scale-invariant feature transformation features for fine-grained matching, the historical video sample most similar to the feature U of the single-modal input is retrieved from the video memory.

[0011] Optionally, the fine-grained matching using scale-invariant feature transform features includes: extracting local key point features of the scale-invariant feature transform of the candidate historical video samples, performing feature matching using fast approximate nearest neighbor matching, and filtering by the nearest nearest neighbor distance ratio test to select the historical video sample with the highest matching score.

[0012] Optionally, constructing the enhanced full-modal input E includes: if the feature U of the single-modal input is an audio modality, then extracting its corresponding video features from the retrieved historical samples and combining them with the feature U of the single-modal input; if the feature U of the single-modal input is a video modality, then extracting its corresponding audio features from the retrieved historical samples and combining them with the feature U of the single-modal input.

[0013] This application also provides an asymmetric edge-cloud collaborative multimodal large language model audio-video question answering system, including an audio-video question answering request access module for receiving audio-video question answering requests, wherein the audio-video question answering request includes video data V, audio data A, and text question Q; an edge-side single-modal reconstruction module, deployed on an edge device, for reconstructing the audio-video question answering request into a single-modal input (U, Q) according to the modality type supported by the edge device, wherein U in the single-modal input includes one modality of video data V and audio data A; and an edge-side single-modal inference module for calling an edge-side small-scale single-modal multimodal large language model on the edge device side to perform local inference on the single-modal input, generating an edge-side initial answer, and recording intermediate information during the edge-side inference process; constructing routing features based on the intermediate information, and according to the routing features... Routing decisions are made based on features to obtain routing labels. If the routing label is 1, the initial answer on the endpoint is returned directly. If the routing label is 0, the features U of the unimodal input and the text question Q are forwarded to the cloud. The cloud-based full-modal inference module is used to calculate a hash key value based on the features U of the unimodal input on the cloud side, and retrieve historical samples with similar features from the cloud memory based on the hash key value, and extract another modal feature that is complementary to the features U of the unimodal input. The features U of the unimodal input are combined with the extracted complementary modal feature to construct an enhanced full-modal input E. A large-scale full-modal multimodal language model is invoked to perform joint inference based on the enhanced full-modal input E and the text question Q to generate a cloud answer, and the cloud answer is returned.

[0014] This application also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the aforementioned asymmetric end-to-cloud collaborative multimodal large language model audio and video question answering method.

[0015] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned asymmetric end-to-cloud collaborative multimodal large language model audio and video question answering method.

[0016] This application also provides a program product consisting of a computer program that, when executed by a processor, implements the aforementioned asymmetric end-to-cloud collaborative multimodal large language model audio and video question answering method.

[0017] This application discloses a method, system, device, medium, and program for audio and video question answering based on an asymmetric edge-cloud collaborative multimodal large language model. The method includes receiving audio and video question answering requests, reconstructing the data according to the modal types supported by the edge device, performing single-modal reasoning and routing decisions on the edge device side, and performing memory retrieval, modality enhancement, and full-modal reasoning on the cloud side. Through the above technical means, this application achieves collaborative operation of lightweight edge-side reasoning, automatic identification of difficult samples, cloud memory modality supplementation, and full-modal enhanced reasoning. Compared with existing symmetric edge-cloud collaborative methods that require both the edge and cloud sides to have complete multimodal processing capabilities, this application forms an asymmetric collaborative structure through an edge-side single-modal model and a cloud-based full-modal model, reducing the computational pressure on the edge side. Compared with existing methods that only rely on confidence for routing, this application integrates question semantic features and answer uncertainty features, improving the reliability of routing decisions. Compared with cloud-based reasoning methods that directly upload complete original audio and video data, this application only uploads lightweight single-modal information when necessary and supplements missing modality information through a cloud memory, reducing transmission overhead and cloud call costs. Therefore, this application can improve the autonomous processing capability of edge devices and the efficiency of edge-cloud collaborative reasoning while ensuring the accuracy of audio and video question answering. Attached Figure Description

[0018] Figure 1 A schematic diagram of the system model for the asymmetric end-to-cloud collaborative multimodal large language model audio and video question answering method provided in the embodiments of this application; Figure 2 A schematic diagram illustrating the specific framework of the asymmetric end-to-cloud collaborative multimodal large language model audio and video question answering method provided in this application embodiment; Figure 3 A flowchart of a multimodal large language model audio and video question answering method based on asymmetric end-to-cloud collaboration provided in this application embodiment.

[0019] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0021] In existing audio and video question answering tasks, inference typically requires simultaneous input of video, audio, and text modalities. However, directly deploying large multimodal models on edge devices faces the challenge of insufficient memory budget due to the massive number of parameters in large-scale models. Furthermore, existing edge-cloud collaborative solutions often require both the edge and cloud sides to have the same modality processing capabilities, which increases the computational burden on the edge. Moreover, existing routing mechanisms often rely on only a single confidence metric, making it difficult to accurately determine the reliability of the edge's answers. This can easily lead to simple samples being incorrectly forwarded to the cloud or difficult samples being incorrectly retained locally.

[0022] refer to Figure 1 The first objective of this invention is to provide an asymmetric, end-to-end collaborative, multimodal large language model audio / video question-answering system. This system addresses the problems of existing small-model-large-model collaborative schemes requiring complete modal data for inference, the typical requirement for identical modal processing capabilities on both the end-to-end and cloud-side sides, and the difficulty of accurately determining the reliability of end-to-end answers using existing routing mechanisms. This system achieves efficient processing of audio / video question-answering tasks through end-to-end single-modal inference, semantically enhanced routing, and cloud-based memory-based modal enhancement. This reduces the computational burden on the end-to-end, decreases data transmission overhead and cloud call frequency, and improves the overall accuracy and inference efficiency of the audio / video question-answering system.

[0023] The technical solution to achieve the first objective of this invention is: an asymmetric end-to-cloud collaborative multimodal large language model audio and video question answering system, including an audio and video question answering request access module, an end-to-end single-modal reasoning module, a semantic enhancement routing module, a cloud memory module, a modality enhancement module, a cloud-based full-modal reasoning module, and a result return module.

[0024] The audio / video question-and-answer request access module is used to receive audio / video question-and-answer requests. Each audio / video question-and-answer request includes video data, audio data, and a text question, which can be represented as x=(V, A, Q), where V represents video modal data, A represents audio modal data, and Q represents a text question.

[0025] The edge-side unimodal inference module is deployed on edge devices for lightweight inference based on a single modality supported by the edge device. This module does not require the edge device to simultaneously process complete audio, video, and text inputs. Instead, it reconstructs the original audio / video question-answering request x=(V, A, Q) into a unimodal input (U, Q), where U∈{A, V}. When U=A, it indicates edge-side audio question-answering based on audio and text questions, with cloud-based audio / video question-answering inference when necessary; when U=V, it indicates edge-side video question-answering based on video and text questions, with cloud-based audio / video question-answering inference when necessary. The edge-side unimodal inference module utilizes a small-scale unimodal multimodal large language model S. UPerform local inference on (U, Q) to generate the initial answer a on the endpoint. edge .

[0026] The semantically enhanced routing module determines the reliability of the initial answer from the client and decides whether the current audio / video question-answering request needs to be forwarded to the cloud. This module defines the routing task as a binary classification decision problem, outputting a routing label r(x). When r(x) = 1, it indicates that the client-side answer is reliable, and the system directly returns the client-side answer; when r(x) = 0, it indicates that the client-side answer is unreliable or the sample is difficult, and the system forwards the request to the cloud for enhanced inference. The semantically enhanced routing module considers both the confidence level of the client-side answer and the prediction of its correctness, rather than relying solely on a single confidence level indicator.

[0027] The cloud-based memory module is deployed on a cloud server to store multimodal feature information of historical audio and video samples. For each complete historical audio and video sample (A, V, Q), the system converts it into a quadruple [Index...] a Index v Feature a Feature v ], where Index a Indicates an audio index, Index v Represents the video index, Feature a Represents audio features. v The video features are represented as follows: Audio features can include frequency peaks and temporal peaks in the audio signal, and the audio index can be generated using an audio fingerprint hashing method; Video features can include perceptual hash features and scale-invariant features, where perceptual hash features are used to represent the global visual similarity of video frames, and scale-invariant features are used to represent local keypoint features. Audio features and video features are organized into audio memory and video memory, respectively, and each memory uses a key-value structure of [HashKey, Value].

[0028] The modality enhancement module is used to supplement missing modalities in the routed single-modal input in the cloud. When the cloud receives the single-modal input U forwarded from the end side, the modality enhancement module calculates the hash key value of the input and retrieves candidate historical samples from the corresponding modality memory. For audio input, the system uses audio fingerprint matching and time offset histogram voting to select the most similar historical audio sample; for video input, the system first uses perceptual hash features for coarse-grained screening, and then uses scale-invariant features, fast approximate nearest neighbor matching, and nearest neighbor distance ratio test for fine-grained matching to select the most similar historical video sample. Subsequently, the system extracts another modality feature from the historical sample that is complementary to the current input and combines it with the current input to obtain the enhanced full-modal input E(U).

[0029] The cloud-based full-modal reasoning module is used for cloud-based reasoning based on the enhanced full-modal input E(U) and the text question Q. This module deploys a large-scale full-modal multimodal language model L, capable of integrating the semantic and temporal relationships between audio, video, and text questions to generate a cloud-based answer a. cloud .

[0030] The result return module is used to output the final question-and-answer result. When the semantic enhancement routing module determines that the end-side answer is reliable, the result return module directly returns the end-side answer 'a'. edge When the semantic enhancement routing module determines that cloud assistance is needed, the result return module returns the answer 'a' generated by the cloud-based full-modal reasoning module. cloud .

[0031] refer to Figure 2 The second embodiment of this application provides an asymmetric, end-to-end collaborative, multimodal large language model audio and video question-answering system. The system includes a user side, an edge device side, and a cloud server side.

[0032] The technical solution to achieve the second objective of this invention is: an asymmetric end-to-end collaborative multimodal large language model audio and video question answering method, comprising the following steps: 1) When the audio / video question and answer request begins, the system receives an audio / video question and answer request, which includes video data V, audio data A, and text question Q, denoted as x=(V, A, Q).

[0033] 2) The system reconstructs the input request based on the modal types supported by the edge device, converting the complete audio / video question-and-answer input x=(V,A,Q) into a single-modal input (U, Q), where U∈{A, V}. If the edge device supports audio inference, then U=A; if the edge device supports video inference, then U=V.

[0034] 3) The edge-side single-modal inference module calls the edge-side small-scale single-modal multimodal large language model S U Local inference is performed on the single-modal input (U,Q) to generate the initial answer a on the endpoint. edge .

[0035] 4) During the process of generating answers on the device side, the system records the intermediate inference information of the device side model, including the hidden state of the text question, the lexical probability of the generated answer, the confidence of the answer, the perplexity, the log-likelihood value, and the entropy.

[0036] 5) The semantically enhanced routing module constructs routing features φ(x) based on intermediate information from the edge-side inference process. These routing features include question semantic features and answer uncertainty features. The question semantic features are calculated from the hidden state of the text question output by the edge-side model encoder; the answer uncertainty features are calculated from the token probabilities during the edge-side answer generation process.

[0037] 6) The semantically enhanced routing module calculates the confidence label of the end-side answer. f(x) Specifically, assuming the answer generated by the endpoint contains M tokens, the system calculates the average confidence score of all generated tokens. When the average token confidence score is greater than a preset threshold τ, l f(x) =1; otherwise, l f(x) =0.

[0038] 7) The semantically enhanced routing module inputs the routing features φ(x) into a lightweight multilayer perceptron classifier to predict the correctness label l of the end-side answer. c(x) When the classifier determines that the end-side answer is correct, l c(x) =1; otherwise, l c(x) =0.

[0039] 8) The semantically enhanced routing module bases its routing on the confidence label. f(x) and correctness label l c(x) Generate route label r(x). If l f(x) =1 and l c(x) If r(x) = 1, then r(x) = 1, indicating that the answer from the client side is reliable; otherwise, r(x) = 0, indicating that the current request requires cloud assistance.

[0040] 9) Determine if the routing label r(x) is 1. If r(x) = 1, the system directly sends the endpoint answer a. edge Return as the final answer; if r(x)=0, the system will forward the lightweight single-modal input U or its features to the cloud and proceed to the next step.

[0041] 10) After the cloud receiver forwards the single-modal input U, it calculates the HashKey corresponding to the input and retrieves the set of candidate historical samples with the same or similar HashKey in the cloud memory module.

[0042] 11) Fine-grained feature matching is performed on the candidate historical sample set in the cloud. If U is an audio modality, the matching score is calculated using audio fingerprint matching and time offset histogram voting, and the historical audio sample with the highest score is selected. If U is a video modality, coarse-grained screening is performed using scale-invariant features, and fine-grained matching is further performed using scale-invariant features, fast approximate nearest neighbor matching, and nearest neighbor distance ratio testing to select the most similar historical video sample.

[0043] 12) The cloud extracts complementary modal features from the most similar historical samples. If the current input U is an audio modality, the video features corresponding to the historical sample are extracted as the complementary modality; if the current input U is a video modality, the audio features corresponding to the historical sample are extracted as the complementary modality.

[0044] 13) The modal enhancement module combines the current single-modal input U with the retrieved complementary modal features to construct the enhanced full-modal input E(U).

[0045] 14) The cloud-based full-modal reasoning module calls the large-scale full-modal multimodal language model L, performs joint reasoning based on the enhanced full-modal input E(U) and the text question Q, and generates the cloud-based answer a. cloud .

[0046] 15) The system will send the answer a from the cloud. cloud The audio / video Q&A process ends when the final answer is returned.

[0047] Step 5) involves the following steps for defining the semantic features of the text question: The edge model encodes the text question Q to obtain the final hidden state of each word in Q; average pooling is performed on the hidden states of all words to obtain the overall semantic representation of the text question; then, a unified-dimensional semantic feature is obtained through linear mapping, random deactivation, and the ReLU activation function. This semantic feature is used to describe the semantic content of the text question and the question-answering requirements.

[0048] The answer uncertainty features in step 5) specifically include answer sequence confidence, answer sequence perplexity, answer sequence log-likelihood, and answer average entropy. Assume the answer generated by the endpoint model is... Composed of 1 word element, denoted as ,in Indicates the first The nth generated word element; assuming the end-side model is at the nth... The probability distribution of the tokens output by each decoding step is as follows: Then generate word elements. The corresponding probability is .

[0049] The answer sequence confidence score is obtained by averaging the probabilities of all generated words. It represents the overall confidence level of the edge model when generating the answer and can be expressed as:

[0050] The perplexity of the answer sequence is calculated based on the negative logarithmic mean of the probabilities of the generated words. It represents the degree of uncertainty the model has when generating the answer. A higher perplexity indicates greater uncertainty in the model, and can be expressed as:

[0051] The log-likelihood value of the answer sequence is obtained by summing the log probabilities of each generated word. It represents the overall probability of the model generating the answer sequence and can be expressed as:

[0052] The average entropy of the answer is obtained by calculating the entropy of the output probability distribution at each decoding step and averaging it over all decoding steps. It is used to represent the uncertainty of the word probability distribution during the generation process and can be expressed as:

[0053] in, Indicates a vocabulary list, This represents any candidate word in the vocabulary. The system assigns confidence scores to the above answer sequence. Answer sequence perplexity Log-likelihood value of the answer sequence and the average entropy of the answer The final routing features are obtained by concatenating them with the semantic features of the question. .

[0054] Step 6) involves averaging the softmax probabilities of each word in the generated answer from the client-side model. If this average is greater than a preset threshold τ, the client-side model is considered to have high confidence in the answer; otherwise, it is considered to have low confidence. The threshold τ can be set or adjusted according to the accuracy and efficiency requirements of different application scenarios.

[0055] The multilayer perceptron classifier in step 7) is used to predict whether the answer at the edge is correct. This classifier may include two fully connected layers and a softmax output layer, where the hidden layer is used to fuse the semantic features of the question and the uncertainty features of the answer, and the output layer is used to obtain a binary classification result of whether the answer is correct or incorrect.

[0056] The cloud-based memory in step 10) is a historical multimodal feature library built offline. For the audio modality, the system extracts the audio frequency peaks and temporal peaks, and generates an audio index based on the audio fingerprint; for the video modality, the system extracts the perceptual hash features and scale-invariant feature transform features of the video frames, and generates a video index based on the visual features. The audio index, video index, audio features, and video features together constitute historical sample entries.

[0057] The audio matching process in step 11) is as follows: audio fingerprint matching is performed on the current audio input and candidate historical audio samples, the consistency of time offset between the two is statistically analyzed, and the matching score is obtained by voting through the time offset histogram. The historical sample corresponding to the highest voting score is selected as the most similar audio sample.

[0058] The video matching process in step 11) is as follows: First, the perceptual hash similarity between the current video frame and the candidate historical video frames is calculated to filter out visually similar candidate samples; then, scale-invariant feature transformation local key point features are extracted, fast approximate nearest neighbor matching is used for feature matching, and the nearest neighbor distance ratio test is used to filter out incorrect matches. Finally, the historical sample with the highest matching score is selected as the most similar video sample.

[0059] The enhanced full-modal input E(U) in step 13) refers to the input obtained by combining the current unimodal input with complementary modal features retrieved from the memory. This input is used to enable the cloud-based full-modal multimodal large language model to obtain more complete audio and video evidence, thereby improving the inference accuracy of audio and video question answering.

[0060] Through the above technical solutions, this invention achieves collaborative operation of lightweight edge-side inference, automatic identification of difficult samples, modality supplementation in the cloud memory, and enhanced full-modality inference. Compared with existing symmetrical edge-cloud collaborative methods that require both the edge and cloud sides to have complete multimodal processing capabilities, this invention reduces the computational burden on the edge by forming an asymmetric collaborative structure through an edge-side single-modal model and a cloud-based full-modal model. Compared with existing methods that only base routing on confidence, this invention integrates question semantic features and answer uncertainty features, improving the reliability of routing decisions. Compared with cloud-based inference methods that directly upload complete original audio and video data, this invention only uploads lightweight single-modal information when necessary and supplements missing modality information through the cloud memory, reducing transmission overhead and cloud call costs. Therefore, this invention can improve the autonomous processing capabilities of edge devices and the efficiency of edge-cloud collaborative inference while ensuring the accuracy of audio and video question answering.

[0061] refer to Figure 3 The third embodiment of this application provides an asymmetric, end-to-end collaborative, multimodal large language model audio and video question answering method. This method can be executed by a processor, which can be located on a terminal or a server. The execution process of this method is as follows: This application discloses an asymmetric end-to-end collaborative multimodal large language model audio and video question answering method, including the following steps: Step s101: Receive an audio / video question-and-answer request, which includes video data V, audio data A, and text question Q; Step s102: Based on the modal types supported by the edge device, reconstruct the audio and video question-and-answer request into a single-modal input (U, Q), where U in the single-modal input includes one modality of video data V and audio data A; Step s103: On the edge device side, a small-scale single-modal multimodal large language model is invoked to perform local reasoning on the single-modal input, generate the initial answer on the edge, and record the intermediate information during the edge reasoning process; based on the intermediate information, routing features are constructed, and routing decisions are made according to the routing features to obtain the routing label; if the routing label is 1, the initial answer on the edge is returned directly; if the routing label is 0, the single-modal input features U and the text question Q are forwarded to the cloud; Step s104: On the cloud side, calculate the hash key value based on the feature U of the single-modal input, and retrieve historical samples with similar features from the cloud memory based on the hash key value. Extract another modal feature that is complementary to the feature U of the single-modal input. Combine the feature U of the single-modal input with the extracted complementary modal feature to construct the enhanced full-modal input E. Call the large-scale full-modal multimodal language model, perform joint reasoning based on the enhanced full-modal input E and the text question Q, generate the cloud answer, and return the cloud answer.

[0062] In one embodiment of this application, constructing routing features based on intermediate information includes: encoding a text question Q, obtaining the word hidden state of the text question Q, and performing average pooling on the word hidden state to obtain question semantic features; calculating the answer uncertainty features of the initial answer on the endpoint, the answer uncertainty features including answer sequence confidence, answer sequence perplexity, answer sequence log likelihood value and answer average entropy; and concatenating the question semantic features and the answer uncertainty features to obtain routing features.

[0063] In one embodiment of this application, routing decisions based on routing features include: calculating the average token confidence of the initial answer on the endpoint, and obtaining a confidence label based on whether the average token confidence is greater than a preset threshold; inputting the routing features into a lightweight multilayer perceptron classifier to predict the correctness label of the initial answer on the endpoint; if both the confidence label and the correctness label are present, the initial answer on the endpoint is determined to be reliable, and the routing label is set to 1; otherwise, the routing label is set to 0.

[0064] In one embodiment of this application, the step of retrieving historical samples with similar features from a cloud memory bank includes: if the feature U of the single-modal input is an audio modality, then using audio fingerprint matching and time offset histogram voting, the historical audio sample most similar to the single-modal input feature U is retrieved from the audio memory bank; if the feature U of the single-modal input is a video modality, then first using perceptual hash features for coarse-grained screening, and then using scale-invariant feature transformation features for fine-grained matching, the historical video sample most similar to the single-modal input feature U is retrieved from the video memory bank.

[0065] In one embodiment of this application, fine-grained matching is performed using scale-invariant feature transform features, including: extracting local key point features of scale-invariant feature transform of candidate historical video samples, performing feature matching using fast approximate nearest neighbor matching, and filtering by nearest nearest neighbor distance ratio test to select the historical video sample with the highest matching score.

[0066] In one embodiment of this application, constructing the enhanced full-modal input E includes: if the feature U of the single-modal input is an audio modality, then extracting its corresponding video features from the retrieved historical samples and combining them with the feature U of the single-modal input; if the feature U of the single-modal input is a video modality, then extracting its corresponding audio features from the retrieved historical samples and combining them with the feature U of the single-modal input.

[0067] This application also provides an asymmetric edge-cloud collaborative multimodal large language model audio and video question answering system, including an audio and video question answering request access module for receiving audio and video question answering requests, which include video data V, audio data A, and text question Q; an edge-side single-modal reconstruction module, deployed on an edge device, for reconstructing the audio and video question answering request into a single-modal input (U, Q) according to the modality type supported by the edge device, wherein U in the single-modal input includes one modality of video data V and audio data A; and an edge-side single-modal inference module for calling a small-scale edge-side single-modal multimodal large language model on the edge device to perform local inference on the single-modal input, generating an initial edge-side answer, and recording intermediate information during the edge-side inference process; and constructing a route based on the intermediate information. The system first identifies features and makes routing decisions based on these features to obtain routing labels. If the routing label is 1, the initial answer is returned directly to the client side. If the routing label is 0, the features U of the single-modal input and the text question Q are forwarded to the cloud. The cloud-based full-modal inference module calculates hash keys based on the features U of the single-modal input on the cloud side, retrieves historical samples with similar features from the cloud memory based on the hash keys, and extracts another modal feature that is complementary to the features U of the single-modal input. The single-modal input features U and the extracted complementary modal features are combined to construct the enhanced full-modal input E. The system then calls a large-scale full-modal multimodal language model to perform joint inference based on the enhanced full-modal input E and the text question Q, generates the cloud answer, and returns the cloud answer.

[0068] It is understood that the above method embodiments correspond to the system embodiments. Therefore, the system embodiments have all the beneficial effects of the method embodiments, and will not be repeated here.

[0069] This application also provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the asymmetric end-to-cloud collaborative multimodal large language model audio and video question answering method provided in any of the preceding embodiments.

[0070] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the asymmetric end-to-cloud collaborative multimodal large language model audio and video question answering method provided in any of the preceding embodiments.

[0071] This application also provides a program product consisting of a computer program that, when executed by a processor, implements the asymmetric end-to-cloud collaborative multimodal large language model audio and video question answering method provided in any of the preceding embodiments.

[0072] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An asymmetric, end-to-end collaborative, multimodal large language model audio and video question answering method, characterized in that, Includes the following steps: Receive an audio / video question-and-answer request, wherein the audio / video question-and-answer request includes video data V, audio data A, and text question Q; Based on the modality types supported by the edge device, the audio and video question-and-answer request is reconstructed into a single-modal input (U, Q), where U in the single-modal input includes one modality of video data V and audio data A; On the edge device side, a small-scale single-modal multimodal large language model is invoked to perform local reasoning on the single-modal input, generating an initial answer on the edge, and recording intermediate information during the edge reasoning process; routing features are constructed based on the intermediate information, and routing decisions are made based on the routing features to obtain a routing label; if the routing label is 1, the initial answer on the edge is returned directly; if the routing label is 0, the features U of the single-modal input and the text question Q are forwarded to the cloud; On the cloud side, a hash key value is calculated based on the feature U of the unimodal input, and historical samples with similar features are retrieved from the cloud memory based on the hash key value. Another modal feature complementary to the feature U of the unimodal input is extracted. The feature U of the unimodal input is combined with the extracted complementary modal feature to construct an enhanced full-modal input E. A large-scale full-modal multimodal language model is invoked to perform joint reasoning based on the enhanced full-modal input E and the text question Q to generate a cloud answer, and the cloud answer is returned.

2. The asymmetric end-to-end collaborative multimodal large language model audio and video question answering method according to claim 1, characterized in that, The construction of routing features based on the intermediate information includes: During the inference of a small-scale multimodal large language model on the edge, the text question Q is over-encoded to obtain the word hidden state of the text question Q, and the word hidden state is averaged and pooled to obtain the question semantic features. Calculate the answer uncertainty features of the initial answer on the endpoint, wherein the answer uncertainty features include answer sequence confidence, answer sequence perplexity, answer sequence log likelihood, and answer average entropy; The routing feature is obtained by concatenating the semantic features of the question with the uncertainty features of the answer.

3. The asymmetric end-to-end collaborative multimodal large language model audio and video question answering method according to claim 2, characterized in that, The routing decision based on the routing features includes: Calculate the average word confidence of the initial answer on the endpoint, and obtain the confidence label based on whether the average word confidence is greater than a preset threshold; The routing features are input into a lightweight multilayer perceptron classifier to predict the correctness label of the initial answer on the edge. If both the confidence and correctness labels are present, the initial answer on the endpoint is determined to be reliable, and the routing label is set to 1; otherwise, the routing label is set to 0.

4. The asymmetric end-to-end collaborative multimodal large language model audio and video question answering method according to claim 1, characterized in that, The steps for retrieving historical samples with similar characteristics from the cloud memory include: If the feature U of the single-modal input is an audio modality, then audio fingerprint matching and time offset histogram voting are used to retrieve the historical audio sample most similar to the feature U of the single-modal input from the audio memory. If the feature U of the single-modal input is a video modality, then coarse-grained screening is first performed using perceptual hash features, followed by fine-grained matching using scale-invariant feature transformation features, and the historical video sample most similar to the feature U of the single-modal input is retrieved from the video memory.

5. The asymmetric end-to-end collaborative multimodal large language model audio and video question answering method according to claim 4, characterized in that, The fine-grained matching using scale-invariant feature transformation features includes: The scale-invariant feature transform local key point features of the candidate historical video samples are extracted, and fast approximate nearest neighbor matching is used for feature matching. The nearest neighbor distance ratio test is used for filtering to select the historical video sample with the highest matching score.

6. The asymmetric end-to-end collaborative multimodal large language model audio and video question answering method according to claim 1, characterized in that, The enhanced full-modal input E includes: If the feature U of the single-modal input is an audio modality, then the corresponding video feature is extracted from the retrieved historical samples and combined with the feature U of the single-modal input. If the feature U of the single-modal input is a video modality, then the corresponding audio features are extracted from the retrieved historical samples and combined with the feature U of the single-modal input.

7. An asymmetric end-to-end collaborative multimodal large language model audio and video question answering system, characterized in that, include: The audio / video question-and-answer request access module is used to receive audio / video question-and-answer requests, which include video data V, audio data A, and text questions Q; An edge-side single-modal reconstruction module, deployed on an edge device, is used to reconstruct the audio and video question-and-answer request into a single-modal input (U, Q) according to the modal types supported by the edge device, wherein U in the single-modal input includes one modality of video data V and audio data A; The edge-side unimodal inference module is used to invoke a small-scale edge-side unimodal multimodal large language model to perform local inference on the unimodal input, generate an initial edge-side answer, and record intermediate information during the edge-side inference process; construct routing features based on the intermediate information, and make routing decisions based on the routing features to obtain routing labels; if the routing label is 1, the initial edge-side answer is returned directly; if the routing label is 0, the features U of the unimodal input and the text question Q are forwarded to the cloud. The cloud-based full-modal reasoning module is used to calculate a hash key value based on the feature U of the single-modal input on the cloud side, and retrieve historical samples with similar features from the cloud memory based on the hash key value, extracting another modal feature complementary to the feature U of the single-modal input; combining the feature U of the single-modal input with the extracted complementary modal feature to construct an enhanced full-modal input E; calling a large-scale full-modal multimodal language model to perform joint reasoning based on the enhanced full-modal input E and the text question Q, generating a cloud-based answer, and returning the cloud-based answer.

8. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the asymmetric end-to-cloud collaborative multimodal large language model audio and video question answering method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the asymmetric end-to-cloud collaborative multimodal large language model audio and video question answering method as described in any one of claims 1 to 6.

10. A program product, characterized in that, The program product consists of a computer program, characterized in that the program, when executed by a processor, implements the asymmetric end-to-cloud collaborative multimodal large language model audio and video question answering method as described in any one of claims 1 to 6.

Images