Mobile edge collaborative multi-modal large language model speculative reasoning method

Abstract

The invention discloses a speculative reasoning method for a multi-modal large language model based on mobile edge collaboration. The speculative reasoning method comprises the following steps: constructing an equipment-side double-clue probe model; task semantic distillation and feature alignment are carried out, joint optimization is carried out on double heads of a double-clue classifier, knowledge of an edge end MLLM is distilled to a double-clue probe model, and an equipment end minimum modal delay decision is executed; asynchronous mode screening and speculative reasoning are realized through a hierarchical gating network; and executing a rollback decision to obtain a corrected final reasoning result. According to the embodiment of the invention, a lightweight dual-clue probe model is constructed at a mobile client, task semantic features are extracted from the deep layer of an edge MLLM, joint estimation of reasoning confidence and modal sufficiency is realized, and a minimum modal delay decision of an equipment end is executed; a hierarchical gating mechanism is adopted at an edge end, a confusion-guided rollback mechanism is introduced, an early reasoning result with insufficient confidence is corrected, and early recognition and efficient reasoning of a minimum full subset in an asynchronous mode are achieved.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and in particular to a mobile edge collaborative multimodal large language model speculative reasoning method. Background Technology

[0002] In recent years, with the rapid development of multimodal perception and generation technologies, Multimodal Large Language Models (MLLMs) have gradually become an important inference core for intelligent mobile terminals, in-vehicle systems, and IoT sensing platforms. By integrating multiple modal inputs such as vision, speech, action, and semantics, MLLMs possess stronger generalization and task transfer capabilities, enabling them to support cognitive understanding and reasoning decisions in complex scenarios. However, due to limitations such as large model size, high computational requirements, and asynchronous modal inputs, real-time inference using MLLMs on mobile terminals still faces significant challenges.

[0003] Currently, mainstream solutions fall into three categories: model compression, cloud inference, and edge collaborative inference. Model compression reduces model complexity through techniques such as structural pruning, distillation, or quantization, but often at the cost of accuracy. Cloud inference relies on centralized computing resources, which, while maintaining high accuracy, leads to significant communication latency and high privacy risks. Edge collaborative inference represents a compromise, leveraging the complementary computing power of terminals and edge nodes to reduce transmission and response latency while ensuring data security. However, traditional edge collaborative frameworks still have two shortcomings: First, lightweight terminal models lack reliable confidence estimation and modality sufficiency judgment, resulting in some inference results that could be completed locally being mistakenly transmitted to the edge, or low-confidence results being incorrectly retained on the terminal. Second, when receiving asynchronous multimodal input, the edge side struggles to determine whether to wait for more slow modal data or proceed with direct inference, thus balancing accuracy and latency.

[0004] To address the aforementioned issues, various improvement schemes have been proposed in this field. For example, AppealNet achieves large-to-small model switching through a confidence threshold, EdgeFM optimizes edge-side inference using semantic distillation, and works such as DynMM and AdaMML introduce modality selection strategies to reduce the use of redundant modalities. However, these methods still assume synchronous input modalities or require offline feature tables and fixed strategies, making it difficult to cope with dynamic network conditions and the asynchronous nature of real-time modalities.

[0005] Therefore, there is an urgent need for a speculative reasoning method that can adaptively balance accuracy and latency in asynchronous multimodal input and dynamic network environments, enabling the terminal to have fast local judgment capabilities and, when necessary, to perform deep reasoning in collaboration with the edge, thereby significantly reducing system latency and transmission overhead while ensuring accuracy.

[0006] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0007] The purpose of this invention is to provide a speculative reasoning method for multimodal large language models in mobile edge collaboration, thereby overcoming, at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0008] This invention first provides a speculative reasoning method for multimodal large language models with mobile edge collaboration, comprising: S1. Construct a lightweight dual-clue probe model for the device side. The dual-clue probe model includes a modal feature extractor, a feature projector, and a dual-clue classifier. S2, use a modal feature extractor to perform task semantic distillation and feature alignment, use a feature projector to jointly optimize the two heads of the dual-clue classifier, distill the knowledge of the edge multimodal large language model MLLM to the dual-clue probe model, perform device-side minimum modal delay decision, and output local inference results or unload asynchronous modal subsets to the edge. S3, after receiving the asynchronous modal subset offloaded by the device at the edge, the asynchronous modal is filtered and dynamically inferred through a hierarchical gating network to obtain the speculative inference result of the edge MLLM; S4 calculates the incremental perplexity and perplexity threshold of the speculative inference result, performs a rollback decision at each step of MLLM token generation, and obtains the corrected final result of edge-end MLLM inference.

[0009] In this invention, in S1, the modal feature extractor is constructed for different modalities supported by the mobile device and the device hardware resources, and is used for extracting modal semantic features on the device side; The feature projector uses a small multilayer perceptron as the feature projector to map the heterogeneous modal features output by the modal feature extractor to the shared embedding space, thereby eliminating feature misalignment caused by modal heterogeneity. The dual-clue classifier includes a parallel Softmax classification head and a semantic contrast classification head, which are used to extract probabilistic cues and semantic cues respectively and achieve fusion.

[0010] In this invention, the Softmax classification head receives fused features from the shared space, inputs them into the probability classification head, calculates the predicted probability of each category using the Softmax function, and calculates the probability marginal score; the semantic contrast classification head inputs fused features and calculates the cosine similarity between the semantic feature embedding and the prototype of each category, and calculates the semantic marginal score.

[0011] In this invention, S2, the process of using a modal feature extractor for task semantic distillation and feature alignment includes: A token filtering projector is constructed using 1D convolution and adaptive pooling to extract task-discriminative embedding features from the edge-end MLLM deep network. Using mean squared error as the loss function, the device-side feature extractor is trained to align with the MLLM embedding output by the token filter projector.

[0012] In this invention, S2 involves jointly optimizing the two heads of the dual-clue classifier using a feature projector, distilling the knowledge from the edge-end multimodal large language model (MLLM) into the dual-clue probe model, performing device-end minimum modality latency decision-making, and outputting local inference results or unloading asynchronous modality subsets to the edge. This process includes: The Softmax classification head is trained using cross-entropy loss; Temperature scaling contrastive loss was used to train the semantic contrastive classification head. Deploy a lightweight dual-thread probe model on the device to mobile devices to perform minimum modal latency decisions, output local inference results, or offload asynchronous modal subsets to the edge.

[0013] In this invention, step S3 includes the following steps: S31, Construct a hierarchical gating network, wherein the hierarchical gating network includes a modality-specific gating module and a modality fusion gating module; S32 uses the edge-end MLLM output as the supervision signal to train a hierarchical gating network; S33 utilizes a hierarchical gating network to filter asynchronous modal subsets and perform dynamic inference, obtaining speculative inference results for edge-end MLLM.

[0014] The present invention further provides a mobile edge-coordinated multimodal large language model speculative reasoning system, comprising: A dual-clue probe model construction unit is used to construct a lightweight dual-clue probe model on the device side. The dual-clue probe model includes a modal feature extractor, a feature projector, and a dual-clue classifier. The distillation unit is used to perform task semantic distillation and feature alignment using a modal feature extractor, and to jointly optimize the two heads of the dual-clue classifier using a feature projector. It distills the knowledge of the edge-end multimodal large language model MLLM to the dual-clue probe model and outputs local inference results or offloads them to the asynchronous modal subset at the edge. The preliminary inference execution unit is used to filter and dynamically infer asynchronous modes by using a subset of asynchronous modes offloaded from the edge receiving device, and obtain the speculative inference result of the edge MLLM through a hierarchical gating network. The final inference execution unit is used to incrementally calculate the perplexity of the speculative inference result. At each step of MLLM token generation, a rollback decision is performed based on the pre-calculated perplexity threshold to obtain the corrected edge-end MLLM inference final result.

[0015] The technical solution provided by this invention may include the following beneficial effects: This invention presents a mobile edge-coordinated multimodal large language model speculative inference method. It constructs a lightweight dual-clue probe model on the mobile client, extracts task semantic features from the deep layers of the edge MLLM, achieves joint estimation of inference confidence and modal sufficiency, performs minimum modal latency decision-making on the device side, and then employs a hierarchical gating mechanism at the edge. Simultaneously, it introduces a perplexity-guided rollback mechanism to correct early inference results with insufficient confidence. Ultimately, it achieves early identification and efficient inference of the "minimum sufficient subset" in asynchronous modality. This effectively reduces system latency, improves inference accuracy, significantly reduces modal utilization, increases the proportion of local inference, and reduces computational costs. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0017] Figure 1 A flowchart illustrating a multimodal large language model speculative reasoning method for mobile edge collaboration in an exemplary embodiment of this disclosure is shown. Figure 2 This is a conceptual diagram of the algorithm in an example of the present invention; Figure 3 This is a speculative reasoning graph of a multimodal large language model for mobile edge collaboration constructed under two mobile devices in the mobile edge collaboration system of this invention. Figure 4 shows the comparison results of multiple baselines under three tasks, three models, and different network conditions; Figure 5 This is a graph showing the comparison of transmission costs and modal utilization rates of multiple baselines under the same task, model, and network conditions in one embodiment. Figure 6 This is a graph showing the comparison of transmission costs and modal utilization of multiple baselines under the same task, model, and network conditions in another embodiment. Detailed Implementation

[0018] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0019] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0020] This example implementation first provides a speculative reasoning method for multimodal large language models in mobile edge collaboration. Please refer to [reference needed]. Figure 1 This method may include: S1-S4, as follows: S1. Construct a lightweight dual-clue probe model for the device side. The dual-clue probe model includes a modal feature extractor, a feature projector, and a dual-clue classifier. S2, using a modal feature extractor to perform task semantic distillation and feature alignment, using a feature projector to jointly optimize the two heads of the dual-clue classifier, distilling the knowledge of the edge-end multimodal large language model MLLM to the device-end dual-clue probe model, and outputting local inference results or unloading asynchronous modal subsets to the edge end; S3, after receiving the asynchronous modal subset offloaded by the device at the edge, the asynchronous modal is filtered and dynamically inferred through a hierarchical gating network to obtain the speculative inference result of the edge MLLM; S4 calculates the incremental perplexity and perplexity threshold of the speculative inference result, performs a rollback decision at each step of MLLM token generation, and obtains the corrected final result of edge-end MLLM inference.

[0021] In this embodiment, a lightweight dual-clue probe model is built on the mobile client to extract task semantic features from the deep layer of edge MLLM, thereby achieving joint estimation of inference confidence and modality sufficiency. Then, a hierarchical gating mechanism is adopted at the edge, and a perplexity-guided rollback mechanism is introduced to correct early inference results with insufficient confidence. Finally, early identification and efficient inference of the "minimum sufficient subset" under asynchronous modality are achieved. This effectively reduces system latency, improves inference accuracy, significantly reduces modality utilization, increases the proportion of local inference, and reduces computational costs.

[0022] The specific process of each step in the above embodiments will be described below.

[0023] S1, Construct a lightweight dual-clue probe model on the device side.

[0024] A lightweight dual-clue probe model is constructed on the device side to realize the extraction of multimodal features and the estimation of inference confidence. The dual-clue probe model includes a modal feature extractor, a feature projector and a dual-clue classifier.

[0025] Modal Feature Extractor: A lightweight feature extraction network is customized for different modalities supported by mobile devices and the hardware resources of the devices, so as to efficiently extract modality-specific semantic features under the resource constraints of the device.

[0026] Feature Projector: A small multilayer perceptron is used as the feature projector to map the heterogeneous modal features output by the modality-specific feature extractor to the shared embedding space, thereby eliminating feature misalignment caused by modality heterogeneity and providing a foundation for cross-modal feature fusion.

[0027] Dual-clue classifier: The dual-clue classifier includes a parallel Softmax classification head and a semantic contrast classification head, used to extract probabilistic cues and semantic cues respectively and achieve fusion. (1) Softmax Classification Head: Receives fused features from the shared embedding space, inputs a probability classification head, and calculates the predicted probability of each class using the Softmax function. Defines the predicted probability of the top-1 class as... The predicted probability for the top-2 category is Calculate the marginal fraction of probability:

[0028] (2) Semantic contrast classification head: Receives fused features from the shared embedding space, inputs the semantic classification head and calculates the semantic feature embedding. With prototypes of each category Cosine similarity:

[0029] First, define the top-1 similarity as... The top-2 similarity is Calculate the semantic marginal score:

[0030] Then, for each category c, maintain a sliding window of recent inference results. The probability threshold corresponding to this category is determined by a binary search algorithm. With semantic threshold To adapt to the differences in boundary uncertainty between different categories in the probability space and semantic space; the threshold is initialized during the training phase and periodically updated during the inference phase based on changes in the distribution of data on the device.

[0031] In S2, the minimum modal delay decision is made at the device side.

[0032] To achieve better device-side minimum modal latency decision-making, a client-specific distillation mechanism is constructed to distill the knowledge from the edge-side MLLM into the device-side dual-threaded probe model, which is executed in two stages: Phase 1, Semantic Distillation and Feature Alignment: Constructing a Token Filtering Projector using 1D Convolution and Adaptive Pooling Extract task-discriminative embedding features from the edge-side MLLM deep network; train the device-side feature extractor using mean squared error (MSE) as the loss function. , so that it is with The output MLLM embedding alignment, the loss function is defined as:

[0033] Phase Two, Dual-Head Joint Optimization: Using cross-entropy loss ,in These are pseudo tags output by MLLM. Train a Softmax head to output probabilities for the Softmax head; Comparison loss using temperature scaling:

[0034] Where N is the batch size. For temperature coefficient, Use the feature projector output to train a semantic contrastive classification head; define the total loss function as follows: ,in We use feature alignment loss weights to achieve collaborative optimization between the two heads.

[0035] Deploy a mobile-customized probe model to mobile devices for minimum modal latency decision-making. After acquiring multimodal input, the device performs local inference and computes the results using a dual-threaded probe model. and The decision is made by combining a class-adaptive threshold. The output result is determined if any of the following conditions are met:

[0036]

[0037] in, As a scaling factor to balance probability and semantic cues.

[0038] If none of the above conditions are met, the current modal subset is selectively unloaded to the edge to obtain an asynchronous modal subset, which then waits for further reasoning at the edge.

[0039] In S3, the edge-end layered gating uses a "select-wait" speculative reasoning mechanism.

[0040] After receiving the asynchronous modal subset offloaded from the device at the edge, the device uses a hierarchical gating network to achieve "selection-wait" modal fusion and speculative reasoning. Specifically: First, a hierarchical gating network is constructed, which includes modality-specific gating modules and modality fusion gating modules.

[0041] Modality-specific gating module: For each arriving mode The task contribution token is extracted using average pooling and a multilayer perceptron network. The cosine similarity between this token and the pre-trained task token is then calculated as the contribution score. And configure two learnable thresholds within it. and ,in Used to trigger the threshold for single-modal inference, while As a threshold for preserving fusion; if Then, the edge-end MLLM is triggered to infer independently based on that modality; if Then add the modality to the candidate modality subset. ,like If so, then discard that mode.

[0042] Modal fusion gating module: Employs a gated recurrent unit (GRU) to achieve incremental fusion of candidate modes. Modal classification in Sort in descending order and initialize the GRU hidden state. For the sorted first Each modality, input to GRU, updates the hidden state. ,calculate The cosine similarity with the task token is used as the fusion score. Configure learnable threshold As a threshold for multimodal fusion inference, if Then, edge-side MLLM based on the current modality subset is triggered. Infer; otherwise, wait for the subsequent modalities to arrive and repeat the above fusion process.

[0043] During training, the output of the edge-end MLLM is used as the supervision signal to train the hierarchical gating network. A composite loss function is defined. ,in The triplet loss is used to align the similarity between valid modalities / fusion features and task tokens, and to separate the similarity between invalid modalities / fusion features and task tokens. For modality-specific gating decisions, binary classification cross-entropy loss. For binary classification cross-entropy loss in modality fusion gated decision-making, The loss weights are set for triples; the threshold is learned on the validation set. , , This ensures that the preset accuracy target is met.

[0044] During inference, modality-specific trigger points are set for different modal granularities (such as video frames and IMU timing segments), and hierarchical gating decisions are performed at each trigger point: if a modality-specific gating triggers single-modal inference, or a modality fusion gating triggers fusion inference, then the edge-side MLLM is called to perform speculative inference.

[0045] S4 executes the rollback mechanism for edge-side MLLM confusion guidance.

[0046] The purpose of S4 is to address potential errors in speculative reasoning in S3 by using MLLM perplexity as a truth-free feedback signal to enable real-time rollback and correction of inference results.

[0047] First, incremental perplexity calculation is performed, and the edge-side MLLM generates a token sequence. At the same time, maintain the cumulative log-likelihood. ,in For the current modal subset participating in reasoning, For the first The token sequence before the step. Incremental calculation of perplexity. When the first generation is generated When updating a token, and update synchronously. .

[0048] Secondly, modality-specific perplexity thresholds are pre-calculated for each modality. On the validation set, it is counted as a "key modality" (contribution score). The maximum reliable perplexity at the highest mode is used as the perplexity threshold for that mode. It is stored in the edge lookup table.

[0049] Finally, a rollback decision is executed. When MLLM generates each new token, the following rollback judgment is performed: Identify the current modality subset Key modes in Retrieve from the lookup table ; like Then determine the current modality subset. Insufficient to support high-accuracy inference, triggering a rollback: Pause the current inference and wait for subsequent modalities to be added. And return to S3 to re-execute the hierarchical gating decision; like If MLLM completes the preset number of token generation steps, the inference will terminate and the final result will be output.

[0050] This embodiment also provides a mobile edge collaborative multimodal large language model speculative reasoning system, including: A dual-clue probe model construction unit is used to construct a lightweight dual-clue probe model on the device side. The dual-clue probe model includes a modal feature extractor, a feature projector, and a dual-clue classifier. The distillation unit is used to perform task semantic distillation and feature alignment using a modal feature extractor, to jointly optimize the two heads of the dual-clue classifier using a feature projector, to distill the knowledge of the edge-end multimodal large language model MLLM to the dual-clue probe model, to perform device-end minimum modal delay decision, and to output local inference results or unload asynchronous modal subsets to the edge. The preliminary inference execution unit is used to filter and dynamically infer asynchronous modes by using a subset of asynchronous modes offloaded from the edge receiving device, and obtain the speculative inference result of the edge MLLM through a hierarchical gating network. The final inference execution unit is used to incrementally calculate the perplexity of the speculative inference result. At each step of MLLM token generation, a rollback decision is performed based on the pre-calculated perplexity threshold to obtain the corrected edge-end MLLM inference final result.

[0051] Regarding the system in the above embodiments, the specific manner in which each module or unit performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0052] The beneficial effects of the mobile edge collaborative multimodal large language model speculative reasoning method of this application will be illustrated below through specific embodiments.

[0053] This implementation uses Python 3.9, PyTorch 1.16.0, and Hugging Face Transformers, and is performed on both an edge server and mobile devices. The edge server is configured with an RTX 3090 GPU, an Intel Xeon Gold 6133 @2.50GHz processor, and 256GB of RAM. Different bandwidth conditions are configured to simulate distributed and asynchronous multimodal arrival. Four decoder-only MLLMs are used: Qwen2.5-Omni-3B / 8GB (…). ), Qwen2.5-Omni-7B / 18GB ( ), Qwen2-VL-2B / 4GB ( Each task is fine-tuned using LoRA; unsupported modalities (such as IMU) are converted into visualizations (e.g., time series plots) and then input through a visual encoder to achieve unified word segmentation and encoding across modalities.

[0054] The system was evaluated on three types of distributed multimodal perception applications: (1) scene recognition ( (2) Human activity recognition: Identify events on the AVE dataset, which contains over 40,000 one-second audio and video clips. ): Analyze human behavior on the MMAct dataset, including accelerometer, IMU and video data. (3) In-vehicle behavior recognition ): Perform in-vehicle behavior classification on the Drive&Act dataset, using RGB and IR video data.

[0055] To verify the real-time inference performance of this invention under distributed asynchronous multimodal input conditions, eight representative MLLM inference strategies were selected as comparative baselines. These baselines cover three heterogeneous paradigms: the first is a conservative strategy that waits for all modalities before inference; the second is an aggressive strategy that uses the fastest modality for immediate inference to pursue low latency; and the third is a strategy that selectively uses some modalities or edge-end collaborative switching to reduce computational and transmission overhead.

[0056] The baseline configurations are as follows: (1) NA (No-adaptation) waits for all distributed modalities to arrive before inference, serving as a baseline control. (2) Min (Minimal Modality Inference) triggers inference immediately using only the first arriving fast modal, such as IMU, to reduce latency but may sacrifice accuracy. (3) STA (Static Selection) uses a fixed combination of modalities for inference to reduce inference costs. (4) AdaMML uses a policy network with Gumbel-Softmax to adaptively select a subset of modalities. (5) DynMM uses a Transformer-based policy network for dynamic modality selection. (6) AppealNet switches between edge-side small models and edge-side MLLMs, using a dual-head structure for classification and confidence estimation respectively. (7) EdgeFM distills the semantic contrastive classifier and decides whether to switch to edge-side MLLMs based on semantic marginal scores. (8) SS (Softmax-based Switching) determines whether to postpone inference requests and offload them to the edge side based on softmax marginal scores.

[0057] Performance comparison experiment results: In terms of the trade-off between system latency and accuracy, eight baselines were compared under three tasks, three models, and different network conditions, as shown in Figure 4. Compared to NA, the method of this invention reduces the average latency by 55.54% on scene recognition, human activity recognition, in-vehicle behavior recognition, and the three models Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and Qwen2-VL-2B. Compared to the Min method, the average accuracy is improved by 24.67%. For example, in the human activity recognition task with 10Mbps bandwidth, the latency of the method of this invention is reduced by 70.38%, 68.43%, 69.72%, 68.84%, 68.78%, 63.13%, and 57.13% respectively compared to NA, STA, AdaMML, DynMM, AppealNet, EdgeFM, and SS.

[0058] In terms of transmission cost, modal utilization, and computational load, such as Figure 5 , Figure 6As shown in Table 1, in the human activity recognition task, the method of this invention achieves the lowest transmission cost while maintaining competitive accuracy. Its transmission cost is reduced by 28%, 55%, 22%, 7%, 44%, 55%, and 55% compared to AppealNet, NA, EdgeFM, SS, STA, AdaMML, and DynMM, respectively. Secondly, the method of this invention significantly reduces modality usage and increases the proportion of local inference, reaching 47.05%; compared to AppealNet, EdgeFM, and SS, it improves local inference capability without waiting for slow modalities by 33.56%, 35.26%, and 14.58%, respectively. Compared to STA and DynMM, the usage of slow modalities (e.g., RGB) is reduced by 49.95%; compared to NA and AdaMML, the combined use of IR-RGB is reduced by 90.01% and 64.33%, respectively. The method of this invention reduces computational cost by 49.56%, 31.76%, and 37.29% compared to NA, DynMM, and SS, respectively.

[0059] Table 1 Computational Costs for Each Method

[0060] In summary, the specific process of the mobile edge collaborative multimodal large language model speculative inference method proposed in this invention is as follows: First, a lightweight dual-clue probe model is constructed on the mobile client. This model includes three core components: a modality-specific feature extractor, a feature projector, and a dual-clue classifier. At the same time, a class-adaptive threshold is introduced, and the task semantic features are extracted from the deep layer of the edge MLLM through client-specific distillation to achieve joint estimation of inference confidence and modality sufficiency. The device-side minimum modality latency decision is executed. If both clues meet the class threshold, the result is output locally; otherwise, the asynchronous modality subset and related inference results are unloaded to the edge. Then, at the edge, a "select first, wait later" inference strategy and a hierarchical gating mechanism are adopted: Modality-specific gating extracts the contribution token of a single modality through average pooling and multilayer perceptron, calculates its cosine similarity with the learned task token, and selects a subset of high-value candidate modalities; Modality fusion gating takes the gating recurrent unit as the core, incrementally fuses candidate modalities in descending order of contribution score, updates the hidden state in real time and calculates the fusion score to determine whether to trigger MLLM inference; At the same time, a perplexity-guided rollback mechanism is introduced to correct early inference results with insufficient confidence, and finally realizes early recognition and efficient inference of "minimum sufficient subset" in asynchronous modality.

[0061] The principle of this invention is inspired by the reflex arc mechanism of the human nervous system. The lightweight detection model on the terminal side corresponds to the "spinal reflex," making rapid local decisions upon sensory arrival, reducing latency. The large multimodal model on the edge side corresponds to "cortical reasoning," handling highly uncertain or complex situations and completing global confirmation. The system maintains real-time performance and reliability in asynchronous multimodal and dynamic networks through a closed loop of "rapid response + deep confirmation." Furthermore, drawing on selective attention and hierarchical decision-making, it employs early modality screening and hierarchical gating, adaptively balancing "waiting for more modalities" and "immediate reasoning"; combined with perplexity rollback, it achieves self-monitoring and error correction. This forms a neuro-heuristic mechanism of "hierarchical processing, rapid feedback, and autonomous correction," significantly improving the optimal trade-off between accuracy and latency in distributed multimodal reasoning.

[0062] It should be noted that although several modules of the system for executing actions are mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for embodiment. Components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0063] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

[0064] It should be noted that the installation of image acquisition and personal identification equipment in public places involved in this application is necessary for maintaining public safety, complies with relevant national regulations, and is accompanied by prominent warning signs. The collected personal images and identification information can only be used for the purpose of maintaining public safety and not for other purposes; or the images, personal identification data, etc. in this application are all legally and compliantly obtained or collected with the individual's separate consent.

Claims

1. A speculative reasoning method for multimodal large language models with mobile edge collaboration, characterized in that, include: S1. Construct a lightweight dual-clue probe model for the device side. The dual-clue probe model includes a modal feature extractor, a feature projector, and a dual-clue classifier. S2, using a modal feature extractor to perform task semantic distillation and feature alignment, using a feature projector to jointly optimize the two heads of the dual-clue classifier, distilling the knowledge of the edge-end multimodal large language model MLLM to the dual-clue probe model, which is used by the device to perform inference and decision-making, and outputting local inference results or unloading asynchronous modal subsets to the edge. S3, after receiving the asynchronous modal subset offloaded by the device at the edge, the asynchronous modal is filtered and dynamically inferred through a hierarchical gating network to obtain the speculative inference result of the edge MLLM; S4 calculates the incremental perplexity and perplexity threshold of the speculative inference result, performs a rollback decision at each step of MLLM token generation, and obtains the corrected final result of edge-end MLLM inference.

2. The multimodal large language model speculative reasoning method for mobile edge collaboration according to claim 1, characterized in that, In S1, the modal feature extractor is constructed for different modalities supported by the mobile device and the device hardware resources, and is used for extracting modal semantic features on the device side; The feature projector uses a small multilayer perceptron as the feature projector to map the heterogeneous modal features output by the modal feature extractor to the shared embedding space, thereby eliminating feature misalignment caused by modal heterogeneity. The dual-clue classifier includes a parallel Softmax classification head and a semantic contrast classification head, which are used to extract probabilistic cues and semantic cues respectively and achieve fusion.

3. The multimodal large language model speculative reasoning method for mobile edge collaboration according to claim 2, characterized in that, The Softmax classification head is used to receive fused features in the shared space, input the probability classification head and calculate the predicted probability of each category through the Softmax function, and calculate the probability margin score; the semantic contrast classification head inputs fused features and calculates the cosine similarity between the semantic feature embedding and the prototype of each category, and calculates the semantic margin score.

4. The multimodal large language model speculative reasoning method for mobile edge collaboration according to claim 2, characterized in that, In S2, the process of semantic distillation and feature alignment using a modality feature extractor includes: A token filtering projector is constructed using 1D convolution and adaptive pooling to extract task-discriminative embedding features from the edge-end MLLM deep network. Using mean squared error as the loss function, the device-side feature extractor is trained to align with the MLLM embedding output by the token filter projector.

5. The multimodal large language model speculative reasoning method for mobile edge collaboration according to claim 4, characterized in that, In S2, the dual heads of the dual-clue classifier are jointly optimized using a feature projector. The knowledge from the edge-end multimodal large language model (MLLM) is distilled into the dual-clue probe model for on-device inference and decision-making. The process of outputting local inference results or unloading asynchronous modal subsets to the edge includes: The Softmax classification head is trained using cross-entropy loss; Temperature scaling contrastive loss was used to train the semantic contrastive classification head. Deploy a lightweight dual-thread probe model on the device to mobile devices to perform minimum modal latency decisions and output local inference results or offload to an asynchronous modal subset at the edge.

6. The multimodal large language model speculative reasoning method for mobile edge collaboration according to claim 5, characterized in that, S3 includes the following steps: S31, Construct a hierarchical gating network, wherein the hierarchical gating network includes a modality-specific gating module and a modality fusion gating module; S32 uses the edge-end MLLM output as the supervision signal to train a hierarchical gating network; S33 utilizes a hierarchical gating network to filter asynchronous modal subsets and perform dynamic inference, obtaining speculative inference results for edge-end MLLM.

7. A mobile edge-coordinated multimodal large language model speculative reasoning system, characterized in that, include: A dual-clue probe model construction unit is used to construct a lightweight dual-clue probe model on the device side. The dual-clue probe model includes a modal feature extractor, a feature projector, and a dual-clue classifier. The distillation unit is used to perform task semantic distillation and feature alignment using a modal feature extractor, and to jointly optimize the two heads of the dual-clue classifier using a feature projector. It distills the knowledge of the multimodal large language model (MLLM) at the edge end to the dual-clue probe model at the device end, and directly outputs the local inference results or unloads the asynchronous modal subset to the edge end. The preliminary inference execution unit is used to filter and dynamically infer asynchronous modes by using a subset of asynchronous modes offloaded from the edge receiving device, and obtain the speculative inference result of the edge MLLM through a hierarchical gating network. The final inference execution unit is used to incrementally calculate the perplexity of the speculative inference result. At each step of MLLM token generation, a rollback decision is performed based on the pre-calculated perplexity threshold to obtain the corrected edge-end MLLM inference final result.

Images