Model collaborative reasoning system and method based on dynamic sketch verification
By introducing bidirectional attention constraints and a dynamic verification strategy driven by confidence difference entropy, the attention focusing mode of the edge model is optimized, which solves the problem of insufficient semantic understanding in collaborative reasoning between edge devices and the cloud, and improves the accuracy and system stability of multimodal association reasoning in scenarios such as vehicle networking.
Patent Information
- Application Number
- CN202511720489.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-07-09
- Filing Date
- 2025-11-21
- Publication Date
- 2026-02-13
AI Technical Summary
Existing collaborative reasoning methods suffer from insufficient semantic understanding capabilities, delays or misjudgments caused by static verification mechanisms, and neglect of attention differences during cross-model collaborative optimization when collaborating between edge devices and the cloud. In particular, they are difficult to achieve efficient and accurate multimodal association reasoning in resource-constrained scenarios such as connected vehicles.
By employing a bidirectional attention constraint mechanism, a confidence difference entropy-driven dynamic verification strategy, and a time-series correlation perturbation sample generation technology, the edge-cloud collaboration efficiency is optimized by dynamically adjusting the attention focus mode of the edge model, thereby achieving long-range semantic capture and time-series data logical consistency.
It significantly improves the long-range semantic capture capability of edge models, achieves synergistic optimization of edge-cloud collaboration efficiency and inference accuracy, and ensures the logical consistency of time-series data and the robustness of the system.
Smart Images
Figure CN121525874A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of distributed artificial intelligence and edge computing, in particular to a model collaborative reasoning system and method based on dynamic sketch verification. BACKGROUND
[0002] With the rapid development of edge computing and distributed artificial intelligence technology, edge device and cloud collaborative reasoning has become a key technology in resource-constrained scenarios such as vehicle networking and industrial Internet of Things. However, the existing collaborative reasoning method still has a significant contradiction between the semantic understanding ability of the edge model and the collaboration efficiency of the cloud. The self-attention mechanism of the lightweight language model (SLM) deployed by the edge device is limited by the computing resources, which makes it difficult to effectively capture long-range semantic dependencies, resulting in insufficient multi-modal correlation reasoning accuracy in complex scenarios. For example, in the vehicle networking environment, the vehicle-mounted model needs to analyze the dynamic correlation of sensor data, driving instructions and road environment, but the attention distribution of the existing SLM often focuses too much on local features, causing misjudgment of key causal relationships.
[0003] The existing edge-cloud collaboration framework mostly adopts a static verification mechanism, which cannot adapt to dynamically changing context environments. When the confidence distribution of the edge model and the cloud large language model (LLM) deviates, a fixed verification threshold can easily cause two problems: redundant verification in low difference scenarios increases the delay, and missed detection in high difference scenarios can cause misexecution of critical instructions. This problem is particularly prominent in vehicle-mounted control systems with strict real-time requirements, and existing solutions cannot achieve dynamic balance between reasoning efficiency and accuracy.
[0004] In addition, in the cross-model collaborative optimization process, traditional knowledge transfer methods often ignore the attention focusing difference between models. Aligning the output layer distribution alone makes it difficult for edge models to effectively learn the deep semantic association patterns of cloud models, especially in time series data processing, random perturbations can disrupt the causal logic of sensor data streams, further exacerbating the collaborative bias between models. For example, in video stream analysis scenarios, uncontrolled inter-frame perturbations can cause action recognition models to confuse the temporal causal relationship between "acceleration" and "brake", severely reducing system reliability.
[0005] To address the above challenges, the present application proposes a model collaborative reasoning system and method based on dynamic sketch verification, which effectively improves the long-range semantic capture ability of edge models by introducing a bidirectional attention constraint mechanism, a confidence difference entropy-driven dynamic verification strategy, and a time series correlation perturbation sample generation technology, achieving collaborative optimization of edge-cloud collaboration efficiency and reasoning accuracy while ensuring the logical consistency of time series data. SUMMARY
[0006] The application aims to provide a model collaborative reasoning system and method based on dynamic sketch verification to solve the problems in the background art.
[0007] To achieve the above-mentioned purpose, the application provides the following technical scheme: a model collaborative reasoning system based on dynamic sketch verification, comprising: a deployment bidirectional attention constraint generator, an integrated attention alignment loss calculation unit, an embedded confidence difference entropy analyzer, a dynamic threshold adjuster and a difficult word element recognizer, further comprising an input space noise injection unit, a feature space adversarial generator and a gradient projection matrix calculation unit.
[0008] Preferably, the gradient projection matrix calculation unit is generated by the following method:
[0009]
[0010] Among them, represents a pseudo-inverse operation, and are the mapping functions of the edge model and the cloud model respectively.
[0011] A model collaborative reasoning method based on dynamic sketch verification, comprising the following steps:
[0012] Step 1: Edge model attention calibration and candidate generation: the edge model SLM encodes the input sequence through a bidirectional attention constraint mechanism to generate an initial candidate word element sequence. In the generation process, the correlation weight of each word element position in the self-attention layer is calculated in real time to reflect the dependence strength between different word elements. An attention distribution alignment loss function is introduced to compare the attention weight distribution difference between the edge model and the cloud large model LLM under the same context, and to calculate the KL divergence of the output probability distribution of the two, so as to dynamically adjust the attention focusing mode of the edge model, so that it approximates the semantic understanding logic of the cloud LLM while maintaining the lightweight reasoning advantage, and improves the reliability of the candidate word element sequence;
[0013] Step 2: Cloud dynamic threshold verification and difficult sample processing: after the cloud receives the candidate word element sequence generated by the edge model, the prediction stability of each word element in the sequence is quantified by the confidence difference entropy. The higher the entropy value, the greater the uncertainty of the word element prediction. Based on the entropy value, the verification threshold is dynamically adjusted. When the entropy value exceeds the preset baseline, the threshold is increased to strictly filter reliable word elements. When the entropy value is lower than the baseline, the threshold is reduced to retain more effective information. For difficult word elements with low confidence determined by the cloud LLM, a joint fine-tuning strategy based on gradient alignment is triggered to enhance the adaptation ability of the edge model to complex scenes and reduce the cross-model reasoning deviation by constraining the gradient direction consistency of the edge model and the cloud LLM at the difficult word elements.
[0014] Step three, cross-space disturbance enhancement and gradient compatibility optimization: to improve the robustness of the collaborative system, noise or time sequence misplacement is actively injected in the input space to simulate input disturbance in real scenarios, and at the same time, adversarial disturbance is applied to the intermediate feature vectors of the edge model in the feature space to ensure that the disturbed samples meet the time correlation constraint. To solve the problem of cross-model gradient incompatibility, the loss gradient of the cloud LLM is mapped to the parameter space of the edge model through Jacobian matrix projection to realize the adaptive conversion of gradient scale and direction, so that the edge model can effectively absorb the optimization signal of the cloud model, and finally improve the stability and precision of the entire collaborative reasoning system in complex input scenarios.
[0015] Preferably, the attention distribution alignment loss function in the generated candidate word sequence satisfies:
[0016]
[0017] wherein, is the number of attention heads, is the weight matrix of the i-th attention head, is a balance coefficient, is the KL divergence.
[0018] Preferably, the dynamic threshold adjustment formula in the cloud verification candidate word sequence is:
[0019]
[0020] wherein, is the baseline threshold, is the adjustment coefficient, is the confidence difference entropy at the current moment, is the reference entropy value.
[0021] Preferably, the joint fine-tuning strategy includes: generating adversarial disturbance samples on the edge side and calculating the loss gradient ; calculating the gradient of the LLM in the cloud, and mapping it to the parameter space of the edge model through the projection matrix
[0022]
[0023] wherein, is a mask matrix used to filter high-order semantic components unique to the LLM.
[0024] Preferably, the time correlation constraint in the adversarial disturbance includes: for video or sensor sequence data, limiting the disturbance operation only to the features within the same time window, and the disturbance amplitude satisfies ; wherein, is the time window. The intensity of the disturbance. For time series correlation coefficients, This is the original feature vector.
[0025] Compared with the prior art, the beneficial effects of the present invention are:
[0026] This application significantly enhances the ability of edge models to capture long-range semantic associations by comparing the attention weight distribution and loss constraints of edge and cloud models, overcoming the bottleneck of insufficient context focus in traditional lightweight models.
[0027] Based on the negative feedback adjustment mechanism of confidence difference entropy, the strictness of cloud verification is adaptively adjusted, which effectively reduces redundant computing overhead while ensuring the accuracy of inference, and achieves a dynamic balance between efficiency and accuracy.
[0028] By constructing perturbation samples in the input and feature space, the robustness of the model is enhanced while strictly constraining the causal relationship of time series data, thus avoiding the problem of logical chain breakage caused by random perturbation in traditional methods.
[0029] By employing gradient projection and parameter space mapping techniques, we can achieve knowledge transfer compatibility between large cloud models and lightweight edge models, and support collaborative reasoning and dynamic system expansion of multimodal data in complex scenarios. Attached Figure Description
[0030] Figure 1 This is a system architecture diagram of the present invention;
[0031] Figure 2 This is a flowchart of the dynamic attention alignment and candidate word generation process of this invention;
[0032] Figure 3 This is a schematic diagram of the cloud-based dynamic verification and gradient projection of the present invention;
[0033] Figure 4 This is a flowchart of the construction process for the temporal correlation perturbation sample of the present invention. Detailed Implementation
[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0035] Example:
[0036] Please see Figures 1-4 The present invention provides a technical solution:
[0037] A model collaborative reasoning system based on dynamic sketch verification, comprising: a deployment bidirectional attention constraint generator, an integrated attention alignment loss calculation unit, a built-in confidence difference entropy analyzer, a dynamic threshold adjuster and a difficult word element recognizer, also containing an input space noise injection unit, a feature space adversarial disturbance generator and a gradient projection matrix calculation unit.
[0038] The system realizes efficient collaborative reasoning of edge and cloud through dynamic attention alignment, entropy value driven verification and gradient compatibility optimization, and specifically includes the following stages: homomorphic encryption and data transmission stage, for sensitive data involving user privacy, the data is encrypted by using homomorphic encryption technology.
[0039] The present application realizes efficient edge-cloud reasoning through a multi-stage collaborative mechanism, first deploying a lightweight model on the edge side, generating candidate word sequences based on bidirectional attention constraints, optimizing the semantic focus mode of the edge model using attention alignment loss functions based on the difference between the attention weights and output distribution of the cloud large model, and improving long-range dependency capture ability; the cloud dynamically adjusts the verification threshold by calculating the confidence difference entropy, strengthens the screening strictness when the entropy value exceeds the baseline, otherwise improves the cooperation efficiency, and triggers the gradient projection strategy for low confidence word elements, maps the semantic gradient of the cloud model to the edge parameter space, realizes cross-model knowledge compatibility optimization; at the same time, noise disturbance of time correlation constraint is injected in the input space, and adversarial disturbance is applied in the feature space, to ensure the causal logic integrity of the sensor data stream; finally, a collaborative reasoning system containing dynamic attention alignment, real-time entropy value verification and disturbance optimization module is constructed, which significantly improves the robustness and real-time performance in cross-model collaboration scenarios;
[0040] The gradient projection matrix calculation unit is generated in the following way:
[0041]
[0042] wherein, denotes the pseudo-inverse operation, and are the mapping functions of the edge model and the cloud model respectively.
[0043] A model collaborative reasoning method based on dynamic sketch verification, comprising the following steps:
[0044] Step one, edge model attention calibration and candidate generation: the edge model SLM encodes the input sequence through a bidirectional attention constraint mechanism to generate an initial candidate token sequence. In the generation process, the correlation weight of each token position in the self-attention layer is calculated in real time to reflect the dependency strength between different tokens. An attention distribution alignment loss function is introduced to compare the differences in attention weight distribution between the edge model and the cloud large model LLM under the same context, and to calculate the KL divergence of the output probability distribution of the two models. By jointly optimizing these two types of losses, the attention focusing mode of the edge model is dynamically adjusted to approximate the semantic understanding logic of the cloud LLM while maintaining the lightweight inference advantage, thereby improving the reliability of the candidate token sequence.
[0045] Step two, cloud dynamic threshold verification and difficult sample processing: after the cloud receives the candidate token sequence generated by the edge model, the prediction stability of each token in the sequence is quantified by the confidence difference entropy. The higher the entropy value, the greater the uncertainty of the token prediction. Based on the entropy value, the verification threshold is dynamically adjusted. When the entropy value exceeds the preset baseline, the threshold is increased to strictly filter reliable tokens. When the entropy value is lower than the baseline, the threshold is reduced to retain more effective information. For difficult tokens with low confidence determined by the cloud LLM, a joint fine-tuning strategy based on gradient alignment is triggered to enhance the adaptation ability of the edge model to complex scenarios and reduce the inference bias across models by constraining the consistency of the gradient direction of the edge model and the cloud LLM at the difficult tokens.
[0046] Step three, cross-space perturbation enhancement and gradient compatibility optimization: to improve the robustness of the collaborative system, noise or timing misplacement is actively injected into the input space to simulate input perturbations in real scenarios. At the same time, an adversarial perturbation is applied to the intermediate feature vector of the edge model in the feature space to ensure that the perturbed samples meet the temporal correlation constraint. To address the cross-model gradient incompatibility issue, the loss gradient of the cloud LLM is mapped to the parameter space of the edge model through Jacobian matrix projection to achieve adaptive conversion of the gradient scale and direction, enabling the edge model to effectively absorb the optimization signals from the cloud model, and ultimately improving the stability and accuracy of the entire collaborative inference system in complex input scenarios.
[0047] In step one, a lightweight language model (SLM) is deployed on the edge device, pre-trained weights are loaded, and a bidirectional attention constraint module is initialized. The multi-modal data (such as vehicle sensor sequences and voice instructions) is input, and the SLM dynamically calculates the correlation weight matrix of each token through the self-attention layer The input sequence is uploaded to the cloud simultaneously to obtain the attention weight matrix and the output probability distribution of the cloud large model (LLM). The alignment loss function is calculated.
[0048] The attention distribution alignment loss function in generating candidate lexical sequences satisfies:
[0049]
[0050] in, For the number of attention heads, For the first The weight matrix of each attention head. For balance coefficient, Let KL divergence be denoted as KL divergence.
[0051] In step two, the cloud calculates the confidence difference entropy for each candidate word:
[0052]
[0053] Adjust the verification threshold based on entropy changes. The formula for adjusting the dynamic threshold in the candidate word sequence verified in the cloud is:
[0054]
[0055] in, The baseline threshold, For adjustment coefficients, The confidence difference entropy at the current moment. As a reference entropy value, when At that time, the judgment For difficult words, joint fine-tuning is triggered;
[0056] Gradient alignment optimization:
[0057] The joint fine-tuning strategy includes: generating adversarial perturbation samples at the edge and calculating the loss gradient. ; Calculate the gradient of LLM on the cloud side Through the projection matrix Map it to the edge model parameter space
[0058]
[0059] Among them, , This is a mask matrix used to filter out high-order semantic components unique to LLM.
[0060] Imposing temporal correlation constraints in adversarial perturbations includes: for video or sensor sequence data, restricting the perturbation operation to act only on features within the same time window, and ensuring that the perturbation amplitude satisfies the following conditions: ;in, For a moment The intensity of the disturbance. For time series correlation coefficients, This is the original feature vector.
[0061] In step three, dynamic verification and joint fine-tuning are performed in the cloud:
[0062] 1) Input spatial perturbation: Injecting temporal misalignment noise into the original data, for example, randomly delaying some frames in the in-vehicle video stream (the delay range is limited to...). , (the interval between adjacent frames).
[0063] 2) Feature space perturbation: in the intermediate feature layers of SLM Apply counter-perturbation The constraints are satisfied:
[0064]
[0065] Ensure that the magnitude of the disturbance is proportional to the change in characteristics at adjacent time points, and avoid disrupting the temporal causal chain.
[0066] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or basic characteristics. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the invention is defined by the appended claims rather than the foregoing description. Therefore, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.
[0067] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A model collaborative reasoning system based on dynamic sketch verification, characterized in that, Comprise: Deploy a bidirectional attention constraint generator, integrate an attention alignment loss calculation unit, build a confidence difference entropy analyzer, a dynamic threshold adjuster and a difficult word unit recognizer, and also contain an input space noise injection unit, a feature space adversarial disturbance generator and a gradient projection matrix calculation unit. 2.The model collaborative reasoning system based on dynamic sketch verification according to claim 1, wherein: The gradient projection matrix calculation unit is generated by: wherein, denotes a pseudo-inverse operation, and are mapping functions for edge model and cloud model, respectively.
3. The model collaborative reasoning method based on dynamic sketch verification according to any one of claims 1-2, characterized in that: Comprise the following steps: Step one, edge model attention calibration and candidate generation: the edge model SLM encodes the input sequence through a bidirectional attention constraint mechanism to generate an initial candidate word sequence. In the generation process, the correlation weight of each word position in the self-attention layer is calculated in real time to reflect the dependence strength between different words. An attention distribution alignment loss function is introduced to compare the attention weight distribution difference between the edge model and the cloud large model LLM under the same context, and to calculate the KL divergence of the output probability distribution of the two, and through the joint optimization of the two types of losses, the attention focusing mode of the edge model is dynamically adjusted to approximate the semantic understanding logic of the cloud LLM while maintaining the lightweight inference advantage, and the reliability of the candidate word sequence is improved; Step two, cloud dynamic threshold verification and difficult sample processing: after the cloud receives the candidate word sequence generated by the edge model, the prediction stability of each word in the sequence is quantified through the confidence difference entropy. The higher the entropy value, the greater the uncertainty of the word prediction. Based on the entropy value, the verification threshold is dynamically adjusted. When the entropy value exceeds the preset baseline, the threshold is increased to strictly filter reliable words. When the entropy value is lower than the baseline, the threshold is reduced to retain more effective information. For difficult words with low confidence determined by the cloud LLM, a joint fine-tuning strategy based on gradient alignment is triggered to enhance the adaptation ability of the edge model to complex scenes and reduce the cross-model inference bias by constraining the gradient direction consistency of the edge model and the cloud LLM at the difficult word; Step three, cross-space disturbance enhancement and gradient compatibility optimization: to improve the robustness of the collaborative system, noise or time sequence misplacement is actively injected in the input space to simulate input disturbance in real scenarios. At the same time, adversarial disturbance is applied to the intermediate feature vectors of the edge model in the feature space to ensure that the disturbance samples meet the time sequence correlation constraint. To solve the cross-model gradient incompatibility problem, the loss gradient of the cloud LLM is mapped to the parameter space of the edge model through Jacobian matrix projection to realize the adaptive conversion of gradient scale and direction, so that the edge model can effectively absorb the optimization signal of the cloud model, and finally improve the stability and precision of the entire collaborative inference system in complex input scenarios.
4. The model collaborative reasoning system and method based on dynamic sketch verification according to claim 3, characterized in that: The attention distribution alignment loss function in the generated candidate word sequence satisfies: wherein, is the number of attention heads, is the weight matrix of the th attention head, is the balancing coefficient, is the KL divergence.
5. The model collaborative reasoning system and method based on dynamic sketch verification according to claim 3, characterized in that: The dynamic threshold adjustment formula for verifying the candidate word sequence in the cloud is: wherein, is a baseline threshold value, is an adjustment coefficient, is a confidence difference entropy at a current time, is a reference entropy value.
6. The model collaborative reasoning system and method based on dynamic sketch verification according to claim 3, characterized in that: The joint fine-tuning strategy includes: edge side generates adversarial perturbation samples and calculates loss gradient ; cloud computing LLM gradient , which is mapped to the edge model parameter space by the projection matrix wherein, is a mask matrix used to filter LLM unique high-order semantic components.
7. The model collaborative reasoning system and method based on dynamic sketch verification of claim 3, wherein: The timing correlation constraint in applying the adversarial perturbation comprises: for video or sensor sequence data, limiting the perturbation operation to only act on features within the same time window, and the perturbation amplitude satisfies ; wherein, is the perturbation intensity at time , is the timing correlation coefficient, is the original feature vector.