A dynamic self-adaptive cloud edge-end fusion method and system for size model co-evolution

Abstract

The application discloses a dynamic self-adaptive cloud edge-end fusion method and system of size model co-evolution, and relates to the technical field of computer application and artificial intelligence. The method comprises the following steps: S1, initializing and maintaining a global prototype set representing semantic category center features on a cloud server; S2, receiving local data features uploaded by multiple edge participants; S3, performing a global prototype guided alignment strategy; S4, performing a preference driven modal adaptive completion; S5, performing a trusted bidirectional knowledge distillation; S6, performing an inference task on the edge side, and dynamically scheduling difficult samples to the cloud for processing according to decoupled uncertainty detection results. The application breaks through the limitation of multi-modal data, successfully mines implicit semantic association and generates high-quality data through global prototype guided alignment (GPGA) and preference driven completion (PDMAC) in the zero-shot alignment and 30% modal missing scene without manual annotation, and significantly improves the robustness of the system.

Description

Technical Field

[0001] This invention relates to the fields of computer applications and artificial intelligence technology, specifically to a dynamic adaptive cloud-edge-device fusion method and system for the co-evolution of large and small models. Background Technology

[0002] In recent years, Transformer-based Large Language Models (LLM) and Multimodal Large Models (MM-LM) have made groundbreaking progress in natural language processing, computer vision, and cross-modal understanding. These models, with their massive parameter scale and pre-trained data, have demonstrated astonishing generalization capabilities and zero-shot inference performance. However, the enormous number of parameters (typically in the billions to trillions) and high computational and storage costs of large models make them difficult to deploy directly on edge devices (such as smartphones, IoT sensors, and autonomous driving terminals) with limited computing power, memory, and energy. Meanwhile, while Lightweight Small Models (SM) can meet real-time requirements, they often exhibit insufficient generalization capabilities and "catastrophic forgetting" when handling complex logical reasoning, long-tailed distributed data, and multimodal fusion tasks.

[0003] To address these contradictions, the "cloud-edge" collaborative architecture has emerged, aiming to combine the powerful inference capabilities of large cloud models with the low latency advantages of small edge models. Traditional cloud-edge collaboration primarily focuses on dynamic offloading during the inference phase, that is, allocating some computing tasks to the cloud for execution based on network conditions and device load. While this approach alleviates single-point resource bottlenecks to some extent, it still faces three core challenges that limit its application in dynamic, open environments:

[0004] Current mainstream architectures are mostly static "train once, deploy permanently" models. Once a large model is deployed in the cloud, it becomes difficult to integrate new domain knowledge generated at the edge at low cost. Meanwhile, small edge models lack a mechanism to continuously acquire the latest general capabilities from the cloud. This one-way, static knowledge flow makes the system unable to adapt to the dynamic shift of data distribution, and frequent full retraining or data uploads bring huge communication overhead and privacy risks.

[0005] The scarcity of multimodal alignment and the lack of modal completeness are critical issues in multimodal applications (such as smart city surveillance and telemedicine). Due to geographical differences, sensor malfunctions, or high annotation costs, edge-side data often suffers from severe modality alignment scarcity and modality completeness gaps. For example, a camera might only have video streams without audio, or text and image data might lack paired labels. Existing multimodal learning methods heavily rely on high-quality aligned data, making effective training difficult in such heterogeneous and incomplete data environments.

[0006] Unidirectional knowledge flow and domain adaptability bottlenecks: Existing collaborative methods mostly focus on unidirectional knowledge distillation from the cloud to the edge, neglecting the feedback enhancement effect of edge-specific domain data (such as defect detection data from a specific factory or medical record data from a specific hospital) on large cloud models. This results in large models performing well on general tasks, but performing poorly on fine-grained tasks in specific vertical domains, creating a dilemma of "insufficient specialization in the cloud and insufficient strength at the edge."

[0007] To address the aforementioned issues, inspired by cutting-edge works such as BoMM, CoEvo, and ACDC, this paper proposes a dynamic adaptive cloud-edge-device fusion method involving the co-evolution of large and small models. This method no longer views cloud-edge collaboration as a simple task offloading, but rather constructs a dynamic, bidirectional evolutionary ecosystem. In this system, the large model in the cloud acts as a "general knowledge base," continuously absorbing domain-specific knowledge from the edge to enhance its specialization, while the small model at the edge acts as an "agile executor," continuously drawing general reasoning capabilities from the cloud to improve its generalization. Summary of the Invention

[0008] This invention aims to address the problems of static rigidity, difficulty in multimodal alignment, poor handling of missing modalities, unidirectional knowledge flow, and high communication overhead in existing cloud-edge collaborative architectures, and provides a dynamic adaptive method and system for the co-evolution of large and small models.

[0009] To address the aforementioned technical problems, this invention provides the following technical solution: a dynamic adaptive cloud-edge-device fusion method based on the co-evolution of large and small models, comprising the following steps:

[0010] S1. Initialize and maintain a global prototype set representing the features of semantic category centers on the cloud server. ;

[0011] S2. Receive local data features uploaded by multiple edge participants, wherein the local data has modality missing and alignment scarce;

[0012] S3. Execute Global Prototype Guided Alignment Strategy (GPGA): Construct a virtual alignment dataset without explicit labels based on the similarity between edge data features and the cost vector of the global prototype.

[0013] S4. Execution Preference-Driven Modal Adaptive Completion (PDMAC): Uses cloud-based generative models to complete missing modalities and filters high-quality generated data based on the Direct Preference Optimization (DPO) mechanism;

[0014] S5. Perform trusted bidirectional knowledge distillation: The cloud transmits the Chain of Thought (CoT) knowledge with confidence scores to the edge to update the edge small model; the edge transmits the double-filtered prediction probability matrix to the cloud to update the adapter module of the cloud large model;

[0015] S6. Perform inference tasks on the edge and dynamically schedule difficult samples to the cloud for processing based on the decoupled uncertainty detection results.

[0016] As a further aspect of the present invention, the Global Prototype Guidance Alignment Strategy (GPGA) specifically includes:

[0017] The global prototype is initialized in the cloud using a small amount of publicly aligned data and a large cloud-based model encoding.

[0018] Feature vectors of misaligned samples are extracted from the edge side, and their cosine distance or Wasserstein distance to the global prototype is calculated to generate a cost vector. ;

[0019] The similarity of cost vectors is compared across participants. When the similarity exceeds a preset threshold τ, it is determined to be a potential alignment pair, as shown in the formula:

[0020] ,in, Let be the similarity vector between the i-th sample in edge device k and the global prototype; Let be the similarity vector between the j-th sample in edge device w and the global prototype;

[0021] As a further aspect of the present invention, the preference-driven modal adaptive completion (PDMAC) specifically includes:

[0022] Constructing a conditional generative model Data on missing modes is generated based on the available modes in the sample.

[0023] Generate preferred output for each missing sample and normal output ;

[0024] Constructing a preference loss function using a large cloud-based model as the reward model The generator parameters are optimized by maximizing the difference in reward between the preferred output and the normal output.

[0025] As a further aspect of the present invention, the edge-to-cloud feedback in the trusted bidirectional knowledge distillation specifically includes:

[0026] The edge side performs multiple Dropout sampling votes on high-value samples in historical interaction records to form a prediction probability matrix P;

[0027] Consistency filtering is used to retain Top-k consistent responses, and confidence filtering is used to retain responses that meet the criteria. > 1 and > Data 2,

[0028] Let be the class probability of the j-th sample in the i-th sampling. The probability of the most likely class in a single sampling. 1 and 2 represents the preset filter threshold;

[0029] The cloud uses the filtered probability matrix to update the Adapter module of the large model through backdistillation, instead of all parameters.

[0030] As a further aspect of the present invention, an adaptive uncertainty detection step is also included:

[0031] For semantic segmentation tasks, uncertainty is decoupled into sample-level uncertainty and pixel-level uncertainty;

[0032] Only pixel regions with high uncertainty or difficult samples are uploaded to the cloud for enhancement processing, while the rest are directly inferred by the edge small model.

[0033] A large-scale model co-evolutionary system, comprising:

[0034] Device-Aware Adaptive Segmenter (DAS): Deployed at the edge to sense the computing power status of devices and dynamically segment inference tasks;

[0035] Dynamic Cloud Enhancement Module (DCAM): Deployed in the cloud, it includes a global prototype manager, a condition generator, and a cloud-based large model for performing modal completion and large model inference.

[0036] Collaborative Fusion Engine (CFE): Deployed on both the cloud and edge sides, it is used to perform bidirectional knowledge distillation, uncertainty detection, and cloud-edge communication scheduling.

[0037] Compared with the prior art, this invention has the following beneficial effects:

[0038] This invention overcomes the limitations of multimodal data by successfully mining implicit semantic associations and generating high-quality data in scenarios with zero-sample alignment and 30% modality loss, using Global Prototype Guided Alignment (GPGA) and Preference-Driven Completion (PDMAC), significantly improving the system's robustness. Secondly, a bidirectional mutually beneficial evolutionary mechanism is constructed: the cloud enhances edge generalization capabilities through CoT distillation (MMLU improvement of 1.5%), while the edge enhances domain expertise in the cloud through reliable probability matrix feedback (medical QA improvement of 2.3%). Furthermore, this invention reduces communication overhead by 30%-45% and optimizes end-to-end latency to 44.20ms through uncertainty decoupling scheduling, uploading only 10% of the most challenging pixel regions, achieving the optimal balance between accuracy and efficiency. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0040] Figure 1 This is a flowchart illustrating a dynamic adaptive cloud-edge-device fusion method based on the co-evolution of large and small models. Detailed Implementation

[0041] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] This invention provides a dynamic adaptive cloud-edge-device fusion method for co-evolution of large and small models. The following describes the technical implementation details in three typical scenarios: multimodal classification (MSCOCO), semantic segmentation (ADE20K), and language reasoning (medical QA).

[0043] Example 1: Multimodal classification task (MSCOCO dataset)

[0044] Scenario setting: Simulate edge data with 30% modality missing + 0% aligned samples (image-text pairs without manual annotation).

[0045] Step 1: Global Prototype Bootstrap Alignment (GPGA)

[0046] 1. Prototype Initialization: Using 10,000 image-text pairs from ImageNet-1K (non-overlapping with COCO) in the cloud, images and text are encoded using fψ, and the average feature vector for each category (COCO has 80 categories) is calculated to initialize the global prototype set. ={o1,o2,...,o80}.

[0047] 2. Cost Vector Calculation: Edge devices extract local image features vi= (xi), calculate the cosine distance between vi and each prototype in O, and generate the cost vector. For example, the cosine distance between the features of an image of a "cat" and the prototype of a "cat" is 0.2, and the distance between the features of an image of a "dog" and the prototype of a "dog" is 0.8.

[0048] 3. Cross-edge matching: Cloud-based aggregation of data from various edge computing platforms. If two images If the similarity exceeds the threshold τ=0.85 (such as the "cat" image at edge A and the "cat" image at edge B), it is determined to be a potential alignment pair.

[0049] Step 2: Preference-Driven Modal Completion (PDMAC)

[0050] 1. Generate Candidates: For the image xi containing the missing text, generate a generator. Two candidate texts are generated:

[0051] : Based on GPGA matching, the "cat" category guides the generation of ("an orange cat lying on the sofa").

[0052] : Unguided free generation ("Animals on furniture").

[0053] 2. Preference Optimization: The large cloud-based model serves as the reward model R, for ( , Scoring (e.g., R(x, )=0.92, R(x, ()=0.65). By minimizing the loss function:

[0054] L_prefer=-E[logσ(0.1×(0.92-0.65))]

[0055] Training the generator is more inclined to generate A high-quality description of the type.

[0056] Step 3: Cloud-to-edge distillation (C2E)

[0057] The cloud will input the completed image and text data into f ψGenerates a thought chain (CoT) containing the reasoning process: "The image contains a furry animal, small in size, which matches the characteristics of 'cat' → predict the category as cat." Only when the confidence S... conf When the value is greater than 0.8, the loss function is:

[0058] L C2E =0.9×(L_KL+0.01L_MSE+0.001L_CE)

[0059] Distilled to the edge of the small model.

[0060]

[0061] Experimental results: In the scenario of zero alignment + 30% missing data, the accuracy of this invention reached 35.21%, which is 4.75% higher than the baseline BoMM (30.46%).

[0062] Example 2: Semantic Segmentation Task (ADE20K Dataset)

[0063] Scenario setting: Low-resolution image (512×512) input from edge device, requiring a balance between accuracy and latency.

[0064] Step 1: Uncertainty Decoupling

[0065] small edge model f θk Infer from the input image and output the probability distribution P for each pixel. seg ∈ℝ^(512×512×150):

[0066] Sample-level uncertainty: Calculate the entropy value of the entire image. If the entropy is >0.7 (such as an image containing a complex background), it is marked as a difficult sample.

[0067] Pixel-level uncertainty: For simple samples, calculate the pixel variance and retain pixels with variance > 0.5 (such as object boundaries).

[0068] Step 2: Dynamic Scheduling and Fusion

[0069] Only the high-uncertainty region (approximately 10% of the entire image) was cropped into a 128×128 sub-image and uploaded to the cloud. After fine-tuning the segmentation using ViT-L / 14 in the cloud, the edge segmentation results were stitched together with the local segmentation results.

[0070]

[0071] Experimental results: When only 10% of the pixels are uploaded, the mIoU reaches 38.15% and the latency is 44.20ms, which is 11.5% lower than that of ACDC (15% upload, 46.51ms).

[0072] Example 3: Language Reasoning and Domain Adaptation (Medical QA)

[0073] Scenario setting: Hospital edge nodes utilize local de-identified medical data to enhance the professionalism of cloud-based models.

[0074] Step 1: Edge-to-Cloud Feedback (E2C)

[0075] 1. Dropout Sampling: The edge model performs 10 Dropout samplings on difficult cases (such as "chest pain with dyspnea"), outputting the probability distribution each time. (For example, the probability of "myocardial infarction" is 0.92, and the probability of "pulmonary embolism" is 0.05).

[0076] 2. Dual filtration:

[0077] Consistency filtering: retain the top-3 results that appear most frequently in 10 samples (e.g., "heart attack" appears 9 times).

[0078] Confidence filtering: Only keep >0.9 and Data with a value >0.85 (e.g., the overall probability of "heart attack" is 0.92).

[0079] Step 2: Cloud Update

[0080] The filtered probability matrix is ​​used in the cloud to update the LLaMA-2-7B Adapter module via backdistillation (only 0.1% of the parameters are updated).

[0081] Experimental results: The cloud platform improved the performance of medical QA tasks by 2.3%, which was significantly higher than the baseline of public data fine-tuning (+0.5%).

[0082] To compare the impact of alignment strategies, removing the GPGA module resulted in a performance decrease of approximately 3.5% in zero-sample alignment scenarios, with an even greater decrease when modality loss was severe, demonstrating its core role in addressing data heterogeneity. The necessity of bidirectional distillation was highlighted: retaining only "cloud -> edge" unidirectional distillation offered no improvement in the domain adaptability of cloud models; retaining only "edge -> cloud" resulted in limited improvement in the generalization ability of edge models. Only bidirectional collaboration could achieve overall optimization, with performance decreases of 1.2% and 0.9%, respectively. Regarding the uncertainty detection mechanism, compared to random sampling and simple entropy detection, the proposed decoupling mechanism achieved a 1.5% higher mIoU under the same upload volume, indicating that distinguishing between sample-level and pixel-level uncertainty is crucial for dense prediction tasks. Finally, preference optimization and completion were discussed: removing DPO preference optimization and directly using generated data for training resulted in a 1.8% performance decrease, demonstrating that noisy data severely interferes with the learning of small models, making preference selection indispensable.

[0083] This study analyzed communication efficiency and convergence. CELSM reduced communication overhead by approximately 30%-45% per round (depending on task complexity) by uploading the probability matrix instead of the original data / gradients and dynamically filtering difficult samples. The convergence curves observed in the experiments were consistent with the theoretical analysis. CELSM's loss decreased rapidly and stabilized after approximately 5-8 iterations, while baseline methods typically required 10-15 iterations to reach a similar level and exhibited greater fluctuations. This verifies that bidirectional evolution accelerates model convergence.

[0084] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A dynamic adaptive cloud-edge-device fusion method for co-evolution of large and small models, characterized in that, Includes the following steps: S1. Initialize and maintain a global prototype set representing the features of semantic category centers on the cloud server. ; S2. Receive local data features uploaded by multiple edge participants, wherein the local data has modality missing and alignment scarce; S3. Execute Global Prototype Guided Alignment Strategy (GPGA): Construct a virtual alignment dataset without explicit labels based on the similarity between edge data features and the cost vector of the global prototype. S4. Execution Preference-Driven Modal Adaptive Completion (PDMAC): Uses cloud-based generative models to complete missing modalities and filters high-quality generated data based on the Direct Preference Optimization (DPO) mechanism; S5. Perform trusted bidirectional knowledge distillation: The cloud transmits the Chain of Thought (CoT) knowledge with confidence scores to the edge to update the edge small model; the edge transmits the double-filtered prediction probability matrix to the cloud to update the adapter module of the cloud large model; S6. Perform inference tasks on the edge and dynamically schedule difficult samples to the cloud for processing based on the decoupled uncertainty detection results.

2. The dynamic adaptive cloud-edge-device fusion method for co-evolution of large and small models according to claim 1, characterized in that, The Global Prototype Guidance Alignment Strategy (GPGA) specifically includes: The global prototype is initialized in the cloud using a small amount of publicly aligned data and a large cloud-based model encoding. Feature vectors of misaligned samples are extracted from the edge side, and their cosine distance or Wasserstein distance to the global prototype is calculated to generate a cost vector. ; The similarity of cost vectors is compared across participants. When the similarity exceeds a preset threshold τ, it is determined to be a potential alignment pair, as shown in the formula: ,in, Let be the similarity vector between the i-th sample in edge device k and the global prototype; Let be the similarity vector between the j-th sample in the edge device w and the global prototype.

3. The dynamic adaptive cloud-edge-device fusion method for co-evolution of large and small models according to claim 1, characterized in that, The Preference-Driven Modal Adaptive Completion (PDMAC) specifically includes: Constructing a conditional generative model Data on missing modes is generated based on the available modes in the sample. Generate preferred output for each missing sample and normal output ; Constructing a preference loss function using a large cloud-based model as the reward model The generator parameters are optimized by maximizing the difference in reward between the preferred output and the normal output.

4. The dynamic adaptive cloud-edge-device fusion method for co-evolution of large and small models according to claim 1, characterized in that, The edge-to-cloud feedback in the trusted bidirectional knowledge distillation specifically includes: The edge side performs multiple Dropout sampling votes on high-value samples in historical interaction records to form a prediction probability matrix P; Consistency filtering is used to retain Top-k consistent responses, and confidence filtering is used to retain responses that meet the criteria. > 1 and > The data for 2, among which, Let be the class probability of the j-th sample in the i-th sampling. The probability of the most likely class in a single sampling. 1 and 2 represents the preset filtering threshold; The cloud uses the filtered probability matrix to update the Adapter module of the large model through backdistillation, instead of all parameters.

5. The dynamic adaptive cloud-edge-device fusion method for co-evolution of large and small models according to claim 1, characterized in that, It also includes an adaptive uncertainty detection step: For semantic segmentation tasks, uncertainty is decoupled into sample-level uncertainty and pixel-level uncertainty; Only pixel regions with high uncertainty or difficult samples are uploaded to the cloud for enhancement processing, while the rest are directly inferred by the edge small model.

6. A large-scale model co-evolutionary system, characterized in that, For implementing the dynamic adaptive cloud-edge-device fusion method of co-evolution of size models as described in any one of claims 1 to 5, the system comprises: Device-Aware Adaptive Segmenter (DAS): Deployed at the edge to sense the computing power status of devices and dynamically segment inference tasks; Dynamic Cloud Enhancement Module (DCAM): Deployed in the cloud, it includes a global prototype manager, a condition generator, and a cloud-based large model for performing modal completion and large model inference. Collaborative Fusion Engine (CFE): Deployed on both the cloud and edge sides, it is used to perform bidirectional knowledge distillation, uncertainty detection, and cloud-edge communication scheduling.

Images