Adaptive resource optimization and model generalization methods in edge-cloud collaborative intelligent systems

Abstract

The present invention discloses an adaptive resource optimization and model generalization method in an end-edge-cloud collaborative intelligent system, comprising the following steps: the end device collects corresponding environmental data and equipment operating parameters, and trains a preset local model based on the collected data; constructs a digital twin model, and calculates the statistical distance between the real-time data stream monitored in real time and the preset data distribution model based on the real-time data stream monitored in real time by the digital twin model. If the statistical distance is greater than the preset value, the active sample selection mechanism is started to optimize and update the local model, and the optimized and updated local model is sent to the edge node; a bidirectional knowledge distillation mechanism is used to compress the local model sent by the end device and the edge node, and a lightweight model is obtained and globally optimized and integrated through a cloud server. The present invention can optimize resource allocation, improve the model generalization capability, and thus improve the data processing efficiency of the system, the fairness of resource allocation, and the accuracy of intelligent decision-making.

Description

Technical Field

[0001] The present invention relates to the field of end-edge-cloud collaborative technology, and in particular to an adaptive resource optimization and model generalization method in an end-edge-cloud collaborative intelligent system. Background Art

[0002] In the context of the rapid development of contemporary intelligent systems, the end-edge-cloud collaborative architecture has become an important way to cope with large-scale data processing and complex decision-making. In this architecture, the end device is responsible for the initial collection and preprocessing of data, the edge node provides fast decision support, and the cloud is responsible for performing high-level data analysis and model training. Despite this, the end-edge-cloud collaborative system still faces many challenges when dealing with non-independent and identically distributed data, unbalanced resource allocation, and insufficient model generalization capabilities, especially in application environments with limited resources and highly diverse data.

[0003] Although edge nodes have advantages in providing proximal computing and decision support, they have limitations in resource allocation and job scheduling. Traditional resource allocation strategies are usually inflexible and cannot adapt to the rapid changes in real-time data characteristics and network status, resulting in inefficient resource allocation and suboptimal job scheduling. In addition, when the cloud center performs data analysis and model training, existing technologies fail to fully utilize the collaborative computing resources of the edge cloud, which not only affects the training efficiency but also limits the generalization potential of the model. At the same time, when processing data from end devices and edge nodes, the cloud center also faces the challenge of adapting to differences in data distribution, which hinders effective knowledge transfer and model optimization. In addition, with the circulation and sharing of data, how to achieve efficient use of data and intelligent decision-making while ensuring data privacy is an urgent problem to be solved in technological development.

[0004] In view of the shortcomings of the existing technology, the present invention proposes a new method of adaptive resource optimization and model generalization in an end-edge-cloud collaborative intelligent system. The method is implemented through the following three key technologies: dynamic data perception and model adaptive adjustment, predictive job scheduling to deal with resource uncertainty, and knowledge distillation and model optimization of end-edge-cloud collaboration. The integrated application of these technologies aims to optimize resource allocation and enhance model generalization capabilities, thereby improving the system's data processing efficiency, the fairness of resource allocation, and the accuracy of intelligent decision-making. The method of the present invention is not only applicable to multiple intelligent fields such as smart cities, smart manufacturing, and smart transportation, but also has significant social value and broad application prospects. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes an adaptive resource optimization and model generalization method in an edge-cloud collaborative intelligent system. In the method, resource allocation can be optimized and model generalization capability can be improved, thereby improving the data processing efficiency of the system, the fairness of resource allocation, and the accuracy of intelligent decision-making. The method of the present invention is not only applicable to multiple intelligent fields such as smart cities, smart manufacturing, and smart transportation, but also has significant social value and broad application prospects.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] The method for adaptive resource optimization and model generalization in the end-edge-cloud collaborative intelligent system is applied to the end-edge-cloud collaborative intelligent system, wherein the end-edge-cloud collaborative intelligent system includes a cloud server and a plurality of edge nodes connected to the cloud server, each of the edge nodes is connected to a plurality of end devices in the area, and the method includes the following steps:

[0008] The terminal device collects local environmental data and equipment operating parameters, trains the preset local model based on the collected data, and obtains a trained local model; constructs a digital twin model, and uses the digital twin model to realize real-time digital mapping of the terminal-edge-cloud collaborative intelligent system. Based on the real-time data stream monitored in real time by the digital twin model, the statistical distance between the real-time data stream monitored in real time and the preset data distribution model is calculated. If the statistical distance is greater than the preset value, the active sample selection mechanism is started to select samples representing the current data distribution characteristics from the real-time data stream, and the local model is optimized and updated according to the selected samples and the causal invariance learning mechanism; the local model of the terminal device is used as the teacher model, and the key representations are extracted from the teacher model through the knowledge distillation technology to form a lightweight terminal-side student model;

[0009] The edge node aggregates and fine-tunes the incremental changes of the end-side student model parameters sent by the end device to form an edge-side student model, and the edge node performs resource status prediction and job scheduling;

[0010] The cloud server collects and aggregates the edge-side student models from each edge node and transmits the extracted global knowledge back to the edge node in the form of parameter updates; the edge node sends the parameter update data to the end device, updates the end-side student model, and further iterates the end-side student model to adapt to the local data characteristics.

[0011] Preferably, the calculation formula of the statistical distance is as follows:

[0012]

[0013] Among them, d stat represents the statistical distance, D real-time is the distribution of real-time data streams, Dpreset is the preset data distribution model, p i and q i are the probabilities of the i-th event in the two distributions,

[0014] The active sample selection mechanism is started to select samples representing the current data distribution characteristics from the real-time data stream. The mathematical expression of the sample selection is as follows:

[0015]

[0016] Among them, x select is the selected sample, g(x) is the fitness function of the sample, and w(x) is the weight function of the sample, which is inversely proportional to the probability of occurrence of the sample in the data stream.

[0017] Preferably, the local model is optimized and updated according to the selected samples and the causal invariance learning mechanism, specifically including the following steps:

[0018] The optimization objective of causal invariance learning can be expressed as:

[0019]

[0020] Among them, θ is the local model parameter, f θ (x) is the predicted output of the local model, y is the true label, is the loss function, λ is the regularization coefficient, is the mutual information, which is used to measure the mutual dependence between the feature representation Φ(x) extracted from the input data x and the causal variable Y,

[0021] The goal of optimization is to minimize the sensitivity to future changes in data distribution while maintaining the model's adaptability to the current data distribution. The process of updating the local model can be expressed as:

[0022]

[0023] Among them, θ new is the updated model parameter, θ old is the current model parameter, η is the learning rate, represents the gradient of the model parameters θ.

[0024] Preferably, the edge node performs resource status prediction and job scheduling, specifically including the following steps:

[0025] The edge node builds a prediction model and pre-collects job requests sent by the end device. The prediction model is used to predict the resource usage and job arrival status in the end-edge-cloud collaborative intelligent system in the future.

[0026] According to the output results of the prediction model and the set of job requests, Bayesian optimization and greedy construction algorithms are used to dynamically adjust the job priority and resource allocation.

[0027] Preferably, the method of dynamically adjusting the priority of jobs and the allocation of resources by using Bayesian optimization and greedy construction algorithms according to the output results of the prediction model and the set of job requests specifically includes the following steps:

[0028] Construct the strategy target, the formula is as follows:

[0029]

[0030]

[0031] Among them, s j is the scheduling time of job j, C j (s j ) is the completion time of job j, D j (s j ) is the delay cost of job j, α is the delay cost weight, S is the set of all possible scheduling times, R(s j ) is the time at which job j is scheduled j The amount of resources required;

[0032] Bayesian optimization is used to optimize the parameters in the job scheduling strategy and determine the optimal scheduling strategy. The formula is as follows:

[0033]

[0034] Among them, θ is a parameter, is the information gain of parameter θ, is the loss function of parameter θ, α and β are parameters for adjusting the information gain and loss function weights;

[0035] Based on the determined optimal scheduling strategy, the job priority and resource allocation are dynamically adjusted. The formula is as follows:

[0036] P t+1 =g(P t ,R t ,J t ) (9)

[0037] Among them, P t is the job priority assignment at time t, R t is the resource state at time t, J t is the set of job requests at time t, and g is the dynamic adjustment function.

[0038] Preferably, the local model of the terminal device is used as the teacher model, and the key representations are extracted from the teacher model through the knowledge distillation technology to form a lightweight terminal-side student model. The terminal device distillation loss function is as follows:

[0039]

[0040] in, is the end device distillation loss function, is the cross entropy loss, y is the true label, is the prediction of the student model, α KL is the weight of the distillation loss, is the KL divergence loss, P' t is the prediction distribution of the teacher model, P' s is the predicted distribution of the student model;

[0041] The loss function on the edge node is as follows:

[0042]

[0043] in, is the loss function on the edge nodes, β reg is the weight of the regularization term, is the regularization loss, used to prevent the model from overfitting;

[0044] The loss function of the cloud server is as follows:

[0045]

[0046] in, is the loss function of the cloud server, N is the number of edge nodes, is the weight of the i-th edge node model, is the loss function of the i-th edge node model, γ is the weight of the diversity loss, is the diversity loss, used to encourage diversity among different models;

[0047] The total loss function of the edge-cloud collaborative intelligent system is as follows:

[0048]

[0049] in, is the total loss function of the entire system, δ is the weight of the communication loss, It is the communication loss, which is used to measure the information loss during the transmission of the model between the edge and the cloud.

[0050] Preferably, the following steps are also included: the end-side student model and the edge-side student model use a multi-branch network structure and random enhancement input to improve training efficiency during the distillation training phase, as follows:

[0051] The multi-branch network structure can be expressed as:

[0052]

[0053] in, is the final prediction, f h (·) is the model of the h-th branch, is the randomly enhanced input data, e h is the enhancement parameter, M is the number of branches, and φ(·) is the aggregation function;

[0054] The end-side student model and the edge-side student model apply random transformation, random rotation, and random scaling operations to the input data during training.

[0055] Preferably, a time mean model is introduced to continuously update the exponential moving average of the end-side student model parameters during the training process through the time mean model, and the formula is as follows:

[0056] θ k+1 =α TMM ·θ k +(1-α TMM )·θ current (12)

[0057] Among them, θ k+1 is the student model parameter for the next time step, θ k is the parameter of the current time step, θ current is the updated value of the model parameters in the current training step, α TMM is a smoothing coefficient between 0 and 1, which determines the influence of historical parameter values ​​on new parameter values. TMM The closer it is to 1, the smoother the parameter update.

[0058] Based on the above technical solution, the beneficial effects of the present invention are:

[0059] 1) Improve model generalization performance: The present invention can dynamically adjust the model training strategy according to the real-time changing data distribution through adaptive data distribution perception and active model adaptation technology. Compared with the traditional end-edge-cloud collaborative learning method, the present invention significantly improves the generalization ability of the model through active sample selection and causal invariant learning in a non-independent and identically distributed data environment. Experimental results show that on the CIFAR-10 dataset, the method of the present invention improves the accuracy by about 2.1% compared with the prior art FedAvg.

[0060] 2) Optimizing resource scheduling efficiency: The predictive job scheduling method under resource uncertainty of the present invention adopts a prediction and optimization framework to effectively deal with the uncertainty of cloud resources. Through Bayesian optimization and greedy construction algorithms, efficient job scheduling is achieved, ensuring high reliability of the system.

[0061] 3) Reduce communication overhead: The knowledge distillation technology of the present invention, which integrates online integration and network collaboration, generates a lightweight model by applying knowledge distillation on terminal devices and edge nodes, and performs global optimization in the cloud, thereby reducing the number of model parameter transmissions.

[0062] 4) Protecting data privacy: The present invention performs data preprocessing and preliminary model training on the end device, which reduces the transmission of raw data and reduces the risk of data leakage. At the same time, the virtual mapping constructed by digital twin technology further ensures the security of data.

[0063] 5) Enhanced system adaptability: The method of the present invention monitors data distribution and system status in real time and quickly responds to environmental changes, effectively adapting to dynamic changes in data distribution and fluctuations in the communication environment. When faced with rapid changes in data distribution, the method of the present invention can quickly adapt to new data distributions and maintain or even improve model performance.

[0064] 6) Improving model training efficiency: The end-edge-cloud collaborative knowledge distillation mechanism of the present invention promotes knowledge transfer between the end-edge-cloud through a multi-branch network structure and random enhanced input, and can achieve collaborative evolution of the model.

[0065] 7) Reducing annotation costs: The present invention uses an active sample selection mechanism to more efficiently utilize data resources with limited annotated data, thereby reducing dependence on a large amount of annotated data.

[0066] 8) Data privacy protection: The present invention performs data preprocessing and preliminary model training on the terminal device, reducing the transmission of raw data, thereby better protecting data privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 The diagram is a schematic diagram of an adaptive resource optimization and model generalization method in an edge-cloud collaborative intelligent system in an embodiment. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0069] like Figure 1As shown, this embodiment provides an adaptive resource optimization and model generalization method in an end-edge-cloud collaborative intelligent system. The method specifically targets the data processing bottleneck, unbalanced resource allocation, and insufficient model generalization performance in the end-edge-cloud collaborative intelligent system. The end-edge-cloud collaborative intelligent system includes a cloud server and several edge nodes connected to the cloud server. Each of the edge nodes is connected to several end devices in the area. The method mainly includes the following three key technical modules:

[0070] 1. Adaptive data distribution perception and active model adaptation: Using advanced digital twin technology, this module realizes real-time digital mapping of the end-edge cloud environment and can dynamically monitor the fluctuations of data distribution. By applying the principle of invariant risk minimization and combining active sample selection with causal invariance learning, this system can effectively identify changes in data distribution and adjust the model accordingly to enhance its generalization performance. In specific operations, the system first monitors the data flow through the digital twin model and compares and analyzes the monitoring results with the preset data distribution model. Once a deviation is found, the system will activate the active sample selection mechanism to select samples that can reflect the characteristics of the current data distribution. Subsequently, the causal invariance learning module will use these samples to optimize the model to adapt to the new data distribution pattern. In this process, the system adopts the theory of structural causal model to extract feature representations that are independent of the data generation mechanism by distinguishing between intervention variables and semantic variables.

[0071] 2. Predictive job scheduling under resource uncertainty: In response to the uncertainty of cloud computing resources, this module develops a new predictive job scheduling strategy. This strategy uses a combination of prediction and optimization, integrating Bayesian optimization and greedy algorithms to efficiently schedule job requests. In the prediction stage, the system will use time series analysis to predict the availability of future resources. In the optimization stage, the system will determine the optimal job scheduling plan based on the prediction results and job characteristics through a greedy algorithm to minimize resource consumption and meet job requirements. At the same time, the system will also comprehensively consider the interdependence and priority between jobs to improve resource utilization and system response speed.

[0072] 3. Knowledge distillation with online integration and network collaboration: This module achieves lightweight model and performance improvement through knowledge distillation technology of end-edge-cloud collaboration. On the end devices and edge nodes, complex models are simplified into lightweight models and optimized through knowledge distillation. The cloud is responsible for the global optimization and fusion of the model. The system adopts a multi-branch network structure and randomized input technology to promote knowledge transfer between the end, edge and cloud, and realize the co-evolution of the model. Specifically, the models on the end devices and edge nodes absorb the key features of the cloud models through the distillation process, and the cloud models achieve global optimization by integrating the knowledge of these lightweight models. This method not only improves the computing efficiency of the end devices, but also enhances the generalization ability of the cloud models.

[0073] Through the collaborative work of these three technical modules, this technical solution can effectively optimize resource allocation in the edge-cloud collaborative system and improve the generalization performance of the model, thereby significantly improving the system's data processing efficiency, the rationality of resource allocation, and the accuracy of intelligent decision-making. The detailed implementation details are as follows.

[0074] 1. Adaptive data distribution awareness and active model adaptation

[0075] In the edge-cloud collaborative intelligent system, the dynamic changes in data distribution have a significant impact on model performance. In order to improve the generalization ability of the model, this paper proposes an adaptive data distribution perception and active model adaptation method. This method monitors the changes in data distribution in real time and dynamically adjusts the local model to adapt to the new data distribution.

[0076] 1.1 Construction and real-time monitoring of digital twin models

[0077] Real-time digital mapping of the edge-cloud environment is achieved by building a digital twin model. A digital twin model (DTM) is a virtual digital copy that can reflect the state and behavior of the physical world in real time. In the present invention, DTM is used to monitor changes in data streams, including statistical characteristics and distribution patterns of data. Specifically, DTM builds a dynamically updated data distribution model by collecting and analyzing data points in real time.

[0078] 1.2 Detection and Analysis of Data Distribution Changes

[0079] The detection of data distribution changes is performed by comparing the real-time monitored data stream with the preset data distribution model. The preset data distribution model is built based on historical data, which can represent the data distribution characteristics of the system under normal operating conditions. By calculating the statistical distance between the real-time data stream and the preset model, it is possible to detect whether the data distribution has changed significantly. The calculation formula for the statistical distance is as follows:

[0080]

[0081] Among them, d stat represents the statistical distance, d real-time is the distribution of real-time data streams, D preset is the preset data distribution model, p i and q i are the probabilities of the ith event in the two distributions respectively.

[0082] 1.3 Active Sample Selection Mechanism

[0083] Once a change in data distribution is detected, the system will activate the active sample selection mechanism. The purpose of this mechanism is to select samples that can represent the current data distribution characteristics from the real-time data stream, and these samples will be used for subsequent model optimization. The sample selection process is based on the unbiased estimation principle in statistics to ensure the representativeness of the selected samples. The mathematical expression of sample selection is as follows:

[0084]

[0085] Among them, x select is the selected sample, g(x) is the fitness function of the sample, and w(x) is the weight function of the sample, which is inversely proportional to the probability of occurrence of the sample in the data stream.

[0086] 1.4 Causal Invariance Learning

[0087] Causal invariance learning is the process of optimizing the model using selected samples. This process is based on causal inference theory and aims to find the invariant factors in the data generation process, namely causal variables. By learning these invariant causal variables, the local model can maintain consistent prediction behavior for inputs under different data distributions. The optimization goal of causal invariance learning can be expressed as:

[0088]

[0089] Among them, θ is the local model parameter, f θ (x) is the predicted output of the local model, y is the true label, is the loss function, λ is the regularization coefficient, is the mutual information, which is used to measure the mutual dependence between the feature representation Φ(x) extracted from the input data x and the causal variable Y. Mutual information measures the mutual dependence between two random variables and is used to quantify the degree of dependence of the model on the causal variable.

[0090] 1.5 Model Update and Adaptive Feedback

[0091] The edge-cloud collaborative intelligent system will update the model parameters based on the results of causal invariance learning. The optimization goal is to minimize the sensitivity to future changes in data distribution while maintaining the model's adaptability to the current data distribution. The process of updating the local model can be expressed as:

[0092]

[0093] Among them, θ new is the updated model parameter, θ old is the current model parameter, η is the learning rate, represents the gradient of the model parameters θ.

[0094] The adaptive data distribution perception and active model adaptation module of the present invention can effectively deal with the problem of data distribution changes in the edge-cloud collaborative intelligent system, and improve the generalization ability and adaptability of the model. The design of this module fully considers the real-time and dynamic nature of the system, and provides a solid technical foundation for realizing the efficient operation of the intelligent system.

[0095] 2. Predictive job scheduling under resource uncertainty

[0096] In the cloud computing environment, resource uncertainty brings great challenges to job scheduling. In order to effectively schedule jobs under such uncertainty, this paper proposes a predictive job scheduling method. This method combines prediction technology and optimization strategy to achieve efficient scheduling of jobs. The following are the technical details of the predictive job scheduling module under resource uncertainty:

[0097] 2.1 Construction of prediction model

[0098] The construction of a prediction model is the basis of predictive job scheduling. The model aims to predict the usage of resources and the arrival of jobs in the future. The prediction model can be constructed using time series analysis, machine learning or other statistical methods. In the present invention, a time series analysis method, such as an autoregressive moving average model or an autoregressive integrated moving average model, is used to predict the availability of resources and the arrival rate of jobs.

[0099]

[0100] in, is the predicted resource state at time t, R t-1 ,R t-2 ,…,R t-n is the resource status at the past n time points, and f is the prediction function.

[0101] 2.2 Pre-collection and presentation of job requests

[0102] Under resource uncertainty, pre-collection of job requests is a key step for effective scheduling. Job requests are collected and represented as J = {j1, j2, …, j m}, where each job j i Contains information such as the job's computing requirements, expected start time, and duration. This information is used to evaluate the job's demand for resources and the job's schedulability.

[0103] 2.3 Prediction-based job scheduling strategy

[0104] The job scheduling strategy is determined based on the output of the prediction model and the set of job requests. The goal of this strategy is to minimize the waste of resources and job delays while meeting job requirements. In the present invention, a method combining a greedy algorithm and Bayesian optimization is used to perform job scheduling.

[0105]

[0106] Among them, s j is the scheduling time of job j, C j (s j ) is the completion time of job j, D j (s j ) is the delay cost of job j, α is the delay cost weight, S is the set of all possible scheduling times, R(s j ) is the time at which job j is scheduled j The amount of resources required.

[0107] 2.4 Bayesian Optimization

[0108] Bayesian optimization is used to optimize the parameters in job scheduling strategies, such as job priority allocation and resource allocation ratio. By building a posterior probability model of parameters and using acquisition functions to guide the optimal selection of parameters, Bayesian optimization can find the optimal scheduling strategy under resource uncertainty conditions.

[0109]

[0110] Among them, θ is a parameter, is the information gain of parameter θ, is the loss function of parameter θ, and α and β are parameters that adjust the information gain and loss function weights.

[0111] 2.5 Dynamic Adjustment of Scheduling Strategies

[0112] The dynamic adjustment of scheduling strategies is to adapt to the real-time changes of resource status and job requests. By real-time monitoring of resource usage and job execution status, job priority and resource allocation are dynamically adjusted to cope with the impact of uncertainty.

[0113] P t+1 =g(P t ,R t ,J t ) (9)

[0114] Among them, P t is the job priority assignment at time t, R t is the resource state at time t, J t is the set of job requests at time t, and g is a dynamic adjustment function. Specifically, when the resource status R is found through the monitoring system t Changes occur, such as a sudden increase in CPU usage or when new job requests are added to the J t When the scheduling system triggers the dynamic adjustment mechanism, the system will evaluate the urgency and resource requirements of each job, and predict future resource trends and job arrivals. Based on this information, the dynamic adjustment function g will calculate an updated job priority assignment P t+1 , this allocation reflects the relative importance and execution possibility of the jobs under the current resource conditions, and thereby increases the job priority of new job requests. For example, suppose that on a cloud platform, there are multiple jobs waiting to be executed, and the resource status indicates that memory resources are sufficient but CPU resources are tight. In this case, the dynamic adjustment function g may decide to lower the priority of jobs that have a higher demand for CPU resources and increase the priority of jobs that rely more on memory resources. In this way, it can be ensured that CPU resources are not overloaded and memory resources are fully utilized.

[0115] The predictive job scheduling module under resource uncertainty can achieve efficient job scheduling under uncertain resource conditions and ensure high reliability of the system. The design of this module takes into account the dynamic changes of resources and the diversity of job requests in the cloud computing environment, and provides effective technical support for realizing resource optimization in intelligent systems.

[0116] 3. Knowledge distillation based on online integration and network collaboration

[0117] Knowledge distillation can be used to compress large and complex models into small and efficient models while maintaining or even improving the performance of the models. The knowledge distillation method proposed in this paper combines online integration and network collaboration to achieve lightweight and performance improvement of the model through knowledge distillation of end devices and edge nodes, and perform global optimization and fusion through the cloud. The following are the technical details of this module:

[0118] 3.1 Knowledge Distillation on Devices

[0119] On the end device, the key knowledge of the model is extracted through knowledge distillation technology. This process involves a teacher-student architecture, where the teacher model is a pre-trained complex model, that is, a local model updated on the end device, and the student model is a lightweight model. The student model is trained by imitating the behavior of the teacher model to gain the ability to understand and predict the data.

[0120]

[0121] in, is the end device distillation loss function, is the cross entropy loss, y is the true label, is the prediction of the student model, α KL is the weight of the distillation loss, is the KL divergence loss, P' t is the prediction distribution of the teacher model, P' s is the predicted distribution of the student model.

[0122] 3.2 Design of multi-branch network structure and random enhancement input

[0123] In order to enhance the adaptability of the model to different data distributions, the student models on the end and edge sides use multi-branch network structures and random augmented inputs to improve training efficiency during the distillation training phase. Specifically, first, each branch (divided into three branches: time, space, and depth in image tasks) focuses on extracting specific pattern information from the input data, enabling the model to understand the data more comprehensively and enhance its ability to process multiple features of the data. The output of the multi-branch network is integrated through an aggregation function to form the final prediction. The multi-branch network structure design can be expressed as:

[0124]

[0125] in, is the final prediction, f h (·) is the model of the h-th branch, is the randomly enhanced input data, e h is the enhancement parameter, M is the number of branches, and φ(·) is the aggregation function. Similarly, the student model applies random transformations, random rotations, and random scaling operations to the input data during training. This data enhancement method enables the student model to learn more robust feature representations and reduce dependence on specific data distributions, thereby improving the generalization performance of the student model in different environments and conditions.

[0126] 3.3 Introduction of the time mean model

[0127] In order to further improve the generalization ability of the model, the present invention introduces the Time Mean Model (TMM). TMM is implemented by continuously updating the exponential moving average of the end-side student model parameters during the distillation training process, which helps to smooth the changes in model parameters, reduce fluctuations during training, and enable the model to learn more stably. The update of TMM can be expressed as:

[0128] θ k+1 =α TMM ·θ k +(1-α TMM )·θ current (12)

[0129] Among them, θ k+1 is the student model parameter for the next time step, θ k is the parameter of the current time step, θ current is the updated value of the model parameters in the current training step, α TMM is a smoothing coefficient between 0 and 1, which determines the influence of historical parameter values ​​on new parameter values. TMM The closer it is to 1, the smoother the parameter update. Note that here θ k Compared with θ in formula (3) and θ in formula (4), new Both refer to the student model parameters of the end-edge-cloud collaborative training. The θ in formula (3) is used to indicate that in the adaptive data distribution perception stage, the end side finds the causal variables of the sample through causal invariance learning, so that the model can learn consistent prediction capabilities under different sample distributions. And the θ in formula (4) new It is used to indicate the active model adaptation process of updating parameters after the model senses changes in data distribution. This process is different from the training phase of the end-side model distillation, but the updated model parameters are all the end-side student model.

[0130] 3.4 Model Optimization on Edge Nodes

[0131] On edge nodes, the model is further optimized using the distilled knowledge uploaded by the end device. Edge nodes have stronger computing power than end devices and can perform more complex optimization tasks. By optimizing the model on edge nodes, the performance of the model for local tasks can be improved, while also providing a better starting point for global optimization in the cloud.

[0132]

[0133] in, is the loss function on the edge nodes, β reg is the weight of the regularization term, is the regularization loss, which is used to prevent the model from overfitting.

[0134] 3.5 Global Model Optimization in the Cloud

[0135] The optimization of cloud models is achieved through network collaboration and fusion. The cloud server collects models from various edge nodes and performs global optimization through ensemble learning. The cloud model not only considers the performance of a single model, but also the complementarity and collaboration between models.

[0136]

[0137] in, is the loss function of the cloud server, N is the number of edge nodes, is the weight of the i-th edge node model, is the loss function of the i-th edge node model, γ is the weight of the diversity loss, is the diversity loss, which is used to encourage diversity among different models.

[0138] 3.6 Knowledge Transfer in Device-Edge-Cloud Collaboration

[0139] The knowledge transfer of device-edge-cloud collaboration is achieved through a two-way distillation mechanism. The models on the end devices and edge nodes absorb the key features of the cloud models through the distillation process, while the cloud models achieve global optimization by integrating the knowledge of these lightweight models.

[0140]

[0141] in, is the total loss function of the entire system, δ is the weight of the communication loss, It is the communication loss, which is used to measure the information loss during the transmission of the model between the edge and the cloud.

[0142] Specifically, the bidirectional distillation mechanism is as follows:

[0143] (I) In the process of knowledge distillation from the edge to the cloud,

[0144] 1) The end device is responsible for collecting and preprocessing raw data, and extracting key representations from the teacher model preset on the end device through knowledge distillation technology to form a lightweight end-side student model. This step is the starting point of the knowledge transfer process, ensuring that useful data features can be captured even on resource-constrained end devices.

[0145] 2) The edge node acts as a bridge between the end device and the cloud. It receives incremental changes in the parameters of the student model from the end device. Based on its own computing power, it can choose to perform the incremental representation of the end-side model after aggregation and upload, form an edge-side student model, and perform more complex data processing tasks, including resource status prediction and job scheduling. The further optimization model in this part means that the edge node can aggregate and fine-tune the incremental changes of the uploaded end-side model to reduce the impact of the distribution offset of the data collected on a single end-side device, in order to form a more robust edge model (local model). What is finally uploaded to the cloud is the incremental representation of the edge student model.

[0146] 3) The cloud model is responsible for global integration and optimization, and forms a comprehensive, globally optimized model by collecting and aggregating model knowledge from various edge nodes. The entire training and uploading process is asynchronous.

[0147] (II) Correspondingly, in the process of transferring knowledge from the cloud to the end side,

[0148] 1) The cloud model is a global aggregation model that aggregates the knowledge from each end-side student model. The cloud analyzes and integrates this information to form a more comprehensive understanding of the entire system.

[0149] 2) The cloud model transmits the extracted global knowledge back to the edge node in the form of parameter updates. By judging the resource operation status and job scheduling strategy, it further sends the data to the end side and updates it to the end-side student model, helping the end-side model to further iterate to adapt to local data characteristics.

[0150] 3.7 Adaptive Learning Rate Adjustment

[0151] In order to adapt to the heterogeneity in the edge-cloud collaborative intelligent system, this paper proposes an adaptive learning rate adjustment strategy. The core idea of ​​this strategy is to monitor the performance of the model in real time during the training process and dynamically adjust the learning rate according to the loss of each edge node, thereby optimizing the training efficiency and convergence speed of the model. The calculation formula is as follows:

[0152]

[0153] Among them, η t is the learning rate at time t, which is dynamically adjusted according to the loss of the model, η0 is the initial learning rate, and λ lr is responsible for controlling the adjustment rate of the learning rate as the loss changes. is the loss function of the edge node at time t. Through this adaptive adjustment, the learning rate η t Able to detect real-time loss of edge nodes Adjustments are made in the next iteration to ensure that the model can quickly respond to changes in data distribution and heterogeneity among nodes during training.

[0154] The knowledge distillation module of the present invention, which integrates online integration and network collaboration, can achieve effective compression and performance improvement of the model in the end-edge-cloud collaborative intelligent system. The design of this module fully considers the heterogeneity and distributed characteristics of the system, and provides important technical support for the efficient operation of the intelligent system.

[0155] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0156] The above are only preferred embodiments of the embodiments of the present application and are not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. Adaptive resource optimization and model generalization method in edge-cloud collaborative intelligent system, characterized by: Applied in an end-edge-cloud collaborative intelligent system, the end-edge-cloud collaborative intelligent system includes a cloud server and a plurality of edge nodes connected to the cloud server, each of the edge nodes is connected to a plurality of end devices in an area, and the method includes the following steps: The terminal device collects local environmental data and equipment operation parameters, trains the preset local model based on the collected data, and obtains the trained local model; constructs a digital twin model, and uses the digital twin model to realize the real-time digital mapping of the terminal-edge-cloud collaborative intelligent system. Based on the real-time data stream monitored in real time by the digital twin model, the statistical distance between the real-time data stream monitored in real time and the preset data distribution model is calculated. If the statistical distance is greater than the preset value, the active sample selection mechanism is started to select samples representing the current data distribution characteristics from the real-time data stream, and the local model is optimized and updated according to the selected samples and the causal invariance learning mechanism; the local model of the terminal device is used as the teacher model, and the key representation is extracted from the teacher model through the knowledge distillation technology to form a lightweight terminal-side student model; wherein, the local model is optimized and updated according to the selected samples and the causal invariance learning mechanism, and specifically includes the following steps: The optimization objective of causal invariance learning is expressed as: Among them, θ is the local model parameter, f θ (x) is the predicted output of the local model, y is the true label, is the loss function, λ is the regularization coefficient, is the mutual information, which is used to measure the mutual dependence between the feature representation Φ(x) extracted from the input data x and the causal variable Y, The goal of optimization is to minimize the sensitivity to future changes in data distribution while maintaining the model's adaptability to the current data distribution. The process of updating the local model is expressed as: Among them, θ new is the updated model parameter, θ old is the current model parameter, η is the learning rate, represents the gradient of the model parameter θ; The edge node aggregates and fine-tunes the incremental changes of the end-side student model parameters sent by the end device to form an edge-side student model, and the edge node performs resource status prediction and job scheduling; The cloud server collects and aggregates the edge-side student models from each edge node and transmits the extracted global knowledge back to the edge node in the form of parameter updates; the edge node sends the parameter update data to the end device, updates the end-side student model, and further iterates the end-side student model to adapt to the local data characteristics.

2. The method for adaptive resource optimization and model generalization in the edge-cloud collaborative intelligent system according to claim 1, characterized in that: The calculation formula of the statistical distance is as follows: Among them, d stat represents the statistical distance, D real-time is the distribution of real-time data streams, D preset is the preset data distribution model, p i and q i are the probabilities of the i-th event in the two distributions, The active sample selection mechanism is started to select samples representing the current data distribution characteristics from the real-time data stream. The mathematical expression of the sample selection is as follows: Among them, x select is the selected sample, g(x) is the fitness function of the sample, and w(x) is the weight function of the sample, which is inversely proportional to the probability of occurrence of the sample in the data stream.

3. The method for adaptive resource optimization and model generalization in the edge-cloud collaborative intelligent system according to claim 1, characterized in that: The edge node performs resource status prediction and job scheduling, which specifically includes the following steps: The edge node builds a prediction model and pre-collects job requests sent by the end device. The prediction model is used to predict the resource usage and job arrival status in the end-edge-cloud collaborative intelligent system in the future. According to the output results of the prediction model and the set of job requests, Bayesian optimization and greedy construction algorithms are used to dynamically adjust the job priority and resource allocation.

4. The method for adaptive resource optimization and model generalization in the edge-cloud collaborative intelligent system according to claim 3 is characterized in that: The method uses Bayesian optimization and greedy construction algorithms according to the output results of the prediction model and the set of job requests to dynamically adjust the job priority and resource allocation, which specifically includes the following steps: Construct the strategy target, the formula is as follows: Among them, s j is the scheduling time of job j, C j (s j ) is the completion time of job j, D j (s j ) is the delay cost of job j, α is the delay cost weight, S is the set of all scheduling times, R(s j ) is the time at which job j is scheduled j the amount of resources required; Bayesian optimization is used to optimize the parameters in the job scheduling strategy and determine the optimal scheduling strategy. The formula is as follows: Among them, θ is a parameter, is the information gain of parameter θ, is the loss function of parameter θ, α and β are parameters for adjusting the information gain and loss function weights; Based on the determined optimal scheduling strategy, the job priority and resource allocation are dynamically adjusted. The formula is as follows: P t+1 =g(P t ,R t ,J t ) Among them, P t is the job priority assignment at time t, R t is the resource state at time t, J t is the set of job requests at time t, and g is the dynamic adjustment function.

5. The method for adaptive resource optimization and model generalization in the edge-cloud collaborative intelligent system according to claim 1, characterized in that: The local model of the end device is used as the teacher model. The key representations are extracted from the teacher model through knowledge distillation technology to form a lightweight end-side student model. The end device distillation loss function is as follows: in, is the end device distillation loss function, is the cross entropy loss, y is the true label, is the prediction of the student model, α KL is the weight of the distillation loss, is the KL divergence loss, P' t is the prediction distribution of the teacher model, P' s is the predicted distribution of the student model; The loss function on the edge node is as follows: in, is the loss function on the edge nodes, β reg is the weight of the regularization term, is the regularization loss, used to prevent the model from overfitting; The loss function of the cloud server is as follows: in, is the loss function of the cloud server, N is the number of edge nodes, is the weight of the i-th edge node model, is the loss function of the i-th edge node model, γ is the weight of the diversity loss, is the diversity loss, used to encourage diversity among different models; The total loss function of the edge-cloud collaborative intelligent system is as follows: in, is the total loss function of the entire system, δ is the weight of the communication loss, It is the communication loss, which is used to measure the information loss during the transmission of the model between the edge and the cloud.

6. The method for adaptive resource optimization and model generalization in the edge-cloud collaborative intelligent system according to claim 5, characterized in that: The following steps are also included: the end-side student model and the edge-side student model use a multi-branch network structure and random enhancement input to improve training efficiency during the distillation training phase, as follows: The multi-branch network structure is expressed as: in, is the final prediction, f h () is the model of the h-th branch, is the randomly enhanced input data, e h is the enhancement parameter, M is the number of branches, and φ() is the aggregation function; The end-side student model and the edge-side student model apply random transformation, random rotation, and random scaling operations to the input data during training.

7. The method for adaptive resource optimization and model generalization in the edge-cloud collaborative intelligent system according to claim 5, characterized in that: It also includes the introduction of a time mean model, which continuously updates the exponential moving average of the end-side student model parameters during the training process. The formula is as follows: i k+1 =a TMM ·i k +(1-a TMM )·θ current Among them, θ k+1 is the student model parameter for the next time step, θ k is the parameter of the current time step, θ current is the updated value of the model parameters in the current training step, α TMM is a smoothing coefficient between 0 and 1, which determines the influence of historical parameter values ​​on new parameter values. TMM The closer it is to 1, the smoother the parameter update.

Images