Cloud edge computing power collaborative data processing method for large model
Through the two-level distributed training and parameter sharing mechanism of the Octopus Bionic Structural Neural Network Model, combined with cloud-edge computing resources, the performance problems of the large model on resource-constrained devices are solved, and efficient large-scale data processing and long sequences are realized Accurate prediction of data.
Patent Information
- Application Number
- CN202411084349.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-12-03
AI Technical Summary
Cloud computing affects real-time application performance due to network latency, while edge computing is difficult to handle large-scale complex computing tasks due to limited device resources, especially in training and inference of large models.
Using the Octopus Bionic Structure Neural Network Model, the two-level distributed training strategy, parameter sharing mechanism and adaptive learning algorithm are used to combine the computing resources of cloud and edge devices to realize parallel model training and parameter updates, and dynamically adjust task allocation and learning. to improve computational efficiency.
It improves the efficiency and resource utilization of large-scale data processing, reduces the latency of cloud computing and resource pressure on edge devices, maintains efficient parallel computing performance, is suitable for long-sequence data processing and avoids historical data forgetting.
Smart Images

Figure CN119065836A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cloud-edge computing technology, and in particular to a cloud-edge computing collaborative data processing method for large models. Background Art
[0002] With the rapid development of information technology, cloud computing and edge computing have become the two pillars of modern data processing. With its powerful remote computing and storage capabilities, cloud computing provides a solid foundation for big data analysis, machine learning, etc. Edge computing, with its geographical location close to the data source and low latency characteristics, provides a new solution for real-time data processing and privacy protection. However, these two computing modes have their own limitations: cloud computing may affect the performance of real-time applications due to network latency, while edge computing is limited by the limited resources of edge devices and is difficult to handle large-scale complex computing tasks.
[0003] Since the Transformer architecture was proposed in 2017, it has made revolutionary progress in the field of natural language processing with its unique self-attention mechanism. This architecture significantly improves the performance of sequence-to-sequence tasks such as machine translation and text summarization by allowing the model to interact directly at all positions within the sequence. However, as the model continues to expand and become more complex, its computing resource requirements on the device gradually increase, which makes it difficult to implement model training and reasoning in resource-constrained devices. Summary of the invention
[0004] The purpose of this invention is to provide a cloud-edge computing power collaborative data processing method for large models. By introducing a two-level distributed training strategy, a parameter sharing mechanism and an adaptive learning algorithm, the computing resources of the cloud and edge devices can be fully utilized to improve the large-scale data processing capabilities and efficiency.
[0005] The purpose of the present invention can be achieved by the following technical solutions:
[0006] A cloud-edge computing power collaborative data processing method for large models includes the following steps:
[0007] Model construction and deployment: Build an octopus bionic structure neural network model and deploy it in multiple processors on the cloud and edge. One processor in the cloud is set as the main processor.
[0008] Parameter initialization: The main processor in the cloud initializes the global parameters of the model and sends the global parameters to other processors in the cloud and edge.
[0009] Adaptive task allocation: Adopt adaptive learning algorithms to make decisions on data processing task allocation between the cloud and edge based on real-time computing resource conditions;
[0010] Model parallel training: Based on the input data, the cloud and edge perform model parallel training based on the task allocation decision results to achieve parameter sharing and updating. During the training process, the edge reconstructs the edge model based on the reshaping factor;
[0011] Model fusion: Fusion of model parameters and feature data trained on the cloud and edge, update of model global parameters, and sending of updated global parameters to edge processors, repeating the adaptive task allocation and model parallel training process until data processing is completed.
[0012] Each layer in the octopus bionic structure neural network model is equivalent to the neurons of an octopus, and the overall architecture adopts the structural shape of an octopus, including an input layer, a feature extraction layer, a feature reading layer and an output layer connected in sequence, wherein the feature extraction layer includes multiple tentacles, each tentacle extracts a kind of feature information, and the feature reading layer calculates the feature vector based on the information extracted by each tentacle of the feature extraction layer.
[0013] The global parameters include the weights and biases of the octopus bionic structure neural network model, which are expressed as:
[0014] Θ total = {W i ,b i ,W o ,b o |i=0,1,…,n}
[0015] Where W i represents the weight matrix of the i-th layer, b i represents the bias vector of the i-th layer, W o and b o Represent the weight matrix and bias vector of the output layer respectively, and n is the number of layers of the model;
[0016] The set of global parameters is divided into two subsets {Θ cloud ,Θ edge}, where the cloud parameter subset Θ cloud The parameters in are sent to other processors in the cloud for training, and the edge parameter subset Θ edge The parameters in the subset are sent to the edge processor for training; the parameters in each subset are divided into different processors for training without duplication.
[0017] The model parallel training is specifically as follows: according to the characteristics of the network structure, the word dimension is used as the standard for segmentation, so that each processor only stores part of the word vector, and the total word vector is obtained by summarizing through the allreduce communication method; the model parameter matrix is divided into multiple processors, each processor processes the corresponding parameters respectively, and the results are summarized through the allgather communication method.
[0018] The edge end reconstructs the edge end model according to the reshaping factor as follows: vertically split the correlation calculation blocks in the model according to the number of processors q at each edge end to obtain the corresponding subset Ω = {A1, A2, ..., A i}, i = 1, 2, ..., q, each edge processor e i There is a remodeling factor e r , which is used to control the network structure reshaping mechanism of the current edge end. The network structure reshaping mechanism is expressed as:
[0019]
[0020] Where N represents the number of submodules in the m-th layer network, P L Indicates the edge end e i The parameters of the mth layer network in , P total represents the total parameters of the edge network, L m Represents the number of modules of the m-th layer network after reshaping.
[0021] The parameters of the octopus bionic structure neural network model are updated by the following formula:
[0022]
[0023] Here, α is a sharing factor between 0 and 1, which is used to balance the update of cloud and edge parameters.
[0024] The specific implementation method of parameter sharing and updating is as follows: the cloud and the edge share parameters through cloud-edge communication, and set the update frequency f, which is calculated by the time interval T of adjacent synchronous parameter updates. When the cloud does not receive the edge E within two time intervals, i When updating parameters, cloud compensation training is started and model training is performed based on historical and cloud parameter sets.
[0025] Let the training scene be S i ,i=1,2,…,a, the training task is T j ,j=1,2,…,a, calculate the scene-task correlation, β represents the scene S i and training task T j Each edge end calculates the correlation between its own training task and scenario, and learns the capabilities of other edge ends through a scenario-task interactive sharing mechanism, wherein the correlation is calculated as follows:
[0026] β=Random_forest(S i ,T j )
[0027] Random_forest represents the scene-task correlation calculation algorithm. First, the original data is sampled with replacement and a subset is constructed. The subset data is the same as the original data, and there can be repeated data between subsets. A tree is constructed based on the subset, and the scene S is selected. i The importance of the features with higher importance is used as the scene S i and Task T i degree of relevance.
[0028] The adaptive task allocation is specifically as follows:
[0029] Define a resource utilization function U(t) to represent the resource utilization of the cloud and edge devices within a specific time:
[0030]
[0031] Where R cloud (t) and R edge (t) represent the resource utilization of the cloud and edge at time t, R total (t) is the total available resources;
[0032] According to the resource utilization rate U(t), the adaptive learning algorithm adjusts the distribution of model tasks in the cloud and edge to optimize the overall computing efficiency. Specifically, the network layer participation training effectiveness parameter τ is set for both the cloud and edge. Each layer of the network is independent of each other. The last layer of the network layer is eliminated by changing τ. When the network layer L i When τ is lower than the preset threshold Γ, the layer is pruned:
[0033]
[0034] Where D represents the decision function and f represents the pruning operation function. According to the decision function F(D), the allocation strategy is completed based on the characteristics of the current device resources and tasks. First, a state table is constructed based on the current resource utilization status, including the currently executed task index, cloud resource utilization status, and the current resource utilization status. and edge utilization The remaining resources of the cloud and edge are calculated. According to the current remaining resources of the cloud and edge, combined with the resource usage of different tasks, the task allocation of the cloud and edge is dynamically adjusted to maximize the number of task executions and minimize the remaining resources of the cloud and edge.
[0035] Use learning rate dynamic scheduling to adjust the entire training of the model:
[0036] l(t+1)=l(t)·Y·P
[0037] Among them, l(t) represents the learning rate at time t, Υ is the factor for adjusting the learning rate, and P is a function representing the model performance;
[0038] Calculate the complexity of the model and adjust the model based on the complexity. Adapt simple reasoning tasks to the structure with fewer model layers and deploy them on the edge. The method for adjusting the model complexity is as follows:
[0039] Θ(t+1)=AC(Θ(t),R cloud (t),R edge (t),T assigned )
[0040] Among them, the parameter Θ(t) represents the parameter set at time t, and AC is the update function, which includes the parameter set at time t, the resource utilization of the cloud and edge, and the task allocation. When the complexity of the edge execution task is low, the number of layers of the edge model is reduced to reduce the parameter complexity and release some edge resource space.
[0041] The described fusion of the model parameters and feature data trained on the cloud and edge is specifically as follows: the model parameters and feature data between the cloud and edge processors are fused, the cloud performs addition and multiplication operations on the received model parameter matrix, obtains the fused global parameters and saves them, and at the same time, fuses and saves the received feature data.
[0042] Compared with the prior art, the present invention has the following beneficial effects:
[0043] (1) The present invention proposes an octopus bionic structure neural network model to achieve the synergy of cloud-edge computing power to improve the utilization efficiency of computing resources. This architecture can fully utilize the powerful computing power of cloud computing and the low latency advantage of edge computing while maintaining the model's efficient parallel computing performance in processing sequence data. The cloud-edge computing power synergy organically combines the computing resources of the cloud and edge devices, places part of the computing tasks on the edge devices, reduces cloud compliance and reduces transmission delays.
[0044] (2) The bionic structure neural network model of the present invention considers historical sequence data based on the Transformer architecture and calculates the correlation between the historical sequence and the current sequence to accurately predict the future sequence state, avoids the forgetting of historical data and reduces data redundancy, and is suitable for processing long sequence data.
[0045] (3) The present invention adopts a two-level training strategy and introduces an adaptive learning algorithm to dynamically adjust task allocation according to the real-time computing resource situation, flexibly allocate computing tasks to cloud and edge devices, realize dynamic computing resource collaboration, make full use of available computing resources, and improve overall computing efficiency.
[0046] (4) The present invention introduces a parameter sharing mechanism to enable the cloud and edge devices to share some parameters, update some parameters of the edge devices locally, reduce parameter transmission between the cloud and edge devices, and improve model reasoning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 is a flow chart of the method of the present invention;
[0048] Figure 2 This is the network structure diagram of the octopus bionic structure neural network model;
[0049] Figure 3 This is a schematic diagram of the model calculation of the octopus bionic structure neural network model;
[0050] Figure 4 This is a model parallel schematic diagram of the octopus bionic structure neural network model;
[0051] Figure 5 Schematic diagram of the parameter sharing mechanism of the octopus bionic structure neural network model. DETAILED DESCRIPTION
[0052] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0053] The present invention introduces a new neural network architecture, which adopts a structure similar to that of a myriad of animals and deploys the model on the cloud side and the edge side, aiming to solve the performance problem of the Transformer model on resource-constrained devices. By combining the powerful computing power of cloud computing and the low latency advantage of edge computing, the architecture optimizes the performance of large-scale data processing tasks. The new network architecture design of the present invention is based on the following core principles: resource optimization, task allocation, data collaboration, dynamic scheduling and model fusion. Resource optimization refers to reducing the complexity and computing requirements of the model through technologies such as dynamically adjusting the model learning rate, so that it can adapt to the resource constraints of edge devices and the cloud. Task allocation refers to intelligently allocating computing tasks between the cloud and the edge according to the computing intensity and real-time requirements of the task. Data collaboration refers to designing efficient data transmission and synchronization mechanisms to ensure data consistency between the cloud and the edge, while taking into account the privacy and security of the data. Dynamic scheduling refers to implementing a dynamic task scheduling strategy to adjust the allocation and execution of computing tasks according to the real-time system load and network conditions. Model fusion refers to realizing the knowledge migration and fusion of the model between the cloud and the edge, so that the edge model can be updated and optimized using the powerful computing resources of the cloud. The present invention is suitable for large-scale data processing and machine learning tasks, provides effective solutions for different practical application scenarios, and is expected to promote the application of edge computing and models in a wider range of fields.
[0054] Specifically, Figure 1 As shown, this embodiment provides a cloud-edge computing power collaborative data processing method for a large model, including the following steps:
[0055] S1, model construction and deployment: Build an octopus bionic structure neural network model and deploy it in multiple processors in the cloud and edge respectively, where one processor in the cloud is set as the main processor.
[0056] The overall operation of the octopus bionic structure neural network model can be understood as the process of an octopus interpreting data. Each layer is equivalent to the octopus's neurons, and the overall architecture adopts the structural shape of an octopus, such as Figure 2As shown in the figure, it includes an input layer, a feature extraction layer, a feature reading layer, and an output layer connected in sequence. The data enters the input layer and is converted into a vector. The vector is input into the feature extraction layer, and then processed by the feature reading layer to obtain the final output. Among them, the feature extraction layer contains k tentacles (modules), and the input is processed by each module to obtain the corresponding features. For example, if an octopus wants to know whether the target in front of it is a dog or a cat, each of its tentacle (module) collects target information, such as contour, color, and semantics. The feature reading layer receives the information collected from each tentacle (module) and completes the feature vector calculation. The weight w represents the importance of each collected information, F represents the contour information, such as w1 represents the importance of the contour information F1 collected by tentacle 1 (module 1), and the bias bias represents the degree to which it is affected by other factors when touching the target. The weight is multiplied by the information and weighted summed with the bias, and the final output is obtained through mechanisms such as activation functions:
[0057]
[0058] Taking advantage of the autonomy and neuronal plasticity of octopus tentacles, and implementing a dynamic reorganization mechanism during the training process, the cloud and edge terminals each perform adaptive network layer reshaping and interactive learning according to different training scenarios and tasks. The present invention aims to improve the utilization efficiency and security of computing resources while optimizing the performance of large-scale data processing tasks. This architecture combines the advantages of the flexible and autonomous octopus bionic architecture with the innovative method of collaborative training between cloud and edge segments based on models to achieve effective deployment and execution of large-scale models.
[0059] S2, parameter initialization: The main processor in the cloud initializes the global parameters of the model and sends the global parameters to other processors in the cloud and edge.
[0060] In this embodiment, a graphics card (GPU) is used as a processor at the cloud and edge to perform information processing.
[0061] A graphics card is specially set up in the cloud as the main GPU, and the global parameter set Θ of the model is total Initialize and pass it to other cloud and edge terminals for subsequent training. The global parameters mainly include the weights and biases of the octopus bionic structure neural network model, which can be expressed by the following formula (2):
[0062] Θ total = {W i ,b i ,W o ,b o|i=0,1,…,n} (2)
[0063] Where W i represents the weight matrix of the i-th layer, b i represents the bias vector of the i-th layer, W o and b o They represent the weight matrix and bias vector of the output layer respectively, and n is the number of layers of the model.
[0064] These parameters are used to calculate the loss gradient for training by minimizing the loss function L(Θ) using gradient descent or other optimization algorithms:
[0065] Θ total' =argmin Θ L(Θ) (3)
[0066] The set of global parameters is divided into two subsets {Θ cloud ,Θ edge}, where the cloud parameter subset Θ cloud The parameters in are sent to other processors in the cloud for training, and the edge parameter subset Θ edge The parameters in the subset are sent to the edge processor for training; the parameters in each subset are divided into different processors for training without duplication, realizing parameter parallelism and improving computing efficiency.
[0067] At the edge computing level, the local parameter set Θ of the model edge is designed as a subset of global parameters in order to perform respective training tasks and inference tasks on edge devices. These local parameters can be expressed by the following formula (4):
[0068] Θ edge = {W i,edge ,b i,edge |i∈S} (4)
[0069] Where S is a set of indices and i represents the layer executed on the edge device.
[0070] S3, adaptive task allocation: Adopts adaptive learning algorithms to make cloud and edge data processing task allocation decisions based on real-time computing resource conditions.
[0071] This architecture uses an adaptive learning algorithm similar to the Octopus learning rule to dynamically adjust task allocation according to the real-time computing resource situation in the current training scenario. This is achieved by defining a resource utilization function U(t), which is used to represent the resource utilization of the cloud and edge devices within a specific time:
[0072]
[0073] Where R cloud (t) and R edge (t) represent the resource utilization of the cloud and edge at time t, R total (t) is the total available resources.
[0074] According to the resource utilization rate U(t), the adaptive learning algorithm adjusts the distribution of model tasks in the cloud and edge to optimize the overall computing efficiency. Specifically: set the network layer participation training effectiveness parameter τ that is applicable to both the cloud and edge. Each layer of the network is independent of each other. By changing τ, the last layer of the network layer is eliminated. When the network layer L i When τ is lower than the preset threshold Γ, the layer is pruned, as shown in formula (6):
[0075]
[0076] Where D represents the decision function and f represents the pruning operation function;
[0077] According to the decision function F(D), the allocation strategy is completed based on the characteristics of the current device resources and tasks; first, a state table is constructed based on the current resource utilization status, including the currently executed task index index, cloud resource utilization status and edge utilization The remaining resources of the cloud and edge are calculated. According to the current remaining resources of the cloud and edge, combined with the resource usage of different tasks, the task allocation of the cloud and edge is dynamically adjusted to maximize the number of task executions and minimize the remaining resources of the cloud and edge.
[0078] In order to make the model perform better inference, this embodiment uses learning rate dynamic scheduling to adjust the entire training of the model:
[0079] l(t+1)=l(t)·Υ·P (7)
[0080] Among them, l(t) represents the learning rate at time t, Υ is the factor for adjusting the learning rate, and P is the function representing the model performance.
[0081] The complexity of the model is calculated based on device resource detection and evaluation, dynamic task allocation, and model learning rate. The model is adjusted based on the complexity, and simple reasoning tasks are adapted to the structure with fewer model layers and deployed on the edge. The method for adjusting the model complexity is as follows:
[0082] Θ(t+1)=AC(Θ(t),R cloud (t),R edge (t),T assigned ) (8)
[0083] Among them, the parameter Θ(t) represents the parameter set at time t, and AC is the update function, which includes the parameter set at time t, the resource utilization of the cloud and edge, and the task allocation. When the complexity of the edge execution task is low, the number of layers of the edge model is reduced to reduce the parameter complexity and release some edge resource space.
[0084] S4, model parallel training: Based on the input data, the cloud and edge perform model parallel training based on the task allocation decision results to achieve parameter sharing and updating. During the training process, the edge reconstructs the edge model based on the reshaping factor.
[0085] like Figure 3 The figure shows the calculation process of the model. After the model performs multiplication calculation based on historical information and noise data, it calculates the correlation with the input data. The specific correlation calculation part is described in detail below.
[0086] like Figure 4 As shown in the figure, the specific parallel training of the model is as follows: according to the characteristics of the network structure, the word dimension is used as the standard for segmentation, so that each graphics card only stores part of the word vector, and the total word vector is obtained by summarizing through the allreduce communication method; the model parameter matrix is divided into multiple graphics cards, the number of divisions is the same as the number of graphics cards, each graphics card processes the corresponding parameters respectively, and the results are summarized through the allgather communication method to ensure that the final result is equal to the calculation result of a single card.
[0087] like Figure 5 As shown, the edge end reconstructs the edge end model according to the reshaping factor as follows: vertically split the correlation calculation blocks in the model according to the number of processors q at each edge end to obtain the corresponding subset Ω = {A1, A2, ..., A i}, i = 1, 2, ..., q, and deploy the segmented correlation calculation module on each GPU. Each GPU executes the correlation between the inputs separately and establishes communication with other GPUs. Each edge processor e i There is a remodeling factor e r , which is used to control the network structure reshaping mechanism of the current edge end. The network structure reshaping mechanism is expressed as:
[0088]
[0089] Where N represents the number of submodules in the m-th layer network, P L Indicates the edge end e i The parameters of the mth layer network in , P total represents the total parameters of the edge network, L m Represents the number of modules of the m-th layer network after reshaping.
[0090] At the same time, the edgei After a round of training, the bias parameters are immediately transmitted to the cloud edge without waiting for other edge terminals to transmit together.
[0091] In order to achieve parameter sharing between the cloud and the edge and complete the parameter update of the entire network architecture, a sharing mechanism is introduced. The Octopus bionic structure neural network model updates parameters through the following formula:
[0092]
[0093] Here, α is a sharing factor between 0 and 1, which is used to balance the update of cloud and edge parameters.
[0094] In order to ensure the consistency of parameters between the cloud and edge devices, a parameter sharing mechanism is introduced. The cloud and edge devices share parameters through cloud-edge communication and set an update frequency f. The update frequency is calculated by the time interval T between adjacent synchronous parameter updates, as shown in formula (11). i When updating parameters, cloud compensation training is started and model training is performed based on historical and cloud parameter sets.
[0095]
[0096] Let the training scene be S i ,i=1,2,…,a, the training task is T j ,j=1,2,…,a, the scene-task correlation is calculated as in formula (12). β represents the scene S i and training task T j Each edge end calculates the correlation between its own training tasks and scenarios, and learns the capabilities of other edge ends through a scenario-task interaction sharing mechanism.
[0097] β=Random_forest(S i ,T j ) (12)
[0098] Random_forest represents the scene-task correlation calculation algorithm. First, the original data is sampled with replacement and a subset is constructed. The subset data is the same as the original data, and there can be repeated data between subsets. A tree is constructed based on the subset, and the scene S is selected. i The importance of the features with higher importance is used as the scene S i and Task T i degree of relevance.
[0099] The cross entropy loss generated during model training is computationally intensive, so it is segmented based on the category dimension, and the final global loss is obtained through communication of intermediate results. Update local parameters and upload them to the cloud, and obtain the latest parameter set on the cloud.
[0100] S5, model fusion: Fusion of model parameters and feature data trained on the cloud and edge, update of model global parameters, and sending of updated global parameters to the edge processor, repeating the adaptive task allocation and model parallel training process until data processing is completed.
[0101] Specifically, the model parameters and feature data between the cloud and edge processors are integrated. The cloud performs addition and multiplication operations on the received model parameter matrix to obtain and save the fused global parameters. At the same time, the received feature data is integrated and saved.
[0102] Through the above specific implementation methods, the cloud-edge computing power collaborative network architecture of the present invention can adapt to different computing environments and resource constraints while maintaining high performance. By allocating tasks and sharing parameters between cloud computing and edge devices, and reconstructing the model, data and model parallelism is implemented, and a federated learning training method is used at the edge to improve the utilization efficiency of computing resources. The adaptive learning algorithm dynamically adjusts task allocation according to the real-time computing resource situation, further improving the overall computing efficiency. This architecture provides an effective solution for large-scale data processing and large-scale model training tasks, and can adapt to different practical application scenarios.
[0103] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A cloud-edge computing power collaborative data processing method for large models, characterized in that: The following steps are involved: Model construction and deployment: Build an octopus bionic structure neural network model and deploy it in multiple processors on the cloud and edge. One processor in the cloud is set as the main processor. Parameter initialization: The main processor in the cloud initializes the global parameters of the model and sends the global parameters to other processors in the cloud and edge. Adaptive task allocation: Adopt adaptive learning algorithms to make decisions on data processing task allocation between the cloud and edge based on real-time computing resource conditions; Model parallel training: Based on the input data, the cloud and edge perform model parallel training based on the task allocation decision results to achieve parameter sharing and updating. During the training process, the edge reconstructs the edge model based on the reshaping factor; Model fusion: Fusion of model parameters and feature data trained on the cloud and edge, update of model global parameters, and sending of updated global parameters to edge processors, repeating the adaptive task allocation and model parallel training process until data processing is completed.
2. According to the large model-oriented cloud-edge computing power collaborative data processing method according to claim 1, it is characterized in that: Each layer in the octopus bionic structure neural network model is equivalent to the neurons of an octopus, and the overall architecture adopts the structural shape of an octopus, including an input layer, a feature extraction layer, a feature reading layer and an output layer connected in sequence, wherein the feature extraction layer includes multiple tentacles, each tentacle extracts a kind of feature information, and the feature reading layer calculates the feature vector based on the information extracted by each tentacle of the feature extraction layer.
3. According to the large model-oriented cloud-edge computing power collaborative data processing method according to claim 1, it is characterized in that: The global parameters include the weights and biases of the octopus bionic structure neural network model, which are expressed as: Θ total ={W i ,b i ,W o ,b o |i=0,1,…,n} Where W i represents the weight matrix of the i-th layer, b i represents the bias vector of the i-th layer, W o and b o Represent the weight matrix and bias vector of the output layer respectively, and n is the number of layers of the model; The set of global parameters is divided into two subsets {Θ cloud ,Θ edge }, where the cloud parameter subset Θ cloud The parameters in are sent to other processors in the cloud for training, and the edge parameter subset Θ edge The parameters in the subset are sent to the edge processor for training; the parameters in each subset are divided into different processors for training without duplication.
4. According to the large model-oriented cloud-edge computing power collaborative data processing method according to claim 1, it is characterized in that: The model parallel training is specifically as follows: according to the characteristics of the network structure, the input word dimension is used as the standard for segmentation, so that each processor only stores part of the word vector, and the total word vector is obtained by summarizing through the allreduce communication method; the model parameter matrix is divided into multiple processors, each processor processes the corresponding parameters respectively, and the results are summarized through the allgather communication method.
5. According to the large model-oriented cloud-edge computing power collaborative data processing method according to claim 1, it is characterized in that: The edge end reconstructs the edge end model according to the reshaping factor as follows: vertically split the correlation calculation blocks in the model according to the number of processors q at each edge end to obtain the corresponding subset Ω = {A1, A2, ..., A i }, i = 1, 2, ..., q, each edge processor e i There is a remodeling factor e r , which is used to control the network structure reshaping mechanism of the current edge end. The network structure reshaping mechanism is expressed as: Where N represents the number of submodules in the m-th layer network, P L Indicates the edge end e i The parameters of the mth layer network in , P total represents the total parameters of the edge network, L m Represents the number of modules of the m-th layer network after reshaping.
6. The cloud-edge computing power collaborative data processing method for large models according to claim 3 is characterized in that: The parameters of the octopus bionic structure neural network model are updated by the following formula: Here, α is a sharing factor between 0 and 1, which is used to balance the update of cloud and edge parameters.
7. The cloud-edge computing power collaborative data processing method for large models according to claim 1 is characterized in that: The specific implementation method of parameter sharing and updating is as follows: the cloud and the edge share parameters through cloud-edge communication, and set the update frequency f, which is calculated by the time interval t of adjacent synchronous parameter updates. When the cloud does not receive the edge E within two time intervals, i When updating parameters, cloud compensation training is started and model training is performed based on historical and cloud parameter sets.
8. The method for large-scale model-oriented cloud-edge computing power collaborative data processing according to claim 1 is characterized in that: The training scene is S i , i = 1, 2, ..., a, the training task is T j , j = 1, 2, ..., a, calculate the scene-task correlation, β represents the scene S i and training task T j Each edge end calculates the correlation between its own training task and scenario, and learns the capabilities of other edge ends through a scenario-task interactive sharing mechanism, wherein the correlation is calculated as follows: β=Random_forest(S i ,T j ) Among them, Random_forest represents the scene-task correlation calculation algorithm.
9. The cloud-edge computing power collaborative data processing method for large models according to claim 1 is characterized in that: The adaptive task allocation is specifically as follows: Define a resource utilization function U(t) to represent the resource utilization of the cloud and edge devices within a specific time: Where R cloud (t) and R edge (t) represent the resource utilization of the cloud and edge at time t, R total (t) is the total available resources; According to the resource utilization rate U(t), the adaptive learning algorithm adjusts the distribution of model tasks in the cloud and edge to optimize the overall computing efficiency. Specifically, the network layer participation training effectiveness parameter τ is set for both the cloud and edge. Each layer of the network is independent of each other. The last layer of the network layer is eliminated by changing τ. When the network layer L i When τ is lower than the preset threshold Γ, the layer is pruned: Where D represents the decision function, and f represents the pruning operation function; the allocation strategy is completed based on the characteristics of the current device resources and tasks according to the decision function F(D); Use learning rate dynamic scheduling to adjust the entire training of the model: l(t+1)=l(t)·Y·P Among them, l(t) represents the learning rate at time t, Υ is the factor for adjusting the learning rate, and P is a function representing the model performance; Calculate the complexity of the model and adjust the model based on the complexity. Adapt simple reasoning tasks to the structure with fewer model layers and deploy them on the edge. The method for adjusting the model complexity is as follows: Θ(t+1)=AC(Θ(t),R cloud (t),R edge (t),T assigned ) Among them, the parameter Θ(t) represents the parameter set at time t; AC is the update function, which includes the parameter set at time t, the resource utilization of the cloud and edge, and the task allocation.
10. The method for large-model cloud-edge computing collaborative data processing according to claim 1, characterized in that: The described fusion of the model parameters and feature data trained on the cloud and edge is specifically as follows: the model parameters and feature data between the cloud and edge processors are fused, the cloud performs addition and multiplication operations on the received model parameter matrix, obtains the fused global parameters and saves them, and at the same time, fuses and saves the received feature data.
Citation Information
Cited By
AI model training method and system
CN119476398A
AR scene dynamic deep learning model training and reasoning method based on federated learning
CN121328741A
Federated learning based ar scene dynamic deep learning model training and inference method
CN121328741B
