Cloud edge size model collaboration method and system based on federal learning, and electronic equipment

Abstract

The invention provides a federated learning-based cloud edge size model collaboration method and system, and electronic equipment, and the method makes full use of cloud edge resources, releases the size model capability: a large-scale model is trained at a cloud end, a small model suitable for a local scene is trained by an edge server and terminal equipment, and through a federated learning mechanism, a large-scale model is trained at the cloud end; and model collaborative optimization among different levels of nodes is realized. Data are available and invisible, data privacy and safety are guaranteed, private data of clients on the edge side do not need to be out of the domain, the cloud side shares the model capacity, the data are available and invisible, and data privacy and safety are guaranteed. The invention provides an efficient, flexible and extensible model training and optimizing method, computing power resources of cloud edges can be fully utilized, user privacy is protected, and a global large model is continuously optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud-edge large and small model collaboration solution design based on federated learning, and specifically to a cloud-edge large and small model collaboration method and system and electronic equipment based on federated learning. Background Art

[0002] In scenarios such as government supervision, scientific research collaboration, and group operations where data is supplied by multiple terminals, the data is private (cannot leave the domain), and the computing power of edge computing devices on the terminal side is insufficient, there is a lack of personalized learning methods that can simultaneously meet the requirements of data privacy protection and adapt to the performance of each terminal device. It is difficult to achieve the integration of common and individual data models. It is difficult for the model to dynamically adapt to the data quality and tasks of each terminal device to achieve personalized learning. There is a lack of edge model update methods driven by large models, and large models are difficult to conduct comprehensive analysis of edge data to generate accurate and effective model calculation results.

[0003] Therefore, the prior art needs to be further developed. Summary of the invention

[0004] The purpose of the present invention is to overcome the above-mentioned technical deficiencies and provide a cloud-edge large and small model collaboration method and system, and electronic device based on federated learning to solve the problems existing in the prior art.

[0005] To achieve the above technical objectives, according to a first aspect of the present invention, the present invention provides a cloud-edge large and small model collaboration method based on federated learning, comprising: S100, determine the nodes participating in federated learning, including cloud servers and edge devices, configure the computing resources and communication capabilities of the nodes, and create virtual objects of the edge devices on the cloud side; S200, the cloud server distributes the initial updated small model to each participating node, and each node updates the edge small model and performs local training based on local data and computing resources; S300, each node performs model training and optimization based on local original data, and sends the updated model parameters to the cloud server; S400, the cloud server generates a new fusion large model by adopting a horizontal and vertical combined federated training mode and a dynamic multi-task data analysis method for the parameters of the small models received from each terminal device; S500 and the new fusion large model are distributed again to each node participating in federated learning through model pruning and distillation slimming and compression technology for the next round of training.

[0006] Specifically, the horizontal and vertical combined federated training mode includes: When the parameters of each edge model are consistent, the horizontal expansion method is used to integrate the parameters; When the parameters of each edge model are inconsistent, the longitudinal modeling method is used to integrate the parameters.

[0007] Specifically, the dynamic multi-task data analysis method includes: In the multi-task learning framework, task weights, model structures or training strategies are dynamically adjusted according to various indicators or conditions during the training process. The dynamic adjustment of training strategies includes: Dynamic weight adjustment dynamically adjusts the weight of each task according to the performance of different tasks during the training process.

[0008] Specifically, dynamically adjusting the weight of each task according to the performance of different tasks during the training process includes: When the performance of a task on the validation set begins to decline, reduce its weight; or use an uncertainty-based weighting strategy to adjust the weight according to the uncertainty of the correlation between tasks.

[0009] Specifically, the dynamic adjustment of the model structure includes: During the training process, the structure of the model is dynamically adjusted according to the correlation between tasks or changes in model performance, including adding or deleting layers, or changing the connection mode between layers. The change in the connection mode between layers includes changing the connection between neurons in each layer and adjacent layers, and reducing connections through convolution, recurrent neural networks, and attention mechanisms.

[0010] Specifically, the method further includes: Dynamically adjust the size and complexity of the model based on the computing resources and training requirements of each node.

[0011] Specifically, the method further includes: Collect relevant data of the CPU, GPU, memory, and disk of each node in real time, and select the usage ratio of each resource as the standard based on industry best practices; secondly, optimize resource usage strategies, such as optimizing memory management strategies, using memory reuse and delayed release technologies to reduce memory usage; finally, use data pipeline technology to parallelize data reading, preprocessing, and training processes.

[0012] Specifically, the utilization ratios of various resources are selected as standards based on industry best practices, including: Select 80% as the standard.

[0013] According to a second aspect of the present invention, a cloud-edge large and small model collaboration system based on federated learning is provided, comprising: The acquisition module is used to determine the nodes participating in federated learning, including cloud servers and edge devices, and configure the computing resources and communication capabilities of the nodes; The control module is used to create virtual objects of edge devices on the cloud side; to control the cloud server to distribute the initial updated small model to each participating node, and each node updates the edge small model and performs local training based on local data and computing resources; to control each node to perform model training and application optimization based on local original data, and then send the updated model parameters to the cloud server; to control the cloud server to generate a new fused large model for the parameters of the small models received from each terminal device, using a federated training mode that combines horizontal and vertical operations and a dynamic multi-task data analysis method; and to distribute the new fused large model to each node participating in federated learning again through model pruning and distillation slimming compression technology for the next round of training.

[0014] According to a third aspect of the present invention, there is provided an electronic device, comprising: a memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above-mentioned cloud-edge size model collaboration method based on federated learning is implemented.

[0015] Beneficial effects: The present invention makes full use of cloud-edge resources to release the capabilities of large and small models: large-scale models are trained on the cloud, while edge servers and terminal devices train smaller models suitable for local scenarios. Through the federated learning mechanism, model collaborative optimization between nodes at different levels is achieved. Data is available but invisible, ensuring data privacy and security: edge-side customer private data does not need to leave the domain, and the cloud-side shared model capabilities achieve the effect of data availability and invisibility, ensuring data privacy and security. The present invention proposes an efficient, flexible and scalable model training and optimization method that can make full use of cloud-edge computing resources, protect user privacy, and continuously optimize the global large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a cloud-edge large and small model collaboration method based on federated learning provided in a specific embodiment of the present invention; Figure 2 It is a schematic diagram of the system composition of a cloud-edge large and small model collaboration system based on federated learning provided in a specific embodiment of the present invention; Figure 3 is a schematic diagram of parameter integration of a horizontal federation provided in a specific embodiment of the present invention; Figure 4 is a schematic diagram of parameter integration of a vertical federation provided in a specific embodiment of the present invention; Figure 5 It is a flowchart of dynamic multi-task training data analysis provided in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is clearly and completely described below in conjunction with the accompanying drawings of the present invention. Based on the embodiments in this application, other similar embodiments obtained by ordinary technicians in this field without making creative work should all fall within the scope of protection of this application. In addition, the directional words mentioned in the following embodiments, such as "up", "down", "left", "right", etc., are only reference to the directions of the accompanying drawings. Therefore, the directional words used are used to illustrate rather than limit the invention.

[0018] The present invention will be further described below in conjunction with the accompanying drawings and preferred embodiments.

[0019] See also Figure 1 The present invention provides a cloud-edge large and small model collaboration method based on federated learning, comprising: S100. Determine the nodes participating in federated learning, including cloud servers and edge devices, configure the computing resources and communication capabilities of the nodes, and create virtual objects of the edge devices on the cloud side.

[0020] It should be noted here that large models require a lot of computing resources and storage space, and terminal devices are often resource-constrained and difficult to deploy large model applications. Therefore, a method of analyzing and making decisions by integrating federated large and small models is proposed. The small models deployed on each terminal device can accurately use customer data, and the large model deployed on the cloud can be used to make the final analysis based on the calculation results of each small model. At the same time, the new large model is trained and optimized. The new large model uses pruning, distillation, compression and other slimming methods to form a new small model and send it to the edge for use.

[0021] Specifically, the purpose of S100 is to initialize the system.

[0022] S200. The cloud server distributes the initial updated small model to each participating node. Each node updates the edge small model and performs local training based on local data and computing resources.

[0023] S300. After each node performs model training and application optimization based on local original data, the updated model parameters are sent to the cloud server.

[0024] S400, the cloud server receives the parameters of the small models of each terminal device, adopts a horizontal and vertical combined federated training mode and a dynamic multi-task data analysis method to generate a new fusion large model.

[0025] Specifically, the horizontal and vertical combined federated training mode includes: When the parameters of each edge model are consistent, the horizontal expansion method is used to integrate the parameters; When the parameters of each edge model are inconsistent, the longitudinal modeling method is used to integrate the parameters.

[0026] It should be noted here that please refer to Figure 3 and Figure 4 , horizontal and vertical combined federated training Figure 3 and Figure 4 As shown, the yellow part represents the parameters of edge model 1, and the green part represents the parameters of edge model 2. When the parameters of each edge model are consistent, the parameter attributes overlap more, and horizontal expansion is used. When the parameters of each edge model are inconsistent, the parameter attributes overlap less, but the data items overlap more, and vertical modeling is used.

[0027] Specifically, the dynamic multi-task data analysis method includes: In the multi-task learning framework, task weights, model structures or training strategies are dynamically adjusted according to various indicators or conditions during the training process. The dynamic adjustment of training strategies includes: Dynamic weight adjustment dynamically adjusts the weight of each task according to the performance of different tasks during the training process.

[0028] Specifically, dynamically adjusting the weight of each task according to the performance of different tasks during the training process includes: When the performance of a task on the validation set begins to decline, reduce its weight; or use an uncertainty-based weighting strategy to adjust the weight according to the uncertainty of the correlation between tasks.

[0029] Specifically, the dynamic adjustment of the model structure includes: During the training process, the structure of the model is dynamically adjusted according to the correlation between tasks or changes in model performance, including adding or deleting layers, or changing the connection mode between layers. The change in the connection mode between layers includes changing the connection between neurons in each layer and adjacent layers, and reducing connections through convolution, recurrent neural networks, and attention mechanisms.

[0030] It should be noted here that dynamic multi-task training data analysis, such as Figure 5As shown in the figure, dynamic multi-task training refers to dynamically adjusting task weights, model structures or training strategies according to certain indicators or conditions in the training process under the framework of multi-task learning. Dynamic training strategies include: dynamic weight adjustment, dynamically adjusting the weight of each task according to the performance of different tasks in the training process. For example, when the performance of a task on the validation set begins to decline, its weight can be reduced to avoid overfitting. The weight can be adjusted according to the uncertainty of the correlation between tasks using a strategy based on uncertainty weighting; dynamic adjustment of model structure, dynamically adjusting the structure of the model according to the correlation between tasks or changes in model performance during training. For example, some layers can be added or deleted, or the connection method of the layers can be changed. Different feature extractors can be dynamically selected and used during training using the adaptive feature sharing method; dynamic adjustment of training strategies: dynamically adjusting training parameters such as learning rate and batch size according to the loss function value, gradient information or other performance indicators during training. Using the early stopping strategy, when the performance of a task on the validation set no longer improves, the training of the task is stopped in advance.

[0031] It is understandable that in the neural network-based model, the network structure consists of multiple model layers from left to right, and each layer is used to extract higher-dimensional target features. The complexity and model size can be reduced by deleting layers, of course, sacrificing certain effects; changing the connection method refers to the connection between neurons in each layer and adjacent layers. The connection can be reduced through convolution, recurrent neural networks, attention mechanisms, etc., thereby further reducing the model size and complexity.

[0032] It is understandable that the dynamic adjustment method of the training strategy, for example, if the loss function value does not drop significantly over a period of time, it may mean that the learning rate is too high or too low. You can reduce the learning rate and try more refined optimization, or increase the learning rate to accelerate convergence. The specific effect needs to be analyzed based on the experimental results. This type of method is a common means of machine learning and is not protected.

[0033] S500 and the new fusion large model are distributed again to each node participating in federated learning through model pruning and distillation slimming and compression technology for the next round of training.

[0034] It should be noted here that the new fusion model is distributed to each participating node again for the next round of training through slimming and compression technologies such as model pruning and distillation. Model pruning is to remove unimportant parameters to reduce the size and complexity of the model, accurately evaluate the importance of parameters, and use advanced methods such as the absolute value of weights and gradient information to deeply analyze the role of each model parameter to provide strong support for optimization decisions. According to the evaluation results, unimportant parameters or connections are pruned. The basic steps of model distillation are to define the fusion model as a teacher model, train it on a large-scale data set, and use the prediction results of this model as an auxiliary target to train the edge model, called the "student model". During the training process, the goal of the student model is to minimize the error with the prediction results of the teacher model and minimize the loss function of the student model itself. The student model can learn the knowledge in the teacher model and apply it to the reasoning task on the new data set.

[0035] Specifically, the method further includes: Dynamically adjust the size and complexity of the model based on the computing resources and training requirements of each node.

[0036] Specifically, the method further includes: Collect relevant data of the CPU, GPU, memory, and disk of each node in real time, and select the usage ratio of each resource as the standard based on industry best practices; secondly, optimize resource usage strategies, such as optimizing memory management strategies, using memory reuse and delayed release technologies to reduce memory usage; finally, use data pipeline technology to parallelize data reading, preprocessing, and training processes.

[0037] Specifically, the utilization ratios of various resources are selected as standards based on industry best practices, including: Select 80% as the standard.

[0038] It is understood that the method further comprises: Dynamic adjustment: dynamically adjust the size and complexity of the model according to the computing resources and training requirements of each node. First, collect relevant data such as CPU, GPU, memory, disk, etc. of each node in real time, and select the utilization ratio of various resources (about 80%) as the standard according to industry best practices; second, optimize resource utilization strategies, such as optimizing memory management strategies, using memory reuse, delayed release and other technologies to reduce memory usage; finally, use data pipeline technology to parallelize data reading, preprocessing and training processes to improve the utilization of computing resources.

[0039] The present invention is applicable to scenarios that require protection of user privacy and use of cloud-edge distributed computing power for model training, such as smart cities, telemedicine, government supervision, etc. In these scenarios, there are a large number of terminal devices and data is widely distributed. Federated learning can achieve efficient use of data and continuous optimization of models.

[0040] It can be understood that the present invention makes full use of cloud-edge resources to release the capabilities of large and small models: large-scale models are trained in the cloud, while edge servers and terminal devices train smaller models suitable for local scenarios, and through the federated learning mechanism, model collaborative optimization between nodes at different levels is achieved. Data is available but invisible, ensuring data privacy and security: edge-side customer private data does not need to leave the domain, and the cloud-side shared model capabilities achieve the effect of data availability and invisibility, ensuring data privacy and security. The present invention proposes an efficient, flexible and scalable model training and optimization method that can make full use of cloud-edge computing resources, protect user privacy, and continuously optimize the global large model.

[0041] See also Figure 2 The present invention provides another embodiment, which provides a cloud-edge size model collaboration system based on federated learning. The cloud-edge size model collaboration system based on federated learning includes: The acquisition module 100 is used to determine the nodes participating in federated learning, including cloud servers and edge devices, and configure the computing resources and communication capabilities of the nodes; Control module 200 is used to create a virtual object of the edge device on the cloud side; to control the cloud server to distribute the initial updated small model to each participating node, and each node updates the edge small model and performs local training based on local data and computing resources; to control each node to perform model training and application optimization based on local original data, and then send the updated model parameters to the cloud server; to control the cloud server to generate a new fused large model for the parameters of the small model received from each terminal device, using a federated training mode that combines horizontal and vertical links and a dynamic multi-task data analysis method; and to distribute the new fused large model to each node participating in federated learning again through model pruning and distillation slimming compression technology for the next round of training.

[0042] It should be noted here that the present invention makes full use of cloud-edge resources to release the capabilities of large and small models: large-scale models are trained on the cloud, while edge servers and terminal devices train smaller models suitable for local scenarios, and through the federated learning mechanism, model collaborative optimization between nodes at different levels is achieved. Data is available but invisible, ensuring data privacy and security: edge-side customer private data does not need to leave the domain, and the cloud-side shared model capabilities achieve the effect of data availability and invisibility, ensuring data privacy and security. The present invention proposes an efficient, flexible and scalable model training and optimization method that can make full use of cloud-edge computing resources, protect user privacy, and continuously optimize the global large model.

[0043] In a preferred embodiment, the present application further provides an electronic device, the electronic device comprising: A memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the cloud-edge size model collaboration method based on federated learning is implemented. The computer device can be broadly a server, a terminal, or any other electronic device with necessary computing and / or processing capabilities. In one embodiment, the computer device may include a processor, a memory, a network interface, a communication interface, etc. connected via a system bus. The processor of the computer device can be used to provide necessary computing, processing and / or control capabilities. The memory of the computer device may include a non-volatile storage medium and an internal memory. An operating system, a computer program, etc. may be stored in or on the non-volatile storage medium. The internal memory may provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the computer device can be used to connect and communicate with external devices through a network. When the computer program is executed by the processor, the steps of the method of the present invention are performed.

[0044] The present invention may be implemented as a computer-readable storage medium having a computer program stored thereon, which causes the steps of the method of an embodiment of the present invention to be executed when executed by a processor. In one embodiment, the computer program is distributed on a plurality of computer devices or processors coupled to a network so that the computer program is stored, accessed and executed in a distributed manner by one or more computer devices or processors. A single method step / operation, or two or more method steps / operations, may be performed by a single computer device or processor or by two or more computer devices or processors. One or more method steps / operations may be performed by one or more computer devices or processors, and one or more other method steps / operations may be performed by one or more other computer devices or processors. One or more computer devices or processors may perform a single method step / operation, or perform two or more method steps / operations.

[0045] It will be appreciated by a person skilled in the art that the method steps of the present invention may be performed by instructing related hardware such as a computer device or a processor through a computer program, and the computer program may be stored in a non-temporary computer-readable storage medium, which causes the steps of the present invention to be performed when the computer program is executed. Depending on the circumstances, any reference to memory, storage, database, or other media herein may include non-volatile and / or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0046] It can be understood that the present invention makes full use of cloud-edge resources to release the capabilities of large and small models: large-scale models are trained in the cloud, while edge servers and terminal devices train smaller models suitable for local scenarios, and through the federated learning mechanism, model collaborative optimization between nodes at different levels is achieved. Data is available but invisible, ensuring data privacy and security: edge-side customer private data does not need to leave the domain, and the cloud-side shared model capabilities achieve the effect of data availability and invisibility, ensuring data privacy and security. The present invention proposes an efficient, flexible and scalable model training and optimization method that can make full use of cloud-edge computing resources, protect user privacy, and continuously optimize the global large model.

[0047] The various technical features described above can be combined arbitrarily. Although all possible combinations of these technical features are not described, any combination of these technical features should be considered to be covered by this specification as long as there is no contradiction in such combination.

[0048] The specific implementation of the present invention described above does not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A cloud-edge large-scale model collaboration method based on federated learning, characterized in that: The method comprises: S100, determine the nodes participating in federated learning, including cloud servers and edge devices, configure the computing resources and communication capabilities of the nodes, and create virtual objects of the edge devices on the cloud side; S200, the cloud server distributes the initial updated small model to each participating node, and each node updates the edge small model and performs local training based on local data and computing resources; S300, each node performs model training and optimization based on local original data, and sends the updated model parameters to the cloud server; S400, the cloud server generates a new fusion large model by adopting a horizontal and vertical combined federated training mode and a dynamic multi-task data analysis method for the parameters of the small models received from each terminal device; S500 and the new fusion large model are distributed again to each node participating in federated learning through model pruning and distillation slimming and compression technology for the next round of training.

2. The cloud-edge large and small model collaboration method based on federated learning according to claim 1 is characterized in that: The horizontal and vertical combined federated training mode includes: When the parameters of each edge model are consistent, the horizontal expansion method is used to integrate the parameters; When the parameters of each edge model are inconsistent, the longitudinal modeling method is used to integrate the parameters.

3. The cloud-edge large and small model collaboration method based on federated learning according to claim 1 is characterized in that: The dynamic multi-task data analysis method comprises: In the multi-task learning framework, task weights, model structures or training strategies are dynamically adjusted according to various indicators or conditions during the training process. The dynamic adjustment of training strategies includes: Dynamic weight adjustment dynamically adjusts the weight of each task according to the performance of different tasks during the training process.

4. The cloud-edge large and small model collaboration method based on federated learning according to claim 3 is characterized in that: The weight of each task is dynamically adjusted according to the performance of different tasks during the training process, including: When the performance of a task on the validation set begins to decline, reduce its weight; or use an uncertainty-based weighting strategy to adjust the weight according to the uncertainty of the correlation between tasks.

5. The cloud-edge large and small model collaboration method based on federated learning according to claim 3 is characterized in that: The dynamically adjusting model structure includes: During the training process, the structure of the model is dynamically adjusted according to the correlation between tasks or changes in model performance, including adding or deleting layers, or changing the connection mode between layers. The change in the connection mode between layers includes changing the connection between neurons in each layer and adjacent layers, and reducing connections through convolution, recurrent neural networks, and attention mechanisms.

6. The cloud-edge large and small model collaboration method based on federated learning according to claim 5, characterized in that: The method further comprises: Dynamically adjust the size and complexity of the model based on the computing resources and training requirements of each node.

7. The cloud-edge large and small model collaboration method based on federated learning according to claim 6, characterized in that: The method further comprises: Collect relevant data of the CPU, GPU, memory, and disk of each node in real time, and select the usage ratio of each resource as the standard based on industry best practices; secondly, optimize resource usage strategies, such as optimizing memory management strategies, using memory reuse and delayed release technologies to reduce memory usage; finally, use data pipeline technology to parallelize data reading, preprocessing, and training processes.

8. The cloud-edge large and small model collaboration method based on federated learning according to claim 7, characterized in that: The usage ratios of various resources are selected as standards based on industry best practices, including: Select 80% as the standard.

9. A cloud-edge large-scale model collaboration system based on federated learning, characterized in that: include: The acquisition module is used to determine the nodes participating in federated learning, including cloud servers and edge devices, and configure the computing resources and communication capabilities of the nodes; The control module is used to create virtual objects of edge devices on the cloud side; it is used to control the cloud server to distribute the initial updated small model to each participating node, and each node updates the small edge model and performs local training based on local data and computing resources; It is used to control each node to perform model training and application optimization based on local original data, and then send the updated model parameters to the cloud server; it is used to control the cloud server to generate a new fused large model for the parameters of the small models received from each terminal device by adopting a horizontal and vertical combined federated training mode and a dynamic multi-task data analysis method; it is used to distribute the new fused large model to each node participating in federated learning again through model pruning and distillation slimming compression technology for the next round of training.

10. An electronic device, characterized in that: include: Memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the cloud-edge size model collaboration method based on federated learning according to any one of claims 1 to 8 is implemented.

Images