Data management method and system for federated cloud

Abstract

The application relates to a data management method and system for a federal cloud, which comprises the following steps: obtaining an identity authentication request and performing identity authentication on a client, granting the client data management authority after the identity authentication of the client is passed, and establishing communication with the client. An updated local model is received and weighted to generate a new global model. The local model is obtained by performing local model training on local data samples by the client and exchanging parameters between different local models to obtain the updated local model. Based on the new global model and the data management authority, a first request instruction is received through the CSP, and the execution authority of the client on the first request instruction is confirmed, and the first request instruction is executed according to the execution authority, and the first request instruction is used to update the server data. A data update request is sent, and the data update request is used to instruct the client to update the data to realize data synchronization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, in particular to a data management method and system for federated cloud. BACKGROUND

[0002] With the explosive growth of modern data, big data has gradually become an important figure in the field of modern technology. Massive data is generated in modern society at all times, and the important thing is to extract effective information from the data and make analysis and intelligent decisions. Among them, data management is the most difficult problem to solve. At the same time, cloud computing is closely related to the communication field because of its service type, and can be expanded to the database as a service or storage, and is the basis for big data. Different cloud environments also stop different delivery data models, such as SaaS, PaaS and IaaS, which provide relationships with software, platforms and infrastructure, among which SaaS provides enable services by providing a large number of integrated features, PaaS enables developers to build applications on the platform, and IaaS has scalability. As a new technology in recent years, federated learning learns a global model by aggregating locally trained models on data generating clients, and can significantly reduce privacy and security risks, providing a new solution for managing and operating data in a collaborative environment.

[0003] In recent years, there have been many attempts to manage big data in a cloud environment, which can be divided into three categories: (1) By proposing a robust reputation management mechanism, encourage cloud providers CPs in the federated cloud to grant resources to users by distinguishing them as good users and malicious users, to achieve resource separation. (2) By proposing a memory data management system to unify online data query and data analysis functions, allowing low-latency storage services and efficient on-site data analysis. (3) Through the implementation of an information flow control model, support application management, and cooperate with middleware. As a new technology that has recently emerged, federated learning also has many application scenarios, such as using federated database computing to calculate the covariance matrix, distributing training data on mobile devices, and learning a shared model by aggregating local computed updates. However, due to the limitations of the scene, the existing method for managing big data in the cloud environment increases the number of communications, thereby reducing the efficiency of data processing and increasing the user's waiting time.

[0004] Therefore, the general method for managing big data in the cloud environment has a large number of communications and low data processing efficiency. SUMMARY

[0005] Therefore, it is necessary to provide a data management method and system for federated cloud, which can combine the existing cloud model with a joint learning scene and utilize the basic capabilities provided by CSP to enhance core capability construction, reduce the required number of communications, and improve the efficiency of data processing.

[0006] Firstly, this application provides a data management method for a federated cloud, applied on a server side, the method comprising:

[0007] Obtain an authentication request and authenticate the client. After the client is successfully authenticated, grant the client data management permissions and establish communication with the client.

[0008] The updated local model is received and weighted to generate a new global model. The local model is trained on local data samples by the client and the parameters between different local models are exchanged to obtain the updated local model.

[0009] Based on the new global model and data management permissions, the CSP receives the first request instruction and confirms the client's execution permission for the first request instruction, and executes the first request instruction according to the execution permission. The first request instruction is used to update the server-side data.

[0010] Send a data update request, which is used to instruct the client to update the data in order to achieve data synchronization.

[0011] In one embodiment, obtaining the authentication request and authenticating the client includes:

[0012] A first key is generated through CSP and the client's usage scenarios are learned. The first key is used to authenticate the client.

[0013] The client is granted a corresponding data management level based on the authentication result, and the data management level is used to determine the scope of server-side data updates;

[0014] When the number of times the client establishes communication with the server exceeds a first threshold, the client is quickly authenticated and navigated to the target file.

[0015] In one embodiment, receiving the updated local model and weighting the local model to generate a new global model includes:

[0016] The global model is sent to the client using the FedAvg algorithm, and the local model in the client is updated to the shared model.

[0017] The shared model is used to train SGD on the client's local data, and the trained local model is obtained.

[0018] The new global model is generated by calculating the weighted sum of the trained local models.

[0019] In one embodiment, the method further includes:

[0020] The local model from the client is received and processed using the CO-OP algorithm. The local model is combined when the age difference between the client's local model and the global model is not less than the lower age limit and not greater than the upper age limit.

[0021] Secondly, this application provides a data management method for a federated cloud, applied to a client, the method comprising:

[0022] An authentication request is sent to the server. After successful authentication, data management permissions are granted, and communication is established with the server.

[0023] Local models are trained on local data samples, and parameters between different local models are exchanged to obtain updated local models. The updated local models are then sent to the server, which weights the updated local models to generate new global models.

[0024] Based on the new global model and data management permissions, a first request instruction is sent through the CSP and the server is instructed to execute the first request instruction according to the execution permission of the first request instruction. The first request instruction is used to update the server data.

[0025] The system receives data update requests and updates the data to achieve data synchronization between the client and the server.

[0026] In one embodiment, the updated local model is local data from a client trained using SGD with a shared model, where the shared model is obtained by updating the global model on the server using the FedAvg algorithm. The method further includes:

[0027] Receive the first key generated by the CSP and send the authentication request according to the use case;

[0028] Based on the authentication result, a corresponding data management level is granted, which is used to determine the scope of server-side data updates;

[0029] When the number of times the client establishes communication with the server exceeds a first threshold, the client will be navigated to the target file after completing identity authentication.

[0030] The local model is sent to the server for CO-OP algorithm processing, and the server combines the local model when the age difference between the local model and the global model is not less than the lower limit and not greater than the upper limit of age.

[0031] Thirdly, this application provides a data management system for a federated cloud, applied on the server side, the system comprising:

[0032] The authentication module is used to obtain authentication requests and authenticate the client, grant the client data management permissions after the client is successfully authenticated, and establish communication with the client.

[0033] The first model training module is used to receive the updated local model and weight it to generate a new global model. The local model is trained on local data samples by the client and the parameters between different local models are exchanged to obtain the updated local model.

[0034] The data management module is used to receive a first request instruction through the CSP based on the new global model and data management permissions, confirm the client's execution permission for the first request instruction, and execute the first request instruction according to the execution permission. The first request instruction is used to update the server-side data.

[0035] The first update module is used to send a data update request, which is used to instruct the client to update the data in order to achieve data synchronization.

[0036] Fourthly, this application provides a data management system for a federated cloud, applied to a client, the system comprising:

[0037] The first sending module is used to send an authentication request to the server. After successful authentication, it is granted data management permissions and establishes communication with the server.

[0038] The second model training module is used to train local models on local data samples and exchange parameters between different local models to obtain an updated local model. The updated local model is then sent to the server, where the server weights the updated local model to generate a new global model.

[0039] The second sending module is used to send a first request instruction through CSP based on the new global model and data management permissions, and instruct the server to execute the first request instruction according to the execution permission of the first request instruction. The first request instruction is used to update the server data.

[0040] The second update module is used to receive data update requests and perform data updates to achieve data synchronization between the client and the server.

[0041] Fifthly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0042] Obtain an authentication request and authenticate the client. After the client is successfully authenticated, grant the client data management permissions and establish communication with the client.

[0043] The updated local model is received and weighted to generate a new global model. The local model is trained on local data samples by the client and the parameters between different local models are exchanged to obtain the updated local model.

[0044] Based on the new global model and data management permissions, the CSP receives the first request instruction and confirms the client's execution permission for the first request instruction, and executes the first request instruction according to the execution permission. The first request instruction is used to update the server-side data.

[0045] Send a data update request, the data update request being used to instruct the client to update data to achieve data synchronization; or

[0046] An authentication request is sent to the server. After successful authentication, data management permissions are granted, and communication is established with the server.

[0047] Local models are trained on local data samples, and parameters between different local models are exchanged to obtain updated local models. The updated local models are then sent to the server, which weights the updated local models to generate new global models.

[0048] Based on the new global model and data management permissions, a first request instruction is sent through the CSP and the server is instructed to execute the first request instruction according to the execution permission of the first request instruction. The first request instruction is used to update the server data.

[0049] The system receives data update requests and updates the data to achieve data synchronization between the client and the server.

[0050] Sixthly, this application provides a computer storage medium storing a computer program, which, when executed by a processor, performs the following steps:

[0051] Obtain an authentication request and authenticate the client. After the client is successfully authenticated, grant the client data management permissions and establish communication with the client.

[0052] The updated local model is received and weighted to generate a new global model. The local model is trained on local data samples by the client and the parameters between different local models are exchanged to obtain the updated local model.

[0053] Based on the new global model and data management permissions, the CSP receives the first request instruction and confirms the client's execution permission for the first request instruction, and executes the first request instruction according to the execution permission. The first request instruction is used to update the server-side data.

[0054] Send a data update request, the data update request being used to instruct the client to update data to achieve data synchronization; or

[0055] An authentication request is sent to the server. After successful authentication, data management permissions are granted, and communication is established with the server.

[0056] Local models are trained on local data samples, and parameters between different local models are exchanged to obtain updated local models. The updated local models are then sent to the server, which weights the updated local models to generate new global models.

[0057] Based on the new global model and data management permissions, a first request instruction is sent through the CSP and the server is instructed to execute the first request instruction according to the execution permission of the first request instruction. The first request instruction is used to update the server data.

[0058] The system receives data update requests and updates the data to achieve data synchronization between the client and the server.

[0059] The aforementioned data management method and system for federated clouds obtains authentication requests from the server and authenticates the client. After successful client authentication, data management permissions are granted to the client, and communication is established. The client updates its local model by training a local model on local data samples and exchanging parameters between the local models. The updated local model is then sent to the server, where it is weighted to generate a new global model. Subsequently, based on the new global model and data management permissions, the CSP receives data update instructions and confirms the client's execution permission for these instructions. The corresponding data update instructions are then executed according to the execution permission to update the server-side data. After the server-side data is updated, it sends data update requests to the clients to ensure that the data of other clients remains synchronized with the updated server-side data. This method proposes an innovative federated learning model, combining all cloud models with a joint learning scenario and other technologies that can be integrated, reducing the number of communications required and providing users with a more efficient and energy-saving system architecture and less waiting time, thereby achieving more efficient data management. Attached Figure Description

[0060] Figure 1 This is a flowchart of a data management method for a federated cloud according to one embodiment of this application;

[0061] Figure 2 This is a flowchart of a data management method for a federated cloud according to one embodiment of this application;

[0062] Figure 3 This is a flowchart of a data management method for a federated cloud according to one embodiment of this application;

[0063] Figure 4 This is a flowchart of a data management method for a federated cloud according to one embodiment of this application;

[0064] Figure 5 This is a schematic diagram of the system architecture of a data management scheme for a federated cloud according to one embodiment of this application;

[0065] Figure 6 This is a flowchart of the server-side data management process for the data management solution used in the federated cloud in this embodiment;

[0066] Figure 7 This is a flowchart illustrating the client-side data management process for the data management solution used in the federated cloud in this embodiment.

[0067] Figure 8 This is a schematic diagram of the data management system architecture for a federated cloud according to one embodiment of this application;

[0068] Figure 9This is a schematic diagram of the data management system architecture for a federated cloud according to one embodiment of this application;

[0069] Figure 10 This is a schematic diagram of the data management system architecture for a federated cloud according to one embodiment of this application;

[0070] Figure 11 This is a schematic diagram of the data management system architecture for a federated cloud according to one embodiment of this application;

[0071] Figure 12 This is an internal structural diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] like Figure 1 As shown, in one embodiment, a data management method for a federated cloud, applied to the server, includes the following steps:

[0074] Step S110: Obtain the authentication request and authenticate the client. After the client is successfully authenticated, grant the client data management permissions and establish communication with the client.

[0075] Specifically, the server receives the authentication request from the client and authenticates the client. After the client is successfully authenticated, the server grants the client data management permissions, enabling the client to establish communication with the server and manage the server's data.

[0076] Step S120: Receive the updated local model and weight it to generate a new global model. The local model is trained on local data samples by the client and the parameters between different local models are exchanged to obtain the updated local model.

[0077] Specifically, the server receives the updated local model from the client and weights it to generate a new global model. The updated local model is obtained by the client training local models on local data samples using its local model and exchanging parameters between different local models.

[0078] Step S130: Based on the new global model and data management permissions, the CSP receives the first request instruction and confirms the client's execution permission for the first request instruction, and executes the first request instruction according to the execution permission. The first request instruction is used to update the server-side data.

[0079] Specifically, based on the new global model generated in step S120 and the data management permissions obtained in step S110, the server receives the task request instruction from the client through the CSP, confirms the execution permission of the task request instruction, and executes the task request instruction according to the execution permission of the task request instruction to update the server data with the corresponding permissions.

[0080] Step S140: Send a data update request. The data update request is used to instruct the client to update the data in order to achieve data synchronization.

[0081] Specifically, when the server updates its data, it sends a data update request to other clients, enabling the client's data to be synchronized with the server's data.

[0082] The aforementioned data management method for federated clouds involves the server obtaining an authentication request and authenticating the client. After successful client authentication, data management permissions are granted, and communication is established. The client updates its local model by training a local model on local data samples and exchanging parameters between these local models. The updated local model is then sent to the server, which weights the local model to generate a new global model. Subsequently, based on the new global model and data management permissions, the CSP receives data update instructions, confirms the client's execution permission for these instructions, and executes the corresponding data update instructions to update the server-side data. After the server-side data update, it sends data update requests to the clients to ensure that other clients' data remains synchronized with the updated server-side data. This method proposes an innovative federated learning model, combining all cloud models with a joint learning scenario and other technologies that can be integrated, reducing the number of communications required and providing users with a more efficient and energy-saving system architecture and less waiting time, thereby achieving more efficient data management.

[0083] like Figure 2 As shown, in one embodiment, a data management method for a federated cloud, applied to the server, includes the following steps:

[0084] Step S210: Generate a first key through CSP and learn the client's usage scenarios. The first key is used to authenticate the client.

[0085] Specifically, the server first generates a key through CSP and learns the client's usage scenarios. The generated key is used to authenticate the client.

[0086] Step S220: Obtain the authentication request and authenticate the client. After the client is successfully authenticated, grant the client data management permissions and establish communication with the client.

[0087] Specifically, the server obtains the authentication request from the client and authenticates the client using the key generated in step S210. After the client is successfully authenticated, the server grants the client data management permissions, enabling the client to establish communication with the server and manage the server's data.

[0088] Step S230: Grant the client the corresponding data management level based on the authentication result. The data management level is used to determine the scope of data updates on the server.

[0089] Specifically, the server grants the client a corresponding data management level based on the client's authentication result in step S220. This data management level determines the scope of the client's data updates to the server.

[0090] Step S240: When the number of times the client establishes communication with the server exceeds the first threshold, perform quick authentication on the client and navigate the client to the target file.

[0091] Specifically, when a client establishes communication with the server and accesses the same file a certain number of times, the server will quickly authenticate the client and navigate to its frequently used target file when authenticating it again.

[0092] Step S250: Receive the updated local model and weight the local model to generate a new global model. The local model is trained on local data samples by the client and the parameters between different local models are exchanged to obtain the updated local model.

[0093] Specifically, the server sends the original global model to the client using the FedAvg algorithm, updates the client's local model to the shared model, then uses the shared model to train the client's local data using SGD to obtain the trained local model, and finally generates a new global model by calculating the weighted sum of the trained local models.

[0094] It should be noted that the server processes the local model received from the client using the CO-OP algorithm, and combines the local model when the age difference between the local model and the global model is not less than the lower age limit and not greater than the upper age limit.

[0095] Step S260: Based on the new global model and data management permissions, the CSP receives the first request instruction and confirms the client's execution permission for the first request instruction, and executes the first request instruction according to the execution permission. The first request instruction is used to update the server-side data.

[0096] Specifically, based on the new global model and data management permissions obtained, the server receives the task request instruction through the CSP and confirms the client's execution permission for the task request instruction and the client's data management level. Based on its data management level and execution permission for the task request instruction, the server executes the task request instruction and updates the server data accordingly.

[0097] Step S270: Send a data update request. The data update request is used to instruct the client to update the data in order to achieve data synchronization.

[0098] Specifically, when the server updates its data, it sends a data update request to other clients, enabling the client's data to be synchronized with the server's data.

[0099] The aforementioned data management method for federated clouds involves the server obtaining an authentication request and authenticating the client. After successful client authentication, data management permissions are granted, and communication is established. The client updates its local model by training a local model on local data samples and exchanging parameters between these local models. The updated local model is then sent to the server, which weights the local model to generate a new global model. Subsequently, based on the new global model and data management permissions, the CSP receives data update instructions, confirms the client's execution permission for these instructions, and executes the corresponding data update instructions to update the server-side data. After the server-side data update, it sends data update requests to the clients to ensure that other clients' data remains synchronized with the updated server-side data. This method proposes an innovative federated learning model, combining all cloud models with a joint learning scenario and other technologies that can be integrated, reducing the number of communications required and providing users with a more efficient and energy-saving system architecture and less waiting time, thereby achieving more efficient data management.

[0100] like Figure 3 As shown, in one embodiment, a data management method for a federated cloud, applied to a client, includes the following steps:

[0101] Step S310: Send an authentication request to the server. After successful authentication, data management permissions are granted, and communication is established with the server.

[0102] Specifically, the client sends an authentication request to the server and is granted data management permissions after successful authentication, enabling the client to establish communication with the server and manage the server's data.

[0103] Step S320: Train a local model on the local data samples and exchange the parameters between different local models to obtain an updated local model. Send the updated local model to the server and weight the updated local model to generate a new global model.

[0104] Specifically, based on the communication established between the client and the server, the client updates its local model by training a local model on local data and exchanging parameters between different local models, and sends the updated local model to the server. Finally, the server weights the updated local model to form a new global model.

[0105] Step S330: Based on the new global model and data management permissions, a first request instruction is sent through the CSP and the server is instructed to execute the first request instruction according to the execution permission of the first request instruction. The first request instruction is used to update the server data.

[0106] Specifically, based on the new global model and its acquired data management permissions, the client sends a task request instruction through the CSP and instructs the server to execute the corresponding task request instruction according to the execution permissions of the task request instruction, so as to update the server data.

[0107] Step S340: Receive a data update request and update the data to achieve data synchronization between the client and the server.

[0108] Specifically, the client receives a data update request after the server has completed its data update, so as to enable other clients to synchronize their data with the server.

[0109] The aforementioned data management method for federated clouds involves the server obtaining an authentication request and authenticating the client. After successful client authentication, data management permissions are granted, and communication is established. The client updates its local model by training a local model on local data samples and exchanging parameters between these local models. The updated local model is then sent to the server, which weights the local model to generate a new global model. Subsequently, based on the new global model and data management permissions, the CSP receives data update instructions, confirms the client's execution permission for these instructions, and executes the corresponding data update instructions to update the server-side data. After the server-side data update, it sends data update requests to the clients to ensure that other clients' data remains synchronized with the updated server-side data. This method proposes an innovative federated learning model, combining all cloud models with a joint learning scenario and other technologies that can be integrated, reducing the number of communications required and providing users with a more efficient and energy-saving system architecture and less waiting time, thereby achieving more efficient data management.

[0110] like Figure 4 As shown, in one embodiment, a data management method for a federated cloud, applied to a client, includes the following steps:

[0111] Step S410: Send an authentication request to the server. After successful authentication, data management permissions are granted, and communication is established with the server.

[0112] Specifically, the client sends an authentication request to the server and receives a key generated by the CSP to authenticate the client. After successful client authentication, the client is granted data management permissions by the server, enabling the client to establish communication with the server.

[0113] Step S420: Based on the authentication result, the corresponding data management level is granted. The data management level is used to determine the scope of server-side data updates.

[0114] Specifically, the client is granted a corresponding data management level based on the server's authentication result, enabling the client to manage updates to server data across different ranges.

[0115] In step S430, when the number of times the client establishes communication with the server exceeds the first threshold, the client is navigated to the target file after completing identity authentication.

[0116] Specifically, when the number of times a client establishes communication with the server and accesses the same file reaches a certain number, the client will be navigated to the frequently used target file after completing authentication.

[0117] Step S440: Train a local model on the local data samples and exchange parameters between different local models to obtain an updated local model. Send the updated local model to the server and weight the updated local model to generate a new global model.

[0118] Specifically, based on the communication established between the client and the server, the client updates its local model by training a local model on local data and exchanging parameters between different local models. The updated local model is then sent to the server, which processes the original global model using the FedAvg algorithm. The client receives the global model processed by the FedAvg algorithm and updates its local model into a shared model. Subsequently, the client's local data is trained using the shared model to obtain a trained local model. Finally, the server weights the trained local model to generate a new global model.

[0119] It should be noted that the server processes the local model received from the client using the CO-OP algorithm, and combines the local model when the age difference between the local model and the global model is not less than the lower age limit and not greater than the upper age limit.

[0120] Step S450: Based on the new global model and data management permissions, a first request instruction is sent through the CSP and the server is instructed to execute the first request instruction according to the execution permission of the first request instruction. The first request instruction is used to update the server data.

[0121] Specifically, based on the new global model and its acquired data management permissions, the client sends a task request instruction through the CSP and instructs the server to execute the corresponding task request instruction according to the execution permissions of the task request instruction, so as to update the server data.

[0122] Step S460: Receive a data update request and update the data to achieve data synchronization between the client and the server.

[0123] Specifically, the client receives a data update request after the server has completed its data update, so as to enable other clients to synchronize their data with the server.

[0124] The aforementioned data management method for federated clouds involves the server obtaining an authentication request and authenticating the client. After successful client authentication, data management permissions are granted, and communication is established. The client updates its local model by training a local model on local data samples and exchanging parameters between these local models. The updated local model is then sent to the server, which weights the local model to generate a new global model. Subsequently, based on the new global model and data management permissions, the CSP receives data update instructions, confirms the client's execution permission for these instructions, and executes the corresponding data update instructions to update the server-side data. After the server-side data update, it sends data update requests to the clients to ensure that other clients' data remains synchronized with the updated server-side data. This method proposes an innovative federated learning model, combining all cloud models with a joint learning scenario and other technologies that can be integrated, reducing the number of communications required and providing users with a more efficient and energy-saving system architecture and less waiting time, thereby achieving more efficient data management.

[0125] like Figures 5 to 7 As shown in one embodiment, a data management scheme for federated clouds proposes an innovative model of federated learning. This model combines all existing cloud models with a joint learning scenario, providing users with a more energy-efficient system architecture and reduced latency. It fills a gap in federated systems by combining existing common scenarios to obtain a new federated scenario. The model can also be trained locally or globally on a server, depending on the situation. The core objective of this scheme is to implement a flexible data management system built on a server and running on the provider's cloud service provider. The main purpose is to set up a collaborative system, a federated scenario, where collaborating CSPs help balance the load of data management and transmission by handling the user authentication load.

[0126] In this embodiment, users can access data through a cloud service provider (CSP), which is authorized to access the organization's servers and achieve instant and faster data management through the cloud platform. User authentication is divided into two parts: (1) a unique key is first provided by the CSP with a key generation system until the connection is established; (2) authentication is performed by the server, and the user's permission level is granted. Based on the authentication result on the CSP platform, the user obtains the corresponding data management level. The entire system is built on a federated learning system among multiple CSPs, implementing an energy and computational efficiency scenario for a single server. It generates a broad model by training distorted local models on local data samples and exchanging parameters between different local models at a certain frequency.

[0127] The entire system model receives various CSP request traffic from users through a central server and supports communication between CSPs and the server. After a secure and fast authentication method, users can select the most suitable CSP to perform the desired task and develop a machine learning system through communication between the CSP and the server. The system will utilize resources provided by CSPs in a PaaS format to process user requests better and faster, and achieve more efficient management in a federated cloud environment.

[0128] Therefore, users can communicate more directly with the server through the secure environment provided by the CSP. Suppose this user is a client k that makes multiple contacts with the server for a specific folder containing different types of data n. The collaborating CSP can learn this user's context to immediately initiate the authentication process and navigate the user to their most frequently used files. This learning approach can be built at the edge of communication between each client and the collaborating CSP, allowing academic servers to reduce the user's computational burden and focus on important research processes. Simultaneously, the system can also have features to skip simple authentication steps.

[0129] In this embodiment, taking the deletion of data on an academic server through access from multiple CSPs as an example, the CSP confirms to the user that it has secure access rights and ownership of the content, and then deletes the data requested by the user. Next, the user's data permissions are executed. If other permissions exist, the user sends an η update request through the CSP to users with other permissions accessing the data or file, indicating that these data or files will be deleted. After confirmation, the data deletion continues. Then, the data management system checks if there are any copies of the data or file. If they are also requested to be deleted, the CSP sends user information indicating that copies exist elsewhere in the file system, and after receiving the user's approval, the file deletion continues.

[0130] In real-world work scenarios, synchronous and asynchronous federated learning algorithms are required. After comparing algorithms in the same direction, the FedAvg algorithm and the CO-OP algorithm are selected.

[0131] The synchronization algorithm uses a federated averaging algorithm, which coordinates training through a central server that hosts a shared global model w. t , where t is the number of communication rounds. Parameters B, E, η, and λ are commonly used parameters during training, where E represents the total number of iterations through local data before global model updates. Specifically, the communication process of the FedAvg algorithm is as follows: The server selects a subset S of clients. t And distribute the current global model to S tAll clients within the shared model are processed. After updating their local models to the shared model, each client divides its local data into batches of size B and performs SGD training. Finally, the client updates its trained local model... The data is uploaded to a central server, which then calculates a weighted average of all received local models to generate a new global model W. t+1 The weighting scheme depends on the number of local training samples, and the core formula is as follows:

[0132]

[0133] The specific algorithm is as follows:

[0134] Server-side operations:

[0135] initw o

[0136] for each round t

[0137] m←max([C·K],1)

[0138] S t =randomsetofmclients

[0139] for each clientin paralleldo

[0140]

[0141] Client operations:

[0142] B←P k

[0143] for each localepochdo

[0144] forb∈Bdo

[0145] w←w-ηζ(w;b)

[0146] return w

[0147] In this context, the K client uses k as the index, B is the local mini-batch size, E is the number of local batches, and η is the learning rate.

[0148] The asynchronous algorithm uses the CO-OP algorithm, which can immediately combine any received client model with the global model. Each client k has an age z associated with its model. k The global model has an age z, and the model age difference zz. kThese are used to calculate the weights when combining models. In an asynchronous framework, some clients train using outdated models, while others train using updated models. In the CO-OP algorithm, for some integer b... l <b u The choice is only when b l ≤zz k ≤b u Only when local models can be combined, lower bound b l and upper limit b u It can be considered an age filter.

[0149] The training process of the CO-OP algorithm is as follows:

[0150] w = w1 = ... = w K ←w0

[0151] z←b l

[0152] z1=…=z K ←0

[0153] Each client executes k independently.

[0154] while true do

[0155] w k ←ClientUpdate(w k )

[0156] ifz-z k <b u then

[0157] w k ←w

[0158] z k ←z

[0159] else ifz-z k <b k then

[0160] continue

[0161] else

[0162] w k ,z k ←UpdateServer(w k ,z k )

[0163] Each client has its own training data and requests the current global model age z and optimization algorithm from the server. The client determines whether its age change meets the constraints. If the local model is outdated, the client will coordinate with the global model and start over; otherwise, the client continues training, but the local models are merged after being uploaded to the server.

[0164] By combining the different capabilities of the FedAvg and CO-OP algorithms, a more efficient model can be obtained. Whether the model is trained locally on the client or as a global model on the server, the choice will be made based on the situation. The specific computation process is as follows:

[0165] Client operations:

[0166] B←P k

[0167] for each localido

[0168] ify-y k <x u then

[0169] w k ←w

[0170] y k ←y

[0171] for batch b∈Bdo

[0172] w←w-ηζ(w;b)

[0173] else

[0174] w k ,y k ←UpdateServer(k,w i )

[0175] return to server w k ,y k

[0176] Server-side operations:

[0177] initw0

[0178] for each round i++

[0179] m←max(S t ,1)

[0180] S i = random set of nc clinets

[0181] Foreash client in parallel do

[0182]

[0183] The algorithm starts with a randomly initialized global model. During the first round of model operation, the central server selects a subset S of clients. i And then distribute the global model at this point to all connected client S i Then, for some integer x i <x u Only when x i ≤yy k ≤x u Local models are merged only at this point. Subsequently, clients update their local models to become a shared model; each client divides its local data into batches of size B and performs local updates. Finally, the clients combine their trained local models into a single batch. k and y k The data is uploaded to a central server, which then generates a new global model w by calculating a weighted sum of all received local models. t+1 The overall weighted scheme depends on the number of local training updates. This algorithm can provide users with less waiting time, and the relevant data will be trained locally or globally based on the priority of each situation.

[0184] The aforementioned data management solution for federated clouds proposes a data management system and an innovative federated learning model. This model combines all cloud models with a joint learning scenario and other technologies that can be integrated, reducing the number of communications required and providing users with a more efficient and energy-saving system architecture and less waiting time, thus achieving more efficient data management. Furthermore, the proposed data management system for federated clouds fills a gap in the field of federated cloud systems by combining the advantages of FedAvg and CO-OP algorithms, enabling models to be trained either locally on the client or globally on the server, depending on the situation.

[0185] like Figure 8 As shown, in one embodiment, a data management system for a federated cloud, applied to the server, includes an authentication module 810, a first model training module 820, a data management module 830, and a first update module 840.

[0186] The authentication module 810 is used to obtain authentication requests and authenticate the client. After the client is successfully authenticated, the module grants the client data management permissions and establishes communication with the client.

[0187] The first model training module 820 is used to receive the updated local model and weight the local model to generate a new global model. The local model is trained on local data samples by the client and the parameters between different local models are exchanged to obtain the updated local model.

[0188] The data management module 830 is used to receive the first request instruction through the CSP based on the new global model and data management permissions, confirm the client's execution permission for the first request instruction, and execute the first request instruction according to the execution permission. The first request instruction is used to update the server-side data.

[0189] The first update module 840 is used to send data update requests, which instruct the client to update the data in order to achieve data synchronization.

[0190] The aforementioned data management system for federated cloud obtains authentication requests from the server and authenticates the client. After successful client authentication, data management permissions are granted to the client, and communication is established. The client then trains a local model on local data samples and exchanges parameters between these local models to update the local model. The updated local model is sent to the server, which weights the local model to generate a new global model. Subsequently, based on the new global model and data management permissions, the CSP receives data update instructions and confirms the client's execution permission for these instructions. The CSP then executes the corresponding data update instructions according to the client's permissions, updating the server-side data. After the server-side data update, it sends data update requests to the clients to ensure that the data on other clients remains synchronized with the updated server-side data. This system proposes an innovative federated learning model, combining all cloud models with a joint learning scenario and other technologies that can be integrated, reducing the number of communications required and providing users with a more efficient and energy-saving system architecture and less waiting time, thereby achieving more efficient data management.

[0191] like Figure 9 As shown, in one embodiment, a data management system for a federated cloud, applied to the server, includes a key generation module 910, an authentication module 920, a permission management module 930, a first navigation module 940, a first model training module 950, a data management module 960, and a first update module 970.

[0192] The key generation module 910 is used to generate a first key through CSP and learn the client's usage scenarios. The first key is used to authenticate the client.

[0193] The authentication module 920 is used to obtain authentication requests and authenticate the client. After the client is successfully authenticated, it grants the client data management permissions and establishes communication with the client.

[0194] The permission management module 930 is used to grant the client the corresponding data management level based on the authentication result. The data management level is used to determine the scope of data updates on the server.

[0195] The first navigation module 940 is used to quickly authenticate the client and navigate the client to the target file when the number of times the client establishes communication with the server exceeds a first threshold.

[0196] The first model training module 950 is used to receive the updated local model and weight the local model to generate a new global model. The local model is trained on local data samples by the client and the parameters between different local models are exchanged to obtain the updated local model.

[0197] The data management module 960 is used to receive the first request instruction through the CSP based on the new global model and data management permissions, confirm the client's execution permission for the first request instruction, and execute the first request instruction according to the execution permission. The first request instruction is used to update the server-side data.

[0198] The first update module 970 is used to send data update requests, which instruct the client to update the data in order to achieve data synchronization.

[0199] The aforementioned data management system for federated cloud obtains authentication requests from the server and authenticates the client. After successful client authentication, data management permissions are granted to the client, and communication is established. The client then trains a local model on local data samples and exchanges parameters between these local models to update the local model. The updated local model is sent to the server, which weights the local model to generate a new global model. Subsequently, based on the new global model and data management permissions, the CSP receives data update instructions and confirms the client's execution permission for these instructions. The CSP then executes the corresponding data update instructions according to the client's permissions, updating the server-side data. After the server-side data update, it sends data update requests to the clients to ensure that the data on other clients remains synchronized with the updated server-side data. This system proposes an innovative federated learning model, combining all cloud models with a joint learning scenario and other technologies that can be integrated, reducing the number of communications required and providing users with a more efficient and energy-saving system architecture and less waiting time, thereby achieving more efficient data management.

[0200] like Figure 10As shown, in one embodiment, a data management system for a federated cloud, applied to a client, includes a first sending module 1010, a second model training module 1020, a second sending module 1030, and a second update module 1040.

[0201] The first sending module 1010 is used to send an authentication request to the server. After successful authentication, it is granted data management permissions and establishes communication with the server.

[0202] The second model training module 1020 is used to train local models on local data samples and exchange parameters between different local models to obtain updated local models. The updated local models are then sent to the server, which weights the updated local models to generate new global models.

[0203] The second sending module 1030 is used to send a first request instruction through CSP based on the new global model and data management permissions, and instruct the server to execute the first request instruction according to the execution permission of the first request instruction. The first request instruction is used to update the server data.

[0204] The second update module 1040 is used to receive data update requests and perform data updates to achieve data synchronization between the client and the server.

[0205] The aforementioned data management system for federated cloud obtains authentication requests from the server and authenticates the client. After successful client authentication, data management permissions are granted to the client, and communication is established. The client then trains a local model on local data samples and exchanges parameters between these local models to update the local model. The updated local model is sent to the server, which weights the local model to generate a new global model. Subsequently, based on the new global model and data management permissions, the CSP receives data update instructions and confirms the client's execution permission for these instructions. The CSP then executes the corresponding data update instructions according to the client's permissions, updating the server-side data. After the server-side data update, it sends data update requests to the clients to ensure that the data on other clients remains synchronized with the updated server-side data. This system proposes an innovative federated learning model, combining all cloud models with a joint learning scenario and other technologies that can be integrated, reducing the number of communications required and providing users with a more efficient and energy-saving system architecture and less waiting time, thereby achieving more efficient data management.

[0206] like Figure 11As shown, in one embodiment, a data management system for a federated cloud, applied to a client, includes a first sending module 1110, an authorization module 1120, a second navigation module 1130, a second model training module 1140, a second sending module 1150, and a second update module 1160.

[0207] The first sending module 1110 is used to send an authentication request to the server. After successful authentication, it is granted data management permissions and establishes communication with the server.

[0208] The permission module 1120 is used to grant the corresponding data management level based on the authentication result. The data management level is used to determine the scope of server-side data updates.

[0209] The second navigation module 1130 is used to navigate to the target file after identity authentication when the number of times the client establishes communication with the server exceeds a first threshold.

[0210] The second model training module 1140 is used to train local models on local data samples and exchange parameters between different local models to obtain updated local models. The updated local models are then sent to the server, which weights the updated local models to generate new global models.

[0211] The second sending module 1150 is used to send a first request instruction through CSP based on the new global model and data management permissions, and instruct the server to execute the first request instruction according to the execution permission of the first request instruction. The first request instruction is used to update the server data.

[0212] The second update module 1160 is used to receive data update requests and perform data updates to achieve data synchronization between the client and the server.

[0213] The aforementioned data management system for federated cloud obtains authentication requests from the server and authenticates the client. After successful client authentication, data management permissions are granted to the client, and communication is established. The client then trains a local model on local data samples and exchanges parameters between these local models to update the local model. The updated local model is sent to the server, which weights the local model to generate a new global model. Subsequently, based on the new global model and data management permissions, the CSP receives data update instructions and confirms the client's execution permission for these instructions. The CSP then executes the corresponding data update instructions according to the client's permissions, updating the server-side data. After the server-side data update, it sends data update requests to the clients to ensure that the data on other clients remains synchronized with the updated server-side data. This system proposes an innovative federated learning model, combining all cloud models with a joint learning scenario and other technologies that can be integrated, reducing the number of communications required and providing users with a more efficient and energy-saving system architecture and less waiting time, thereby achieving more efficient data management.

[0214] In one embodiment, a computer device is provided, which may be a smart terminal, and its internal structure diagram may be as follows: Figure 12 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a data management method for a federated cloud.

[0215] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0216] In one embodiment, a computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0217] In one embodiment, a computer storage medium stores a computer program that, when executed by a processor, implements the steps described in the above method embodiments.

[0218] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0219] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0220] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0221] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A data management method for federated clouds, characterized in that, Applied to the server side, the method includes: Obtain an authentication request and authenticate the client. After the client is successfully authenticated, grant the client data management permissions and establish communication with the client. The updated local model is received and weighted to generate a new global model. The local model is trained on local data samples by the client and the parameters between different local models are exchanged to obtain the updated local model. Based on the new global model and data management permissions, the CSP receives the first request instruction and confirms the client's execution permission for the first request instruction, and executes the first request instruction according to the execution permission. The first request instruction is used to update the server-side data. Send a data update request, which instructs the client to update the data to achieve data synchronization; The step of obtaining the authentication request and authenticating the client includes: A first key is generated through CSP and the client's usage scenarios are learned. The first key is used to authenticate the client. The client is granted a corresponding data management level based on the authentication result, and the data management level is used to determine the scope of server-side data updates; When the number of times the client establishes communication with the server exceeds a first threshold, the client is quickly authenticated and navigated to the target file; The method further includes: The local model from the client is received and processed using the CO-OP algorithm. The local model is combined when the age difference between the client's local model and the global model is not less than the lower age limit and not greater than the upper age limit.

2. The data management method for federated cloud according to claim 1, characterized in that, The step of receiving the updated local model and weighting the local model to generate a new global model includes: The global model is sent to the client using the FedAvg algorithm, and the local model in the client is updated to the shared model. The shared model is used to train SGD on the client's local data, and the trained local model is obtained. The new global model is generated by calculating the weighted sum of the trained local models.

3. A data management method for federated clouds, characterized in that, Applied to a client, the method includes: An authentication request is sent to the server. After successful authentication, data management permissions are granted, and communication is established with the server. Local models are trained on local data samples, and parameters between different local models are exchanged to obtain updated local models. The updated local models are then sent to the server, which weights the updated local models to generate new global models. Based on the new global model and data management permissions, a first request instruction is sent through the CSP and the server is instructed to execute the first request instruction according to the execution permission of the first request instruction. The first request instruction is used to update the server data. Receive data update requests and update the data to achieve data synchronization between the client and the server; The updated local model is the client's local data trained using SGD with a shared model, and the shared model is obtained by updating the global model on the server using the FedAvg algorithm. The method further includes: Receive the first key generated by the CSP and send the authentication request according to the use case; Based on the authentication result, a corresponding data management level is granted, which is used to determine the scope of server-side data updates; When the number of times the client establishes communication with the server exceeds a first threshold, the client will be navigated to the target file after completing identity authentication. The local model is sent to the server for CO-OP algorithm processing, and the server combines the local model when the age difference between the local model and the global model is not less than the lower limit and not greater than the upper limit of age.

4. A data management system for a federated cloud, characterized in that, The system, applied to the server side, includes: The authentication module is used to obtain authentication requests and authenticate the client, grant the client data management permissions after the client is successfully authenticated, and establish communication with the client. The first model training module is used to receive the updated local model and weight it to generate a new global model. The local model is trained on local data samples by the client and the parameters between different local models are exchanged to obtain the updated local model. The data management module is used to receive a first request instruction through the CSP based on the new global model and data management permissions, confirm the client's execution permission for the first request instruction, and execute the first request instruction according to the execution permission. The first request instruction is used to update the server-side data. The first update module is used to send a data update request, which is used to instruct the client to update the data in order to achieve data synchronization. This includes obtaining the authentication request and authenticating the client, including: A first key is generated through CSP and the client's usage scenarios are learned. The first key is used to authenticate the client. The client is granted a corresponding data management level based on the authentication result, and the data management level is used to determine the scope of server-side data updates; When the number of times the client establishes communication with the server exceeds a first threshold, the client is quickly authenticated and navigated to the target file; The system also includes: The local model from the client is received and processed using the CO-OP algorithm. The local model is combined when the age difference between the client's local model and the global model is not less than the lower age limit and not greater than the upper age limit.

5. A data management system for a federated cloud, characterized in that, The system, applied to a client, includes: The first sending module is used to send an authentication request to the server. After successful authentication, it is granted data management permissions and establishes communication with the server. The second model training module is used to train local models on local data samples and exchange parameters between different local models to obtain an updated local model. The updated local model is then sent to the server, where the server weights the updated local model to generate a new global model. The second sending module is used to send a first request instruction through CSP based on the new global model and data management permissions, and instruct the server to execute the first request instruction according to the execution permission of the first request instruction. The first request instruction is used to update the server data. The second update module is used to receive data update requests and perform data updates to achieve data synchronization between the client and the server. The updated local model is the client's local data trained using SGD with a shared model. The shared model is obtained by updating the global model on the server using the FedAvg algorithm. The system also includes: Receive the first key generated by the CSP and send the authentication request according to the use case; Based on the authentication result, a corresponding data management level is granted, which is used to determine the scope of server-side data updates; When the number of times the client establishes communication with the server exceeds a first threshold, the client will be navigated to the target file after completing identity authentication. The local model is sent to the server for CO-OP algorithm processing, and the server combines the local model when the age difference between the local model and the global model is not less than the lower limit and not greater than the upper limit of age.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 2 or 3.

7. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 2 or 3.

Images