Adaptive privacy online federated learning method and system based on reinforcement learning

Abstract

The invention discloses an adaptive privacy online federated learning method and system based on reinforcement learning, and relates to the technical field of federated learning, and the method comprises the steps: initializing a global federated learning model, and issuing the model to a client; the clients are trained by using local data based on the global federated learning model to obtain a local federated learning model, the variable quantity of the local federated learning model is calculated, the updated global federated learning model and the number of the clients capable of being sampled are input into a DDQN model, and the DDQN model is updated by using a gradient descent method; and calculating a sampling interval and a client sampling number by using the updated DDQN model, and issuing the sampling interval, the client sampling number and the updated global federated learning model to the client to complete online federated learning. According to the method, through user sampling and aggregation interval control, the communication efficiency of the online federated learning system is remarkably improved, and privacy consumption is reduced; the effectiveness of the algorithm is remarkably improved, and the overall operation efficiency of the system is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and in particular to an adaptive privacy online federated learning method and system based on reinforcement learning. Background Art

[0002] With the continuous development of various digital technologies in today's modern society, data has become an inseparable part of everyone's life. The constantly generated data is closely related to everyone's privacy. The emergence of machine learning, deep learning and other methods based on various data has brought many conveniences to various fields such as modern life and industry. Data is the support of various deep learning and other technologies, but privacy constraints in many scenarios limit the use of data. Data is often distributed in the hands of users or continuously generated by users. The data may usually contain users' private information or business secrets, etc. Users are often reluctant to share the original data with the model trainer. Since it is difficult for model trainers to obtain or use enough users' private data, the performance of machine learning models in some specific scenarios will also be affected. In addition, even if the model trainer obtains the user's private data, it is difficult to guarantee that the user's private data will not be leaked during the model training process. Federated learning, as a new distributed machine learning paradigm, allows multiple clients to jointly train models under the organization of a central server, and the training data does not leave the client. Compared with traditional centralized machine learning, the central server in federated learning does not need to collect all users' raw data, and can relatively protect users' privacy. However, current studies still show that using only federated learning may still cause users' privacy to be leaked during training.

[0003] Differential privacy has a strict mathematical definition and quantifiable privacy guarantees. Using differential privacy to prevent privacy leaks in federated learning is a common method. In differential privacy, privacy protection issues in data streams are usually handled based on the following granularities: event-level privacy, user-level privacy, and w-event-level privacy. Among them, event-level privacy protects the user's privacy on a single event in the data stream; user-level privacy protects the user's privacy on the entire data stream; w-event-level privacy can provide privacy guarantees for users on a window of size w on an infinitely long data stream.

[0004] The differential privacy used in existing federated learning algorithms mostly uses event-level privacy or user-level privacy. However, in the scenario of online federated learning, the privacy protection of online federated learning protection algorithms using event-level privacy is too loose. If the privacy budget is too high, the overall privacy budget will be too high, and the privacy protection ability will be greatly reduced. If the privacy budget is too low, the utility will be seriously reduced. User-level privacy is difficult to achieve on data streams of infinite length due to its limited total privacy budget. w-event-level privacy can be regarded as a compromise between user-level privacy and event-level privacy, which can balance between better privacy protection ability and utility. There are few studies on federated learning algorithms for w-event-level privacy. When directly applying traditional w-event-level privacy algorithms based on statistical data release to federated learning, there are still problems with communication efficiency and utility. User sampling and aggregation interval control in traditional w-event-level privacy need to be further improved. Summary of the invention

[0005] In view of the problem in the above-mentioned prior art that online federated learning is difficult to balance privacy protection requirements and effectiveness, the present invention is proposed.

[0006] Therefore, the technical problem solved by the present invention is to achieve online federated learning with adjustable privacy protection granularity and efficient communication through user sampling and aggregation interval control.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: an adaptive privacy online federated learning method based on reinforcement learning, comprising: initializing a global federated learning model, and sending the initialized global federated learning model to a client; the client uses local data to train based on the global federated learning model to obtain a local federated learning model, and calculates the change of the local federated learning model; based on the change of the local federated learning model, the change of the global federated learning model is calculated, and the global federated learning model is updated according to the change of the global federated learning model; the updated global federated learning model and the number of clients that can be sampled are input into the DDQN model, and the DDQN model is updated using the gradient descent method; the sampling interval and the number of client samplings are calculated using the updated DDQN model, and the sampling interval and the number of client samplings and the updated global federated learning model are sent to the client to complete the online federated learning.

[0008] As a preferred solution of the adaptive privacy online federated learning method based on reinforcement learning of the present invention, the optimization objective function of the global federated learning model is as follows:

[0009]

[0010] Where T represents the total training time or rounds, t represents the current time, N represents the number of clients, j represents the client, and m represents the initialized global federated learning model. represents the loss function, represents the local model that is finally used for decision making at time t+1 after training. represents a set of training data provided at training time t; the constraints of the global federated learning model are as follows:

[0011]

[0012] Among them, t represents a fixed time, w represents the length of the sliding time window, i represents the time index, and U i and U j They represent the client sets sampled at time i and time j respectively, N represents the number of clients, and T represents the total number of training rounds.

[0013] As a preferred solution of the adaptive privacy online federated learning method based on reinforcement learning of the present invention, wherein: updating the global federated learning model includes training the local federated learning model using the gradient descent method and calculating the change of the local federated learning model; perturbing the change of the local federated learning model using the Laplace mechanism, and uploading the perturbed change of the local federated learning model to the central server; calculating the change of the global federated learning model using the federated average algorithm, and updating the global federated learning model.

[0014] As a preferred solution of the adaptive privacy online federated learning method based on reinforcement learning of the present invention, the gradient descent method is used to update the local federated learning model, and the change of the local federated learning model is calculated as follows: the client uses the gradient descent method to train the local federated learning model based on the data obtained from the previous model aggregation moment to the current moment, and obtains the change of the local federated learning model.

[0015] As a preferred solution of the adaptive privacy online federated learning method based on reinforcement learning of the present invention, wherein: perturbing the change amount of the local federated learning model includes, when the client is selected to upload the model parameters, checking whether the current selected time meets the set conditions; if so, recording the current time as the most recent global federated learning model update time, and using the Laplace mechanism to perturb the change amount of the local federated learning model, and obtaining the perturbed change amount of the local federated learning model as follows:

[0016]

[0017] Among them, Lap(a,b) represents the sampling of Laplace distribution with mean a and scaling factor b, η represents the learning rate, C represents the gradient clipping threshold, and d represents the parameter dimension of the local federated learning model. Represents the change in the local federated learning model after the disturbance.

[0018] As a preferred solution of the adaptive privacy online federated learning method based on reinforcement learning of the present invention, wherein: using the federated average algorithm to calculate the change amount of the global federated learning model, and updating the global federated learning model includes, the calculation formula of the change amount of the global federated learning model is as follows:

[0019]

[0020] Among them, Δ t Represents the change of the global federated learning model, U t represents the client set sampled at time t, j represents the client index, Represents the change of the local federated learning model after the disturbance; the change of the global federated learning model is added to the global federated learning model to update the global federated learning model.

[0021] As a preferred solution of the adaptive privacy online federated learning method based on reinforcement learning of the present invention, sending the updated DDQN model to the client includes calculating the reinforcement learning reward and the converted cumulative reward respectively according to the change amount of the global federated learning model; the calculation formula of the reinforcement learning reward is as follows:

[0022] r k =e||Δ k ||-1

[0023] Among them, r k represents the reward for the kth round of reinforcement learning training, Δ k It represents the change of the global federated learning model corresponding to the kth round of reinforcement learning; the calculation formula for the converted cumulative reward is as follows:

[0024]

[0025] Among them, R k represents the cumulative reward from the first round to the kth round of reinforcement learning, K represents the total number of global updates, k represents the index of the global update round, γ represents the decay coefficient, Δ k Represents the change in the global federated learning model corresponding to the kth global update; records the environment change-behavior-reward value tuple in the playback buffer of the DDQN model, and extracts data from the recorded tuple to train the DDQ model. Training the DDQN model includes initializing a state s containing the change in the global federated learning model and the number of clients that can be adopted k , and randomly select an action a that includes the sampling interval and the number of client samples, and then change the state s k And action a is input to the DDQN model, and the optimal value function of the DDQN model is defined as follows:

[0026]

[0027] Among them, Q * (s k ,a) represents the optimal value function, that is, in a given state s k and the maximum expected cumulative reward that can be obtained by executing the optimal strategy when taking action a, Q * (s k+1 ,a) means in state s k+1 and the maximum expected cumulative reward that can be obtained by taking action a, represents the average of all strategies, r k represents the reward of the kth round of reinforcement learning training, γ represents the decay coefficient, and s k represents the state, and a represents the action; the optimization objectives of the DDQN model are as follows:

[0028] l k (θ k )=(Y k -Q(s k+1 ,a;θ k )) 2

[0029] Among them, l k (θ k ) represents the loss function of reinforcement learning, Y k represents the target value, θ k represents the DDQN model parameters, Q(s k+1 ,a;θ k ) represents the action value function, which is used to evaluate the weight θ k Next state k+1 The expected cumulative reward after executing action a;

[0030]

[0031] Among them, Y k represents the target value, r k represents reward, γ represents attenuation coefficient, a represents action, Q(s k+1 ,a;θ k ) represents the action value function, which is used to evaluate the weight θ k Next state k+1 The expected cumulative reward after executing action a, s k represents the state of the kth round of reinforcement learning, s k+1 represents the state of the k+1th round of reinforcement learning, θ′ k Indicates freezing parameters to increase the stability of action value estimation; the calculation formula for updating DDQN model parameters is as follows:

[0032]

[0033] Among them, θ k represents the k-th round DDQN model training parameters, θ k+1 represents the k+1th round DDQN model parameters, η′ represents the learning rate, and Y k represents the learning rate, Indicates the derivative of the parameters; using the updated DDQN model, calculate the optimal value action, obtain the sampling interval and number of sampling clients for the next global update, and determine whether the DDQN training meets the termination condition. If so, send information to all clients to terminate reinforcement learning.

[0034] Another object of the present invention is to provide an adaptive privacy online federated learning system based on reinforcement learning, comprising: an initialization module, used to initialize a global federated learning model, and send the initialized global federated learning model to a client; a training module, used for the client to train based on the global federated learning model using local data to obtain a local federated learning model, and calculate the change of the local federated learning model; a calculation module, used to calculate the change of the global federated learning model based on the change of the local federated learning model, and update the global federated learning model according to the change of the global federated learning model; an update module, used to input the updated global federated learning model and the number of clients that can be sampled into a DDQN model, and update the DDQN model using a gradient descent method; a sending module, used to calculate the sampling interval and the number of client samplings using the updated DDQN model, and send the sampling interval and the number of client samplings and the updated global federated learning model to the client to complete the online federated learning.

[0035] A computer device includes a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the above-mentioned adaptive privacy online federated learning method based on reinforcement learning are implemented.

[0036] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned adaptive privacy online federated learning method based on reinforcement learning are implemented.

[0037] The beneficial effects of the present invention are as follows: the present invention adopts the w-event-level privacy model to provide granularly adjustable longitudinal privacy protection for online federated learning, thereby enhancing the privacy security of the online federated learning system; through user sampling and aggregation interval control, the communication efficiency of the online federated learning system is significantly improved, and the privacy consumption is reduced; through reinforcement learning, the user sampling and aggregation intervals in the online federated learning process are adaptively controlled, which significantly improves the effectiveness of the algorithm and improves the overall operation efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0039] Figure 1 This is a schematic diagram of the structure of a terminal in the first embodiment of the present invention;

[0040] Figure 2 It is a schematic diagram of a flow chart in the first embodiment of the present invention;

[0041] Figure 3 The figure is a schematic diagram of the device interaction process in one embodiment of the present invention.

[0042] Figure 4 A schematic diagram of the structure of a computer device in a third embodiment of the present invention; DETAILED DESCRIPTION

[0043] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0045] Example 1

[0046] Reference Figures 1 to 3 , as an embodiment of the present invention, provides an adaptive privacy online federated learning method based on reinforcement learning.

[0047] First, the adaptive privacy online federated learning method based on reinforcement learning provided in this application can be applied to Figure 1 In the terminal shown. Figure 1 As shown, the terminal may include one or two ( Figure 1Only one is shown in the figure) processor and memory for storing data, wherein the processor may include but is not limited to a processing system such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device and an input and output device for communication functions. It can be understood by ordinary technicians in this field that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0048] The memory can be used to store computer programs, such as the computer program corresponding to the adaptive privacy online federated learning method based on reinforcement learning in this embodiment. The processor executes various functional applications and data processing by running the computer program stored in the memory, that is, implementing the above method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage systems, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0049] The transmission device is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0050] like Figures 2-3 As shown, the embodiment of the present invention provides an adaptive privacy online federated learning method based on reinforcement learning, and the method is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0051] S1: Initialize the global federated learning model and send the initialized global federated learning model to the client.

[0052] Specifically, the central server and the client initialize the settings of online federated learning. The central server initializes the global federated learning model m and sends the global federated learning model to all clients.

[0053] Furthermore, the central server is the coordinator and leader of the entire federated learning, responsible for controlling the participation of each participant in the entire online federated learning at each moment, and aggregating the model parameters uploaded by the participants to obtain the global federated learning model of the online federated learning.

[0054] Furthermore, there are multiple participants in online federated learning. Each participant has a private privacy data stream that continuously generates data. The participants need to use the federated learning algorithm on the data stream to train the model sent by the central server and use the differential privacy mechanism to protect the privacy of the uploaded model.

[0055] S2: The client uses local data to train based on the global federated learning model, obtains the local federated learning model, and calculates the change in the local federated learning model.

[0056] Specifically, the client sets the local last sampling time record t′, and the client's local data participates in the training of the federated learning model in an online form. Each client j (j = 1, 2, ..., N, N is the number of clients) provides a set of training data at each time t of training. Used to train federated learning models After T moments, the optimization objective function is as follows:

[0057]

[0058] Where T represents the total training time or rounds, t represents the current time, N represents the number of clients, j represents the client, and m represents the initialized global federated learning model. represents the loss function, represents the local model that is finally used for decision making at time t+1 after training. Represents a set of training data provided at training time t.

[0059] The constraints of the local federated learning model are as follows:

[0060]

[0061] Among them, t represents a fixed time, w represents the length of the sliding time window, i represents the time index, and U i and U j They represent the client sets sampled at time i and time j respectively, N represents the number of clients, and T represents the total number of training rounds.

[0062] Specifically, the gradient descent method is used to train the local federated learning model, and the change of the local federated learning model is calculated; the Laplace mechanism is used to perturb the change of the local federated learning model, and the perturbed change of the local federated learning model is uploaded to the central server; the federated average algorithm is used to calculate the change of the global federated learning model, and the global federated learning model is updated.

[0063] Furthermore, after receiving the global federated learning model sent by the central server, all clients will update their local federated learning models and use the data in the local data stream at the current moment to train the local model at each moment. The client uses the gradient descent method to train the local federated learning model based on the data obtained from the previous model aggregation moment to the current moment to obtain the change in the local federated learning model.

[0064] Furthermore, all clients will update their local federated learning models after receiving the global federated learning model sent by the central server.

[0065] Furthermore, the local federated learning model is updated using the gradient descent method, and the change in the local federated learning model is calculated as follows: The client uses the gradient descent method to train the local federated learning model based on the data obtained from the previous model aggregation moment to the current moment, and obtains the change in the local federated learning model.

[0066] S3: Based on the change in the local federated learning model, the change in the global federated learning model is calculated, and the global federated learning model is updated according to the change in the global federated learning model.

[0067] Specifically, the perturbation of the local federated learning model change includes, when the client is selected to upload the model parameters, checking whether the current selected time satisfies the set condition tt′≤w-1; if satisfied, updating the time, using the Laplace mechanism to perturb the local federated learning model change, and obtaining the perturbed local federated learning model change as follows:

[0068]

[0069] Among them, Lap(a,b) represents the sampling of Laplace distribution with mean a and scaling factor b, η represents the learning rate, C represents the gradient clipping threshold, and d represents the parameter dimension of the local federated learning model. Represents the change in the local federated learning model after the disturbance.

[0070] Furthermore, after receiving the local federated learning model change from the client, the central server uses the federated average algorithm to calculate the global federated learning model change. The calculation formula for the global federated learning model change is as follows:

[0071]

[0072] Among them, Δ t Represents the change of the global federated learning model, U t represents the client set sampled at time t, j represents the client index, Represents the change in the local federated learning model after the disturbance.

[0073] Furthermore, the global federated learning model changes are added to the global federated learning model using m t =m t-1 +Δ t , update the global federated learning model, where m t represents the global federated learning model at time t, m t-1 represents the global federated learning model at the previous moment, Δ t Represents the change in the global federated learning model.

[0074] S4: Input the updated global federated learning model and the number of clients that can be sampled into the DDQN model, and update the DDQN model using the gradient descent method.

[0075] Specifically, the central server uses the change in the global federated learning model and the number of users who can currently participate in federated learning as the input of the DDQN model. Before inputting the global federated learning model parameters into the DDQN model, the principal component analysis is used to reduce the parameter quantity of the model. The principal component analysis matrix can be obtained based on the model parameters at the first model aggregation (the principal component analysis matrix can be obtained based on the local model parameters at the first local federated learning model aggregation). The output calculated using the DDQN model is the sampling interval τ for this round. t and the number of client samples N t , sample the corresponding number of clients and wait for τ t Get the random sample N after time t A client model.

[0076] Furthermore, according to the change in the global federated learning model, the reinforcement learning reward and the converted cumulative reward are calculated respectively; the calculation formula of the reinforcement learning reward is as follows:

[0077] r k =e||Δ k ||-1

[0078] Among them, r k represents the reward for the kth round of reinforcement learning training, Δ k Represents the change in the global federated learning model corresponding to the kth round of reinforcement learning.

[0079] Furthermore, the calculation formula for the converted cumulative rewards is as follows:

[0080]

[0081] Among them, R k represents the cumulative reward from the first round to the kth round of reinforcement learning, K represents the total number of global updates, k represents the index of the global update round, γ represents the decay coefficient, Δ k Represents the change in the global federated learning model corresponding to the kth global update.

[0082] Furthermore, the environment change-behavior-reward value tuple is recorded in the playback buffer of the DDQN model, and the data DDQ model is extracted from the recorded tuple. Training the DDQN model includes initializing a state s containing the global federated learning model change amount and the number of clients that can be adopted k , and randomly select an action a that includes the sampling interval and the number of client samples, and then change the state s k And action a is input to the DDQN model, and the optimal value function of the DDQN model is defined as follows:

[0083]

[0084] Among them, Q * (s k ,a) represents the optimal value function, that is, in a given state s k and the maximum expected cumulative reward that can be obtained by executing the optimal strategy when taking action a, Q * (s k+1 ,a) means in state s k+1 and the maximum expected cumulative reward that can be obtained by taking action a, represents the average of all strategies, r k represents the reward of the kth round of reinforcement learning training, γ represents the decay coefficient, and s k represents the state and a represents the action.

[0085] Furthermore, the optimization objectives of the DDQN model are as follows:

[0086] l k (θ k )=(Y k -Q(s k+1 ,a;θ k )) 2

[0087] Among them, l k (θ k ) represents the loss function of reinforcement learning, Y k represents the target value, θ k represents the DDQN model parameters, Q(sk+1 ,a;θ k ) represents the action value function, which is used to evaluate the weight θ k Next state k+1 The expected cumulative reward after performing action a.

[0088]

[0089] Among them, Y k represents the target value, r k represents reward, γ represents attenuation coefficient, a represents action, Q(s k+1 ,a;θ k ) represents the action value function, which is used to evaluate the weight θ k Next state k+1 The expected cumulative reward after executing action a, s k represents the state of the kth round of reinforcement learning, s k+1 represents the state of the k+1th round of reinforcement learning, θ′ k Denotes freezing parameters to increase the stability of action value estimation.

[0090] Furthermore, after several rounds of OFL (Online Federated Learning) training, the DRL (Deep Reinforcement Learning) optimizer has sampled some action states, and the DDQN model uses the sampled action states for reinforcement learning training and solves it using the gradient descent method.

[0091] Furthermore, the calculation formula for updating the DDQN model parameters is as follows:

[0092]

[0093] Among them, θ k represents the k-th round DDQN model training parameters, θ k+1 represents the k+1th round DDQN model parameters, η′ represents the learning rate, and Y k represents the learning rate, represents the derivative of a parameter.

[0094] Furthermore, the central server sends the latest global federated learning model to all clients, returns to S2 and repeats the execution. One execution from S2 to S4 is regarded as an online federated learning aggregation. After each round, it is checked whether the online federated learning continues. If not, the center sends information to all clients to terminate the federated learning.

[0095] Example 2

[0096] Reference Figure 1 to Figure 3 , which is an embodiment of the present invention, provides an adaptive privacy online federated learning system based on reinforcement learning, including:

[0097] The initialization module is used to initialize the global federated learning model and send the initialized global federated learning model to the client;

[0098] The training module is used by the client to train the global federated learning model using local data to obtain the local federated learning model and calculate the change amount of the local federated learning model;

[0099] A calculation module, used to calculate the change amount of the global federated learning model based on the change amount of the local federated learning model, and update the global federated learning model according to the change amount of the global federated learning model;

[0100] The update module is used to input the updated global federated learning model and the number of clients that can be sampled into the DDQN model, and update the DDQN model using the gradient descent method;

[0101] The sending module is used to calculate the sampling interval and the number of client samples using the updated DDQN model, and send the sampling interval and the number of client samples and the updated global federated learning model to the client to complete the online federated learning.

[0102] For the specific definition of the adaptive privacy online federated learning system based on reinforcement learning, please refer to the definition of the adaptive privacy online federated learning method based on reinforcement learning above, which will not be repeated here. Each module in the above-mentioned adaptive privacy online federated learning system based on reinforcement learning can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0103] Example 3

[0104] Reference Figure 4 , which is the third embodiment of the present invention. Based on the first two embodiments, the embodiment of the present invention provides a computer device, which may be a server, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium.

[0105] The database of the computer device is used to store action detection data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in any of the above-mentioned sparse tensor operation acceleration method embodiments are implemented.

[0106] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0107] In one embodiment, an embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in any of the above-mentioned sparse tensor operation acceleration method embodiments are implemented.

[0108] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0109] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An adaptive privacy online federated learning method based on reinforcement learning, characterized in that: include: Initialize the global federated learning model and send the initialized global federated learning model to the client; The client uses local data for training based on the global federated learning model to obtain a local federated learning model, and calculates a change in the local federated learning model; Based on the change amount of the local federated learning model, the change amount of the global federated learning model is calculated, and the global federated learning model is updated according to the change amount of the global federated learning model; Input the updated global federated learning model and the number of clients that can be sampled into the DDQN model, and use the gradient descent method to update the DDQN model; The updated DDQN model is used to calculate the sampling interval and the number of client samples, and the sampling interval and the number of client samples and the updated global federated learning model are sent to the client to complete the online federated learning.

2. The method for adaptive privacy-preserving online federated learning based on reinforcement learning as claimed in claim 1, characterized in that: The optimization objective function of the global federated learning model is as follows: Where T represents the total training time or rounds, t represents the current time, N represents the number of clients, j represents the client, and m represents the initialized global federated learning model. represents the loss function, represents the local model that is finally used for decision making at time t+1 after training. represents a set of training data provided at training time t; The constraints of the global federated learning model are as follows: Among them, t represents a fixed time, w represents the length of the sliding time window, i represents the time index, and U i and U j They represent the client sets sampled at time i and time j respectively, N represents the number of clients, and T represents the total number of training rounds.

3. The method for adaptive privacy-preserving online federated learning based on reinforcement learning as claimed in claim 2, characterized in that: The updating of the global federated learning model includes: Use the gradient descent method to train the local federated learning model and calculate the change in the local federated learning model; Using the Laplace mechanism to perturb the change amount of the local federated learning model, and uploading the perturbed change amount of the local federated learning model to the central server; The federated average algorithm is used to calculate the change in the global federated learning model and update the global federated learning model.

4. The method for adaptive privacy-preserving online federated learning based on reinforcement learning as claimed in claim 3, characterized in that: The method of using the gradient descent method to update the local federated learning model and calculating the change in the local federated learning model is that the client uses the gradient descent method to train the local federated learning model based on data obtained from the previous model aggregation moment to the current moment to obtain the change in the local federated learning model.

5. The method for adaptive privacy-preserving online federated learning based on reinforcement learning as claimed in claim 4, characterized in that: The perturbation of the change amount of the local federated learning model includes, when the client is selected to upload the model parameters, checking whether the current selected time meets the set conditions; If satisfied, the current time is recorded as the most recent global federated learning model update time, and the local federated learning model change is perturbed using the Laplace mechanism. The perturbed local federated learning model change is obtained as follows: Among them, Lap(a,b) represents the sampling of Laplace distribution with mean a and scaling factor b, η represents the learning rate, C represents the gradient clipping threshold, and d represents the parameter dimension of the local federated learning model. Represents the change in the local federated learning model after the disturbance.

6. The method for adaptive privacy-preserving online federated learning based on reinforcement learning as claimed in claim 5, characterized in that: The method of calculating the change of the global federated learning model using the federated average algorithm and updating the global federated learning model includes: The calculation formula for the change in the global federated learning model is as follows: Among them, Δ t Represents the change of the global federated learning model, U t represents the client set sampled at time t, j represents the client index, Represents the change in the local federated learning model after the disturbance; The global federated learning model changes are added to the global federated learning model to update the global federated learning model.

7. The method for adaptive privacy-preserving online federated learning based on reinforcement learning as claimed in claim 6, characterized in that: Sending the updated DDQN model to the client includes: According to the changes in the global federated learning model, the reinforcement learning rewards and the converted cumulative rewards are calculated respectively; The calculation formula of the reinforcement learning reward is as follows: Among them, r k represents the reward for the kth round of reinforcement learning training, Δ k Indicates the change in the global federated learning model corresponding to the kth round of reinforcement learning; The calculation formula for the converted cumulative reward is as follows: Among them, R k represents the cumulative reward from the first round to the kth round of reinforcement learning, K represents the total number of global updates, k represents the index of the global update round, γ represents the decay coefficient, Δ k represents the change in the global federated learning model corresponding to the kth global update; The environment change-behavior-reward value tuple is recorded in the playback buffer of the DDQN model, and data is extracted from the recorded tuple to train the DDQ model, wherein the training of the DDQN model includes initializing a state s containing the global federated learning model change amount and the number of clients that can be adopted k , and randomly select an action a that includes the sampling interval and the number of client samples, and then change the state s k And action a is input to the DDQN model, and the optimal value function of the DDQN model is defined as follows: Among them, Q * (s k ,a) represents the optimal value function, that is, in a given state s k and the maximum expected cumulative reward that can be obtained by executing the optimal strategy when taking action a, Q * (s k+1 ,a) means in state s k+1 and the maximum expected cumulative reward that can be obtained by taking action a, represents the average of all strategies, r k represents the reward of the kth round of reinforcement learning training, γ represents the decay coefficient, and s k represents a state, and a represents an action; The optimization objectives of the DDQN model are as follows: l k (i k )=(Y k -Q(s k+1 ,a;θ k )) 2 Among them, l k (θ k ) represents the loss function of reinforcement learning, Y k represents the target value, θ k represents the DDQN model parameters, Q(s k+1 ,a;θ k ) represents the action value function, which is used to evaluate the weight θ k Next state k+1 The expected cumulative reward after executing action a; Among them, Y k represents the target value, r k represents reward, γ represents attenuation coefficient, a represents action, Q(s k+1 ,a;θ k ) represents the action value function, which is used to evaluate the weight θ k Next state k+1 The expected cumulative reward after executing action a, s k represents the state of the kth round of reinforcement learning, s k+1 represents the state of the k+1th round of reinforcement learning, θ′ k Represents freezing parameters to increase the stability of action value estimation; The calculation formula for updating the DDQN model parameters is as follows: Among them, θ k represents the k-th round DDQN model training parameters, θ k+1 represents the k+1th round DDQN model parameters, η′ represents the learning rate, and Y k represents the learning rate, represents the derivative of the parameter; Use the updated DDQN model to calculate the optimal value action, obtain the sampling interval and number of sampling clients for the next global update, and determine whether the DDQN training meets the termination conditions. If so, send information to all clients to terminate reinforcement learning.

8. A system using the adaptive privacy online federated learning method based on reinforcement learning as described in any one of claims 1 to 7, characterized in that: include: The initialization module is used to initialize the global federated learning model and send the initialized global federated learning model to the client; A training module, used for the client to train the local data based on the global federated learning model to obtain the local federated learning model and calculate the change amount of the local federated learning model; A calculation module, used to calculate the change amount of the global federated learning model based on the change amount of the local federated learning model, and update the global federated learning model according to the change amount of the global federated learning model; The update module is used to input the updated global federated learning model and the number of clients that can be sampled into the DDQN model, and update the DDQN model using the gradient descent method; The sending module is used to calculate the sampling interval and the number of client samples using the updated DDQN model, and send the sampling interval and the number of client samples and the updated global federated learning model to the client to complete the online federated learning.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the adaptive privacy online federated learning method based on reinforcement learning described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the adaptive privacy online federated learning method based on reinforcement learning described in any one of claims 1 to 7 are implemented.

Images