Federal reinforcement learning-based data enhancement method and related equipment

Abstract

According to the federal reinforcement learning-based data enhancement method and the related equipment provided by the invention, optimization of training performance and the number of enhanced samples is realized, and the method comprises the following steps: acquiring a first model parameter of at least one initial model; processing the first model parameter based on the generative adversarial network, and determining a first enhanced data set; processing the at least one initial model through a federated average algorithm to obtain a global model; determining an enhanced data strategy based on a Markov decision process, the first model parameter and a first sample number of the first enhanced data set; according to a second model parameter of the global model and an enhanced data strategy, training the initial model to obtain a local model; obtaining a strategy reward according to the test accuracy of the global model on the test set and the second sample number; based on a multi-agent enhanced optimization algorithm, taking the initial model, the enhanced data strategy, the local model and the strategy reward as training samples, and storing the training samples in a sample pool; and iteratively training the local model according to the training sample.

Description

Technical Field

[0001] The present application relates to the field of power safety detection, and in particular to a data enhancement method and related equipment based on federated reinforcement learning. Background Art

[0002] In the field of wireless communications, Automatic Modulation Classification (AMC) is crucial for spectrum monitoring, cognitive radio, and electronic warfare. In response to the shortcomings of traditional AMC methods in computational complexity and feature extraction capabilities, as well as the data privacy leakage and "data island" problems faced by deep learning (DL) methods, Federated Learning (FL) came into being.

[0003] FL allows edge devices to independently train models using local data, and only upload model parameters rather than raw data to the central server for aggregation, thereby effectively ensuring data security and privacy. However, since the data distribution between edge devices usually presents non-independent and identically distributed (Non-IID) characteristics, this not only increases the difficulty of FL model training, but also becomes an obstacle to improving AMC performance. Summary of the invention

[0004] In view of this, the purpose of this application is to propose a data enhancement method and related equipment based on federated reinforcement learning.

[0005] As one aspect of the present application, a data enhancement method based on federated reinforcement learning is provided, comprising:

[0006] Acquire a first model parameter of at least one initial model; each initial model in the at least one initial model corresponds to a client; the first model parameter is a signal modulation mode data set;

[0007] Processing the first model parameters based on a generative adversarial network to determine a first enhanced data set;

[0008] Processing the at least one initial model through a federated averaging algorithm to obtain a global model;

[0009] Determining an enhanced data strategy based on a Markov decision process, the first model parameter, and a first sample number of the first enhanced data set; the first sample number is used to characterize the number of enhanced samples added to each client of at least one client;

[0010] For each of the at least one initial model, training the initial model according to the second model parameter of the global model and the enhanced data strategy to obtain a local model;

[0011] Obtaining a policy reward according to the test accuracy of the global model on the test set and the second sample quantity; the second sample quantity is used to represent the number of enhanced samples added to the at least one client;

[0012] Based on a multi-agent reinforcement optimization algorithm, the at least one initial model, the enhanced data strategy, the local model, and the strategy reward are stored in a sample pool as training samples;

[0013] The local model is iteratively trained according to the training samples.

[0014] Optionally, in response to the iterative training of the local model reaching a preset condition, the iteration is stopped; the preset condition includes at least one of the following: the number of iterations reaches a first preset threshold and the strategy reward reaches a second preset threshold.

[0015] Optionally, for each of the at least one initial model, training the initial model according to the second model parameter of the global model and the enhanced data strategy to obtain a local model includes:

[0016] Determining a second enhanced data set according to the enhanced data strategy;

[0017] The initial model is trained according to the second model parameters and the second enhanced data set to obtain the local model.

[0018] Optionally, the number of samples in the second enhanced data set is less than or equal to that in the first enhanced data set.

[0019] Optionally, the first model parameters are obtained by compression and decomposition based on a tensor decomposition algorithm.

[0020] Optionally, before obtaining the first model parameter of at least one initial model, the method further includes:

[0021] Initialize the initial model.

[0022] As a second aspect of the present application, a data enhancement device based on federated reinforcement learning is provided, comprising: an acquisition module, a determination module, a processing module, a storage module and a training module;

[0023] The acquisition module is used to acquire a first model parameter of at least one initial model; each of the at least one initial model corresponds to a client; the first model parameter is a signal modulation mode data set;

[0024] The determination module is used to process the first model parameters based on a generative adversarial network to determine a first enhanced data set;

[0025] The processing module is used to process the at least one initial model through a federated average algorithm to obtain a global model;

[0026] The determination module is further used to determine an enhanced data strategy based on a Markov decision process, the first model parameter, and a first sample number of the first enhanced data set; the first sample number is used to characterize the number of enhanced samples added to each client of the at least one client;

[0027] The training module is used for, for each of the at least one initial model, training the initial model according to the second model parameter of the global model and the enhanced data strategy to obtain a local model;

[0028] The processing module is further used to obtain a policy reward according to the test accuracy of the global model on the test set and the second sample quantity; the second sample quantity is used to represent the number of enhanced samples added to the at least one client;

[0029] The storage module is used to store the at least one initial model, the enhanced data strategy, the local model and the strategy reward as training samples in a sample pool based on a multi-agent reinforcement optimization algorithm;

[0030] The training module is also used to iteratively train the local model according to the training samples.

[0031] Optionally, the processing module is further used to stop iteration in response to the iterative training of the local model reaching a preset condition; the preset condition includes at least one of the following: the number of iterations reaches a first preset threshold and the strategy reward reaches a second preset threshold.

[0032] Optionally, the training module is specifically used to determine a second enhanced data set according to the enhanced data strategy;

[0033] The initial model is trained according to the second model parameters and the second enhanced data set to obtain the local model.

[0034] Optionally, the number of samples in the second enhanced data set is less than or equal to that in the first enhanced data set.

[0035] Optionally, the first model parameters are obtained by compression and decomposition based on a tensor decomposition algorithm.

[0036] Optionally, the data enhancement device based on federated reinforcement learning also includes an initialization module.

[0037] The initialization module is used to initialize the initial model.

[0038] Optionally, the initial model is a convolutional neural network model.

[0039] As the third aspect of the present application, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data enhancement method based on federated reinforcement learning when executing the program.

[0040] As the fourth aspect of the present application, a non-transitory computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned data enhancement method based on federated reinforcement learning provided in the present application.

[0041] As can be seen from the above, the data enhancement method and related equipment based on federated reinforcement learning provided by the present application can increase the amount of data for model training by generating the first enhanced data set generated by the adversarial network, thereby increasing the diversity of data and improving the generalization ability of the model. The initial models of each client are fused into a global model using the federated average algorithm, which can integrate the knowledge of multiple clients while protecting data privacy and improve the performance of the global model. Furthermore, based on the Markov decision process, the first model parameters and the first sample number of the first enhanced data set, the enhanced data strategy is dynamically determined. This strategy can intelligently adjust the generation and use of enhanced data according to the performance of the current model and the size of the data set to achieve the best training effect. Then, the strategy reward is evaluated by the test accuracy of the global model and the number of enhanced samples, which helps to optimize the data enhancement strategy and ensure that the cost of data enhancement is controlled while improving the model performance. Further, using the optimization algorithm of multi-agent reinforcement learning, the models of multiple clients can be trained at the same time, so that each client can learn from the experience of the global model and other clients, thereby improving the learning efficiency and performance of the entire system. Finally, the client's initial model, enhanced data strategy, local model and strategy reward are stored in the sample pool as training samples, and the local model is iteratively trained based on these samples to continuously optimize the model performance. Through federated learning, the model training process is dispersed to each client, avoiding centralized storage and processing of sensitive data, effectively protecting data privacy, and balancing the model performance and data enhancement costs to ensure that the demand for computing resources and data storage is reduced while improving the model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 A schematic diagram of the structure of a communication system provided in an embodiment of the present application;

[0044] Figure 2 A flowchart of a data enhancement method based on federated reinforcement learning provided in an embodiment of the present application;

[0045] Figure 3 A flowchart of another data enhancement method based on federated reinforcement learning provided in an embodiment of the present application;

[0046] Figure 4 A flowchart of another data enhancement method based on federated reinforcement learning provided in an embodiment of the present application;

[0047] Figure 5 A flowchart of another data enhancement method based on federated reinforcement learning provided in an embodiment of the present application;

[0048] Figure 6 A flowchart of another data enhancement method based on federated reinforcement learning provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of the composition of a data enhancement device based on federated reinforcement learning provided in an embodiment of the present application;

[0050] Figure 8 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0052] It should be noted that, in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a concrete way. Unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meaning understood by people with ordinary skills in the field to which the present application belongs. SUMMARY OF THE INVENTION

[0054] In the related art, the AMC method is mainly based on likelihood (Likelihood-Based, referred to as LB) and feature-Based (Feature-Based, referred to as FB). Although the LB method is optimal in the Bayesian sense, it is computationally intensive and has poor robustness to model mismatch. Although the FB method has low computational requirements, it often has limited feature extraction capabilities. Deep learning can effectively improve recognition performance by extracting high-dimensional features of debugging mode data. However, traditional DL methods upload the perception parameters collected by edge nodes to the central unit, and then perform global model training in the central unit. This direct data upload method may lead to data privacy leakage, which in turn leads to the occurrence of the "data island" problem. In order to protect the privacy of perception data, federated learning methods are widely used in the AMC field. Unlike direct data sharing in centralized training, federated learning trains data locally, and then uploads local model parameters to the central server for model aggregation. Data privacy can be well protected by uploading parameters. Due to the heterogeneity of local nodes, the data distribution between nodes is usually Non-IID, which significantly affects the training performance of federated learning and has become one of the bottleneck problems that plague AMC.

[0055] The inventors of this application found that in order to alleviate the impact of non-independent and identically distributed data on the performance of federated learning, related studies mainly use four methods: data enhancement, multi-task learning (MTL), client selection and transfer learning (TL). However, these methods have several limitations in practical applications: First, these methods generally rely on the known distribution of client data, which is often difficult to meet in practical applications, limiting their applicability and effectiveness. Secondly, although these methods have made certain progress in optimizing training performance, such as through data enhancement, multi-task learning, client selection and transfer learning, they are powerless in improving sample efficiency, especially when data is scarce or extremely unevenly distributed. The generalization ability and accuracy of the model are severely challenged. In addition, although the model upload mechanism in the federated learning framework promotes the update of the global model, it also raises the potential risk of privacy leakage, because the local model may contain sensitive information and preferences of the user, which poses a serious threat to data privacy protection.

[0056] In order to solve the above problems, the present application provides a data enhancement method based on federated reinforcement learning. By generating a first enhanced data set generated by a generative adversarial network, the amount of data for model training can be increased, thereby increasing the diversity of data and improving the generalization ability of the model. The initial models of each client are fused into a global model using a federated average algorithm, which can integrate the knowledge of multiple clients while protecting data privacy and improve the performance of the global model. Furthermore, based on the Markov decision process, the first model parameters and the first sample number of the first enhanced data set, the enhanced data strategy is dynamically determined. This strategy can intelligently adjust the generation and use of enhanced data according to the performance of the current model and the size of the data set to achieve the best training effect. Then, the strategy reward is evaluated by the test accuracy of the global model and the number of enhanced samples, which helps to optimize the data enhancement strategy and ensure that the cost of data enhancement is controlled while improving the model performance. Further, using the optimization algorithm of multi-agent reinforcement learning, the models of multiple clients can be trained at the same time, so that each client can learn from the experience of the global model and other clients, thereby improving the learning efficiency and performance of the entire system. Finally, the client's initial model, enhanced data strategy, local model and strategy reward are stored in the sample pool as training samples, and the local model is iteratively trained based on these samples to continuously optimize the model performance. Through federated learning, the model training process is dispersed to each client, avoiding centralized storage and processing of sensitive data, effectively protecting data privacy, and balancing the model performance and data enhancement costs to ensure that the demand for computing resources and data storage is reduced while improving the model accuracy.

[0057] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application are described in detail below.

[0058] Application Scenario

[0059] Figure 1 This is a schematic diagram of the structure of a communication system provided in an embodiment of the present application. Figure 1 As shown, the communication system includes a local device 101, a central server 102, a local server 103, a local server 104 and a local server 105, wherein the local device 101 and the central server 102 can be connected via a wired or wireless network, and the central server 102 and the local server 103 can be connected via a wired or wireless network.

[0060] The local device 101 may be a device with wireless transceiver functions, such as a laptop, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a mobile phone, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present invention does not limit the specific type of the local device 101.

[0061] In some embodiments, the local device 101 is a device that directly interacts with the user, which can receive and send data, and transmit the data to the local server 103. The local device 101 can collect the original modulated signal data and perform necessary preprocessing, such as feature extraction, signal cleaning, etc.

[0062] In some embodiments, the central server 102 is responsible for managing and maintaining the entire communication system. The central server 102 includes node computers and communication lines for information exchange, and is responsible for communication processing such as data transmission, switching, processing and exchange. In a specific workflow, the central server 102 is responsible for managing the initialization and update of the global model. It aggregates model parameters from local servers 103, local servers 104 and local servers 105, generates an updated global model, and can distribute the global model parameters to each local server so that they can be trained and optimized locally.

[0063] In some embodiments, in the wireless communication system, the local server 103 is responsible for collecting and processing local data, and then uses the local data to train the local model and update it according to the global model parameters provided by the central server. At the same time, the local server 103 can feed back the trained local model parameters or performance indicators to the central server 102 to update and optimize the global model.

[0064] In some embodiments, the central server 102 is defined as a server, and the local servers (e.g., local servers 103, 104, and 105) are defined as clients. In the data interaction process, the central server 102 plays the role of the server, and the local servers play the role of the clients, and data is transmitted between the two.

[0065] In some embodiments, the central server 102 and the local server 103 may be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Optionally, the server can also be implemented on a cloud platform, for example, the cloud platform can include private clouds, public clouds, hybrid clouds, community clouds, distributed clouds, inter-clouds, and multi-clouds, etc., or any combination thereof. The embodiments of the present application are not limited to this.

[0066] It should be understood that Figure 1 is an exemplary structural diagram, Figure 1 The communication system shown includes an unlimited number of devices, for example, an unlimited number of local devices 101 and an unlimited number of local servers 103. Figure 1 In addition to the equipment shown, Figure 1 The communication system shown may also include other devices, which is not limited thereto.

[0067] Combine the following Figure 1 The data enhancement method based on federated reinforcement learning according to the exemplary embodiment of the present application is described by using the application scenario. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in this regard. On the contrary, the implementation of the present application can be applied to any applicable scenario.

[0068] Figure 2 A flowchart of a data enhancement method based on federated reinforcement learning provided in an embodiment of the present application. Figure 2As shown, the data enhancement method based on federated reinforcement learning provided by this application specifically includes the following steps:

[0069] S201. Obtain a first model parameter of at least one initial model.

[0070] Each initial model in the at least one initial model corresponds to a client.

[0071] In some embodiments, in an application of automatically identifying a signal modulation mode, a first model parameter of an initial model is a data set related to the signal modulation mode, which contains signal samples of different modulation modes and is used to train the initial model to identify the signal modulation mode.

[0072] In some embodiments, the initial model is a convolutional neural network model (CNN). In the related deep learning research work, the complexity of the network structure is very high due to the pursuit of powerful performance. However, the huge network parameters bring a heavy burden to the communication link in FL. The huge model parameters will seriously limit the training performance of the model and even make it non-convergent. In order to solve the above problems, a tensor decomposition algorithm, namely Tucker decomposition, can be used to compress the four-dimensional parameters of the CNN model, thereby reducing the training pressure brought by the high-dimensional model parameters and reducing the communication overhead during the FL training process.

[0073] It should be noted that Tucker decomposition is a high-order extension of principal component analysis (PCA). The essence of Tucker decomposition is to decompose high-dimensional vectors into core vectors and factor vectors corresponding to different dimensions (three factor vectors in the third-order tensor decomposition). Each factor vector can be regarded as a linear transformation on different dimensions. Tucker decomposition can be expressed as the following expression:

[0074]

[0075] Among them, Y is the core matrix after decomposition, and A, B, and C are factor matrices of different dimensions.

[0076] In some embodiments, Figure 3 As shown, before S201, the data enhancement method based on federated reinforcement learning provided in the embodiment of the present application further includes the following S200:

[0077] S200: Initialize the initial model.

[0078] In some embodiments, the weights and biases of the initial model are initialized with random parameters to ensure that each neuron has a different weight in the initial stage, thereby avoiding that all neurons have the same output and update gradient at the beginning of training, which helps the model learn different features.

[0079] Exemplarily, when constructing a neural network model, the weights and bias parameters of the model are randomly assigned values ​​by uniform distribution initialization, Gaussian distribution initialization, or Xavier initialization.

[0080] S202: Process the first model parameters based on the generative adversarial network to determine a first enhanced data set.

[0081] In some embodiments, a generative adversarial network (GAN) is used to enhance the data set of signal modulation methods. GAN can generate new signal samples that simulate the distribution of real signals, thereby expanding the training data set and improving the generalization ability of the model.

[0082] It should be noted that GAN consists of a generator and a discriminator. The generator generates new data samples, and the discriminator determines whether the samples are real or generated. In this way, enhanced data similar to real data can be generated.

[0083] S203, processing at least one initial model through a federated average algorithm to obtain a global model.

[0084] In some embodiments, in a federated learning framework, a federated averaging algorithm (FedAvg) updates a global model by calculating the average of all client model parameters. Therefore, the federated averaging algorithm is used to aggregate the initial model parameters of multiple clients to obtain a global model.

[0085] S204: Determine an enhanced data strategy based on the Markov decision process, the first model parameter, and the first number of samples of the first enhanced data set.

[0086] The first sample quantity is used to characterize the enhanced sample quantity added to each client in the at least one client.

[0087] In some embodiments, in the framework of reinforcement learning, the Markov Decision Process (MDP) is a mathematical model used to describe how an agent takes actions in an environment to maximize cumulative rewards. In multi-agent reinforcement learning (MARL), each agent (client) has a current state, namely the actor state, which reflects the environmental information when the agent makes a decision. The actor state satisfies the following expression:

[0088]

[0089] in, represents the first model parameters of client i at decision time t, which reflects the training status of the client's current model. It represents the number of first samples added to client i at decision time t, which reflects the data enhancement efforts made by the client to improve model performance.

[0090] Actor actions (enhanced data strategies) represent the behaviors that the agent can take in the current state. In this scenario, Actor actions can be Among them: Action 0 means that no enhancement data is added to client i, and actions 1 to 10 are different enhancement data addition amounts or strategies. The Critic state is used to evaluate the value of the Actor's actions and help the Actor make better decisions. In this scenario, the Critic state satisfies the following expression:

[0091]

[0092] Compared with the Actor state, the Critic state not only contains the first model parameters and the first sample number of the current agent, but also takes into account the enhanced sample conditions of other agents, so that the Critic can more comprehensively evaluate the impact of the current action on the entire federated learning system. In addition, in order to protect the privacy of the model, the model parameters of other agents are not available to the current agent. The Critic action evaluates its value based on the observed state, including the enhanced sample conditions of all agents and the Actor action. The Critic action satisfies the following expression:

[0093]

[0094] in, represents the action chosen by agent i at decision time t.

[0095] S205. For each initial model in the at least one initial model, train the initial model according to the second model parameter of the global model and the enhanced data strategy to obtain a local model.

[0096] In some embodiments, the second model parameters are parameters of the global model, which are distributed to each client so that they can further train the model on local data. At the same time, the augmented data strategy is determined by the MARL agent, which guides how many augmented samples should be added to each client during training to improve the generalization ability of the model and control the cost of augmented data. During the training of the local model, the client's data is always kept local, and only the update of the model parameters is shared with the central server or other clients, thereby protecting data privacy.

[0097] It should be understood that the federated learning system can use global information and local data to train local models with better performance, which help improve the performance of the entire federated learning system at the global level. At the same time, by introducing enhanced data strategies, the system can take into account the cost and efficiency of data enhancement while improving model performance.

[0098] In some embodiments, Figure 4 As shown, S205 can be specifically implemented as follows S2051-S2052:

[0099] S2051. Determine a second enhanced data set according to the enhanced data strategy.

[0100] In some embodiments, at decision time t, the Actor makes a decision based on the local observation Select the enhanced data strategy for the current moment Enhanced data-based strategy Client i adds enhanced data to the first enhanced data set to determine a second enhanced data set.

[0101] It should be understood that the number of samples of the second enhanced data set is less than or equal to that of the first enhanced data set.

[0102] S2052: Train the initial model according to the second model parameters and the second enhanced data set to obtain a local model.

[0103] In some embodiments, the second model parameters are distributed from the global model to each client, which represent the current state of the global model. The client uses these parameters to further train the initial model on the local data and the second enhanced data set. After training, the client obtains a local model with better performance. This model not only reflects the characteristics of the local data, but also incorporates the global information introduced by the enhanced data strategy.

[0104] S206. Obtain a strategy reward based on the test accuracy of the global model on the test set and the number of second samples.

[0105] The second sample quantity is used to characterize the enhanced sample quantity added to at least one client.

[0106] In some embodiments, according to the description of MDP in S204 above, the MDP process also includes a strategy reward, and the agent i is in state Next select action and receive strategic rewards from the environment Strategy Rewards It is divided into two parts, one is the test accuracy function f(A) of the global model on the test set, and the other is the number function g(N aug ). Strategy Rewards Satisfies the following expression:

[0107]

[0108] Among them, m1 and m2 are positive constants greater than 1, is the average training accuracy, N aug is the second sample size, N GAN is the number of samples in the first augmented dataset, ω1 and ω2 are A and N respectively aug The weighted coefficient is used to represent the ratio of the importance of the two optimization objectives. The setting of the above strategy reward helps the agent to reasonably control the cost of enhanced data on the basis of pursuing high test accuracy, and achieve the joint optimization of test accuracy and the number of enhanced samples.

[0109] In some embodiments, the optimization goal is to maximize the test accuracy function f(A) and the enhancement sample number function g(N aug ), the objective function satisfies the following expression:

[0110]

[0111] 0≤N aug ≤N GAN ;

[0112]

[0113] It should be noted that reinforcement learning (RL) is a computational paradigm that aims to learn the optimal strategy in different states by interacting with the environment, thereby maximizing the cumulative reward of an episode. The dynamic learning characteristics of RL make it particularly suitable for solving non-convex problems. In RL, MDP is a widely used modeling framework. It is a control process defined as a four-tuple <state (S), action (A), reward (R), transition probability (P)>. At the decision time t of RL, the agent makes a decision based on the current environment state S. t Select an action t , then transfer to the next state S t+1 And get a reward t The optimization goal of the agent is to maximize the cumulative reward of an episode, and the cumulative reward satisfies the following expression:

[0114]

[0115] Among them, γ is a discount factor, which is used to weigh the importance of immediate rewards and future rewards. Q-learning is a classic offline value function-based optimization algorithm that can maximize the cumulative reward of an episode in RL. The core of Q-learning is the Q value update based on the Bellman Equation. The Bellman Equation describes the recursive relationship between the state-action value function (Q function), which is expressed as the following expression:

[0116]

[0117] Where ρ is the learning rate, Q(s t ,a t ) is used to describe the agent in state s t Take action a t In Q-learning, Q values ​​are stored in a table and updated iteratively according to the above Bellman equation.

[0118] S207. Based on the multi-agent reinforcement optimization algorithm, at least one initial model, enhanced data strategy, local model and strategy reward are used as training samples and stored in the sample pool.

[0119] In some embodiments, the sample pool (Replay Buffer) is used in reinforcement learning to store the experience (i.e., the four-tuple of state, action, reward, and next state) generated during the interaction between the agent and the environment. After each agent completes an episode of training, their initial model parameters, enhanced data strategy, local model parameters, and strategy rewards are encapsulated into a training sample and stored in the sample pool. These samples will be used in the subsequent reinforcement learning training process.

[0120] In some embodiments, after each agent completes the state transfer, client i downloads the actions and enhanced sample situations N selected by other agents from the blockchain. t (aug), blockchain provides a decentralized and secure environment for recording and sharing this information. As in S204 above, the client generates a global observation based on local information and information shared by other agents. This global observation includes the client's local observation of the current environment state and the actions of other agents downloaded from the blockchain (enhanced data strategy) and enhanced sample conditions. Similarly, the client generates the next global observation It reflects the expected state of the environment after executing the action. The client creates a transition sample that contains the current global observation action Rewards Received and the next global observation This transfer sample is the key record of the interaction between the agent and the environment, containing all the information needed for learning. The samples are stored in the sample pool, thereby improving learning efficiency and stability and enhancing the decision-making ability of the intelligent agent in complex environments.

[0121] It should be noted that, unlike Single-Agent Reinforcement Learning (SARL), MARL extends the principles of RL to multi-agent scenarios, focusing on the joint learning of a group of agents that coexist and may compete in a shared environment. In order to alleviate the convergence problem caused by partial observability, the MARL research of related technologies has proposed three main learning schemes: fully centralized learning (FCL), independent learning (IL), and centralized training and decentralized execution (CTDE). FCL allows agents to share state information, thereby solving the non-steady-state problem, but when the number of agents is huge, the communication overhead will increase significantly; in IL, the agent chooses actions based only on local observations, so the non-steady-state problem is serious; CTDE combines the advantages of FCL and IL, solves the communication overhead problem of FCL through local execution, and handles the non-steady-state problem of IL through centralized learning. Similar to the Q-learning described in S206 above, VDN is a value-based algorithm that decomposes the joint action-value function into separate components. VDN converts the total value function Q tot (s,a) is decomposed into a single value function Q i (s,a), the total value function satisfies the following expression:

[0122]

[0123] Where B is the number of agents. To approximate the global state, VDN uses historical state information to estimate the value decomposition. This approach allows agents to learn the global optimal strategy through collaboration when only knowing local observations.

[0124] S208. Iteratively train the local model according to the training samples.

[0125] In some embodiments, in a multi-agent environment, the agent randomly selects samples from the sample pool to train a local multi-agent deep deterministic policy gradient (MADDPG) network, i.e., a local local model. MADDPG is an algorithm that combines deep learning and policy gradients. It extends the deep deterministic policy gradient algorithm to multi-agent scenarios to iteratively optimize the policy. In each iteration, the agent updates its policy network based on the samples drawn from the sample pool to better adapt to the environment and improve the cumulative reward.

[0126] The data enhancement method based on federated reinforcement learning provided by the present application can increase the amount of data for model training by generating the first enhanced data set generated by the adversarial network, thereby increasing the diversity of data and improving the generalization ability of the model. The initial models of each client are fused into a global model using the federated average algorithm, which can integrate the knowledge of multiple clients while protecting data privacy and improve the performance of the global model. Furthermore, based on the Markov decision process, the first model parameters and the first sample number of the first enhanced data set, the enhanced data strategy is dynamically determined. This strategy can intelligently adjust the generation and use of enhanced data according to the performance of the current model and the size of the data set to achieve the best training effect. Then, the strategy reward is evaluated by the test accuracy of the global model and the number of enhanced samples, which helps to optimize the data enhancement strategy and ensure that the cost of data enhancement is controlled while improving the performance of the model. Further, using the optimization algorithm of multi-agent reinforcement learning, the models of multiple clients can be trained at the same time, so that each client can learn from the experience of the global model and other clients, thereby improving the learning efficiency and performance of the entire system. Finally, the client's initial model, enhanced data strategy, local model and strategy reward are stored in the sample pool as training samples, and the local model is iteratively trained based on these samples to continuously optimize the model performance. Through federated learning, the model training process is dispersed to each client, avoiding centralized storage and processing of sensitive data, effectively protecting data privacy, and balancing the model performance and data enhancement costs to ensure that the demand for computing resources and data storage is reduced while improving the model accuracy.

[0127] In some embodiments, Figure 5 As shown, the data enhancement method based on federated reinforcement learning provided in the embodiment of the present application also includes the following S209:

[0128] S209: In response to the iterative training of the local model reaching a preset condition, stop iteration.

[0129] The preset condition includes at least one of the following: the number of iterations reaches a first preset threshold and the strategy reward reaches a second preset threshold.

[0130] In some embodiments, before starting iterative training, two preset conditions are set to determine when to stop the iterative process. These conditions are intended to balance the sufficiency and efficiency of training and ensure that the model does not perform unnecessary iterations after reaching a certain performance level. The training samples in the sample pool are used as the first model parameters, and the steps of iterative S201 to S207 are repeated. When the number of iterations of the agent reaches a first preset threshold, or the reward obtained in the environment through its strategy reaches or exceeds a second preset threshold, the training is stopped.

[0131] like Figure 6As shown, the embodiment of the present application provides a flow chart of a data enhancement method based on federated reinforcement learning. The present application obtains a new multi-agent reinforcement learning optimization algorithm (MARLO for short). First, the client trains a convolutional neural network model (initial model) locally. For a specific description, reference may be made to the relevant description of S201 above, which is not described in detail in this application. Then, the initial model of each client is uploaded to the blockchain, and processed by a federated average algorithm on the central server to obtain a global model. For a specific description, reference may be made to the relevant description of S203 above, which is not described in detail in this application. Further, according to the deep deterministic policy gradient and based on the Markov decision process, an enhanced data strategy (action) is determined. For a specific description, reference may be made to the relevant description of S204 above, which is not described in detail in this application. Finally, the enhanced data generated based on the generative adversarial network is added to the client according to the enhanced data strategy, and the local model of the client is trained based on the new enhanced data set (second enhanced data set). For a specific description, reference may be made to the relevant description of S202 and S205 above, which is not described in detail in this application.

[0132] It should be understood that modulation recognition is indispensable in modern communication systems. The traditional method uploads the modulation data distributed in the network nodes to the central node for centralized training. This direct data upload method will lead to data privacy leakage, so federated learning is widely used in the AMC field. However, in practical applications, especially at edge nodes, the data between nodes are often not independent and identically distributed, and this distribution method will seriously limit the training performance of FL. In order to effectively meet this challenge, this application uses a federated learning mechanism to allow edge nodes to collaboratively train models while keeping the original data locally, effectively ensuring the security of local signal data.

[0133] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0134] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0135] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a data enhancement device based on federated reinforcement learning.

[0136] refer to Figure 7 , the data enhancement device based on federated reinforcement learning includes: an acquisition module 701, a determination module 702, a processing module 703, a storage module 704 and a training module 705;

[0137] The acquisition module 701 is used to acquire a first model parameter of at least one initial model; each initial model in the at least one initial model corresponds to a client; the first model parameter is a signal modulation mode data set;

[0138] The determination module 702 is used to process the first model parameters based on a generative adversarial network to determine a first enhanced data set;

[0139] The processing module 703 is used to process the at least one initial model through a federated average algorithm to obtain a global model;

[0140] The determination module 702 is further used to determine an enhanced data strategy based on a Markov decision process, the first model parameter, and a first sample number of the first enhanced data set; the first sample number is used to characterize the number of enhanced samples added to each client of the at least one client;

[0141] The training module 705 is used to, for each of the at least one initial model, train the initial model according to the second model parameter of the global model and the enhanced data strategy to obtain a local model;

[0142] The processing module 703 is further used to obtain a policy reward according to the test accuracy of the global model on the test set and the second sample number; the second sample number is used to represent the number of enhanced samples added to the at least one client;

[0143] The storage module 704 is used to store the at least one initial model, the enhanced data strategy, the local model and the strategy reward as training samples in a sample pool based on a multi-agent reinforcement optimization algorithm;

[0144] The training module 705 is also used to iteratively train the local model according to the training samples.

[0145] In some embodiments, the processing module 703 is also used to stop iteration in response to the iterative training of the local model reaching a preset condition; the preset condition includes at least one of the following: the number of iterations reaches a first preset threshold and the strategy reward reaches a second preset threshold.

[0146] In some embodiments, the training module 705 is specifically used to determine a second enhanced data set according to the enhanced data strategy;

[0147] The initial model is trained according to the second model parameters and the second enhanced data set to obtain the local model.

[0148] In some embodiments, the number of samples of the second enhanced dataset is less than or equal to the first enhanced dataset.

[0149] In some embodiments, the first model parameters are obtained by compression decomposition based on a tensor decomposition algorithm.

[0150] In some embodiments, the data enhancement device based on federated reinforcement learning also includes an initialization module 706.

[0151] The initialization module 706 is used to initialize the initial model.

[0152] In some embodiments, the initial model is a convolutional neural network model.

[0153] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0154] The device of the above embodiment is used to implement the corresponding data enhancement method based on federated reinforcement learning in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0155] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data enhancement method based on federated reinforcement learning described in any of the above embodiments is implemented.

[0156] Figure 8 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0157] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0158] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0159] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0160] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0161] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0162] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0163] The electronic device of the above embodiment is used to implement the corresponding data enhancement method based on federated reinforcement learning in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0164] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the data enhancement method based on federated reinforcement learning as described in any of the above embodiments.

[0165] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0166] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the data enhancement method based on federated reinforcement learning as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0167] Based on the same inventive concept, corresponding to the data enhancement method based on federated reinforcement learning described in any of the above embodiments, the present disclosure also provides a computer program product, which includes a computer program. In some embodiments, the computer program is executable by one or more processors so that the processor executes the data enhancement method based on federated reinforcement learning. Corresponding to the execution subject corresponding to each step in each embodiment of the data enhancement method based on federated reinforcement learning, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0168] The computer program product of the above embodiment is used to enable the processor to execute the data enhancement method based on federated reinforcement learning as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0169] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0170] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0171] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0172] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A data enhancement method based on federated reinforcement learning, characterized in that: The method comprises: Acquire a first model parameter of at least one initial model; each initial model in the at least one initial model corresponds to a client; the first model parameter is a signal modulation mode data set; Processing the first model parameters based on a generative adversarial network to determine a first enhanced data set; Processing the at least one initial model through a federated averaging algorithm to obtain a global model; Determining an enhanced data strategy based on a Markov decision process, the first model parameter, and a first sample number of the first enhanced data set; the first sample number is used to characterize the number of enhanced samples added to each client of at least one client; For each of the at least one initial model, training the initial model according to the second model parameter of the global model and the enhanced data strategy to obtain a local model; Obtaining a policy reward according to the test accuracy of the global model on the test set and the second sample quantity; the second sample quantity is used to represent the number of enhanced samples added to the at least one client; Based on a multi-agent reinforcement optimization algorithm, the at least one initial model, the enhanced data strategy, the local model, and the strategy reward are stored in a sample pool as training samples; The local model is iteratively trained according to the training samples.

2. The method according to claim 1, characterized in that: The method further comprises: In response to the iterative training of the local model reaching a preset condition, the iteration is stopped; the preset condition includes at least one of the following: the number of iterations reaches a first preset threshold and the policy reward reaches a second preset threshold.

3. The method according to claim 1, characterized in that: The step of training each of the at least one initial model according to the second model parameter of the global model and the enhanced data strategy to obtain a local model comprises: Determining a second enhanced data set according to the enhanced data strategy; The initial model is trained according to the second model parameters and the second enhanced data set to obtain the local model.

4. The method according to claim 3, characterized in that The number of samples of the second enhanced data set is less than or equal to the first enhanced data set.

5. The method according to claim 1, characterized in that The first model parameters are obtained by compression and decomposition based on a tensor decomposition algorithm.

6. The method according to claim 1, characterized in that Before obtaining the first model parameter of at least one initial model, the method further includes: Initialize the initial model.

7. The method according to any one of claims 1 to 6, characterized in that: The initial model is a convolutional neural network model.

8. A data enhancement device based on federated reinforcement learning, characterized in that: The device comprises: an acquisition module, a determination module, a processing module, a storage module and a training module; The acquisition module is used to acquire a first model parameter of at least one initial model; each of the at least one initial model corresponds to a client; the first model parameter is a signal modulation mode data set; The determination module is used to process the first model parameters based on a generative adversarial network to determine a first enhanced data set; The processing module is used to process the at least one initial model through a federated average algorithm to obtain a global model; The determination module is further used to determine an enhanced data strategy based on a Markov decision process, the first model parameter, and a first sample number of the first enhanced data set; the first sample number is used to characterize the number of enhanced samples added to each client of the at least one client; The training module is used for, for each of the at least one initial model, training the initial model according to the second model parameter of the global model and the enhanced data strategy to obtain a local model; The processing module is further used to obtain a policy reward according to the test accuracy of the global model on the test set and the second sample quantity; the second sample quantity is used to represent the number of enhanced samples added to the at least one client; The storage module is used to store the at least one initial model, the enhanced data strategy, the local model and the strategy reward as training samples in a sample pool based on a multi-agent reinforcement optimization algorithm; The training module is also used to iteratively train the local model according to the training samples.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

Images