Adaptive federated learning method and system based on digital twinning in edge computing environment

Abstract

The application belongs to the technical field of edge computing and federated learning, and specifically discloses an adaptive federated learning method and system based on digital twinning in an edge computing environment, which comprises the following steps: obtaining state information of each device at the current moment and model parameters obtained in local training of the device at the current moment, and aggregating global model parameters x t from model parameters uploaded by part of the industrial internet of things devices selected at the last moment; using a trained deep reinforcement learning agent model to optimize the configuration of each device at the next moment according to the current state information of all devices, and preferentially selecting the devices for aggregation of global model parameters at the next moment according to the configuration result of the bandwidth ratio h n . The problems of Non-IID and resource allocation existing in federated learning under digital twinning are solved, real-time online optimization of the system is supported, and the robustness of the system under the condition of unfriendly channel state is improved.

Description

Technical Field

[0001] This invention belongs to the field of edge computing and federated learning technology, and specifically discloses an adaptive federated learning method and system based on digital twins in an edge computing environment. Background Technology

[0002] Traditional cloud computing can no longer meet the stringent latency requirements of Industrial Internet of Things (IIoT) applications. Edge computing (EC), as a promising technology, can extend traditional cloud services to edge networks closer to end devices, making it suitable for network services with lower latency requirements. Meanwhile, in machine learning (ML)-based IIoT, the implementation of edge intelligent services relies on real-time status processing and monitoring of a large number of devices. However, due to the random communication latency and the rapid increase in operating devices in IIoT, edge servers struggle to perform online optimization by analyzing the operational environment, such as channel status information of IoT devices.

[0003] Digital Twin (DT), as an emerging technology, provides a bridge between the real-time physical state and virtual space of the Industrial Internet of Things (IIoT). Typically, servers possess sufficient resources to maintain the ever-changing digital objects of IIoT devices in real time. Specifically, a digital object is created in virtual space through software definition and sensor perception; it is a timely digital representation of the state, characteristics, and evolution of a physical entity. DT's excellent state awareness and real-time analysis capabilities significantly improve the efficiency of control decisions. However, DT is a data-driven approach, relying on the analysis of massive amounts of data from distributed IoT devices. Nevertheless, due to considerations of business competition and privacy protection, manufacturers are reluctant to exchange private data from their IoT devices in virtual space. Therefore, the existence of data silos poses a challenge to using DT to build digital objects for physical systems.

[0004] Federated Learning (FL), as a novel machine learning (ML) technology, achieves a new application paradigm of "data available but not visible" and "data does not move but the model moves" by exchanging model parameters without uploading data to a central server. In DT's IoT systems, federated learning enables the construction of intelligent models by making flexible decisions based on the variable state information of the industrial IoT system. Therefore, introducing FL into DT-based IIoT can not only improve control efficiency but also increase manufacturers' willingness to participate in global model training, thereby obtaining more accurate digital objects within DT.

[0005] However, in the EC environment, the non-independent and identically distributed (Non-IID) data across devices and limited edge resources make it very difficult to maintain virtual objects in the digital space through FL-integrated DT technology. Meanwhile, in the edge environment, DTs can interact to form a DT Edge Network (DTEN), which works in real time with IIoT devices, providing consistent feedback. Therefore, dynamic optimization of physical entities can be achieved by optimizing the DTs within the DTEN. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes an adaptive federated learning method based on digital twins in edge computing environments.

[0007] The method includes:

[0008] Obtain the current state information of each industrial IoT device or its digital twin, as well as the model parameters currently trained locally on the device. The status information includes the actual CPU frequency γ of the device. n Transmission power p n The transmission rate v between the device and the base station n ;

[0009] Based on the model parameters uploaded by the selected industrial IoT devices at the previous moment. Aggregation yields global model parameters x t ;

[0010] Using a trained deep reinforcement learning (DRL) agent model, the CPU frequency γ of each device is optimized for the next time step based on the current state information of all devices. n Transmission power p n and bandwidth ratio h n ;

[0011] The optimization objective of the deep reinforcement learning agent model is to minimize the global loss function of model training and the total resource consumption under the corresponding constraints.

[0012] At the same time, according to the bandwidth ratio h n The configuration results are preferred for devices used in the next time step for global model parameter aggregation. The preferred condition is that, in the current time step's configuration results, the bandwidth allocated to the device is higher than that allocated to the device in the next time step.

[0013] Repeat the above process, performing multiple resource allocations and model parameter aggregations, until the conditions for ending federated learning are met.

[0014] Furthermore, the learning agent model is a model constructed and trained based on the Deep Deterministic Policy Gradient (DDPG) method.

[0015] Furthermore, the learning agent model includes an action network. and critics network Q θ The parameters are respectively and θ.

[0016] Furthermore, in the training of the learning agent model, online training is carried out using digital twins of each device.

[0017] Furthermore, the objective function P1 of the learning agent model and its constraints can be expressed as:

[0018]

[0019]

[0020]

[0021]

[0022]

[0023] in:

[0024]

[0025] N is the number of IIoT devices, ξ j It belongs to The sample, For the local dataset of the nth IIoT device, f n (x n ξ j ) represents the sample ξ in the nth IIoT device. j loss function on x n This represents the local model parameters on the nth IIoT device;

[0026] It is a given resource.

[0027] Furthermore, Z=2 indicates that both time and energy resources are considered. The given resources are represented by the total energy and total time consumption in time period t as follows:

[0028]

[0029] Among them, for IIoT devices U n e n This represents the number of CPU cycles required to execute one sample of data, expressed by U.n CPU performance determines, U n The computational energy consumption in one iteration is:

[0030]

[0031] Where κ1 is U n The calculated energy consumption coefficient, and U n The calculation time is:

[0032]

[0033] IoT devices U in each global aggregation n The transmission energy consumption is

[0034]

[0035] Uplink transmission time is

[0036]

[0037] Furthermore, in the training of the learning agent model, the reward function used for updating the action network parameters is:

[0038]

[0039] φ(t)=(α t +β t ) / 2-1

[0040] Where Φ is a constant, and r t ∈(-1,0],α t β represents the model accuracy. t This indicates the ratio of total resource consumption.

[0041] Furthermore, the parameters of the critic network are updated using gradient descent.

[0042] Furthermore, the gradient function minimization problem for updating the critic network parameters can be expressed as follows:

[0043]

[0044]

[0045] Where S represents the number of random samples in the deep deterministic policy gradient method. Let be the sample set, where sj, aj, and rj represent the state space, action space, and reward function of the action network in the j-th random sample of the set, and γ represents the reward discount factor.

[0046] The present invention also discloses an adaptive federated learning system based on digital twins, applied in an edge computing environment, including industrial IoT devices and servers, wherein the servers are equipped with digital twins of each industrial IoT device and trained deep reinforcement learning agent models.

[0047] The system performs adaptive federated learning between industrial IoT devices and servers using the aforementioned method.

[0048] This invention proposes an adaptive resource-efficient FL algorithm based on DT (Automatic Depth-Adaptive) for use in IIoT wireless FL networks (WFLNs). This algorithm mitigates the negative impact of Non-IID data and reduces resource consumption costs. Based on the impact of wireless parameters (such as bandwidth ratio and transmission power) on FL training performance under fixed Non-IID conditions in WFLNs, and leveraging the advantages of adaptive resource allocation in heterogeneous IIoT devices, this invention transforms the DT-based FL resource constraint (DTFL-RC) problem into a joint minimization problem of FL model loss and resource consumption under highly coupled variables of bandwidth ratio and transmission power. For the DTFL-RC problem, AFDT employs a method based on Deep Reinforcement Learning (DRL) to adaptively adjust CPU frequency, transmission power, and bandwidth ratio. This solves the Non-IID and resource allocation problems inherent in DT-based FLs, while supporting real-time online system optimization and improving the system's robustness under unfavorable channel conditions. The AFDT algorithm mitigates the negative impact of Non-IID data and improves the utilization of limited resources by adaptively adjusting the CPU frequency, transmission power, and bandwidth ratio of IIoT devices. Through extensive simulation results, the AFDT algorithm, compared with three benchmark algorithms based on DT in the Industrial Internet of Things, can significantly save communication wheel costs by up to 74.23%, 69.51%, and 60.94% respectively in the worst-case Non-IID scenario. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of a heterogeneous IIoT system model in an embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram of the federated learning process in a WFLN network.

[0051] Figure 3 is The overall framework diagram of the AFDT algorithm in this embodiment of the invention.

[0052] Figure 4 This is a schematic diagram of the ring network region in an embodiment of the present invention.

[0053] Figure 5This is a graph showing the changes in reward and loss functions with the number of iterations when training DRL with different datasets in an embodiment of the present invention.

[0054] Figure 6 This is a comparison chart of model accuracy between different models under different Non-IID data settings in embodiments of the present invention.

[0055] Figure 7 This is a comparison chart of communication costs between different models in this embodiment of the invention.

[0056] Figure 8 This diagram illustrates the resource consumption and DT deviation of the AFDT algorithm under different Non-IID data settings in this embodiment of the invention, as well as the variation of communication rounds. Detailed Implementation

[0057] For ease of understanding, Table 1 provides the important symbols and their definitions used in this embodiment:

[0058] Table 1: List of Main Symbols

[0059]

[0060] like Figure 1 As shown, the exemplary heterogeneous IIoT system model based on DT in this embodiment has three components: IIoT devices, edge servers, and digital twins (DTs) corresponding to the IIoT devices. IIoT devices (such as excavators, vehicles, industrial robots, etc.) use... This indicates that each IIoT device possesses a local dataset for use with... It indicates that the size is D n These IoT devices are interconnected with the base stations (BSs) of the edge servers via wireless networks. This indicates that DT can map the physical state of IIoT devices to virtual space to enable real-time interaction with IIoT devices. IIoT device U n DT model n It is from its vicinity Established, specifically, B m By collecting and processing its existing key physical states, DT dynamically displays its current and historical behavior in digital form at time slot t. n It can be represented as

[0061]

[0062] in γ t n , IIoT devices U n The training status, computing power (i.e., CPU frequency), computing power deviation, and energy consumption.

[0063] In the context of IoT, this embodiment considers a Field-Based (FL) system that ensures industrial production safety. Within the FL framework, IIoT devices (such as excavators, vehicles, and industrial robots) can make satisfactory intelligent control and predictive maintenance decisions in collaboration with Base-Based Systems (BSs). In FL-based IIoT, when an industrial task needs to be completed, these IIoT devices collaboratively train a global model without needing to use their local datasets. Upload to BS. For example, such as Figure 1 As shown, when heterogeneous industrial equipment needs to complete a quality inspection task, each device trains a local detection model based on its own collected dataset. Each device uploads its local model parameters, rather than its local dataset, to the BS (Base Station). The BS performs global aggregation on the local models, updates the global detection model parameters, and broadcasts the updated global parameters to each device for the next local training. Each industrial device repeats these processes until the global model converges. Therefore, in FL-based IIoT, all... The global loss function on is defined as:

[0064]

[0065] Where ξ j It belongs to For the given samples, the objective in this embodiment is to minimize the global loss function:

[0066]

[0067] The resource consumption model for industrial IoT driven by digital twins based on federated learning, established in this embodiment, is as follows:

[0068] (1) Wireless Transmission: For local model uploads, the total bandwidth for all IIoT devices communicating with the BSS via the communication network in DTEN is H. Taking Orthogonal Frequency-Division Multiple Access (OFDMA) communication as an example, let... Because of the bandwidth ratio, therefore, U n In B m The uplink data rate at this location is:

[0069]

[0070] Among them, o n For U n With B m The channel gain between pn For U n The transmission power is N0, where N0 is the power spectral density of Gaussian noise.

[0071] (2) Local computing resource consumption: For IIoT devices U n e n This represents the number of CPU cycles required to execute one sample of data, expressed by U. n The CPU performance determines this. n The CPU frequency is denoted as γ. n Therefore, U n The computational energy consumption in one iteration is:

[0072]

[0073] Where κ1 is U n The calculated energy consumption coefficient, and U n The calculation time is:

[0074]

[0075] (3) Transmission resource consumption: For the uplink, the IoT device U in each global aggregation n The transmission energy consumption is

[0076]

[0077] Where, |x n |For U n The size of the local model parameters, and the uplink transmission time are

[0078]

[0079] Since the BS has sufficient resources and its energy consumption has little impact on the system, this embodiment only considers the resource consumption of IIoT devices. Furthermore, the downlink bandwidth of the BS is much greater than the uplink bandwidth of the IIoT devices; therefore, we ignore the downlink broadcast time of the BS in DTEN. Also note that DTEN can be extended to a scenario where a macro base station (MBS) contains multiple small base stations (SBSs), and the data transmission rate expression from SBSs to MBS is similar to that from IIoT devices to SBSs (i.e., Equation (4)). For ease of analysis, this embodiment only considers the scenario of a single BS; therefore, v will be used in the following discussion. n To represent v n,m .

[0080] Due to the inherent instability of wireless transmission, errors can occur in each data uplink. In the uplink phase, this implementation treats the local model parameters transmitted to the server as a single data packet and uses a cyclic redundancy check (CRC) mechanism to check for transmission errors. Specifically, the local model parameter x... n Packet errors generated during wireless transmission to the BS can be described as follows:

[0081]

[0082] Where m is the waterfall threshold.

[0083] In DTEN, if any local FL model Including errors, BS will not use it to update the global model. Therefore, we can use the global FL model x n Written as:

[0084]

[0085] in And there are

[0086] In this embodiment, some assumptions are made regarding the objective function:

[0087] Assumption 1. For the objective function F(x), F n (x), It is L-smooth, and L > 0. That is, for any... We have Where ||·|| is the L2 norm.

[0088] Assumption 2. For the objective function F(x), F n (x), It is l-strongly convex, and l > 0. That is, for any... We have

[0089] Assumption 3. For any IIoT devices For a given mini-batch ξ n The expectation of this stochastic gradient The average value of the gradient And there are Meanwhile, the upper bound of the stochastic gradient variance is: χ represents the degree of Non-IID. When the data distribution is IID, we have χ = 0. Otherwise, when the data is Non-IID, we have χ > 0.

[0090] Theorem 1. Let Hypothesis 1 through Hypothesis 3 be true and we have γ = max{8ν,τ}, and the learning rate The upper bound of this convergence rate is as follows:

[0091]

[0092] in The detailed proof is provided in Appendix A.

[0093] Remark 1. Theorem 1 shows that the extended form of FL can achieve The convergence rate, which is related to using a decaying learning rate. The standard gradient descent (SGD) algorithm is the same. This means that in WFLN, we can mitigate the effects of Non-IID and thus minimize the global loss by choosing appropriate bandwidth and transmission power.

[0094] In FL-based IIoT, industrial equipment is typically heterogeneous and resource-constrained. For example... Figure 2 As illustrated in the example, IIoT devices with powerful computing capabilities and abundant communication resources will transmit local model parameters to the BS faster than IIoT devices with weaker computing capabilities and poorer communication resources. Meanwhile, considering that the local data possessed by those less powerful devices can significantly contribute to global model training, aggregating only the fastest devices in each global aggregation is an unwise decision. Furthermore, global aggregation only occurs after all device participants (i.e., at time t' when U1's transmission is complete), which may reduce the training efficiency of the global model. After introducing power and communication resource optimizations for the devices, they can participate in global model aggregation approximately simultaneously in time slot t. Therefore, for heterogeneous industrial devices in an FL-based IIoT environment, an adaptive resource allocation strategy is needed. This can be achieved by appropriately configuring the CPU frequency γ. n Transmission power p n and bandwidth ratio h n To achieve this.

[0095] In DTEN, each IIoT device needs to update the local model parameters of FL. and its device status in time slot t They are used for updating the global model and synchronization, respectively. Given a FL (Follow-up) task in an IIoT scenario, this embodiment divides the FL time period into T ≥ 1 communication time slots. For There exist τ≥1 local training instances. At each t, ​​each IIoT device U n of and The value of should be optimized to minimize the global loss function while achieving optimal resource allocation under resource constraints. It is worth noting that... and The value changes only during the global aggregation phase and remains unchanged during the local training phase. In this embodiment, it is assumed that the allocated bandwidth is greater than the threshold h if and only if... min ,Right now IIoT devices U n Only then will it be selected for local model training in round t+1. Therefore, we can formulate the DTFL-RC problem as:

[0096]

[0097]

[0098]

[0099]

[0100]

[0101] in Given resources (such as time, energy, and memory), this invention considers Z different types of resources. Taking the consumption of two resources, time and energy, i.e., Z=2, as an example, the total energy and total time consumption for time period t are given in this embodiment as follows:

[0102]

[0103] Formula (12a) means that resource consumption cannot exceed a given resource limit. Otherwise, the iteration process of FL will terminate, and formula (12b) is the CPU frequency γ. n The constraint condition, namely the CPU frequency γ n It should be between the minimum CPU frequency γ min and maximum CPU frequency γ max Between. Formula (12c) represents the transmission power p. n The constraint condition, namely the transmission power p n It must be greater than the minimum transmission power p min and less than the maximum transmission power p max Formula (12d) indicates that the sum of all allocated bandwidth ratios is 1.

[0104] At each time slot t, due to the CPU frequency γ n Transmission power p n and bandwidth ratio h nThe determination of P1 is a stochastic process, making it difficult for traditional convex optimization algorithms to solve for P1. Furthermore, as shown in Remark 1 and the preceding content, the transmission power p... n and bandwidth ratio h n These factors jointly determine the global loss and resource assumptions. Therefore, bandwidth ratio and transmission power are highly coupled in P1. This complex coupling makes solving P1 difficult. Leveraging the model-free learning advantage of DRL in handling stochastic optimization problems, this embodiment uses DRL to solve P1.

[0105] To address the P1 problem, this embodiment proposes an AFDT algorithm for the Industrial Internet of Things (IIoT), with the specific framework as follows: Figure 3 As shown in the diagram, the DTs' states (such as data transmission rate, CPU frequency, and model parameters) are transmitted to a learning agent, which is constructed using a DRL method called Deep Deterministic Policy Gradient (DDPG). Then, the optimal decision is derived based on DDPG to minimize the global model loss and improve resource utilization.

[0106] First, the Markov decision process based on DT in this embodiment can be described as follows:

[0107] 1) State Space: To solve P1, this embodiment uses DRL to explore the action space and represents problem P1 as a Markov Decision Process (MDP), with the state space s t Constructed from DTs. At time slot t, we can construct the state space s. t Defined as:

[0108]

[0109]

[0110]

[0111]

[0112] Where, x t This represents the global model parameters at time slot t. Additionally, the actual CPU speed of the IIoT device is considered. With the CPU speed issued There is a deviation between them, and random noise is used in the state space. To approximate this phenomenon, there are noise The range of fluctuations can be set manually.

[0113] 2) Action Space: Action space is defined as a t ={γ t ,p t ,h t}, which contains the optimization variables for P1. Among them,

[0114] 3) Rewards: The immediate reward consists of two parts: the model accuracy and the total resource consumption ratio after each global aggregation. This embodiment defines the model accuracy and the total resource consumption ratio as α. t =Acc t / Acc target and Among them Acc target This refers to the model's target accuracy. In time slot t, the reward function is defined as follows:

[0115]

[0116] Where Φ is a constant, and r t ∈(-1,0], because φ(t)∈(-1,0). If α t →1 or β t →1, FL training will stop, therefore, the cumulative reward can be written in the following form:

[0117]

[0118] Here, γ is the discount factor for future rewards.

[0119] Now, regarding Φ in formula (16) φ(t) To explain, we can obtain r from formula (16). t The value depends on Acc t and Φ φ(t) This can incentivize DRL agents to make decisions with higher model accuracy and lower resource consumption. In P1, our goal is to maximize model accuracy and minimize resource consumption; therefore, in equation (15), for Acc... t and Give them the same weight.

[0120] To solve problem P1, this embodiment first presents the detailed process of AFDT as shown in Algorithm 1 and Algorithm 2. Next, as... Figure 1 As shown, these DTs (Devices and Devices) interact with the DRL agent instead of physical objects, thereby achieving optimal decision-making in P1. DT-based FL (Flexible Interaction) not only achieves the same training results as the real environment but also has lower resource consumption. The initial phase initializes global model parameters x0, CPU frequency γ0, transmission power p0, bandwidth ratio h0, and device settings. Acc0 = 0, ensuring that all devices participate in the first global aggregation (line 23-31 of Algorithm 1).

[0121] Time period The DRL agent, composed of relevant parameters, is backed up to the server, and the corresponding DTs replicas are updated (Algorithm 1, line 30). Then, the DRL agent executes the action network. Obtain the action space (see lines 11-19 of Algorithm 1). Whether a device participates in the next global aggregation is based on h. n and h min The value is compared to determine the value (Algorithm 1, lines 37-42). Next, the selected IIoT device will be determined based on its local data. Train the local model and use these selected devices to obtain the next global model (Algorithm 1, lines 2-8 and 31). Finally, calculate the resource consumption and global model accuracy, if α... t →1 or β t →1, training of FL will stop (Algorithm 1, lines 32-34).

[0122]

[0123]

[0124]

[0125] In this embodiment, the determination of CPU frequency, transmission power, and bandwidth ratio constitutes a continuous variable MDP, which DQN and DDQN cannot solve. DDPG, as one of the most advanced action-commentator methods, inherits the advantages of DDQN while solving the continuous MDP problem. Therefore, DDPG is used in the DRL agent to achieve online learning and solve the continuous MDP problem. The DRL agent contains two network entities (i.e., the action network). and critics network Q θ ), parameters are and θ.

[0126] Algorithm 2 details the training process of the DRL agent.

[0127]

[0128]

[0129] At the start of the training process, random initialization is performed. Q θ The relevant parameters are initialized, along with other parameters (lines 1-3 in Algorithm 2). After the above initialization is completed, the relevant parameters are broadcast to each DTs for local training (line 5 in Algorithm 2). Next, the current state s is... t Feedback to the action network Thus, we obtain the action a. tThen, the device trains a local model based on the action. Afterwards, it obtains the instantaneous reward r. t The state is determined by s t Convert to s t+1 (Lines 8-13 in Algorithm 2). Note that the parameter only changes when the transition buffer P is full. Only when θ is updated (lines 15-22 in Algorithm 2). Considering the huge state space, which is not conducive to DRL training, we use PCA to reduce s t Dimensionality and resource consumption.

[0130] We will only analyze the computational complexity of the core part of Algorithm 2. For lines 9-11 of Algorithm 2, each set has... Each device performs T global aggregations. The maximum value is N, therefore its computational complexity is O(N). For lines 15-23 of Algorithm 2, there are T global aggregations in each set, and in each... In this process, there can be at most I training iterations; therefore, its computational complexity is O(I). From the above analysis, it can be seen that for set Y, the computational complexity of Algorithm 2 is O(n). For Algorithm 1, step τ is performed between the two global aggregations. Therefore, for each device, the computational complexity of the local update process is O(n).

[0131] To evaluate the performance of the AFDT algorithm, this embodiment considers the following: Figure 4 The diagram shows a ring network area with a radius of 500m. 100 IIoT devices are evenly distributed within this area and connected to a central base station. AFDT is compared with three representative benchmark algorithms to verify its performance under different Non-IID data settings and resource capacities.

[0132] Control parameters and experimental environment: The experimental parameters related to wireless transmission are listed in Table 2. Here, the coefficient m is set to 0.01, the learning rate η = η′ = 0.01, and the transaction... The capacity is P = 10, Φ = 100, and a future reward discount factor γ = 0.9 is set. We consider two resource types, Z = 2 (i.e., energy resources and time resources), both of which are consumed during each local update and global aggregation. These experiments were conducted by opening multiple processes on a single host, with one process representing a single client. The backend algorithm was developed in Python 3.8 and tested on a single processor (Intel Core™ i7-11700 @ 2.50GHz).

[0133] Table 2: Parameter values ​​for wireless transmission

[0134]

[0135] Datasets: Three real-world datasets were used in the experiments: MNIST, FasdionMNIST, and CIFAR-10. MNIST and FasdionMNIST each contain 6,000,0028*28 training images and 10,000 test images across 10 classes; CIFAR-10 contains 600,00032*32 color images, with 50,000 training images and 10,000 test images. Furthermore, the learning rate for training MNIST and FasdionMNIST with the CNN was 0.001, while the learning rate for training CIFAR-10 was set to 0.01. The local epoch count and batch size for each dataset were 5 and 10, respectively.

[0136] Benchmarks: In this example, AFDT is compared with three reference benchmarks, as described below:

[0137] •FDTA: The parameters are the same as the AFDT algorithm, except that a fixed bandwidth ratio, CPU frequency and transmission power are used. The BS will randomly select δ×U devices to participate in FL with δ=0.5.

[0138] •FDTB: Compared to the AFDT algorithm, this method only uses an adaptive bandwidth ratio, while using a fixed CPU frequency and transmission power.

[0139] • DTFavor: Favor is a device selection algorithm based on DDQN. When there are a total of 100 devices, 10 devices are selected. The DT-based wireless environment is added to Favor to form DTFavor. It has a fixed bandwidth ratio, CPU frequency and transmission power.

[0140] DRL agent training: The performance of the DRL agent is first demonstrated by examining the training loss and reward. Actor network in DRL. and critics network Q θ The model consists of a two-layer MLP network with 100 hidden states. Figure 5 Figures (a) to (c) illustrate the DRL training process on different datasets. As can be seen from the figures, both the reward and loss tend to converge after approximately 20 iterations, demonstrating the effectiveness of the DRL agent. Based on this observation, it can be seen that the proposed DRL agent converges quickly and can serve as an effective mechanism.

[0141] Efficiency of Different Non-IID Data Settings: This example compares the model accuracy of AFDT with the three benchmarks mentioned above on different Non-IID data settings. The Non-IID data settings range from 0.2 to 1 to obtain the corresponding model accuracy. Specifically, we use μ to represent four different Non-IID data settings: for example, μ = 0.2 means that 20% of the data on each device belongs to the same tag, and the remaining 80% belongs to other tags. Similarly, μ = 0.5, μ = 0.8, and μ = 1 can also be defined.

[0142] like Figure 6 As shown, the accuracy of all schemes increases with the number of communication rounds. For all three datasets, it can be observed from the graph that the accuracy required to reach convergence and the number of communication rounds decrease and increase respectively with increasing μ. This phenomenon indicates that the higher the level of Non-IID data, the worse the overall model, which corresponds to the actual situation. It is worth noting that as the value of μ increases, AFDT and FDTB outperform FDTA and DTFavor in most cases, especially on the CIFAR-10 dataset. This is because, compared to FDTA and DTFavor which choose fixed CPU frequencies, transmission power, and communication bandwidth, AFDT and FDTB choose more flexible CPU frequencies, transmission power, and communication bandwidth ratios. Furthermore, AFDT also outperforms FDTB in all Non-IID data settings because FDTB only adjusts the bandwidth ratio of all devices, while AFDT adjusts all CPU frequencies, transmission power, and bandwidth ratios of all devices.

[0143] Performance analysis of different Non-IID data settings at a given model accuracy: This example compares the communication costs of AFDT with DTFA, DTFB, and DTFavor at different μ values ​​for a given model accuracy. To make the experimental results more convincing, five simulations were performed in this example to obtain the average number of communication rounds, as shown in the following figures. Figure 7 As shown in the figure, it can be seen that, under the same model accuracy, the communication round cost of AFDT is consistently lower than that of DTFA, DTFB, and DTFavor. Specifically, when Non-IID is set to μ=1, compared with DTFA, DTFB, and DTFavor, AFDT saves 74.23%, 69.51%, and 60.94% of the communication loop cost on the three datasets, respectively. Furthermore, it can be noted from the figure that, in most cases, as the value of μ increases, AFDT can save a higher proportion of the communication cost, because the larger μ is, the lower the probability of model convergence. These results verify the advantage of the AFDT algorithm over DTFA, DTFB, and DTFavor algorithms in reducing communication loop cost.

[0144] AFDT Performance Analysis and DT Bias Comparison under the Same Resource Consumption: This section presents the resource consumption and DT bias analysis of AFDT under different μ values ​​for a given number of communication rounds. Only results from the MNIST dataset are shown here. Similarly, the average resource cost is obtained through 5 simulations, and the results are as follows... Figure 8 As shown in the figure. Simulation results show that the training process of the DRL agent on the FashionMNIST and CIFAR-10 datasets is exactly the same as that on MNIST, and the results also show similar trends.

[0145] It can be seen that, under different μ values, for a given number of communication rounds, the time and energy costs of AFDT are consistently lower than those of DTFA, DTFB, and DTFavor. Note that among all schemes, DTFA consistently has the highest time and energy costs because, in most cases, the number of devices participating in each round of communication is the largest. On the MNIST dataset, another interesting phenomenon is that DTFavor has a small time cost but a huge energy cost. This phenomenon can be explained by the fact that a small number of devices with fixed CPU frequencies, transmit power, and bandwidth ratios participate in each round of communication. This is because, during training, the time cost depends more on the number of devices, while the energy cost depends more on the values ​​of CPU frequency, transmit power, and bandwidth ratio. Furthermore, the time and energy costs of DTFB are almost the same as those of AFDT, because the bandwidth ratio not only determines the number of devices participating in each round of communication but also affects the energy cost during training. Additionally, as... Figure 8 As shown in (c), the DT bias affects the performance of AFDT in all μ cases. The DT bias has the greatest impact on the performance of AFDT when μ = 1, and this impact gradually occurs earlier as the value of μ increases.

[0146] The method and system in this embodiment utilize resource-efficient adaptive federated learning (AFDT) for Non-IID data across heterogeneous devices in the Industrial Internet of Things (IIoT). By selecting appropriate wireless parameters to minimize the global loss of WFLN (Wide-Field-Federated Learning), the impact of Non-IID can be mitigated, and resource optimization can be achieved through reasonable configuration of wireless parameters. Based on this, this embodiment transforms the DTFL-RC problem into a joint minimization problem of FL model loss and resource consumption, and proposes the AFDT algorithm to solve this problem. Simulation results verify that AFDT outperforms multiple benchmark tests in DT-based IIoT.

Claims

1. An adaptive federated learning method based on digital twins in an edge computing environment, characterized in that, include: Obtain the current state information of each industrial IoT device or its digital twin, as well as the model parameters obtained during local training on the device. The status information includes the actual CPU frequency of the device. Transmission power The transmission rate between the device and the base station ; Based on the model parameters uploaded by the selected industrial IoT devices at the previous moment. Aggregation yields global model parameters ; Using a pre-trained deep reinforcement learning agent model, the CPU frequency of each device is optimized for the next time step based on the current state information of all devices. Transmission power and bandwidth ratio In the training of the deep reinforcement learning agent model, online training is carried out using digital twins of each device; the deep reinforcement learning agent model is a model constructed and trained based on the deep deterministic policy gradient method. The optimization objective of the deep reinforcement learning agent model is to minimize the global loss function of model training and the total resource consumption under the corresponding constraints. The objective function P1 and its constraints of the deep reinforcement learning agent model are expressed as follows: ； in: , N is the number of IIoT devices. It belongs to The sample, For the first Local datasets of IIoT devices Indicates the first Among the IIoT devices, the sample loss function on, Indicates the first Local model parameters on each IIoT device; It is a given resource; Z = 2 indicates that both time and energy resources are considered. The given resources are represented by the total energy and total time consumption in time period t as follows: ， Among them, for IIoT devices , This represents the number of CPU cycles required to execute one sample of data, determined by... The CPU performance determines The computational energy consumption in one iteration is: ， in, for The calculated energy consumption coefficient, and The calculation time is: ， IoT devices in each global aggregation The transmission energy consumption is: ， Uplink transmission time is: ; At the same time, according to the bandwidth ratio The configuration results are preferred for devices used in the next time step for global model parameter aggregation. The preferred condition is that, in the current time step's configuration results, the bandwidth allocated to the device is higher than that allocated to the device in the next time step. , This represents the minimum threshold for allocated bandwidth. Repeat the above process, performing multiple resource allocations and model parameter aggregations, until the conditions for ending federated learning are met.

2. The method according to claim 1, characterized in that, The deep reinforcement learning agent model includes an action network. and the network of critics The parameters are respectively and .

3. The method according to claim 2, characterized in that, In the training of the deep reinforcement learning agent model, the reward function used for updating the action network parameters is: in, Let be a constant, then , Indicates model accuracy. This indicates the ratio of total resource consumption.

4. The method according to claim 2, characterized in that, The parameters of the critic network are updated using gradient descent.

5. The method according to claim 4, characterized in that, The gradient function minimization problem for updating the parameters of the commentator network can be represented as follows: ； Where S represents the number of random samples in the deep deterministic policy gradient method. For the sampled set, This represents the state space, action space, and reward function of the action network in the j-th random sample of the set. This represents the reward discount factor.

6. An adaptive federated learning system based on digital twins, applied in an edge computing environment, comprising industrial IoT devices and servers, characterized in that, The server contains digital twins of various industrial IoT devices and pre-trained deep reinforcement learning agent models. The system performs adaptive federated learning between industrial IoT devices and servers in accordance with the method described in any one of claims 1-5.

Images