Communication resource allocation method and system based on meta-federal reinforcement learning

Abstract

The invention discloses a communication resource allocation method and system based on meta-federal reinforcement learning. The method comprises the following steps: a communication scheduling device extracts transmission state characteristics from collected data of a current service and inputs the characteristics into a pre-trained meta-federal near-end strategy optimization model; according to a resource allocation strategy output by the meta-federal near-end strategy optimization model, bandwidth resources are allocated to the service; wherein the meta-federal near-end strategy optimization model is obtained by performing meta-training by using local historical service data and a resource allocation strategy through an edge server, then performing meta-adaptation by using the local historical service data and the resource allocation strategy through a plurality of communication scheduling devices, and performing online training. And performing federal aggregation on the model parameters of the multiple online trained meta-federal near-end strategy optimization models to obtain the meta-federal near-end strategy optimization model. According to the invention, the distribution of communication resources can be automatically adjusted to adapt to the dynamic change of a network environment, and the communication efficiency and the network stability are improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a communication resource allocation method and system for meta-federated reinforcement learning. Background Art

[0002] As the scale of power grids expands and the level of intelligence increases, the management of power grid communication resources becomes increasingly complex. Traditional resource management methods often have difficulty coping with the dynamic changes and uncertainties of communication resources in large-scale power grids, requiring more intelligent and flexible management strategies.

[0003] Moreover, with the access of new energy sources and the fluctuation of grid load, the demand for grid communication resources is also changing. Traditional resource scheduling methods often have difficulty responding to these changes in real time, resulting in low resource utilization efficiency and reduced grid stability. Therefore, a method for dynamically mobilizing grid communication resources that can perceive environmental changes in real time and make optimal decisions is needed.

[0004] The power grid communication dispatch method based on dynamic programming uses the optimization technology of staged decision-making to decompose the complex main problem into smaller sub-problems and gradually solve them to achieve the global optimal solution. Dynamic programming is suitable for solving problems with overlapping sub-problems and optimal substructure properties. Dynamic programming can flexibly handle changes and uncertainties in different time periods and is suitable for communication dispatch in dynamic environments.

[0005] In large-scale power grids, the number of nodes, time periods, and possible decision combinations will increase dramatically, resulting in an exponential growth in the state space. This makes the memory and computing time requirements of dynamic programming algorithms uncontrollable. Dynamic programming usually requires the storage of a large number of intermediate results (or states), which consumes a lot of memory when facing large problems and may cause the system to crash or fail to execute. Summary of the invention

[0006] In view of this, the purpose of the present invention is to propose a communication resource allocation method and system based on meta-federated reinforcement learning. Through the edge server and multiple communication scheduling devices jointly training the meta-federated proximal strategy optimization model, that is, through the meta-federated reinforcement learning method, the system can automatically adjust the allocation of communication resources to adapt to the dynamic changes of the network environment and improve communication efficiency and network stability.

[0007] Based on the above purpose, the present invention provides a communication resource allocation method based on meta-federated reinforcement learning, comprising:

[0008] The communication scheduling device extracts transmission status features from the collected data of the current service, and inputs the extracted transmission status features into the pre-trained meta-federation proximal strategy optimization model;

[0009] The communication scheduling device allocates bandwidth resources to the service according to the resource allocation policy output by the meta-federation proximal policy optimization model;

[0010] Among them, the meta-federation proximal policy optimization model is obtained by meta-training the edge server using local historical business data and resource allocation strategies, and then meta-adapting and online training are performed through multiple communication scheduling devices using local historical business data and resource allocation strategies, and then the model parameters of multiple online trained meta-federation proximal policy optimization models are federated and aggregated.

[0011] Preferably, the meta-federation proximal strategy optimization model is trained specifically according to the following method:

[0012] The edge server trains the meta-federation proximal policy optimization model using local historical service data and resource allocation strategy to obtain a meta-trained meta-federation proximal policy optimization model; and sends the meta-trained meta-federation proximal policy optimization model to each communication scheduling device;

[0013] The communication scheduling device trains the meta-federation proximal policy optimization model after meta-training by using local historical service data and resource allocation strategy to obtain the meta-federation proximal policy optimization model after meta-adaptation;

[0014] The communication scheduling device performs online training on the meta-adapted meta-federation proximal policy optimization model, and uploads model parameters of the meta-federation proximal policy optimization model after online training to the edge server;

[0015] The edge server performs federation aggregation on the model parameters uploaded by each communication scheduling device to obtain a global model parameter, and sends the global model parameter to each communication scheduling device;

[0016] The communication scheduling device uses the global model parameters to update the parameters of the local meta-federation proximal policy optimization model to obtain a global meta-federation proximal policy optimization model as the meta-federation proximal policy optimization model finally obtained by training.

[0017] Preferably, the communication scheduling device performs online training on the meta-adapted meta-federation proximal policy optimization model, specifically:

[0018] The communication scheduling device extracts transmission status features from the collected data of the current service, and inputs the extracted transmission status features into the meta-adapted meta-federation proximal policy optimization model; then, according to the resource allocation strategy output by the meta-federation proximal policy optimization model, after allocating bandwidth resources to the service, a reward value is calculated according to the subsequent transmission status features of the service; and the model parameters of the meta-federation proximal policy optimization model are fine-tuned according to the calculated reward value.

[0019] Preferably, the collected business data is multimodal, including: text data and image data; and

[0020] The specific feature of the transmission status of the collected business data extraction is:

[0021] Extracting features of the text data of the business through a first LSTM network;

[0022] Extracting features of the image data of the business through a second LSTM network;

[0023] The features of the text data and the features of the image data of the service are merged to obtain the transmission status features of the data of the service.

[0024] The present invention also provides a communication resource allocation system based on meta-federated reinforcement learning, comprising: an edge server, and a plurality of communication scheduling devices; wherein:

[0025] The communication scheduling device is used to extract transmission status features from the collected data of the current service, input the extracted transmission status features into a pre-trained meta-federation proximal policy optimization model, and allocate bandwidth resources to the service according to the resource allocation strategy output by the meta-federation proximal policy optimization model;

[0026] Among them, the meta-federation proximal policy optimization model is obtained by meta-training the edge server using local historical business data and resource allocation strategies, and then meta-adapting and online training are performed through multiple communication scheduling devices using local historical business data and resource allocation strategies, and then the model parameters of multiple online trained meta-federation proximal policy optimization models are federated and aggregated.

[0027] The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the steps of the above-mentioned communication resource allocation method based on meta-federated reinforcement learning when executing the computer program.

[0028] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned communication resource allocation method based on meta-federated reinforcement learning.

[0029] In the technical solution of the present invention, the communication scheduling device extracts the transmission state characteristics from the collected data of the current service, and inputs the extracted transmission state characteristics into the pre-trained meta-federation proximal policy optimization model; the communication scheduling device allocates bandwidth resources for the service according to the resource allocation strategy output by the meta-federation proximal policy optimization model; wherein the meta-federation proximal policy optimization model is obtained by the edge server using the local historical service data and resource allocation strategy for meta-training, and then multiple communication scheduling devices use the local historical service data and resource allocation strategy for meta-adaptation, and online training, and then the model parameters of the multiple online trained meta-federation proximal policy optimization models are federated and aggregated. The present invention uses the edge server and multiple communication scheduling devices to jointly train the meta-federation proximal policy optimization model, that is, through the meta-federation reinforcement learning method, the system can automatically adjust the allocation of communication resources to adapt to the dynamic changes of the network environment and improve communication efficiency and network stability.

[0030] Moreover, even when the number of nodes, time periods, and possible decision combinations increase dramatically, the edge servers and multiple communication scheduling devices use local data for training without the need to obtain all global data, thus avoiding the need to store a large number of intermediate results (or states) and avoid the problem of system crash or inability to execute due to the consumption of a large amount of memory in the face of large problems;

[0031] On the other hand, when edge servers and multiple communication scheduling devices use local data for training, they do not need to share their data directly, thereby protecting local data privacy and using collective knowledge to update the model. In other words, different nodes can share experience data but not local data, and can benefit from the knowledge of other nodes while learning individually, thereby improving overall performance and protecting local privacy.

[0032] In addition, by performing collaborative training on multiple nodes, the robustness and generalization ability of the model can be improved, avoiding the risks caused by single point failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A schematic diagram of the architecture of a communication resource allocation system based on meta-federated reinforcement learning provided by an embodiment of the present invention;

[0035] Figure 2 A flow chart of a training method for a meta-federation proximal policy optimization model provided by an embodiment of the present invention;

[0036] Figure 3 A flow chart of a communication resource allocation method based on meta-federated reinforcement learning provided by an embodiment of the present invention;

[0037] Figure 4a , 4b A schematic diagram of the structure of the first and second LSTM networks provided in an embodiment of the present invention;

[0038] Figure 5 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0040] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present invention should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0041] The inventors of the present invention consider using the idea of ​​meta-federated reinforcement learning to optimize the allocation and scheduling of power grid communication resources. In this method, different power grid nodes can share experience data, but not local data. While learning individually, they can also benefit from the knowledge of other nodes, thereby improving overall performance and protecting local privacy. Through the meta-federated reinforcement learning method, the system can automatically adjust the allocation of communication resources to adapt to the dynamic changes in the power grid environment, improve communication efficiency and network stability. This method helps to achieve efficient operation and management of smart grids.

[0042] In addition, in the management of power grid communication resources, various types of data are involved, such as text, image data, etc. These data come from different perception modes and are of great significance for the comprehensive understanding and optimization of power grid communication resources. However, how to effectively fuse these multimodal data and use them to make decisions is a problem that needs to be solved urgently. Therefore, when extracting the transmission state characteristics of the data, the present invention adopts a method of extracting transmission state characteristics for business data of different modes respectively, and then fusing them, which overcomes the limitations of traditional methods in processing heterogeneous data, so as to fully understand and optimize power grid communication resources.

[0043] The technical solutions of the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0044] like Figure 1 As shown, a communication resource allocation system based on meta-federated reinforcement learning proposed in an embodiment of the present invention can be applied to a power grid communication dispatching system, including: an edge server 101, and a plurality of communication dispatching devices 102 communicating with the edge server 101;

[0045] Among them, the communication scheduling device 102 extracts transmission status features from the collected data of the current service, inputs the extracted transmission status features into the pre-trained meta-federation proximal policy optimization model, and allocates bandwidth resources to the service according to the resource allocation strategy output by the meta-federation proximal policy optimization model to achieve communication scheduling.

[0046] Among them, the meta-federation proximal policy optimization model is obtained by meta-training by the edge server 101 using local historical business data and resource allocation strategies, and then meta-adapting and online training are performed by multiple communication scheduling devices 102 using local historical business data and resource allocation strategies, and then the model parameters of multiple online trained meta-federation proximal policy optimization models are federated and aggregated.

[0047] Specifically, the edge server 101 trains the meta-federation proximal policy optimization model by using local historical service data and resource allocation strategy during the meta-training of the meta-federation proximal policy optimization model; after obtaining the meta-trained meta-federation proximal policy optimization model, the meta-trained meta-federation proximal policy optimization model is sent to each communication scheduling device 102;

[0048] The communication scheduling device 102 is also used to train the meta-trained meta-federation proximal policy optimization model using local historical service data and resource allocation strategy during the meta-adaptation process of the meta-federation proximal policy optimization model to obtain the meta-adapted meta-federation proximal policy optimization model; and to perform online training on the meta-adapted meta-federation proximal policy optimization model, and upload the model parameters of the online trained meta-federation proximal policy optimization model to the edge server 101;

[0049] Specifically, when the communication scheduling device 102 performs online training on the meta-adapted meta-federation proximal policy optimization model, it extracts transmission state features from the collected data of the current service, and inputs the extracted transmission state features into the meta-adapted meta-federation proximal policy optimization model; then, according to the resource allocation strategy output by the meta-federation proximal policy optimization model, after allocating bandwidth resources to the service, it calculates a reward value according to the subsequent transmission state features of the service; and fine-tunes the model parameters of the meta-federation proximal policy optimization model according to the calculated reward value.

[0050] Furthermore, the edge server 101 is further used to perform federation aggregation on the model parameters uploaded by each communication scheduling device 102 to obtain a global model parameter, and send the global model parameter to each communication scheduling device 102;

[0051] The communication scheduling device 102 uses the global model parameters to update the parameters of the local meta-federation proximal policy optimization model to obtain a global meta-federation proximal policy optimization model.

[0052] After the communication scheduling device 102 obtains the global meta-federation proximal policy optimization model, communication scheduling can be performed through the meta-federation proximal policy optimization model: the transmission state characteristics are extracted from the collected data of the current service, the extracted transmission state characteristics are input into the meta-federation proximal policy optimization model, and bandwidth resources are allocated to the service according to the resource allocation strategy output by the meta-federation proximal policy optimization model;

[0053] More preferably, the communication scheduling device 102 can also perform online fine-tuning on the global meta-federation proximal policy optimization model: after allocating bandwidth resources to the service according to the resource allocation strategy output by the meta-federation proximal policy optimization model, the communication scheduling device 102 calculates a reward value according to the subsequent transmission state characteristics of the service; and adjusts the model parameters of the meta-federation proximal policy optimization model according to the calculated reward value.

[0054] In a specific implementation, the meta-federation proximal policy optimization model is a neural network model of an Actor-Critic network; the power grid communication resource scheduling problem can be modeled as a multi-agent Markov decision chain, which is first converted into a sequential decision problem through a Markov decision process, and the communication scheduling device is used as an agent, and combined with the meta-federation proximal policy optimization model (i.e., Actor-Critic network) algorithm, it is modeled as a tuple: Among them, I meta is the number of communication scheduling devices, O meta It is the state set of all communication scheduling devices, that is, the transmission state characteristics of the service data of all communication scheduling devices, and the transmission state characteristics may include: bandwidth utilization, packet loss rate, error rate, network congestion; is the action set of each communication scheduling device, R meta is a reward function, which is related to the state set of all communication scheduling devices and the action set of each communication scheduling device, and is defined as the total number of communication bandwidth resources, P meta is the state transfer function, which is related to the state of the communication scheduling device and the action taken. The state transfer formula is: in, represents O at time t meta , represents O at time t+1 meta , represents the action set at time t.

[0055] Based on this, the present invention proposes a specific training method flow of a meta-federation proximal strategy optimization model, such as Figure 2 As shown, the following steps are included:

[0056] Step S201: meta-training stage;

[0057] In the meta-training stage, the edge server 101 uses local historical business data and resource allocation strategies to train the meta-federation proximal policy optimization model to obtain a meta-trained meta-federation proximal policy optimization model; then, the edge server 101 sends the meta-trained meta-federation proximal policy optimization model to each communication scheduling device 102.

[0058] In one iteration of the training process, the edge server 101 obtains data of the service transmitted at time k from the local historical service data, and extracts transmission state features from the obtained data; the edge server 101 inputs the extracted transmission state features into the meta-federation proximal policy optimization model; the edge server 101 calculates the difference between the resource allocation policy output by the meta-federation proximal policy optimization model and the resource allocation policy at time k+1 in the historical resource allocation policy; according to the calculated difference, the model parameters of the meta-federation proximal policy optimization model are adjusted by applying the gradient descent method to minimize the difference; wherein, the gradient descent method calculation formula is: in, represents the data set of historical business data, f1 represents the meta-federation proximal strategy optimization model, and f1 * represents the calculated gradient, and ▽l represents the gradient calculation operation;

[0059] After several iterations, the cumulative loss is: Among them, F is the parameter of the meta-learner (i.e., the meta-federated proximal policy optimization model), is the transmission state feature of all inputs, 1 is the first iteration of the model, and N is the total number of iterations.

[0060] The training process of this step is completed through multiple iterations. Considering the completeness of the training, in the training process of this step, the historical business data of various types of business and the resource allocation strategy can be used to train the meta-federation proximal policy optimization model. For example, data of N types of business are randomly extracted from the historical business data, and K samples are extracted from each type of business to train the meta-federation proximal policy optimization model.

[0061] After the entire training process is completed, through the formula: F1 * The model parameters of the updated meta-learner (i.e., the meta-federated proximal policy optimization model) are used to initialize the model parameter descent direction in the meta-training phase.

[0062] Step S202: meta-adaptation stage;

[0063] In the meta-adaptation stage, the communication scheduling device 102 uses local historical service data and resource allocation strategies to train the meta-trained meta-federation proximal policy optimization model to obtain the meta-adapted meta-federation proximal policy optimization model;

[0064] In one iteration of the training process in the meta-adaptation phase, the communication scheduling device 102 obtains data of the service transmitted at time k from the local historical service data, and extracts transmission state features from the obtained data; the communication scheduling device 102 inputs the extracted transmission state features into the meta-federation proximal policy optimization model; the communication scheduling device 102 calculates the difference between the resource allocation strategy output by the meta-federation proximal policy optimization model and the resource allocation strategy at time k+1 in the historical resource allocation strategy; according to the calculated difference, the model parameters of the meta-federation proximal policy optimization model are adjusted by applying the gradient descent method to minimize the difference; wherein, the gradient descent method updates the formula F2 is the model parameter of the initial meta-federation proximal strategy optimization model in the meta-adaptation phase; The model parameters of the meta-federated proximal strategy optimization model are updated for the meta-adaptation phase;

[0065] The meta-adaptation phase training process of this step is completed through multiple iterations. Considering the completeness of the training, in the training process of this step, the historical business data of various types of business and the resource allocation strategy can be used to train the meta-federation proximal policy optimization model. For example, data of N types of business are randomly extracted from the historical business data, and K samples are extracted from each type of business to train the meta-federation proximal policy optimization model.

[0066] Step S203: local online training phase;

[0067] In the local online training phase, the communication scheduling device 102 performs online training on the meta-adapted meta-federation proximal policy optimization model: extracting transmission state features from the collected data of the current service, inputting the extracted transmission state features into the meta-adapted meta-federation proximal policy optimization model, and after allocating bandwidth resources to the service according to the resource allocation strategy output by the meta-federation proximal policy optimization model, calculating the reward value according to the subsequent transmission state features of the service; adjusting the model parameters of the meta-federation proximal policy optimization model according to the calculated reward value. After multiple iterations of adjusting the meta-adapted meta-federation proximal policy optimization model, the online trained meta-federation proximal policy optimization model is obtained.

[0068] Specifically, the communication scheduling device 102 initializes the local agent's meta , and then extract the transmission state characteristics of the collected business data at the current time t as the observed state at the current time t Input the meta-federation proximal strategy optimization model; and output according to the time t of the meta-federation proximal strategy optimization model As an action taken in the current state, bandwidth resources are allocated to the service; that is, Where μ represents the state transfer function, represents the set of possible actions, Indicates the action taken in the current state, that is, the allocation strategy for bandwidth resources; after allocating bandwidth resources to the service, the data of the service collected at time t+1 is extracted to obtain the state observed at time t+1 According to the state at time t+1 Calculate the reward value R meta ;

[0069] Among them, the expression of the reward function is:

[0070] Among them, r t represents the basic reward value, γ represents the reward discount factor, which can be set to 0.45, for example; F * represents the current model parameters of the meta-federation proximal policy optimization model, Denoted as critic network parameters, Indicates action output. After the action is completed, bandwidth resource B1 is allocated to the service and the next state is entered.

[0071] Then according to the formula Update the model parameters of the meta-federation proximal policy optimization model; where ▽L represents the operation of updating the model, and N represents the total number of iterations;

[0072] The above process is repeated until the stop condition is reached. Afterwards, the communication scheduling device 102 uploads the model parameter θ1 of the meta-federation proximal policy optimization model after online training to the edge server 101 .

[0073] Step S204: global aggregation stage;

[0074] In the global aggregation stage, the edge server 101 performs federation aggregation on the model parameters uploaded by each communication scheduling device 102 to obtain a global model parameter, and sends the global model parameter to each communication scheduling device;

[0075] Specifically, after the edge server 101 collects the model parameters uploaded by each communication scheduling device 102, it starts the federation aggregation process: The model parameters of the federated aggregation model are calculated, that is, the global model parameters θ FL ,θ i is the model parameter uploaded by the i-th communication scheduling device, b i is the weight factor of the i-th communication scheduling device;

[0076] The edge server 101 starts the federation model sharing process and distributes the model parameters θ of the federation aggregation model to each communication scheduling device 102. FL .

[0077] Step S205: model parameter updating stage;

[0078] In the model parameter updating phase, the communication scheduling device 102 uses the global model parameter θ FL The parameters of the local meta-federated proximal policy optimization model are updated to obtain the global meta-federated proximal policy optimization model as the final trained meta-federated proximal policy optimization model.

[0079] The meta-federation proximal strategy optimization model trained by the above method is used for communication resource scheduling. Compared with the traditional scheduling method, this method can more flexibly cope with the dynamic changes of network communication resources, thereby improving the efficiency of resource scheduling. Moreover, meta-learning significantly improves the learning efficiency by allowing the machine learning algorithm to automatically learn how to learn. Traditional machine learning methods require a large amount of data and specific feature engineering to achieve efficient learning, while the meta-learning process of meta-training and meta-adaptation can quickly learn and generalize to new tasks under limited data and features. By utilizing the characteristics of reinforcement learning, the optimal resource allocation strategy can be found through continuous learning and optimization. This can not only ensure the full utilization of power grid communication resources, but also avoid waste and idleness of resources. At the same time, this method can also achieve dynamic balance and optimal allocation of resources according to the actual needs and resource conditions of the network.

[0080] Based on the above communication resource allocation system and the trained meta-federation proximal strategy optimization model, the embodiment of the present invention proposes a communication resource allocation method based on meta-federation reinforcement learning. The specific process is as follows: Figure 3 As shown, the following steps are included:

[0081] Step S301: The communication scheduling device 102 extracts transmission status characteristics from the collected data of the current service;

[0082] Specifically, the inventors of the present invention consider that in network communication resource management, multiple types of data are involved, such as text, image data, etc. These data come from different perception modalities and are of great significance for the comprehensive understanding and optimization of network communication resources. However, how to effectively integrate these multimodal data and use them for decision making is an urgent problem to be solved;

[0083] Based on this, the present invention inputs data into an improved fusion long short-term memory network (LSTM), and inputs the features of each modality into the corresponding LSTM layer, that is, the data of each modality is input into the corresponding LSTM network. For example, the features of the text data of the business are extracted through the first LSTM network, and the features of the image data of the business are extracted through the second LSTM network; then the features of the text data of the business and the features of the image data are fused to obtain the transmission status features of the data of the business.

[0084] Among them, a specific structure of the first LSTM network can be as follows Figure 4a As shown, it includes: an input layer 401, an embedding layer 402, an LSTM layer 403, and an output layer 404;

[0085] The input layer 401 is used to pre-process the input text data, such as word segmentation, and output the word segmentation results of the text data.

[0086] Embedding Layer 402: Converts the input word segmentation result into a dense vector representation, captures the semantic relationship between words, and outputs the word vector.

[0087] LSTM layer 403: consists of multiple LSTM units, and is used to process the word sequence in the word vector through the LSTM layer, capture the time dependency, and output a two-dimensional array of feature word vectors; each LSTM unit contains three main gating mechanisms: Forget Gate: determines how much past information is retained, Input Gate: determines how much current input information is added to the unit state, and Output Gate: determines how much current unit state is output to the next layer;

[0088] The output layer 404 is used to upgrade the two-dimensional array output by the LSTM layer 403 to the final three-dimensional text feature map.

[0089] A specific structure of the second LSTM network can be as follows Figure 4b As shown, it includes: an input layer 411, a convolutional layer 412, a feature map layer 413, an LSTM layer 414, a fully connected layer (DenseLayer) 415, and an output layer 416;

[0090] The input layer 411 is used to pre-process the input image data and convert it into a standard size suitable for subsequent processing, and output a standard pixel matrix as a 3-dimensional array (for example, the shape of an RGB image is (height, width, number of channels)).

[0091] The convolutional layer 412 is used to extract the spatial features of the image using multiple convolutional layers and output a feature map. The feature map output by the convolutional layer can be regarded as a high-dimensional representation of the image. The convolutional layer extracts local features through a convolution operation, and is usually followed by an activation function (such as ReLU) and a pooling layer (such as maximum pooling) to reduce the dimension.

[0092] The feature map layer 413 is used to flatten the feature map into a one-dimensional vector. In a time series task, the feature map can be segmented by time steps.

[0093] The LSTM layer 414 is used to process the feature sequence of the one-dimensional vector output by the feature map layer 413, capture the time dependency, and output the primary feature map of the image modality. Each LSTM unit manages the information flow through its gating mechanism (forget gate, input gate, output gate).

[0094] The fully connected layer (Dense Layer) 415 is used to map the primary feature map of the image modality output by the LSTM layer to the target output space;

[0095] The output layer 416 is used to output the final feature map of the image modality based on the primary feature map mapped to the target output space.

[0096] In the improved LSTM network, C t-1 is the state of the neuron at the previous moment; Y t-1 is the output of the neuron at the previous moment; Z t is the input at the current moment. The forget gate determines the cell state C at the previous moment. t-1 How much can be retained until the current moment C t , the input of the forget gate includes the output Y at the previous moment t-1 and the current load Z t , and finally the formula is:

[0097] Where W f is the weight matrix of the forget gate; b f represents the deviation vector; n σ represents the activation function sigmoid; f t It indicates the probability that the last layer of neurons are forgotten. The value range is [0, 1], 0 means completely discarded, 1 means completely retained, and t indicates the number of inputs.

[0098] The input gate determines which new inputs can be stored in the neuron. Neurons are mainly divided into i t and v t The calculation process is: t =σ(W i ·[Y t-1 ,Z t ]+b i )-ζV(W t+1 ; μ), v t =tanh(W c ·[Y t-1 ,Z t ]+bc )+W I ,

[0099] Among them, W i and W c is the weight matrix of the input gate; b i and b c is the deviation vector. t and v t Indicates the proportion of load information that needs to be retained at present, W I is the hidden matrix function parameter, ζ is the matrix coefficient, V(W t+1 ; μ) is expressed as the activation value μ and the matrix W t+1 Perform concatenation and product calculation, add it to the neuron state retained at the previous moment, and generate the updated neuron state information C t Neurons are mainly divided into i t and v t , the calculation method is shown in the formula: C t =f t ·C t-1 +i t ·v t ;

[0100] The output gate controls the output of the neuron state and transfers the state to the next neuron. The calculation method of the output gate is shown as follows: t =σ(W o ·[Y t-1 ,Z t ]+b o );

[0101] Among them, W o is the weight matrix of the output gate; b o is the deviation vector; Y t is the output of the neuron at the current moment. The output Y obtained by the calculation of the last layer t = cos(o t tanh(C t )); The output result passes through the ReLU activation function and the softmax function;

[0102] Specifically, the text data T is input into the text network (the first LSTM network) to obtain the extracted text features: Among them, the extracted eigenvalue Ux (t) is a hidden neuron, W a t , are the neuron parameters for text data processing, b T is the offset;

[0103] Image data P is input into the image network (second LSTM network) to obtain the extracted image features. Among them, j is the convolution kernel of size a×b.

[0104] The text features will be obtained respectively and image features Enter the fusion formula: Fusion is performed; where [] represents the concatenation operation; and then fused through the fully connected layer FC to obtain W T is the text feature extraction LSTM network parameter, W P are the image feature extraction LSTM network parameters; It is the final fusion result of the fusion of text features and image features, that is, the features of the text and image data of the fused business, which reflects the transmission status of the data of the business, thereby obtaining the transmission status features of the data of the business.

[0105] Step S302: the communication scheduling device 102 inputs the extracted transmission state features into the pre-trained meta-federation proximal strategy optimization model;

[0106] Step S303: the communication scheduling device 102 allocates bandwidth resources to the service according to the resource allocation policy output by the meta-federation proximal policy optimization model.

[0107] In the technical solution of the present invention, the communication scheduling device extracts the transmission state characteristics from the collected data of the current service, and inputs the extracted transmission state characteristics into the pre-trained meta-federation proximal policy optimization model; the communication scheduling device allocates bandwidth resources for the service according to the resource allocation strategy output by the meta-federation proximal policy optimization model; wherein the meta-federation proximal policy optimization model is obtained by the edge server meta-training using the local historical service data and resource allocation strategy, and then the model parameters of the multiple meta-adapted meta-federation proximal policy optimization models are federated and aggregated after meta-adaptation by multiple communication scheduling devices. The present invention uses the edge server and multiple communication scheduling devices to jointly train the meta-federation proximal policy optimization model, that is, through the meta-federation reinforcement learning method, the system can automatically adjust the allocation of communication resources to adapt to the dynamic changes of the network environment and improve communication efficiency and network stability.

[0108] Moreover, even when the number of nodes, time periods, and possible decision combinations increase dramatically, the edge servers and multiple communication scheduling devices use local data for training without the need to obtain all global data, thus avoiding the need to store a large number of intermediate results (or states) and avoid the problem of system crash or inability to execute due to the consumption of a large amount of memory in the face of large problems;

[0109] On the other hand, when edge servers and multiple communication scheduling devices use local data for training, they do not need to share their data directly, thereby protecting local data privacy and using collective knowledge to update the model. In other words, different nodes can share experience data but not local data, and can benefit from the knowledge of other nodes while learning individually, thereby improving overall performance and protecting local privacy.

[0110] In addition, by performing collaborative training on multiple nodes, the robustness and generalization ability of the model can be improved, avoiding the risks caused by single point failures.

[0111] Figure 5 The hardware architecture diagram of the computer device 1300 according to the communication resource allocation method based on meta-federated reinforcement learning according to the embodiment of the present application is schematically shown. In this embodiment, the computer device 1300 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. For example, it can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Figure 5 As shown, the computer device 1300 includes at least but is not limited to: a memory 1310, a processor 1320, and a network interface 1330 which can communicate with each other via a system bus. Among them:

[0112] The memory 1310 includes at least one type of computer-readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 1310 may be an internal storage module of the computer device 1300, such as a hard disk or a memory of the computer device 1300. In other embodiments, the memory 1310 may also be an external storage device of the computer device 1300, such as a plug-in hard disk equipped on the computer device 1300, a smart memory card (Smart Media Card, referred to as SMC), a secure digital (Secure Digital, referred to as SD) card, a flash card, etc. Of course, the memory 1310 may also include both the internal storage module of the computer device 1300 and its external storage device. In this embodiment, the memory 1310 is generally used to store an operating system and various application software installed on the computer device 1300, such as program code of a communication resource allocation method based on meta-federated reinforcement learning, etc. In addition, the memory 1310 can also be used to temporarily store various data that have been output or are to be output.

[0113] In some embodiments, the processor 1320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 1320 is generally used to control the overall operation of the computer device 1300, such as performing control and processing related to data interaction or communication with the computer device 1300. In this embodiment, the processor 1320 is used to run the program code stored in the memory 1310 or process data.

[0114] The network interface 1330 may include a wireless network interface or a wired network interface, and the network interface 1330 is generally used to establish a communication link between the computer device 1300 and other computer devices. For example, the network interface 1330 is used to connect the computer device 1300 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 1300 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0115] It should be pointed out that Figure 5 Only a computer device having components 1310 - 1330 is shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0116] In this embodiment, the communication resource allocation method based on meta-federated reinforcement learning stored in the memory 1310 can also be divided into one or more program modules and executed by one or more processors (processor 1320 in this embodiment) to complete the embodiment of the present application.

[0117] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0118] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present invention, the above embodiments or technical features in different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0119] In addition, to simplify the description and discussion, and in order not to obscure the present invention, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the devices may be shown in the form of block diagrams to avoid obscuring the present invention, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the present invention will be implemented (i.e., these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present invention, it will be apparent to those skilled in the art that the present invention may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0120] Although the invention has been described in conjunction with specific embodiments of the invention, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0121] The embodiments of the present invention are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A communication resource allocation method based on meta-federated reinforcement learning, characterized in that: include: The communication scheduling device extracts transmission status features from the collected data of the current service, and inputs the extracted transmission status features into the pre-trained meta-federation proximal strategy optimization model; The communication scheduling device allocates bandwidth resources to the service according to the resource allocation policy output by the meta-federation proximal policy optimization model; Among them, the meta-federation proximal policy optimization model is obtained by meta-training the edge server using local historical business data and resource allocation strategies, and then meta-adapting and online training are performed through multiple communication scheduling devices using local historical business data and resource allocation strategies, and then the model parameters of multiple online trained meta-federation proximal policy optimization models are federated and aggregated.

2. The method according to claim 1, characterized in that The meta-federation proximal strategy optimization model is specifically trained according to the following method: The edge server trains the meta-federation proximal policy optimization model using local historical service data and resource allocation strategy to obtain a meta-trained meta-federation proximal policy optimization model; And send the meta-federation proximal strategy optimization model after meta-training to each communication scheduling device; The communication scheduling device trains the meta-federation proximal policy optimization model after meta-training by using local historical service data and resource allocation strategy to obtain the meta-federation proximal policy optimization model after meta-adaptation; The communication scheduling device performs online training on the meta-adapted meta-federation proximal policy optimization model, and uploads model parameters of the meta-federation proximal policy optimization model after online training to the edge server; The edge server performs federation aggregation on the model parameters uploaded by each communication scheduling device to obtain a global model parameter, and sends the global model parameter to each communication scheduling device; The communication scheduling device uses the global model parameters to update the parameters of the local meta-federation proximal policy optimization model to obtain a global meta-federation proximal policy optimization model as the meta-federation proximal policy optimization model finally obtained by training.

3. The method according to claim 2, characterized in that The communication scheduling device performs online training on the meta-adapted meta-federation proximal strategy optimization model, specifically: The communication scheduling device extracts transmission status features from the collected data of the current service, and inputs the extracted transmission status features into the meta-adapted meta-federation proximal policy optimization model; then, according to the resource allocation strategy output by the meta-federation proximal policy optimization model, after allocating bandwidth resources to the service, a reward value is calculated according to the subsequent transmission status features of the service; and the model parameters of the meta-federation proximal policy optimization model are fine-tuned according to the calculated reward value.

4. The method according to claim 1, characterized in that: The collected business data is multimodal, including: text data and image data; and The specific feature of the transmission status of the collected business data extraction is: Extracting features of the text data of the business through a first LSTM network; Extracting features of the image data of the business through a second LSTM network; The features of the text data and the features of the image data of the service are merged to obtain the transmission status features of the data of the service.

5. A communication resource allocation system based on meta-federated reinforcement learning, characterized in that: include: An edge server, and a plurality of communication scheduling devices; wherein, The communication scheduling device is used to extract transmission status features from the collected data of the current service, input the extracted transmission status features into a pre-trained meta-federation proximal policy optimization model, and allocate bandwidth resources to the service according to the resource allocation strategy output by the meta-federation proximal policy optimization model; Among them, the meta-federation proximal policy optimization model is obtained by meta-training the edge server using local historical business data and resource allocation strategies, and then meta-adapting and online training are performed through multiple communication scheduling devices using local historical business data and resource allocation strategies, and then the model parameters of multiple online trained meta-federation proximal policy optimization models are federated and aggregated.

6. The system according to claim 5, characterized in that The edge server is specifically used to train the meta-federation proximal policy optimization model using local historical service data and resource allocation strategy to obtain the meta-trained meta-federation proximal policy optimization model; and send the meta-trained meta-federation proximal policy optimization model to each communication scheduling device; as well as The communication scheduling device is also used to train the meta-federation proximal policy optimization model after meta-training by using local historical service data and resource allocation strategy to obtain the meta-federation proximal policy optimization model after meta-adaptation; and performing online training on the meta-adapted meta-federation proximal policy optimization model, and uploading model parameters of the meta-federation proximal policy optimization model after online training to the edge server; The edge server is also used to perform federation aggregation on the model parameters uploaded by each communication scheduling device to obtain a global model parameter, and send the global model parameter to each communication scheduling device; as well as The communication scheduling device is further used to use the global model parameters to update the parameters of the local meta-federation proximal policy optimization model to obtain a global meta-federation proximal policy optimization model.

7. The system according to claim 6, characterized in that The communication scheduling device is specifically used to extract transmission state characteristics from the collected data of the current service when online training is performed on the meta-adapted meta-federation proximal policy optimization model, and input the extracted transmission state characteristics into the meta-adapted meta-federation proximal policy optimization model; then, according to the resource allocation strategy output by the meta-federation proximal policy optimization model, after allocating bandwidth resources to the service, calculate the reward value according to the subsequent transmission state characteristics of the service; and fine-tune the model parameters of the meta-federation proximal policy optimization model according to the calculated reward value.

8. The system according to claim 5, characterized in that The communication scheduling device specifically extracts features from the collected multimodal business data: extracts features of the text data of the business through a first LSTM network; extracts features of the image data of the business through a second LSTM network; and fuses the features of the text data and the features of the image data of the business to obtain transmission status features of the data of the business.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it is used to implement the steps of the communication resource allocation method based on meta-federated reinforcement learning as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed by at least one processor so that the at least one processor executes the steps of the communication resource allocation method based on meta-federated reinforcement learning as described in any one of claims 1 to 4.

Images