Metareinforcement learning-based communication network rapid adaptive control strategy generation method

Abstract

The invention discloses a communication network rapid adaptive control strategy generation method based on meta reinforcement learning, and relates to the field of communication network adaptive control, and the method comprises the steps: collecting and processing a physical layer signal and a network layer index, and obtaining a standardized multi-dimensional time sequence matrix; inputting the standardized multi-dimensional time sequence matrix into a hybrid encoder formed by CNN-LSTM, dynamically weighting through a self-attention mechanism, and outputting an environment feature vector; inputting the environment feature vector into a neural differential equation network, generating a strategy parameter increment and updating a strategy network parameter; and constructing a dynamic reward function based on physical layer features in the environment feature vector, and performing detection to obtain an adjusted dynamic reward value. According to the method, continuous time evolution of strategy parameters is realized through a neural differential equation network, and a dual-channel ResNet-ODE architecture is combined with a gating fusion mechanism, so that the problem of oscillation caused by discrete updating of a traditional reinforcement learning strategy is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication network adaptive control, in particular to a meta-reinforcement learning based communication network fast adaptive control strategy generation method. BACKGROUND

[0002] In the field of communication network adaptive control, traditional methods mainly rely on rule-based static strategies or classical reinforcement learning algorithms. In recent years, deep reinforcement learning has been introduced into network resource allocation, traffic scheduling and other scenarios due to its decision-making ability in complex environments. The adaptive control scheme based on deep Q network in the prior art can adapt to network state changes to a certain extent by training the policy network offline and fine-tuning in the online stage. However, this method usually faces the problem of insufficient strategy generalization ability. When the network environment changes suddenly, the strategy needs to be retrained for a long time before it converges. The prior art uses a single network structure to process network state information, which lacks the ability to model cross-layer joint modeling of physical layer signals and network layer indicators, resulting in limited adaptability of the strategy in complex dynamic environments.

[0003] The prior art has the disadvantage that it fails to effectively solve the contradiction between cross-layer state feature fusion and fast strategy adaptation. Traditional methods usually use separate feature extraction paths or simple concatenation fusion when processing physical layer signals and network layer indicators, ignoring the dynamic coupling relationship between the two in the time-frequency domain. This processing method not only increases the model complexity, but also causes the strategy to consume a large amount of computational resources to retrain when facing new scenarios. The prior art lacks explicit constraints on network stability during strategy updating, which may cause strategy shock or convergence to a suboptimal solution. When the network load changes suddenly, the traditional RL strategy may pursue short-term rewards excessively and destroy global stability. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a meta-reinforcement learning based communication network fast adaptive control strategy generation method to solve the key contradiction between low cross-layer state fusion efficiency and slow strategy adaptation speed in traditional methods.

[0006] To solve the above technical problems, the present application provides the following technical solutions: In a first aspect, the present application provides a meta-reinforcement learning based communication network fast adaptive control strategy generation method, which comprises collecting and processing physical layer signals and network layer indicators to obtain a standardized multi-dimensional time series matrix; inputting the standardized multi-dimensional time series matrix into a hybrid encoder composed of CNN-LSTM, dynamically weighting through self-attention mechanism, and outputting an environment feature vector; Input the environmental feature vector into the neural differential equation network to generate a strategy parameter increment and update the strategy network parameters; A dynamic reward function is constructed based on the physical layer features in the environmental feature vector and detected to obtain an adjusted dynamic reward value, the updated strategy network parameters are compressed into a lightweight strategy model through knowledge distillation, and the lightweight strategy model is distributed to the edge nodes through an encrypted channel; The strategy execution data of the edge nodes are collected and screened, stored in the scenario memory bank, the central meta-learner parameters are updated through federated learning, the Lyapunov index is obtained based on the strategy execution data, the lightweight strategy model is analyzed and detected, and the environmental code is regenerated to trigger the conservative update mode.

[0007] As a preferred scheme of the communication network fast adaptive control strategy generation method based on meta-reinforcement learning, wherein: the physical layer signals and network layer indicators are collected and processed to obtain a standardized multi-dimensional time series matrix, including the following steps, The sampling rate of the original signal is collected through the RFSoC chip in the base station AAU to obtain a time-aligned baseband sample stream, the baseband sample stream performs FFT to obtain the channel frequency response, and the instantaneous signal-to-noise ratio and the Doppler shift are calculated; The DPDK captures the time delay, throughput and packet loss rate from the network card to obtain a set of hardware-timestamped indicators, synchronizes the physical layer feature triple and the network layer indicators to the same time reference through the PTP protocol to obtain an aligned mixed data block, and processes each feature dimension in the mixed data block to obtain a two-dimensional data matrix; The two-dimensional data matrix is added with timestamp metadata and transmitted to the preprocessing server through the PCIe 4.0 channel to obtain a standardized multi-dimensional time series matrix.

[0008] As a preferred scheme of the communication network fast adaptive control strategy generation method based on meta-reinforcement learning, wherein: the standardized multi-dimensional time series matrix is input into a hybrid encoder composed of CNN-LSTM, and the environmental feature vector is output through dynamic weighting based on the self-attention mechanism, including the following steps, The input two-dimensional data matrix is processed through three layers of cavity convolution to obtain a spatio-temporal feature tensor, the spatio-temporal feature tensor is input into a bidirectional LSTM to obtain a time series enhanced feature, Based on the time series enhanced feature, a query matrix, a key matrix and a value matrix are generated, and a four-head attention mechanism is used for dynamic weighting to obtain a two-dimensional feature tensor; The two-dimensional feature tensor is spliced to form a multi-head attention splicing feature matrix, and a compressed spatio-temporal feature vector is obtained through a linear transformation layer; The compressed spatio-temporal feature vector is divided into a physical layer feature encoding segment and a network layer feature encoding segment to obtain an environmental feature vector.

[0009] As a preferred embodiment of the fast adaptive control policy generation method for communication networks based on meta-reinforcement learning described in this invention, the method includes the following steps: inputting environmental feature vectors into a neural differential equation network, generating policy parameter increments, and updating the policy network parameters. The physical layer feature coding segment and the network layer feature coding segment are processed by Min-Max normalization and Gaussian normalization respectively, and the standardized environmental feature vector is output. The microsecond-level UNIX timestamp feature is embedded at the end of the standardized environmental feature vector, and the time-series enhanced feature vector is output. The physical layer feature encoding segment in the temporal enhancement feature vector is input into the short-term ResNet-ODE network. The Euler method is used to solve the short-term differential change of the policy parameters with a step size of 50ms, and the short-term policy parameter increment is output. The network layer feature encoding segments in the temporal enhancement feature vector are input into the long-term ResNet-ODE network. The Runge-Kutta4 method is used to solve the long-term trend of the policy parameters with an adaptive step size, and the long-term policy parameter increment is output. Based on the real-time weights of the physical layer feature encoding segments and the network layer feature encoding segments, the short-term policy parameter increment and the long-term policy parameter increment are fused through the sigmoid gating mechanism, and the fused policy parameter increment is output. Based on the KL divergence of the probability distributions of the old and new policy outputs, the effective learning rate is calculated and the increment of the fusion policy parameters is scaled to output the adjusted policy parameter update amount. The adjusted policy parameter update is constrained by the L2 norm, and the security policy parameter increment is output. The security policy parameter increment is then added to the current policy network parameter to generate the updated policy network parameter.

[0010] As a preferred embodiment of the fast adaptive control strategy generation method for communication networks based on meta-reinforcement learning described in this invention, the method includes the following steps: constructing a dynamic reward function based on physical layer features in the environmental feature vector and performing detection to obtain an adjusted dynamic reward value. The physical layer feature coding segment in the decoupled environment feature vector is decomposed into four types of structured data: spectral features, signal-to-noise ratio dynamics, Doppler features, and interference fingerprints, and the physical layer feature dictionary is output. The performance reward is calculated based on the spectral features in the physical layer feature dictionary, the stability penalty is calculated based on the dynamic signal-to-noise ratio and Doppler features, the spectral efficiency reward is calculated based on the spectral features, and the performance reward, stability penalty and spectral efficiency reward are fused by dynamic weighting to obtain the original reward value.

[0011] A pre-trained LSTM predictor is used to detect the bias in the original reward value, and the reward value is halved to output the adjusted dynamic reward value.

[0012] As a preferred embodiment of the fast adaptive control policy generation method for communication networks based on meta-reinforcement learning described in this invention, the updated policy network parameters are compressed into a lightweight policy model through knowledge distillation and distributed to edge nodes via an encrypted channel, including the following steps: The updated policy network parameters are used to build a teacher model, which infers on the state sample set and outputs the action probability distribution and action value function. The action probability distribution and action value function output by the teacher model are used as supervision signals to train a lightweight convolutional neural network-long short-term memory network hybrid student model and output the student model parameters. The student model parameters are quantized into 16-bit floating-point quantization and converted into 8-bit integers. The quantized compressed parameters are output. The compressed parameters are then subjected to a secure hash algorithm, and the digital signature and hash digest value are output. The digital signature and hash digest value are used to generate ciphertext data. The session key is encrypted using the public key, and the encrypted data packet is output. This data is then converted into a transmission stream. After receiving the transmission stream, the edge node verifies the signature, decrypts the data, and loads the compression parameters in sequence.

[0013] As a preferred embodiment of the fast adaptive control policy generation method for communication networks based on meta-reinforcement learning described in this invention, the method includes the following steps: collecting and filtering policy execution data from edge nodes, storing it in a context memory, and triggering federated learning to update the parameters of the central meta-learner. The DPDK kernel bypass technique is used to capture the original state-action-reward data stream of edge node policy execution, output the original dataset with timestamps, perform sliding window statistical tests on the original dataset, output the cleaned dataset, obtain the temporal difference error of each sample in the cleaned data, and obtain the dataset with priority labeling. The data is divided into three layers of high, medium and low priority weights, and a structured context memory is output. The proportion of high-priority samples in the memory is monitored and detected in real time. Trigger signals and candidate sample sets are output. Edge nodes calculate the gradient of network parameters based on candidate samples. Gaussian noise is added to achieve differential privacy protection and encrypted gradient increments are output. The encrypted gradients are modularly aggregated using the SecAgg protocol to output the global aggregated gradient. The global gradient is then pruned to update the meta-learner parameters, and the next-generation policy network parameters are output.

[0014] As a preferred embodiment of the fast adaptive control policy generation method for communication networks based on meta-reinforcement learning described in this invention, the method includes the following steps: obtaining the Lyapunov exponent based on policy execution data, analyzing and detecting the lightweight policy model, regenerating the environment code, and triggering a conservative update mode. The system collects time-series data such as latency, throughput, and packet loss rate of the strategy execution in real time from edge nodes, outputs the original state dataset, performs latency embedding and dimensional expansion on the original state data, and outputs the reconstructed phase space trajectory matrix. The Jacobian matrix sequence is subjected to QR decomposition iteration to obtain the Lyapunov exponent. The policy gradient is then pruned using the L2 norm to output the safety parameter increment. The safety increment is processed with the decaying learning rate to update the policy network and trigger the conservative update mode.

[0015] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the fast adaptive control strategy generation method for communication networks based on meta-reinforcement learning as described in the first aspect of the present invention.

[0016] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the fast adaptive control strategy generation method for communication networks based on meta-reinforcement learning as described in the first aspect of the present invention.

[0017] The beneficial effects of this invention are as follows: it realizes the continuous-time evolution of policy parameters through a neural differential equation network, adopts a dual-channel ResNet-ODE architecture combined with a gating fusion mechanism, solves the oscillation problem caused by the discrete update of traditional reinforcement learning policies, achieves smooth parameter adaptation, and integrates performance reward, stability penalty and spectral efficiency reward based on physical layer features of the dynamic reward function, and breaks through the limitation of single optimization of network layer indicators by online correction through a long short-term memory network predictor. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Fig. 1 This is a flowchart of a fast adaptive control strategy generation method for communication networks based on meta-reinforcement learning.

[0020] Fig. 2 This is a schematic diagram of environmental feature vectors. Detailed Implementation

[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0023] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0024] Example 1, referring to Figs. 1-2 This is one embodiment of the present invention, which provides a method for generating fast adaptive control policies for communication networks based on meta-reinforcement learning, including the following steps: S1. Collect physical layer signals and network layer indicators and process them to obtain a standardized multidimensional time series matrix.

[0025] S1.1. The original signal is acquired by sampling the RFSoC chip in the base station AAU to obtain a time-aligned baseband sample stream. The baseband sample stream is subjected to FFT to obtain the channel frequency response, and the instantaneous signal-to-noise ratio and Doppler frequency shift are calculated.

[0026] Specifically, the expression is, ; in, For channel frequency response, It is a time-domain baseband signal.

[0027] ; in, For instantaneous signal-to-noise ratio, This represents the noise power spectral density.

[0028] ; in, For Doppler frequency shift, The sampling interval is... For the first The received value of each pilot symbol. The number of pilot symbols, To indicate the first The received value of each pilot symbol is a complex conjugate.

[0029] S1.2 DPDK captures latency, throughput, and packet loss rate from the network card to obtain a set of indicators with hardware timestamps. It then synchronizes the physical layer feature triplet with the network layer indicator set to the same time base via the PTP protocol to obtain an aligned mixed data block. Each feature dimension in the mixed data block is processed to obtain a two-dimensional data matrix.

[0030] Furthermore, the data plane development toolkit captures latency, throughput, and packet loss rate metrics from the network interface card to obtain a set of network performance metrics with hardware timestamps. Using a precise time protocol, it synchronizes the physical layer feature triplet (comprising channel frequency response, instantaneous signal-to-noise ratio, and Doppler shift) with the set of network performance metrics to obtain a time-aligned hybrid data block. Each feature dimension in the hybrid data block is then standardized. Specifically, this includes performing a fast Fourier transform on the channel frequency response to obtain spectral features, calculating sliding window statistics for the instantaneous signal-to-noise ratio, performing a short-time Fourier transform analysis on the Doppler shift, and performing Z-score normalization on latency, throughput, and packet loss rate. After these processes, a two-dimensional data matrix is ​​obtained, with rows representing time points and columns representing feature dimensions.

[0031] S1.3 adds timestamp metadata to the two-dimensional data matrix and transmits it to the preprocessing server through the PCIe 4.0 channel to obtain a standardized multidimensional time series matrix.

[0032] Furthermore, a corresponding microsecond-level UNIX timestamp is appended to each row of data in the two-dimensional data matrix to form an extended data matrix with timestamps. This extended data matrix is ​​then transmitted from the data acquisition device to the preprocessing server via a high-speed data transmission channel using the PCI Express 4.0 bus interface. Upon receiving the data, the preprocessing server performs Mel-frequency cepstral coefficient transformation on the channel frequency response characteristics in the two-dimensional data matrix, logarithmic transformation on the instantaneous signal-to-noise ratio characteristics, normalization on the Doppler shift characteristics, and Min-Max standardization on the latency, throughput, and packet loss rate characteristics. After these processes, a three-dimensional time series matrix containing time, feature, and standardized values ​​is finally obtained. The time dimension corresponds to the sampling time, the feature dimension includes physical layer features and network layer indicators, and the numerical range is uniformly standardized to the [0,1] interval, resulting in a standardized multidimensional time series matrix.

[0033] S2. Input the standardized multidimensional time series matrix into the hybrid encoder composed of CNN-LSTM, dynamically weight it through the self-attention mechanism, and output the environmental feature vector.

[0034] S2.1 The input two-dimensional data matrix is ​​processed through three layers of dilated convolution to obtain a spatiotemporal feature tensor. The spatiotemporal feature tensor is then input into a bidirectional LSTM to obtain temporal enhancement features.

[0035] Furthermore, the two-dimensional data matrix first undergoes a dilated convolutional layer with a kernel size of 3×3 and an inflation rate of 1, outputting a 64-channel primary feature map. This primary feature map is then input into a second dilated convolutional layer, where the kernel size remains 3×3 but the inflation rate is adjusted to 2, outputting a 128-channel intermediate feature map. The intermediate feature map is then input into a third dilated convolutional layer, where the kernel size remains 3×3 but the inflation rate is increased to 4, ultimately outputting a 256-channel spatiotemporal feature tensor. This spatiotemporal feature tensor is then input into a bidirectional long short-term memory network (LSTM), which contains two hidden layers, each with 128 hidden units. After processing the spatiotemporal feature tensor via forward and backward propagation, the outputs from both directions are concatenated along the feature dimension to obtain a 512-dimensional temporally enhanced feature vector. This temporally enhanced feature vector contains both historical and future trend information, fully representing the spatiotemporal evolution of the network state.

[0036] S2.2. Generate query matrix, key matrix and value matrix based on temporal enhancement features, and dynamically weight them using a four-head attention mechanism to obtain a two-dimensional feature tensor.

[0037] Furthermore, the temporal augmentation feature vector generates a query matrix, a key matrix, and a value matrix through three independent linear transformation layers. The query and key matrices are 512×128 in dimension, and the value matrix is ​​also 512×128. Each of these matrices is then divided into four 128×128 sub-matrices, each corresponding to an attention head. Scaling dot product attention is performed on each attention head, specifically: the query sub-matrix is ​​multiplied by the transpose of the key sub-matrix, then divided by 8 for scaling, normalized using the softmax function, and multiplied by the value sub-matrix, resulting in four 128-dimensional weighted attention outputs. The outputs of the four attention heads are concatenated along the feature dimension to form a 512-dimensional composite feature vector. This composite feature vector is then compressed into a 128-dimensional two-dimensional feature tensor through a linear projection layer. This two-dimensional feature tensor retains the key information of the original temporal augmentation features and achieves dynamic reconstruction of the feature space through a multi-head attention mechanism.

[0038] S2.3. The two-dimensional feature tensors are concatenated to form a multi-head attention concatenated feature matrix, and the compressed spatiotemporal feature vector is obtained through a linear transformation layer.

[0039] Furthermore, the 128-dimensional feature vectors output by the four attention heads are concatenated along their feature dimensions to form a 512-dimensional multi-head attention concatenated feature matrix. This multi-head attention concatenated feature matrix is ​​input into a linear transformation layer, which contains a trainable weight matrix and a bias vector. The weight matrix has a dimension of 512×128, and the bias vector has a dimension of 128. Matrix multiplication and vector addition operations compress the 512-dimensional features to 128 dimensions. The compressed spatiotemporal feature vector retains the key features extracted by the multi-head attention mechanism while reducing the feature dimension to improve subsequent processing efficiency. The spatiotemporal feature vector is divided into a physical layer feature coding segment and a network layer feature coding segment. The physical layer feature coding segment contains 64-dimensional spectral features and channel state information, while the network layer feature coding segment contains 64-dimensional network performance indicators and traffic features. The physical layer feature coding segment and the network layer feature coding segment together constitute the final environmental feature vector.

[0040] S2.4 Divide the compressed spatiotemporal feature vector into physical layer feature coding segments and network layer feature coding segments to obtain the environmental feature vector.

[0041] Furthermore, the compressed spatiotemporal feature vector is equally divided along its feature dimensions. The first 64 dimensions serve as the physical layer feature encoding segment, containing spectral features, dynamic signal-to-noise ratio features, and Doppler shift features weighted by multi-head attention. The latter 64 dimensions serve as the network layer feature encoding segment, containing latency features, throughput features, and packet loss rate features filtered by the attention mechanism. The physical layer and network layer feature encoding segments are arranged consecutively in dimension, together forming a 128-dimensional environment feature vector. The physical layer feature encoding segment in the environment feature vector is used for subsequent dynamic reward function construction, while the network layer feature encoding segment is used for policy parameter evolution calculation. By preserving the cross-layer correlation between physical layer signals and network layer indicators in the original data, the environment feature vector provides the neural differential equation network with input features containing complete spatiotemporal characteristics.

[0042] S3. Input the environmental feature vector into the neural differential equation network to generate policy parameter increments and update the policy network parameters.

[0043] S3.1. For the physical layer feature coding segment and the network layer feature coding segment, perform Min-Max normalization and Gaussian normalization respectively, and output the standardized environmental feature vector. Embed the microsecond-level UNIX timestamp feature at the end of the standardized environmental feature vector and output the time-enhanced feature vector.

[0044] Furthermore, the maximum and minimum values ​​of the spectral feature components are calculated. Each feature value is subtracted from its minimum value and then divided by its range, mapping it to the [0,1] interval. The same operation is performed on the signal-to-noise ratio dynamic feature components and the Doppler shift feature components. The network layer feature coding segment is processed using Gaussian normalization. Specifically, the mean and standard deviation of the delay feature components, throughput feature components, and packet loss rate feature components are calculated, and each feature value is subtracted from its mean and then divided by its standard deviation. The normalized physical layer feature coding segment and the network layer feature coding segment are then reassembled into a 128-dimensional normalized environmental feature vector. A 64-bit microsecond-level UNIX timestamp feature is added to the end of the normalized environmental feature vector, forming a 192-dimensional time-enhanced feature vector. The first 128 dimensions of the time-enhanced feature vector are the normalized cross-layer network features, and the output time-enhanced feature vector is then generated.

[0045] S3.2 Input the physical layer feature encoding segment in the temporal enhancement feature vector into the short-term ResNet-ODE network, use the Euler method to solve the short-term differential change of the policy parameters with a step size of 50ms, and output the short-term policy parameter increment.

[0046] Furthermore, the physical layer feature encoding segment from the temporal augmentation feature vector is input into the short-term ResNet-ODE network. This network contains three residual blocks, each consisting of two convolutional layers and skip connections. The convolutional kernel size is 3×3, and the stride is 1. The short-term ResNet-ODE network solves the differential equation of the policy parameters using the Euler method with a fixed stride of 50 milliseconds. At the current time point, it calculates the gradient of the physical layer feature encoding segment with respect to the policy parameters. The gradient is multiplied by the fixed stride and then superimposed onto the current policy parameters, outputting the short-term differential change of the policy parameters within the next 50 millisecond time window. The short-term policy parameter increment includes the changes in the weights and biases of each layer of the policy network, with the dimension consistent with the original policy parameters. During the training phase, the short-term ResNet-ODE network uses the mean squared error loss function and updates the network parameters through the backpropagation algorithm, so that the predicted changes in policy parameters can accurately reflect the impact of rapid fluctuations in physical layer features on the policy. The trained short-term ResNet-ODE network can respond to channel state changes in real time, providing high-frequency parameter adjustment suggestions for policy updates.

[0047] S3.3 Input the network layer feature coding segment in the temporal enhancement feature vector into the long-term ResNet-ODE network, use the Runge-Kutta4 method with adaptive step size to solve the long-term trend of policy parameters, output the long-term policy parameter increment, and according to the real-time weights of the physical layer feature coding segment and the network layer feature coding segment, fuse the short-term policy parameter increment and the long-term policy parameter increment through the sigmoid gating mechanism to output the fused policy parameter increment.

[0048] Furthermore, the network layer feature encoding segments in the temporal augmentation feature vector are input into the long-term ResNet-ODE network. This network contains four residual blocks, each consisting of three convolutional layers and skip connections. The convolutional kernel size is 5×5, and the stride is 2. The long-term ResNet-ODE network solves the differential equation of the policy parameters with an adaptive stride using the Runge-Kutta4 method. It automatically adjusts the integration stride within each time interval and obtains the long-term trend of the policy parameters by calculating a weighted average through four intermediate gradients. It outputs the long-term differential change of the policy parameters within the next 500 millisecond time window. Physical layer feature encoding segments and network layer feature encoding segments generate real-time weight coefficients through independent fully connected layers. The weight coefficients are input into a sigmoid function to generate a gating value, which ranges from 0 to 1. The short-term policy parameter increment and the long-term policy parameter increment are multiplied by the corresponding gating value and the complement gating value, respectively, and then weighted and summed to output the fused policy parameter increment. The fused policy parameter increment includes both a fast response to physical layer mutations and retains the stability constraints of the long-term trend of the network layer. The long-term ResNet-ODE network adopts a smooth L1 loss function during the training phase and optimizes the network parameters through an adaptive moment estimation algorithm, so that the predicted long-term ResNet-ODE trend of policy parameters conforms to the macroscopic change law of network performance indicators.

[0049] S3.4. Based on the KL divergence of the probability distributions of the old and new policy outputs, calculate the effective learning rate and scale the increment of the fusion policy parameters, and output the adjusted policy parameter update amount.

[0050] Specifically, the expression is, ; in, For effective learning rate, Based on the learning rate, The sensitivity coefficient, Let KL divergence be the probability distribution output by the old and new strategies. This is the old strategy. This is a new strategy.

[0051] S3.5 Apply L2 norm constraints to the adjusted policy parameter update amount, output the security policy parameter increment, and add the security policy parameter increment to the current policy network parameter to generate the updated policy network parameter.

[0052] Furthermore, the adjusted policy parameter update amount first calculates the L2 norm. When the L2 norm value exceeds a preset threshold, the policy parameter update amount is scaled proportionally to the threshold range, and the security policy parameter increment is output. Each component of the security policy parameter increment satisfies the constraint conditions to ensure that the parameter update magnitude is controlled. The security policy parameter increment is added element-wise to the current policy network parameters to generate the updated policy network parameters. The updated policy network parameters retain the structure and dimensions of the original policy network, but the weights and biases have been adjusted according to environmental characteristics and temporal evolution. The updated policy network parameters will replace the original parameters for subsequent policy decisions and serve as the benchmark value for the next round of parameter updates. The L2 norm constraint mechanism effectively prevents abrupt changes in policy parameters and ensures the smoothness and stability of policy updates.

[0053] S4. Construct a dynamic reward function based on the physical layer features in the environmental feature vector and perform detection to obtain the adjusted dynamic reward value.

[0054] S4.1 Decompose the physical layer feature coding segment in the decoupled environment feature vector into four types of structured data: spectral features, signal-to-noise ratio dynamics, Doppler features, and interference fingerprints, and output the physical layer feature dictionary.

[0055] Furthermore, the physical layer feature coding segment in the decoupled environment feature vector is decomposed into four independent components through feature segmentation: the first 16-dimensional feature components serve as spectral features, including the Mel-cephem coefficients of the channel frequency response; the subsequent 12-dimensional feature components serve as signal-to-noise ratio dynamics, including the instantaneous signal-to-noise ratio and its first-order difference; the next 8-dimensional feature components serve as Doppler features, including the Doppler frequency shift and its short-time Fourier transform energy spectrum; and the last 4-dimensional feature components serve as interference fingerprints, including the PCA dimensionality reduction features of interference from adjacent base stations. These four types of structured data are organized in key-value pair format. The spectral features, signal-to-noise ratio dynamics, Doppler features, and interference fingerprints together constitute the physical layer feature dictionary. The spectral features in the physical layer feature dictionary are used to calculate performance rewards, the signal-to-noise ratio dynamics are used to construct stability penalty terms, the Doppler features participate in the calculation of spectral efficiency rewards, and the interference fingerprints are used to identify external interference sources. The physical layer feature dictionary stores cross-layer network features in a structured manner.

[0056] It should be noted that the physical layer feature coding segment in the decoupled environment feature vector is decomposed into four independent components through feature segmentation: the first 16-dimensional feature components are spectral features, including the Mel-cephem coefficients of the channel frequency response; the next 12-dimensional feature components are signal-to-noise ratio dynamics, including the instantaneous signal-to-noise ratio and its first-order difference; the next 8-dimensional feature components are Doppler features, including the Doppler frequency shift and its short-time Fourier transform energy spectrum; and the last 4-dimensional feature components are interference fingerprints, including the PCA dimensionality reduction features of interference from adjacent base stations. The four types of structured data are organized in key-value pair form to constitute the physical layer feature dictionary, with a total of 40 dimensions. The total dimension of the physical layer feature coding is 64 dimensions, and the remaining 24 dimensions are reserved for subsequent expansion or other feature representations to support more complex environment modeling and cross-layer feature fusion.

[0057] S4.2 Calculate the performance reward based on the spectral features in the physical layer feature dictionary, calculate the stability penalty based on the dynamic signal-to-noise ratio and Doppler features, calculate the spectral efficiency reward based on the spectral features, and fuse the performance reward, stability penalty and spectral efficiency reward through dynamic weighting to obtain the original reward value.

[0058] Specifically, the performance expression is as follows: ; in, The original reward value. This represents the current measured throughput. For the target throughput of the slice, For the current end-to-end delay, This represents the maximum allowable delay.

[0059] Specifically, the stability penalty expression is as follows: ; in, As a stability penalty, The instantaneous rate of change of the signal-to-noise ratio. For Doppler frequency shift, For frequency shift alarm threshold, The penalty coefficient is... This is an indicator function.

[0060] Specifically, the expression for the spectral efficiency reward is as follows: ; in, As a reward for spectrum efficiency, Total bandwidth For subcarriers Channel response, For subcarriers noise power, For subcarriers Frequency offset, For the total number of subcarriers, For subcarrier indexing.

[0061] S4.3. Use a pre-trained LSTM predictor to detect the deviation of the original reward value, perform a halving correction on the reward value, and output the adjusted dynamic reward value.

[0062] Furthermore, the pre-trained Long Short-Term Memory (LSTM) predictor receives the original reward value as input, processes the time-series data through two stacked LSM layers, and outputs the predicted reward value. The absolute deviation between the original and predicted reward values ​​is obtained. When the deviation exceeds a preset threshold, the original reward value is multiplied by a correction factor of 0.5, outputting an adjusted dynamic reward value. The adjusted dynamic reward value retains the sign of the original reward value but reduces its magnitude, preventing drastic parameter fluctuations in the policy network due to abnormal reward signals. In the offline phase, the LSM predictor is trained using historical policy execution data. The input is the reward value sequence of the past 10 time steps, and the output is the predicted reward value for the current time step. The loss function uses mean squared error. The trained LSM predictor can identify abnormal fluctuations in the reward signal, providing a benchmark reference for online reward correction. The reward correction mechanism effectively smooths the policy update process and improves the stability of reinforcement learning training.

[0063] S5. The updated policy network parameters are compressed into a lightweight policy model through knowledge distillation and distributed to edge nodes via an encrypted channel.

[0064] S5.1. Construct a teacher model using the updated policy network parameters, perform reasoning on the state sample set, and output the action probability distribution and action value function. Use the action probability distribution and action value function output by the teacher model as supervision signals to train a lightweight convolutional neural network-long short-term memory hybrid student model and output the student model parameters.

[0065] Furthermore, the updated policy network parameters are used as teacher model parameters. The teacher model receives the state sample set as input and calculates the probability distribution and corresponding action value function for each selectable action through forward propagation. The action probability distribution is generated through a softmax layer with the same dimension as the action space. The action value function is output through a fully connected layer, representing the expected long-term reward of each action. The action probability distribution and action value function output by the teacher model are used as supervision signals and input into a lightweight convolutional neural network-long short-term memory network hybrid student model for knowledge distillation training. The lightweight convolutional neural network-long short-term memory network hybrid student model contains three layers of dilated convolutions and a single layer of bidirectional long short-term memory network, with convolution kernel sizes of 3×3, 5×5, and 3×3, respectively. During training, the KL divergence of the action probability distribution and the mean squared error of the action value function are minimized, with loss function weights of 0.7 and 0.3, respectively. The parameters of the lightweight convolutional neural network-long short-term memory network hybrid student model are updated through backpropagation algorithm, and the final output is a student model with less than 1% of the parameters of the teacher model.

[0066] S5.2 Perform 16-bit floating-point quantization and 8-bit integer conversion on the student model parameters, output the quantized compressed parameters, perform a secure hash algorithm on the compressed parameters, and output the digital signature and hash digest value.

[0067] Furthermore, the student model parameters are first converted to a 16-bit floating-point format, with 1 sign bit, 5 exponent bits, and 10 mantissa bits, retaining 4 decimal places of precision. The 16-bit floating-point parameters are then converted to 8-bit integers through uniform quantization. The quantization range is divided into 256 equal intervals between the minimum and maximum parameter values, with each interval mapping to an integer value from 0 to 255. The quantized compressed parameters are then hashed using a secure hash algorithm 3-256. The input includes the compressed parameter binary stream, timestamp, and node identifier, and the output is a 256-bit hash digest value. The hash digest value is signed using an elliptic curve digital signature algorithm, and a 64-byte digital signature is generated using a 256-bit private key. The digital signature and hash digest value together constitute the parameter integrity verification credential, ensuring that the parameters have not been tampered with during transmission. The quantization and compression process reduces the number of student model parameters by 75%, the hash signature mechanism provides transmission security, and the error introduced by the 16-bit floating-point quantization is within a controllable range. In actual tests, the impact on the accuracy of policy decisions is less than 2%.

[0068] S5.3. Generate ciphertext data from the digital signature and hash digest value, encrypt the session key using the public key, output the encrypted data packet, convert it into a transmission stream, and after the edge node receives the transmission stream, it sequentially verifies the signature, decrypts the data, and loads the compression parameters.

[0069] Furthermore, the digital signature and hash digest value, along with compression parameters, constitute a data block. This data block is encrypted using the Advanced Encryption Standard-256 algorithm to generate ciphertext data. The encryption key is a 256-bit randomly generated session key, which is encrypted using an elliptic curve cryptography algorithm and further encrypted with the edge node's 2048-bit RSA public key. The encrypted session key is then output. The encrypted data packet consists of ciphertext data, the encrypted session key, and a protocol header. The header includes the packet length, version number, and checksum. The encrypted data packet is encapsulated into a transmission stream using the User Data Packet Protocol (UDP), with a 16-bit Cyclic Redundancy Check (CRC) added to ensure transmission integrity. After receiving the transmission stream, the edge node first verifies the CRC, then decrypts the session key using its RSA private key, and finally decrypts the ciphertext data using the Advanced Encryption Standard-256 algorithm. The decrypted data block separates the digital signature and hash digest value. The signature validity is verified using an elliptic curve digital signature algorithm. After confirming data integrity, the compression parameters are loaded into a lightweight convolutional neural network-long short-term memory hybrid student model. This entire process ensures the secure transmission of policy parameters from the central node to the edge node, preventing man-in-the-middle attacks and data tampering.

[0070] S6. Collect and filter the policy execution data of the edge nodes, store it in the context memory, and trigger federated learning to update the parameters of the central meta-learner.

[0071] S6.1. Capture the original state-action-reward data stream of edge node policy execution using DPDK kernel bypass technology, output the original dataset with timestamps, perform sliding window statistical tests on the original dataset, output the cleaned dataset, obtain the temporal difference error of each sample in the cleaned data, and obtain the dataset with priority labeling.

[0072] Furthermore, the data plane development toolkit directly captures the state, action, and reward data streams generated during the policy execution process of edge nodes from the network interface card through kernel bypass technology. It adds a timestamp with microsecond precision to each data record, forming a timestamped raw dataset. This timestamped raw dataset is input into the sliding window statistical testing module, with the window width set to 50 samples. The mean and standard deviation of each feature within each window are calculated, and outliers falling outside three standard deviations are removed. The cleaned dataset is then output into the temporal difference error calculation module. Based on the difference between the estimated value function of the current policy network and the actual reward value, the temporal difference error of each sample is calculated. The error value is normalized to the interval [0,1]. The temporal difference error is used as a priority weight to label the corresponding sample, forming a priority-labeled dataset. In the priority-labeled dataset, samples with a temporal difference error greater than 0.5 are classified as high-priority samples, samples with an error value between 0.2 and 0.5 are medium-priority samples, and samples with an error value less than 0.2 are low-priority samples. The priority label is used for sample selection and weighted aggregation in the subsequent federated learning process.

[0073] S6.2. Divide the data into three layers of high, medium and low priority according to priority weight, output a structured context memory bank, monitor the proportion of high priority samples in the memory bank in real time and perform detection, output trigger signal and candidate sample set, edge nodes calculate the gradient of strategy network parameters based on candidate samples, add Gaussian noise to achieve differential privacy protection, and output encrypted gradient increment.

[0074] Furthermore, the priority-labeled dataset is processed by a stratified sampling module, dividing it into three levels based on the temporal difference error value: high-priority layer (error > 0.5), medium-priority layer (0.2 ≤ error ≤ 0.5), and low-priority layer (error < 0.2). Data from each level is stored in a circular buffer, collectively forming a structured context memory. The context memory monitoring module continuously counts the proportion of high-priority samples. When the proportion exceeds 15%, a trigger signal is generated, and Top-K samples are extracted from the high-priority layer to form a candidate sample set. After receiving the candidate sample set, the edge nodes calculate the loss through forward propagation via the policy network. The loss function is used to obtain the original gradient through backpropagation. Gaussian noise with a mean of 0 and a standard deviation of 0.1 is added to achieve differential privacy protection. The noisy gradient is encrypted using a secure multi-party computation protocol, and the encrypted gradient increment is output. The encrypted gradient increment contains the weights and biases of each layer of the policy network. The dimension is consistent with the original parameters, but the values ​​are homomorphically encrypted. The gradient encryption process uses the Paillier encryption algorithm with a public key length of 2048 bits to ensure that the original gradient information is not leaked during federated aggregation. The hierarchical partitioning mechanism of the structured context memory allows important samples to participate in training first, improving the efficiency of federated learning.

[0075] S6.3. Perform modulus aggregation on the encrypted gradients using the SecAgg protocol, output the global aggregated gradient, prune the global gradients, update the meta-learner parameters, and output the next-generation policy network parameters.

[0076] Furthermore, the secure aggregation protocol receives encrypted gradient increments uploaded by edge nodes and performs modular addition aggregation operations in the encrypted state. The modulus is a Mersenne prime with a value of 2^64 minus 1. During the aggregation process, the gradient data of individual nodes is not decrypted, and the global aggregated gradient in the encrypted state is directly output. The global aggregated gradient is restored to the plaintext gradient using the Paillier homomorphic decryption algorithm. The L2 norm pruning technique is used to limit the magnitude of the gradient vector to within the range of 0.01, and a secure global gradient is output. The secure global gradient is input into the meta-learner parameter update module and weighted and superimposed with the current policy network parameters. The weight coefficients are dynamically adjusted by the adaptive learning rate algorithm. The base learning rate is set to 0.001 and dynamically scaled according to the KL divergence. After the updated parameters are verified by the parameter verification module for numerical range and structural consistency, the next-generation policy network parameters are output. The next-generation policy network parameters will serve as the benchmark model for the next round of federated learning. At the same time, a lightweight policy is generated through the knowledge distillation process. The secure aggregation protocol ensures that the local gradient information of edge nodes is completely confidential during the aggregation process, and only global statistical characteristics are exposed.

[0077] S7. Based on the policy execution data, obtain the Lyapunov index, analyze the lightweight policy model and perform testing, regenerate the environment code, and trigger the conservative update mode.

[0078] S7.1. Collect time-series data such as latency, throughput, and packet loss rate of strategy execution from edge nodes in real time, output the original state dataset, perform latency embedding and dimensional expansion on the original state data, and output the reconstructed phase space trajectory matrix.

[0079] Furthermore, edge nodes use the data plane development toolkit to collect real-time metrics such as latency, throughput, and packet loss rate during strategy execution. Each data point is appended with a timestamp with nanosecond precision to form an original state dataset. This original state dataset is input into the phase space reconstruction module, where each feature dimension undergoes independent latency embedding processing. The embedding dimension is set to 3, and the time delay parameter is set to 5 sampling intervals. The feature vectors after latency embedding are reduced to 3 dimensions through principal component analysis. The features of each dimension are concatenated to form a phase space trajectory matrix. The rows of the phase space trajectory matrix represent time points, the columns represent reconstructed state variables, and the third dimension represents principal component coefficients. The phase space trajectory matrix has a dimension of N×3×3, where N is the number of time points, which can fully characterize the dynamic evolution characteristics of the network state. The reconstruction process strictly follows the Takens embedding theorem to ensure that the phase space trajectory is equivalent to the original system dynamics. The phase space trajectory matrix will be used for subsequent Lyapunov exponent calculation and stability analysis.

[0080] S7.2 Perform QR decomposition iteration on the Jacobian matrix sequence to obtain the Lyapunov exponent, perform L2 norm pruning on the policy gradient, output the safety parameter increment, process the safety increment with the decaying learning rate, update the policy network, and trigger the conservative update mode.

[0081] Furthermore, the Jacobian matrix sequence undergoes QR decomposition iteration through a Gram-Schmidt orthogonalization process. Each iteration retains the logarithmic average of the diagonal elements of the R matrix. After 100 iterations, the maximum Lyapunov exponent is output. The policy gradient is processed using L2 norm pruning. When the magnitude of the gradient vector exceeds a threshold, it is scaled proportionally to within the threshold range, and a safety parameter increment is output. The safety parameter increment is multiplied by the decaying learning rate, which is dynamically adjusted according to the Lyapunov exponent. The weighted safety parameter increment is added to the current policy network parameters to generate updated policy network parameters. When the Lyapunov exponent exceeds the maximum Lyapunov exponent, a conservative update mode is triggered. In this mode, the learning rate decays to 10% of the normal value, and the gradient pruning threshold is reduced to 30% of the original value. The conservative update mode continues until the Lyapunov exponent falls back to the stable range, ensuring the stability of the policy update process. The Lyapunov exponent calculation module and the policy update module form a closed-loop control, adjusting the learning rate and gradient pruning intensity in real time.

[0082] It should be noted that QR decomposition iteration is performed on the Jacobian matrix sequence to obtain the Lyapunov exponent (measured decrease of 66.7%, from 0.12±0.05 to 0.04±0.02). L2 norm pruning is performed on the policy gradient (gradient variance is reduced by 73.3%, from 0.45 to 0.12). The safety parameter increment is output. The safety increment is processed with the decaying learning rate to update the policy network (policy update oscillations are reduced by 62%, and the number of interruptions is reduced by 83%), triggering a conservative update mode (business interruption probability is reduced from 0.3% to 0.05%).

[0083] This embodiment also provides a computer device applicable to the method for generating fast adaptive control strategies for communication networks based on meta-reinforcement learning, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the method for generating fast adaptive control strategies for communication networks based on meta-reinforcement learning as proposed in the above embodiment.

[0084] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0085] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the method for generating fast adaptive control strategies for communication networks based on meta-reinforcement learning as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0086] In summary, this invention achieves continuous-time evolution of policy parameters through a neural differential equation network. It adopts a dual-channel ResNet-ODE architecture combined with a gating fusion mechanism to solve the oscillation problem caused by discrete updates of traditional reinforcement learning policies, thus achieving smooth parameter adaptation. The dynamic reward function constructed based on physical layer features integrates performance rewards, stability penalties, and spectral efficiency rewards. Online correction is achieved through a long short-term memory network predictor, overcoming the limitations of single-index optimization in network layers.

[0087] Example 2, referring to Table 1, is the second embodiment of the present invention. To further verify the technical solution of the present invention, experimental simulation data of the fast adaptive control strategy generation method for communication networks based on meta-reinforcement learning are given.

[0088] Experiment preparation and implementation process This embodiment was implemented in a test environment consisting of 5G base stations and edge computing nodes. The test environment included 3 base stations and 12 edge nodes, covering three typical scenarios: urban commercial areas, residential areas, and industrial areas. Physical layer signal acquisition used a Xilinx Zynq UltraScale+ RFSoC chipset, configured with 100MHz bandwidth and 30kHz subcarrier spacing for 5G NR signals. Network layer performance data acquisition used an Intel XXV710 network card in conjunction with DPDK 20.11 to achieve microsecond-level packet capture accuracy.

[0089] The testing process is divided into four stages: data acquisition, feature processing, policy generation and distribution, and federated update. During the data acquisition stage, RFSoC captures the baseband signal at a sampling rate of 122.88MHz, calculates the channel frequency response using a 2048-point FFT, and simultaneously collects latency, throughput, and packet loss rate metrics per millisecond using DPDK. Physical layer and network layer data are synchronized via the PTPv2 protocol, with time deviation controlled within ±100ns. The preprocessing server is equipped with an NVIDIA A100 GPU and receives data using a PCIe 4.0×16 interface.

[0090] In the feature processing stage, three layers of dilated convolutions (with dilation rates of 1, 2, and 4) are used to extract spatial features. The kernel size of each convolution is 3×3, and the number of output channels is 64, 128, and 256, respectively. The number of bidirectional LSTM units is 512. The self-attention mechanism uses 4 heads, each with a dimension of 128. The short-term ResNet-ODE uses the Euler method with a fixed stride of 50ms, while the long-term ResNet-ODE uses the RK4 method with an adaptive stride, ranging from 10 to 500ms.

[0091] In the policy generation phase, initial reward weights are set as follows: performance reward 0.5, stability penalty 0.3, and spectral efficiency reward 0.2. The knowledge distillation process uses ResNet-34 as the teacher model and a 4-layer CNN + single-layer LSTM structure as the student model. In the federated learning phase, the privacy budget ε = 0.5, the gradient pruning threshold is 1.0, and the learning rate decay coefficient is 0.95.

[0092] The test lasted 72 hours, with a complete policy update cycle executed every hour, and key performance indicators recorded. The comparison system used a traditional Q-learning algorithm and a static policy configuration scheme, running on the same hardware platform. To eliminate the impact of environmental fluctuations, the three schemes were executed alternately on the same group of base stations and edge nodes using a round-robin approach, with each scheme executing for 20 minutes each time.

[0093] The performance comparison data is shown in Table 1 below:

[0094] Performance Comparison Data Table 1 Data Analysis and Innovative Description The test data shows that the proposed solution significantly outperforms traditional methods in several key metrics. In terms of latency performance, the average latency is reduced by 47.8% compared to Q-learning and by 65% ​​compared to static strategies. More importantly, the latency fluctuation coefficient is controlled at 12.3%, indicating that the neural differential equation network's ability to model network dynamics far surpasses that of traditional methods. Throughput stability reaches 94.7%, thanks to the dynamic weighted fusion of physical and network layer features via the self-attention mechanism, representing a 15% improvement over Q-learning.

[0095] The policy update efficiency demonstrates a significant advantage, with an update time of 1.8 seconds, 3.6 times faster than Q-learning. This is attributed to the efficient feature extraction capability of the CNN-LSTM hybrid encoder. Anomaly recovery time is only 3.2 seconds, 79% faster than the static policy, proving that the conservative update mode of Lyapunov exponential analysis effectively prevents policy oscillations. The record-breaking spectral efficiency of 4.8 bps / Hz comes from the refined modeling of physical layer features by the dynamic reward function, representing a 23% improvement over traditional methods.

[0096] In terms of resource utilization, the 35.2% CPU utilization rate demonstrates the effectiveness of the knowledge distillation and compression strategy, reducing it by 27.7% compared to Q-learning. The 87.5% hit rate of the memory bank validates the advantages of the priority-based federated learning sample selection mechanism. The 15-minute federated update cycle is 50% faster than traditional methods, thanks to the efficient gradient aggregation of the SecAgg protocol.

[0097] The novelty of the technology is mainly reflected in three aspects: First, the neural differential equation network models policy parameter updates as a continuous dynamic system, and by fusing short-term and long-term ResNet-ODE, it solves the limitation of traditional reinforcement learning in temporal scale coupling. Test data shows that this method improves policy adaptability by 42%. Second, the normalization processing of multi-dimensional time series combined with the self-attention mechanism realizes the dynamic association of cross-layer features, which is verified by throughput stability metrics. Finally, the conservative update mode based on the Lyapunov exponent achieves fast convergence while maintaining policy stability, which is reflected in the anomaly recovery time and latency fluctuation coefficient.

[0098] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for generating fast adaptive control strategies for communication networks based on meta-reinforcement learning, characterized in that: This includes collecting and processing physical layer signals and network layer metrics to obtain a standardized multidimensional time series matrix; The standardized multidimensional time series matrix is ​​input into a hybrid encoder composed of CNN-LSTM, and dynamically weighted through a self-attention mechanism to output an environmental feature vector; The environmental feature vector is input into the neural differential equation network to generate policy parameter increments and update the policy network parameters. A dynamic reward function is constructed based on the physical layer features in the environmental feature vector and detected to obtain the adjusted dynamic reward value. The updated policy network parameters are compressed into a lightweight policy model through knowledge distillation and distributed to edge nodes through an encrypted channel. Collect and filter policy execution data from edge nodes, store it in the context memory, trigger federated learning to update the parameters of the central meta-learner, obtain the Lyapunov exponent based on the policy execution data, analyze and test the lightweight policy model, regenerate the environment code, and trigger the conservative update mode.

2. The method for generating fast adaptive control strategies for communication networks based on meta-reinforcement learning as described in claim 1, characterized in that: The process of acquiring and processing physical layer signals and network layer metrics to obtain a standardized multidimensional time series matrix includes the following steps. The sampling rate of the original signal is acquired by the RFSoC chip in the base station AAU to obtain the time-aligned baseband sample stream. The baseband sample stream is then subjected to FFT to obtain the channel frequency response, and the instantaneous signal-to-noise ratio and Doppler frequency shift are calculated. DPDK captures latency, throughput, and packet loss rate from the network card to obtain a set of metrics with hardware timestamps. It then synchronizes the physical layer feature triplet with the network layer metric set to the same time base via the PTP protocol to obtain an aligned hybrid data block. Each feature dimension in the hybrid data block is processed to obtain a two-dimensional data matrix. Timestamp metadata is added to the two-dimensional data matrix and transmitted to the preprocessing server via the PCIe 4.0 channel to obtain a standardized multidimensional time series matrix.

3. The method for generating fast adaptive control strategies for communication networks based on meta-reinforcement learning as described in claim 2, characterized in that: The standardized multidimensional time series matrix is ​​input into a hybrid encoder composed of CNN-LSTM, dynamically weighted through a self-attention mechanism, and outputs an environmental feature vector. Includes the following steps, The input two-dimensional data matrix is ​​processed through three layers of dilated convolution to obtain a spatiotemporal feature tensor. The spatiotemporal feature tensor is then input into a bidirectional LSTM to obtain temporal enhancement features. Based on temporal enhancement features, query matrix, key matrix and value matrix are generated respectively, and dynamic weighting is performed using a four-head attention mechanism to obtain a two-dimensional feature tensor; Two-dimensional feature tensors are concatenated to form a multi-head attention concatenated feature matrix, which is then passed through a linear transformation layer to obtain a compressed spatiotemporal feature vector. The compressed spatiotemporal feature vector is divided into physical layer feature coding segments and network layer feature coding segments to obtain the environmental feature vector.

4. The method for generating fast adaptive control strategies for communication networks based on meta-reinforcement learning as described in claim 3, characterized in that: The environmental feature vector is input into a neural differential equation network to generate policy parameter increments and update the policy network parameters, including the following steps. The physical layer feature coding segment and the network layer feature coding segment are processed by Min-Max normalization and Gaussian normalization respectively, and the standardized environmental feature vector is output. The microsecond-level UNIX timestamp feature is embedded at the end of the standardized environmental feature vector, and the time-series enhanced feature vector is output. The physical layer feature encoding segment in the temporal enhancement feature vector is input into the short-term ResNet-ODE network. The Euler method is used to solve the short-term differential change of the policy parameters with a step size of 50ms, and the short-term policy parameter increment is output. The network layer feature encoding segments in the temporal enhancement feature vector are input into the long-term ResNet-ODE network. The Runge-Kutta4 method is used to solve the long-term trend of the policy parameters with an adaptive step size, and the long-term policy parameter increment is output. Based on the real-time weights of the physical layer feature encoding segments and the network layer feature encoding segments, the short-term policy parameter increment and the long-term policy parameter increment are fused through the sigmoid gating mechanism, and the fused policy parameter increment is output. Based on the KL divergence of the probability distributions of the old and new policy outputs, the effective learning rate is calculated and the increment of the fusion policy parameters is scaled to output the adjusted policy parameter update amount. The adjusted policy parameter update is constrained by the L2 norm, and the security policy parameter increment is output. The security policy parameter increment is then added to the current policy network parameter to generate the updated policy network parameter.

5. The fast adaptive control strategy generation method for communication networks based on meta-reinforcement learning as described in claim 4, characterized in that: A dynamic reward function is constructed based on physical layer features in the environmental feature vector, and detection is performed to obtain an adjusted dynamic reward value. This includes the following steps: The physical layer feature coding segment in the decoupled environment feature vector is decomposed into four types of structured data: spectral features, signal-to-noise ratio dynamics, Doppler features, and interference fingerprints, and the physical layer feature dictionary is output. The performance reward is calculated based on the spectral features in the physical layer feature dictionary, the stability penalty is calculated based on the dynamic signal-to-noise ratio and Doppler features, and the spectral efficiency reward is calculated based on the spectral features. The performance reward, stability penalty, and spectral efficiency reward are then fused using dynamic weights to obtain the original reward value. A pre-trained LSTM predictor is used to detect the deviation of the original reward value, and the reward value is halved to correct it, outputting the adjusted dynamic reward value.

6. The method for generating fast adaptive control strategies for communication networks based on meta-reinforcement learning as described in claim 5, characterized in that: The updated policy network parameters are compressed into a lightweight policy model through knowledge distillation and distributed to edge nodes via an encrypted channel, including the following steps. The updated policy network parameters are used to build a teacher model, which infers on the state sample set and outputs the action probability distribution and action value function. The action probability distribution and action value function output by the teacher model are used as supervision signals to train a lightweight convolutional neural network-long short-term memory network hybrid student model and output the student model parameters. The student model parameters are quantized into 16-bit floating-point quantization and converted into 8-bit integers. The quantized compressed parameters are output. The compressed parameters are then subjected to a secure hash algorithm, and the digital signature and hash digest value are output. The digital signature and hash digest value are used to generate ciphertext data. The session key is encrypted using the public key, and the encrypted data packet is output. This data is then converted into a transmission stream. After receiving the transmission stream, the edge node verifies the signature, decrypts the data, and loads the compression parameters in sequence.

7. The method for generating fast adaptive control strategies for communication networks based on meta-reinforcement learning as described in claim 6, characterized in that: Collect and filter policy execution data from edge nodes, store it in the context memory, and trigger federated learning to update the parameters of the central meta-learner. This includes the following steps: The DPDK kernel bypass technique is used to capture the original state-action-reward data stream of edge node policy execution, output the original dataset with timestamps, perform sliding window statistical tests on the original dataset, output the cleaned dataset, obtain the temporal difference error of each sample in the cleaned data, and obtain the dataset with priority labeling. The data is divided into three layers of high, medium and low priority weights, and a structured context memory is output. The proportion of high-priority samples in the memory is monitored and detected in real time. Trigger signals and candidate sample sets are output. Edge nodes calculate the gradient of network parameters based on candidate samples. Gaussian noise is added to achieve differential privacy protection and encrypted gradient increments are output. The encrypted gradients are modularly aggregated using the SecAgg protocol to output the global aggregated gradient. The global gradient is then pruned to update the meta-learner parameters, and the next-generation policy network parameters are output.

8. The method for generating fast adaptive control strategies for communication networks based on meta-reinforcement learning as described in claim 7, characterized in that: The Lyapunov index is obtained based on policy execution data, and a lightweight policy model is analyzed and tested. Environment coding is then regenerated, triggering a conservative update mode. This process includes the following steps: The system collects time-series data such as latency, throughput, and packet loss rate of the strategy execution in real time from edge nodes, outputs the original state dataset, performs latency embedding and dimensional expansion on the original state data, and outputs the reconstructed phase space trajectory matrix. The Jacobian matrix sequence is subjected to QR decomposition iteration to obtain the Lyapunov exponent. The policy gradient is then pruned using the L2 norm to output the safety parameter increment. The safety increment is processed with the decaying learning rate to update the policy network and trigger the conservative update mode.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the fast adaptive control strategy generation method for communication networks based on meta-reinforcement learning as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the fast adaptive control strategy generation method for communication networks based on meta-reinforcement learning as described in any one of claims 1 to 8.

Images