A network information age optimization method and system based on meta-deep reinforcement learning

Abstract

The present invention discloses a network information age optimization method and system based on meta-deep reinforcement learning. This relates to the field of wireless network communication technology and includes establishing a video transmission model, collecting current state data, initial reward information, and action information; obtaining meta-features through a meta-feature extraction function; using an actor policy network to combine the current policy and meta-features to obtain the next action, which is then sent back to the video transmission model to provide the next state data and reward information; evaluating the effect of the next action selected by the actor policy network through a critic value network to calculate the actor policy network policy gradient; simultaneously updating the critic value network value function, setting a shearing probability ratio mechanism, and performing policy gradient shearing on the network; and dynamically updating network parameters in conjunction with a meta-learning algorithm. The present invention improves the overall utilization of network resources and reduces data transmission failures and retransmissions caused by network congestion.

Description

Technical Field

[0001] The present invention relates to the field of wireless network communication technology, and more particularly to a network information age optimization method and system based on meta-deep reinforcement learning. Background Art

[0002] Industry 4.0 represents a new stage in industrial development, involving the integrated application of technologies such as automation, data exchange, cloud computing, the Internet of Things (IoT), and artificial intelligence (AI). First proposed in Germany in 2011, the concept aims to enhance the intelligence and flexibility of the manufacturing industry. The core of Industry 4.0 is the creation of smart factories, where network systems and physical processes are integrated, devices are interconnected through the IoT, and cloud computing provides powerful data processing capabilities. This environment requires network communications to meet extremely high standards in terms of real-time performance and reliability to support real-time monitoring and control of production lines and machines.

[0003] In recent years, with the surge in IoT devices and data traffic, traditional network architectures have struggled to meet the stringent real-time and reliability requirements of Industry 4.0. In industrial environments, network congestion and electromagnetic interference are particularly prominent, severely impacting network performance and data timeliness. Furthermore, industrial applications place extremely high demands on the timeliness of network information; any delay in information can lead to decreased production efficiency or even safety incidents. Summary of the Invention

[0004] In view of this, the present invention provides a network information age optimization method and system based on meta-deep reinforcement learning to solve the problems existing in the background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A network information age optimization method based on meta-deep reinforcement learning, comprising:

[0007] Establishing a video transmission model and collecting current state data, initial reward information, and action information of the video transmission model;

[0008] The current state data is mapped to the feature space through a meta-feature extraction function to obtain meta-features. The Actor strategy network combines the current strategy with the meta-features to obtain the next action, and the next action is sent back to the video transmission model. The video transmission model provides the next state data and the next reward information, and evaluates the effect of the next action based on the advantage function and the target value.

[0009] The Actor Policy Network policy gradient is calculated by evaluating the effect of the next action selected by the Actor Policy Network through the Critic Value Network, which is used to guide the Actor Policy Network to update information. The Critic Value Network updates the Critic Value Network value function based on the next reward information and the discount factor, and sets a clipping probability ratio mechanism to perform policy gradient clipping on the Critic Value Network and the Actor Policy Network.

[0010] According to the value function of the Critic value network, the policy gradient of the Actor policy network and the next step reward information, the parameters of the Critic value network and the Actor policy network are dynamically updated in combination with the meta-learning algorithm.

[0011] Preferably, the obtaining of meta-features specifically includes: the input feature is X, whose shape is N×d, where N is the number of samples and d is the feature dimension, and the input feature is mapped to the feature space through the meta-feature extraction function g(·); Z=g(X;θg), where Z is the meta-feature and θg is the parameter of the meta-feature extraction function.

[0012] Preferably, the evaluating the effect of the next action according to the advantage function and the target value specifically includes:

[0013] δ t =r t +γV(s t+1 )-V(s t );

[0014] Among them, δ t is the TD error at time t, r t is the reward, γ is the discount factor, V(s t ) and V(s t+1 ) are the value function predictions of the current state and the next state respectively;

[0015] A t =δ t +(γλ)δ t+1 +…+(γλ) T-t+1 δ T-1 ;

[0016] Among them, the advantage function A t represents the difference between the immediate reward obtained after taking an action and its corresponding state value function at time step t; λ is a parameter used for GAE to balance variance and bias, and T is the last time step in the decision sequence;

[0017] Use the collected data to calculate the advantage function for each step: A(s,a)=Q(s,a)-V(s t );

[0018] Where Q(s,a) is the state-action value function, that is, the expected total reward that can be obtained by taking action a in state s;

[0019] Calculate the target value Vtarget for each state: Vtarget = r t +γV(s t+1 ).

[0020] Preferably, the reward information is calculated based on a reward function, and the reward function includes:

[0021]

[0022] Where n is the index of the sensor, k is the time step, delay is the delay of the current video block obtained from the environment, and sensor_selection is the currently selected sensor;

[0023] If the age of a sensor's information exceeds the set threshold, a violation is considered to have occurred. I is the indicator function, A_DIM is the number of sensors, and it returns 1 if the age exceeds the threshold, otherwise it returns 0; tau[n] is the age threshold of the nth sensor;

[0024] A multi-objective optimization problem is defined, with the goals of minimizing the total delay of video transmission, minimizing the information age of the sensor, minimizing the number of violations, minimizing the number of rebuffering events, minimizing the sudden change in video quality, and maximizing the accuracy of the video. The optimization problem is divided into six sub-problems, which are defined by the following mathematical expression:

[0025] R total =R delay +R age +R quality +R violation +R rebuffer +R smooth ;

[0026] Among them, R delay is the delay penalty formula, R age is the information age penalty formula, R quality is the video quality reward formula, R violation is the violation penalty formula, R rebuffer is the heavy buffer penalty formula, R smooth Penalty formula for smooth switching;

[0027] Weighted sum scalarization: Each objective is multiplied by a weight, and then all weighted objectives are summed:

[0028]

[0029] Among them, f i (x) is the i-th objective function, w i is the weight of the i-th objective function, and x is the decision variable.

[0030] Preferably, the shear probability ratio mechanism specifically includes:

[0031]

[0032] Among them, π θ and denote the current strategy and the old strategy respectively, a is the action selected in state s;

[0033] This function avoids excessive adjustments during the policy update process by clipping the probability ratio mechanism, limiting the ratio to fluctuate within the range of [1-∈, 1+∈], where ∈ is the clipping threshold and π θ is the probability of the current strategy, is the probability of the old strategy.

[0034] Preferably, the method further includes calculating the loss of the Critic value network and the Actor strategy network, specifically including:

[0035] Actor policy network loss:

[0036]

[0037] Among them, π θ is the probability of the current strategy, is the probability of the old strategy, A t is the advantage function, ∈ is the clipping threshold;

[0038] Critic value network loss:

[0039]

[0040] Among them, V target =r+γV(s′),V φ (s t ) is the predicted value of the current Critic value network, r is the reward, γ is the discount factor, and V(s′) is the value estimate of the next state.

[0041] Preferably, the method for updating the parameters of the Critic value network and the Actor strategy network is a synchronous update method or an asynchronous update method.

[0042] Preferably, before updating the parameters of the Critic value network and the Actor strategy network, a normalization process is further performed, specifically including:

[0043] The data is recorded as M, M={m1,m2,...,m n};

[0044] Calculate the mean of the data Among them, μ is the mean, m i is the i-th data point, N is the total number of data points;

[0045] Calculate the standard deviation of data

[0046] Let Y be less than μ-kσ or greater than μ+kσ, and the data point Y={y|y<μ-2σor y>μ+2σ}, where Y is the data in X that meets the conditions, Y={y1,y2,...,y n}, for each data point y i Normalize it to convert it into Among them, z i is the standard deviation of the ith data point.

[0047] A network information age optimization system based on meta-deep reinforcement learning, including:

[0048] A model building module, which builds a video transmission model and collects current state data, initial reward information, and action information of the video transmission model;

[0049] An evaluation module maps the current state data to a feature space using a meta-feature extraction function to obtain meta-features. The Actor Strategy Network combines the current strategy with the meta-features to obtain a next action, which is then sent back to the video transmission model. The video transmission model provides the next state data and next reward information, and evaluates the effect of the next action based on the advantage function and the target value.

[0050] The comprehensive update module uses the Critic value network to evaluate the effect of the next action selected by the Actor policy network and calculates the Actor policy network policy gradient to guide the Actor policy network to update information. The Critic value network updates the Critic value network value function based on the next reward information and the discount factor, and sets a shearing probability ratio mechanism to perform policy gradient shearing on the Critic value network and the Actor policy network; according to the Critic value network value function, the Actor policy network policy gradient and the next reward information, the parameters of the Critic value network and the Actor policy network are dynamically updated in combination with the meta-learning algorithm.

[0051] Through the above technical solutions, it can be seen that compared with the existing technology, the present invention discloses a network information age optimization method and system based on meta-deep reinforcement learning, which significantly improves the efficiency and quality of video transmission in the industrial Internet environment. Through intelligent routing selection and system optimization, it effectively reduces latency and improves the processing accuracy of video data, meeting the high standards of Industry 4.0 for real-time and reliability. Integrating meta-learning into each link can effectively improve the adaptability and stability of the video transmission model in different network environments, ensuring that the model can run efficiently in a real industrial IoT environment and can quickly adapt to new tasks and environments. This method improves the overall utilization of network resources and reduces the need for data transmission failures and retransmissions due to network congestion. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0053] Figure 1 A flow chart of the method provided by the present invention;

[0054] Figure 2 This is a structural diagram provided by the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0056] The embodiment of the present invention discloses a network information age optimization method based on meta-deep reinforcement learning, which is characterized in that: Figure 1 Shown, including:

[0057] Establish a video transmission model and collect the current state data, initial reward information and action information of the video transmission model;

[0058] The current state data is mapped into the feature space through the meta-feature extraction function to obtain the meta-features. The Actor Policy Network combines the current policy and the meta-features to obtain the next action and sends the next action back to the video transmission model. The video transmission model provides the next state data and the next reward information, and evaluates the effect of the next action based on the advantage function and the target value.

[0059] The Actor Policy Network policy gradient is calculated by evaluating the effect of the next action selected by the Actor Policy Network through the Critic Value Network. This is used to guide the Actor Policy Network to update information. The Critic Value Network updates the Critic Value Network value function based on the next reward information and the discount factor. At the same time, a clipping probability ratio mechanism is set to perform policy gradient clipping on the Critic Value Network and the Actor Policy Network.

[0060] According to the value function of the Critic value network, the policy gradient of the Actor policy network and the next reward information, the parameters of the Critic value network and the Actor policy network are dynamically updated in combination with the meta-learning algorithm.

[0061] Reward information refers to the immediate return fed back by the environment at each time step, but its uses are different. The Critic network uses reward information to update the value function, which helps the Critic network accurately evaluate the value of states and actions.

[0062] The Actor network uses reward information to calculate the policy gradient, which helps the Actor network optimize the policy so that it can obtain the Actor policy network in the future state: the input layer accepts the state of the environment, passes through two layers of 128-unit fully connected layers (each layer has ReLU activation function), and the output layer generates the probability distribution of the action through the Softmax activation function (for discrete action space) or the linear activation function to generate the specific action value (for continuous action space).

[0063] Critic Value Network: The input layer receives the state of the environment, passes through two fully connected layers with 128 units (each layer uses ReLU activation function), and the output layer generates the value evaluation of the state through a linear activation function. (This can lead to higher cumulative returns)

[0064] In a specific embodiment, meta-learning is a method for improving the adaptability and generalization ability of a model. The present invention applies a meta-learning algorithm in meta-feature extraction, meta-strategy adjustment, and meta-optimization. Obtaining meta-features specifically includes: the input feature is X, whose shape is N×d, where N is the number of samples and d is the feature dimension, and the input feature is mapped to the feature space through the meta-feature extraction function g(·); Z = g(X; θg), where Z is the meta-feature and θg is the parameter of the meta-feature extraction function.

[0065] Meta-policy adjustment: In the actor policy network and critic value network, meta-learning methods are used to dynamically adjust the policy to quickly adapt to different network conditions.

[0066] Meta-optimization: During the training process, a meta-learning optimization algorithm is used to dynamically adjust the learning rate and parameter update strategy to improve the convergence speed and stability of the model.

[0067] In a specific embodiment, the temporal difference (TD) error is used to measure the difference between the actual reward and the predicted reward after the policy is executed. This error reflects the accuracy of the current policy's estimate of the value of the environment state. In PPO, the TD error is used to calculate the advantage function, which helps evaluate the relative advantages of taking a specific action in a specific state. Through the use of the advantage function, PPO can effectively update the policy and drive the learning process towards a better solution. The effect of the next action evaluated based on the advantage function and the target value specifically includes:

[0068] δ t =r t +γV(s t+1 )-V(s t );

[0069] Among them, δ t is the TD error at time t, r t is the reward, γ is the discount factor, V(s t ) and V(s t+1 ) are the value function predictions of the current state and the next state respectively;

[0070] A t =δ t +(γλ)δ t+1 +…+(γλ) T-t+1 δ T-1 ;

[0071] Among them, the advantage function A t represents the difference between the immediate reward obtained after taking an action and its corresponding state value function at time step t; λ is a parameter used for GAE to balance variance and bias, and T is the last time step in the decision sequence;

[0072] Use the collected data to calculate the advantage function for each step: A(s,a)=Q(s,a)-V(s t );

[0073] Where Q(s,a) is the state-action value function, that is, the expected total reward that can be obtained by taking action a in state s;

[0074] Calculate the target value Vtarget for each state: Vtarget = r t +γV(s t+1 ).

[0075] In a specific embodiment, the reward information is calculated based on a reward function. The reward function is designed to improve information timeliness and optimize network performance. The reward information is calculated based on the reward function. The reward function includes:

[0076]

[0077] Where n is the index of the sensor, k is the time step, delay is the delay of the current video block obtained from the environment, and sensor_selection is the currently selected sensor;

[0078] If the age of a sensor's information exceeds the set threshold, a violation is considered to have occurred. I is the indicator function, A_DIM is the number of sensors, and it returns 1 if the age exceeds the threshold, otherwise it returns 0; tau[n] is the age threshold of the nth sensor;

[0079] A multi-objective optimization problem is defined, with the goals of minimizing the total delay of video transmission, minimizing the information age of the sensor, minimizing the number of violations, minimizing the number of rebuffering events, minimizing the sudden change in video quality, and maximizing the accuracy of the video. The optimization problem is divided into six sub-problems, which are defined by the following mathematical expression:

[0080] R total =R delay +R age +R quality +R violation +R rebuffer +R smooth ;

[0081] Among them, R delay is the delay penalty formula, R age is the information age penalty formula, R quality is the video quality reward formula, R violation is the violation penalty formula, R rebuffer is the heavy buffer penalty formula, R smooth Penalty formula for smooth switching;

[0082] Delay penalty formula: Rdelay = -w1·delay; w1 is the weight coefficient of delay, and delay is the delay time of the video block obtained from the environment.

[0083] Information age penalty formula: R age = -w2·total_age; w2 is the weight coefficient of information age, and total_age is the sum of the current information ages of all sensors.

[0084] Video quality reward formula: R quality= w3·video_quality_reward; w3 is the weight coefficient for video quality, and video_quality_reward is the reward value obtained from the HD_REWARD array based on the currently selected video bitrate. HD_REWARD = [1, 2, 3, 12, 15, 20] # High-definition video reward.

[0085] Violation penalty formula: R violation = -λ·current_violation; λ is the penalty coefficient for violation, and current_violation represents the number of sensors exceeding the threshold in the current time step.

[0086] Rebuffering penalty formula: R rebuffer =-rebuf_penalty·sleep_time; rebuf_penalty is the penalty coefficient for rebuffering, and sleep_time is the delay time due to waiting for buffering.

[0087] Smooth switching penalty formula:

[0088] smooth_penalty is the penalty coefficient for smooth switching, which is used to reduce sudden changes in video quality.

[0089] In multi-objective optimization, different objectives may conflict with each other. For example, improving video quality (increasing bit rate) may increase transmission delay and increase the possibility of rebuffering, while minimizing delay and rebuffering may limit the highest video quality you can choose. Scalarization technology attempts to synthesize multiple optimization objectives into a single scalar objective function, which can be solved using standard optimization methods. For applications with clear priorities and the need for fast response, such as certain manufacturing and automation processes, scalarization technology may be more suitable because it can quickly provide decision support and simplify control logic. In the application field of the present invention, it is reasonable to regard information age as the most important objective, and at the same time, the remaining objectives need to be constrained, that is, the weight of information age is maximized.

[0090] Each goal is multiplied by a weight, and then all weighted goals are summed. The weight reflects the relative importance of different goals. Weighted sum scalarization: Each goal is multiplied by a weight, and then all weighted goals (six sub-problems) are summed:

[0091]

[0092] Among them, f i (x) is the i-th objective function, w i is the weight of the i-th objective function, and x is the decision variable.

[0093] In a specific embodiment, the shear probability ratio mechanism specifically includes:

[0094]

[0095] Among them, π θ and denote the current strategy and the old strategy respectively, a is the action selected in state s;

[0096] The clipping probability ratio mechanism is a technique used to avoid excessive policy updates during policy gradient optimization. Specifically, the mechanism clips the policy update ratio to limit the range of the ratio between the new policy and the old policy, thereby avoiding excessive adjustments during the policy update process, which can lead to model instability. This function avoids excessive adjustments during the policy update process by clipping the probability ratio mechanism, limiting the ratio to the range [1-∈, 1+∈], where ∈ is the clipping threshold and π θ is the probability of the current strategy, is the probability of the old policy. Clipping the probability ratios avoids making drastic adjustments during policy updates. By limiting the probability ratios within a certain range, the PPO algorithm can reduce fluctuations in policy updates caused by a single or small number of examples. These extreme examples can have a significant impact on the gradient without clipping. By clipping the influence of these extreme probability ratios, PPO helps reduce variance in the learning process and improves overall policy stability.

[0097] In a specific embodiment, the loss calculation of the Critic value network and the Actor strategy network is also included, specifically including:

[0098] Actor Policy Network Loss: PPO uses a clipping probability ratio mechanism to avoid excessive adjustments during policy updates:

[0099]

[0100] Among them, π θ is the probability of the current strategy, is the probability of the old strategy, A t is the advantage function, ∈ is the clipping threshold;

[0101] Critic Value Network Loss: The goal of the value function is to estimate the value of the state, usually by minimizing the mean squared error between the predicted value and the actual return:

[0102]

[0103] Among them, V target =r+γV(s′),V φ (st ) is the predicted value of the current Critic value network, r is the reward, γ is the discount factor, and V(s′) is the value estimate of the next state.

[0104] It also includes an entropy term (used to encourage strategy exploration). The entropy reward term is used to encourage strategy exploration and maintain strategy diversity. Specifically:

[0105] H(π θ )=E t [entropy(π θ (a t |s t ))];

[0106] Total loss function: L(θ,φ)=L CLIP (θ)+c1L VF (φ)-c2H(π θ ), where c1 and c2 are weight hyperparameters that control the impact of value function loss and entropy reward, respectively.

[0107] In a specific embodiment, the method for updating the parameters of the Critic value network and the Actor strategy network is a synchronous update method or an asynchronous update method.

[0108] Gradient descent is the primary method for optimizing model parameters. During the learning process, the parameters of the Critic Value Network and the Actor Policy Network are optimized by calculating the gradient of the loss function and updating the parameters. These updates can be synchronous or asynchronous:

[0109] Synchronous Update: All parameters are updated together after each time step.

[0110] Asynchronous updates: Parameters in different parts can be updated independently at different times. Asynchronous updates are a method for improving training efficiency and stability. In asynchronous reinforcement learning algorithms, multiple agents learn in parallel, each interacting with the environment and independently updating their own policies and value networks. These updates are then asynchronously applied to the global model. This approach accelerates learning while reducing the risk of overfitting through diverse experience.

[0111] The parameters of the actor policy network and the critic value network are updated within a single computing environment. In the traditional PPO algorithm, the parameters of the actor and critic networks are updated within a single computing environment. These network parameters are stored and updated to the same global network after each iteration, ensuring that the learning results obtained from experimental data can be effectively applied to the iterative optimization of the policy. The introduction of meta-learning methods allows the grouped delayed update mechanism to be combined with meta-learning. Parameter updates are not performed immediately after each new data is collected, but after sufficient data has been collected, which is divided into different groups and randomly sampled. This ensures that updates are based on reliable and stable data trends and facilitates exploration.

[0112] Adaptive learning rate management is an important part of meta-learning. It dynamically adjusts the learning rate based on the model's performance during training: when the model's performance improves, the learning rate is reduced to fine-tune the model parameters; when the model's performance stagnates or deteriorates, the learning rate is increased to escape possible local optima.

[0113] Specific adaptive learning rate adjustment formula:

[0114] Among them, η t is the learning rate of the tth iteration, η0 is the initial learning rate, τ is the time constant, L i is the loss function for the i-th iteration. θ L i is the loss function L of the i-th iteration i The gradient of the parameter θ, ▽ θ L i-1 is the loss function L of the i-1th iteration i-1 The gradient of the parameter θ.

[0115] In asynchronous updating, each parallel worker node autonomously trains its own actor and critic network and asynchronously transmits the learned updates to the global network. The global network integrates the learning results of all nodes and updates the global model parameters. This asynchronous approach allows multiple agents to execute in parallel in different copies of the environment. These agents can independently update the global network based on their collected data without waiting for other agents to complete their episodes. This improves sample utilization efficiency, reduces update variance, and enhances the stability of the learning process. Asynchronous versions of PPO tend to use larger batch sizes and longer update cycles, which facilitate the use of more stable policy gradient estimates.

[0116] Forward propagation and backpropagation are used to update the Actor policy network and Critic value network models, calculate losses, and optimize network parameters through the gradient descent algorithm to improve the accuracy of the model's policy evaluation and decision-making. The selection should be based on actual conditions.

[0117] In a specific embodiment, the neural network input data is normalized using the standard deviation Z-score. Ordinary normalization is not suitable for data with a large span or abnormal data. The purpose of normalization is to process state (s_batch) and reward (r_batch) data, as well as action probabilities. Before updating the parameters of the Critic value network and the Actor policy network, normalization is also performed, specifically including:

[0118] The data is recorded as M, M={m1,m2,...,m n};

[0119] Calculate the mean of the data Among them, μ is the mean, m i is the i-th data point, N is the total number of data points;

[0120] Calculate the standard deviation of data

[0121] Let Y be less than μ-kσ or greater than μ+kσ, and the data point Y={y|y<μ-2σor y>μ+2σ}, where Y is the data in X that meets the conditions, Y={y1,y2,...,y n}, for each data point y i Normalize it to convert it into Among them, z i is the standard deviation of the i-th data point. This can effectively reduce the impact of noise and outliers in the data while ensuring data consistency and standardization, thereby improving data processing efficiency and model accuracy.

[0122] In specific embodiment 1, it specifically includes:

[0123] Build a network simulation environment: For industrial IoT scenarios, use the ns3 simulation tool to build a multi-node video transmission model involving routers, servers, edge servers, and multiple network nodes, taking into account the complex transmission paths between nodes.

[0124] Real-time collection of network status data: In a simulated network environment, real-time monitoring of network performance, including end-to-end delay, throughput t, bandwidth B of each link, and packet loss rate P.

[0125] Customized link packet loss model: Design and implement a customized packet loss model to address network congestion and electromagnetic interference in industrial scenarios, simulating data packet loss in specific network environments.

[0126] Taking into account factors such as network congestion and electromagnetic interference in industrial scenarios, a customized link packet loss model is used to simulate packet loss in the network. This customized model will determine whether to drop packets based on the current time, simulated electromagnetic interference, and packet size (all packets larger than K bytes will be discarded. The threshold K can be adjusted as needed to simulate specific network environments and conditions).

[0127] Electromagnetic Interference (EMI):

[0128] Use a uniformly distributed random variable to simulate sporadic electromagnetic interference that occasionally causes packet loss.

[0129] Assume X is a uniformly distributed random variable, X~U(0,1), representing whether electromagnetic interference will affect packet transmission at any given point in time. We define a threshold θ, assuming θ = 0.05, which means there is a 5% probability that electromagnetic interference will significantly affect the network. When X ≤ θ, electromagnetic interference will increase the probability of packet loss ∈. Otherwise, electromagnetic interference has no effect on the packet loss rate, meaning the increased packet loss rate is zero.

[0130]

[0131] Network congestion:

[0132] The system determines whether it is a network peak period based on the current time. If so, it increases the probability of packet loss to simulate the situation where the network is more prone to packet loss during high traffic periods.

[0133] Packet size:

[0134] If the packet size exceeds a certain threshold K, the possibility of packet loss increases, simulating that large packets are more susceptible to impact when the network environment is poor or the load is high. Assume that the threshold K = 1000

[0135] Logic: If the size of the packet C(p) exceeds 1000 bytes, then this function returns true, indicating that the packet should be discarded.

[0136]

[0137] Here, C(p) is an indicator function that takes the value 1 (packet loss) when the size of p exceeds 1000 bytes, and 0 (no packet loss) otherwise.

[0138] P total =P base +PEMI +P otherfactors ;

[0139] The total packet loss probability P total is the basic packet loss probability P base (Pc) plus the increased packet loss probability P caused by electromagnetic interference EMI , and the packet loss probability P increased by network congestion otherfactors .

[0140] Introducing randomness to simulate network fluctuations: Use normally distributed random variables to simulate randomness in data transmission, simulate network fluctuations by dynamically adjusting the sending interval or size of data packets, and introduce randomness in data transmission delay and data rate to indirectly simulate network fluctuations.

[0141] normal distribution:

[0142] μ is the mean, which indicates the center of the distribution. σ is the standard deviation, which indicates the width of the distribution and the degree of dispersion of the data.

[0143] In the network simulation environment described above, the download time T is calculated based on the current network conditions and video quality, and the network status is returned. If the packet payload plus the data already sent exceeds the video segment size, the time required to download the remaining portion is calculated. The delay time, wait time, and video segment size for downloading the video segment are returned.

[0144] Network information processing and conversion: Process input vectors and state data, apply fully connected layers, convolutional layers, and coordinate attention mechanisms, and optimize input data processing for the Actor Policy Network and Critic Value Network.

[0145] Convert the information form that the neural network model can accept into the feature vector X that the neural network model can accept i (i=x1,x2,x3).

[0146] Network Policy and Value Evaluation: A critic value network and an actor policy network are established to evaluate the value of policies and make network scheduling decisions, respectively. The critic value network evaluates the value of the current policy and helps predict future rewards. The actor policy network generates specific network scheduling decisions, such as packet priority assignment and bandwidth allocation. A critic value network is constructed to evaluate the value of the current policy. This network takes a state vector as input, processes it using convolutional and fully connected layers, and outputs an estimated value for the state.

[0147] Learning process of optimization strategy:

[0148] The PPO algorithm optimizes the learning process through temporal difference error and clipping probability ratio mechanisms. A meta-learning mechanism is introduced to train multiple sub-models in different simulation environments. The meta-learning framework then extracts and fuses the knowledge of these sub-models to improve the accuracy and stability of policy execution.

[0149] Reward function design: Design a reward function with the goals of improving information timeliness and optimizing network performance, focusing on solving complex network scheduling problems through multi-objective optimization.

[0150] Adjustment of learning rate and update mechanism: Implement adaptive learning rate management and grouped delayed update mechanism to optimize the update process of network parameters and improve the learning efficiency and stability of the model.

[0151] Data standardization: Implement Z-score normalization to reduce the impact of noise and outliers in the data, and improve the efficiency of data processing and the accuracy of the model.

[0152] Asynchronous update mechanism: Asynchronous training is performed in multiple parallel working nodes to improve sample utilization efficiency and the stability of the learning process.

[0153] A network information age optimization system based on meta-deep reinforcement learning, such as Figure 2 Shown, including:

[0154] Model building module, builds a video transmission model, collects the current state data, initial reward information and action information of the video transmission model;

[0155] The evaluation module maps the current state data to the feature space through the meta-feature extraction function to obtain meta-features. The actor policy network combines the current policy and meta-features to obtain the next action and sends the next action back to the video transmission model. The video transmission model provides the next state data and next reward information, and evaluates the effect of the next action based on the advantage function and target value.

[0156] The comprehensive update module uses the Critic value network to evaluate the effect of the next action selected by the Actor policy network and calculates the policy gradient of the Actor policy network, which is used to guide the Actor policy network to update information. The Critic value network updates the Critic value network value function based on the next reward information and the discount factor, and sets the shearing probability ratio mechanism to perform policy gradient shearing on the Critic value network and the Actor policy network; according to the Critic value network value function, the Actor policy network policy gradient and the next reward information, the parameters of the Critic value network and the Actor policy network are dynamically updated in combination with the meta-learning algorithm.

[0157] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0158] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A network information age optimization method based on meta-deep reinforcement learning, characterized in that: include: Establishing a video transmission model and collecting current state data, initial reward information, and action information of the video transmission model; The current state data is mapped to the feature space through a meta-feature extraction function to obtain meta-features. The Actor strategy network combines the current strategy with the meta-features to obtain the next action, and the next action is sent back to the video transmission model. The video transmission model provides the next state data and the next reward information, and evaluates the effect of the next action based on the advantage function and the target value. The Actor Policy Network policy gradient is calculated by evaluating the effect of the next action selected by the Actor Policy Network through the Critic Value Network, which is used to guide the Actor Policy Network to update information. The Critic Value Network updates the Critic Value Network value function based on the next reward information and the discount factor, and sets a clipping probability ratio mechanism to perform policy gradient clipping on the Critic Value Network and the Actor Policy Network. Dynamically update the parameters of the Critic value network and the Actor policy network based on the Critic value network value function, the Actor policy network policy gradient, and the next step reward information in combination with the meta-learning algorithm; The reward information is calculated based on a reward function, which includes: Where n is the index of the sensor, k is the time step, delay is the delay of the current video block obtained from the environment, and sensorselection is the currently selected sensor; If the age of a sensor's information exceeds the set threshold, a violation is considered to have occurred. I is the indicator function, A_DIM is the number of sensors, and it returns 1 if the age exceeds the threshold, otherwise it returns 0; tau[n] is the age threshold of the nth sensor; A multi-objective optimization problem is defined. The goal is to minimize the total delay of video transmission, minimize the information age of the sensor, minimize the number of violations, minimize the rebuffering events, minimize the sudden change of video quality and maximize the accuracy of the video. The optimization problem is divided into six sub-problems, which are defined by the following mathematical expression: R total =R delay +R age +R quality +R violation +R rebuffer +R smooth ; Among them, R delay is the delay penalty formula, R age is the information age penalty formula, R quality is the video quality reward formula, R violation is the violation penalty formula, R rebuffer is the heavy buffer penalty formula, R smooth Penalty formula for smooth switching; Weighted sum scalarization: Each objective is multiplied by a weight, and then all weighted objectives are summed: Among them, f i (x) is the i-th objective function, w i is the weight of the i-th objective function, and x is the decision variable.

2. The network information age optimization method based on meta-deep reinforcement learning according to claim 1 is characterized in that: The method of obtaining meta-features specifically includes: inputting a feature X having a shape of N×d, where N is the number of samples and d is the feature dimension, and mapping the input feature to a feature space through a meta-feature extraction function g(·); Z=g(X;θg), where Z is the meta-feature and θg is a parameter of the meta-feature extraction function.

3. The network information age optimization method based on meta-deep reinforcement learning according to claim 1 is characterized in that: The evaluation of the effect of the next action according to the advantage function and the target value specifically includes: δ t =r t +γV(s t+1 )-V(s t ); Among them, δ t is the TD error at time t, r t is the reward, γ is the discount factor, V(s t ) and V(s t+1 ) are the value function predictions of the current state and the next state respectively; A t =d t +(gl)d t+1 +…+(cl) T-t+1 d T-1 ; Among them, the advantage function A t represents the difference between the immediate reward obtained after taking an action and its corresponding state value function at time step t; λ is a parameter used for GAE to balance variance and bias, and T is the last time step in the decision sequence; Use the collected data to calculate the advantage function for each step: A(s,a)=Q(s,a)-V(s t ); Where Q(s,a) is the state-action value function, that is, the expected total reward that can be obtained by taking action a in state s; Calculate the target value Vtarget for each state: Vtarget = r t +γV(s t+1 ).

4. The network information age optimization method based on meta-deep reinforcement learning according to claim 1 is characterized in that: The shear probability ratio mechanism specifically includes: Among them, π θ and denote the current strategy and the old strategy respectively, a is the action selected in state s; The clipping probability ratio mechanism is used to avoid excessive adjustments during the strategy update process, limiting the ratio to fluctuate within the range of [1-∈, 1+∈], where ∈ is the clipping threshold and π θ is the probability of the current strategy, is the probability of the old strategy.

5. The network information age optimization method based on meta-deep reinforcement learning according to claim 4 is characterized in that: It also includes loss calculation for the Critic value network and the Actor strategy network, specifically including: Actor policy network loss: Among them, π θ is the probability of the current strategy, is the probability of the old strategy, A t is the advantage function, ∈ is the clipping threshold; Critic value network loss: Among them, V target =r+γV(s′),V φ (s t ) is the predicted value of the current Critic value network, r is the reward, γ is the discount factor, and V(s′) is the value estimate of the next state.

6. The network information age optimization method based on meta-deep reinforcement learning according to claim 1 is characterized in that: The method for updating the parameters of the Critic value network and the Actor strategy network is a synchronous update method or an asynchronous update method.

7. The network information age optimization method based on meta-deep reinforcement learning according to claim 6 is characterized in that: Before updating the parameters of the Critic value network and the Actor strategy network, normalization processing is also performed, specifically including: The data is recorded as M, M={m1,m2,...,m n }; Calculate the mean of the data Among them, μ is the mean, m i is the i-th data point, N is the total number of data points; Calculate the standard deviation of data Let Y be less than μ-kσ or greater than μ+kσ, and the data point Y={y|y<μ-2σor y>μ+2σ}, where Y is the data in X that meets the conditions, Y={y1,y2,...,y n }, for each data point y i Normalize it to convert it into Among them, z i is the standard deviation of the ith data point.

8. A network information age optimization system based on meta-deep reinforcement learning, applying a network information age optimization method based on meta-deep reinforcement learning according to any one of claims 1 to 7, characterized in that: include: A model building module, which builds a video transmission model and collects current state data, initial reward information, and action information of the video transmission model; An evaluation module maps the current state data to a feature space using a meta-feature extraction function to obtain meta-features. The Actor Strategy Network combines the current strategy with the meta-features to obtain a next action, which is then sent back to the video transmission model. The video transmission model provides the next state data and next reward information, and evaluates the effect of the next action based on the advantage function and the target value. The comprehensive update module uses the Critic value network to evaluate the effect of the next action selected by the Actor policy network and calculates the Actor policy network policy gradient to guide the Actor policy network to update information. The Critic value network updates the Critic value network value function based on the next reward information and the discount factor, and sets a shearing probability ratio mechanism to perform policy gradient shearing on the Critic value network and the Actor policy network; according to the Critic value network value function, the Actor policy network policy gradient and the next reward information, the parameters of the Critic value network and the Actor policy network are dynamically updated in combination with the meta-learning algorithm.

Images