Network information age optimization method and system based on meta-deep reinforcement learning

Abstract

The invention discloses a network information age optimization method and system based on meta-deep reinforcement learning. The method comprises the following steps: establishing a video transmission model, and collecting current state data, initial reward information and action information; the method comprises the following steps: acquiring meta-features through a meta-feature extraction function, acquiring a next-step action by an Actor strategy network in combination with a current strategy and the meta-features, sending the next-step action back to a video transmission model to give next-step state data and reward information, evaluating a next-step action effect selected by the Actor strategy network through a Critic value network, and calculating to obtain an Actor strategy network strategy gradient. Meanwhile, updating a value function of the Critic value network, setting a shearing probability ratio mechanism, and performing strategy gradient shearing on the network; and dynamically updating network parameters in combination with a meta-learning algorithm. According to the invention, the overall utilization rate of network resources is improved, and data transmission failure and retransmission caused by network congestion are reduced.

Description

Technical Field

[0001] The present invention relates to the field of wireless network communication technology, and more specifically to a network information age optimization method and system based on meta-deep reinforcement learning. Background Art

[0002] Industry 4.0 represents a new stage of industrial development, involving the comprehensive application of technologies such as automation, data exchange, cloud computing, the Internet of Things (IoT), and artificial intelligence (AI). This concept was first proposed in Germany in 2011 with the goal of improving the intelligence and flexibility of the manufacturing industry. The core of Industry 4.0 is to create smart factories, in which network systems and physical processes are integrated with each other, devices are interconnected through the Internet of Things, and powerful data processing capabilities are provided through cloud computing. This environment requires network communications to meet extremely high standards in both real-time and reliability to support real-time monitoring and control of production lines and machines.

[0003] In recent years, with the surge in IoT devices and data traffic, traditional network architectures have been unable to meet the strict real-time and reliability requirements of Industry 4.0. Especially in industrial environments, network congestion, electromagnetic interference and other issues are more prominent, which seriously affect the performance of the network and the timeliness of data. In addition, industrial applications have very high requirements for the timeliness of network information, and any delay in information may lead to a decrease in production efficiency or safety accidents. Summary of the invention

[0004] In view of this, the present invention provides a network information age optimization method and system based on meta-deep reinforcement learning to solve the problems existing in the background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A network information age optimization method based on meta-deep reinforcement learning, comprising:

[0007] Establishing a video transmission model, and collecting current state data, initial reward information, and action information of the video transmission model;

[0008] The current state data is mapped to the feature space through the meta-feature extraction function to obtain the meta-feature, and the Actor strategy network combines the current strategy with the meta-feature to obtain the next action, and the next action is sent back to the video transmission model. The video transmission model gives the next state data and the next reward information, and evaluates the effect of the next action according to the advantage function and the target value;

[0009] The policy gradient of the Actor policy network is calculated by evaluating the effect of the next action selected by the Actor policy network through the Critic value network, which is used to guide the Actor policy network to update information. The Critic value network updates the Critic value network value function based on the next reward information and the discount factor, and sets a shearing probability ratio mechanism to perform policy gradient shearing on the Critic value network and the Actor policy network.

[0010] According to the value function of the Critic value network, the policy gradient of the Actor policy network and the next step reward information, the parameters of the Critic value network and the Actor policy network are dynamically updated in combination with the meta-learning algorithm.

[0011] Preferably, the obtaining of meta-features specifically includes: the input feature is X, whose shape is N×d, where N is the number of samples and d is the feature dimension, and the input feature is mapped to the feature space through the meta-feature extraction function g(·); Z=g(X;θg), where Z is the meta-feature and θg is the parameter of the meta-feature extraction function.

[0012] Preferably, the evaluating the effect of the next action according to the advantage function and the target value specifically includes:

[0013] δ t =r t +γV(s t+1 )-V(s t );

[0014] Among them, δ t is the TD error at time t, r t is the reward, γ is the discount factor, V(s t ) and V(s t+1 ) are the value function predictions of the current state and the next state respectively;

[0015] A t =δ t +(γλ)δ t+1 +…+(γλ) T-t+1 δ T-1 ;

[0016] Among them, the advantage function A t represents the difference between the immediate reward obtained after taking an action and its corresponding state value function at time step t; λ is a parameter used for GAE to balance variance and bias, and T is the last time step in the decision sequence;

[0017] Use the collected data to calculate the advantage function for each step: A(s,a)=Q(s,a)-V(s t );

[0018] Among them, Q(s,a) is the state-action value function, that is, the expected total reward that can be obtained by taking action a in state s;

[0019] Calculate the target value Vtarget for each state: Vtarget = r t +γV(s t+1 ).

[0020] Preferably, the reward information is calculated based on a reward function, and the reward function includes:

[0021]

[0022] Where n is the index of the sensor, k is the time step, delay is the delay of the current video block obtained from the environment, and sensor_selection is the currently selected sensor;

[0023] If the age of a sensor's information exceeds a set threshold, a violation is considered to have occurred. I is the indicator function, A_DIM is the number of sensors, returns I if the age exceeds the threshold, otherwise returns 0; tau[n] is the age threshold of the nth sensor;

[0024] Define a multi-objective optimization problem, the goal is to minimize the total delay of video transmission, minimize the information age of the sensor, minimize the number of violations, minimize the rebuffering events, minimize the sudden change of video quality and maximize the accuracy of the video. The optimization problem is divided into six sub-problems, which are defined by the following mathematical expression:

[0025] R total =R delay +R age +R quality +R violation +R rebuffer +R smooth ;

[0026] Among them, R delay is the delay penalty formula, R age is the information age penalty formula, R quality is the video quality reward formula, R violation is the penalty formula for violations, R rebuffer is the heavy buffer penalty formula, R smooth Penalty formula for smooth switching;

[0027] Weighted sum scalarization: Each objective is multiplied by a weight, and then all weighted objectives are summed:

[0028]

[0029] Among them, f i (x) is the i-th objective function, w i is the weight of the i-th objective function, and x is the decision variable.

[0030] Preferably, the shear probability ratio mechanism specifically includes:

[0031]

[0032] Among them, π θ and denote the current strategy and the old strategy respectively, a is the action selected in state s;

[0033] This function avoids excessive adjustments during the policy update process by clipping the probability ratio mechanism, limiting the ratio to vary within the range of [1-,1+], where ∈ is the clipping threshold and π θ is the probability of the current strategy, is the probability of the old strategy.

[0034] Preferably, it also includes calculating the loss of the Critic value network and the Actor strategy network, specifically including:

[0035] Actor strategy network loss:

[0036]

[0037] Among them, π θ is the probability of the current strategy, is the probability of the old strategy, A t is the advantage function, ∈ is the clipping threshold;

[0038] Critic value network loss:

[0039]

[0040] Among them, V target =r+γV(s′),V φ (s t ) is the predicted value of the current Critic value network, r is the reward, γ is the discount factor, and V(s′) is the value estimate of the next state.

[0041] Preferably, the method for updating the parameters of the Critic value network and the Actor strategy network is a synchronous update method or an asynchronous update method.

[0042] Preferably, before updating the parameters of the Critic value network and the Actor strategy network, a normalization process is further performed, specifically including:

[0043] Let the data be M, M={m1,m2,...,m n};

[0044] Calculate the mean of the data Among them, μ is the mean, m i is the i-th data point, and N is the total number of data points;

[0045] Calculate the standard deviation of the data

[0046] Let Y be less than μ-kσ or greater than μ+kσ. Let Y={y|y<μ-2σor y>μ+2σ}, where Y is the data in X that meets the conditions, Y={y1,y2,...,y n}, for each data point y i Normalize it to convert it into Among them, z i is the standard deviation of the ith data point.

[0047] A network information age optimization system based on meta-deep reinforcement learning, comprising:

[0048] A model building module, which builds a video transmission model and collects current state data, initial reward information and action information of the video transmission model;

[0049] An evaluation module maps the current state data to a feature space through a meta-feature extraction function to obtain meta-features. The Actor strategy network combines the current strategy with the meta-features to obtain a next action, and sends the next action back to the video transmission model. The video transmission model gives the next state data and the next reward information, and evaluates the effect of the next action according to the advantage function and the target value.

[0050] The comprehensive update module evaluates the effect of the next action selected by the Actor policy network through the Critic value network and calculates the Actor policy network policy gradient, which is used to guide the Actor policy network to update information. The Critic value network updates the Critic value network value function based on the next reward information and the discount factor, and sets a shearing probability ratio mechanism to perform policy gradient shearing on the Critic value network and the Actor policy network; according to the Critic value network value function, the Actor policy network policy gradient and the next reward information, the parameters of the Critic value network and the Actor policy network are dynamically updated in combination with the meta-learning algorithm.

[0051] It can be seen from the above technical solutions that compared with the prior art, the present invention discloses a network information age optimization method and system based on meta-deep reinforcement learning, which significantly improves the efficiency and quality of video transmission in the industrial Internet environment. Through intelligent routing selection and system optimization, it effectively reduces delays and improves the processing accuracy of video data, meeting the high standards of Industry 4.0 for real-time and reliability. Integrating meta-learning into each link can effectively improve the adaptability and stability of the video transmission model in different network environments, ensuring that the model can run efficiently in a real industrial IoT environment and can quickly adapt to new tasks and environments. This method improves the overall utilization of network resources and reduces the need for data transmission failures and retransmissions due to network congestion. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0053] Figure 1 A flow chart of the method provided by the present invention;

[0054] Figure 2 A schematic diagram of the structure provided by the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The embodiment of the present invention discloses a network information age optimization method based on meta-deep reinforcement learning, characterized in that: Figure 1 As shown, including:

[0057] Establish a video transmission model and collect the current state data, initial reward information and action information of the video transmission model;

[0058] The current state data is mapped to the feature space through the meta-feature extraction function to obtain the meta-feature. The Actor strategy network combines the current strategy and the meta-feature to obtain the next action, and sends the next action back to the video transmission model. The video transmission model gives the next state data and the next reward information, and evaluates the effect of the next action based on the advantage function and the target value.

[0059] The policy gradient of the Actor policy network is calculated by evaluating the effect of the next action selected by the Actor policy network through the Critic value network, which is used to guide the Actor policy network to update information. The Critic value network updates the Critic value network value function based on the next reward information and the discount factor, and sets the shear probability ratio mechanism to perform policy gradient shear on the Critic value network and the Actor policy network.

[0060] According to the value function of the Critic value network, the policy gradient of the Actor policy network and the next reward information, the parameters of the Critic value network and the Actor policy network are dynamically updated in combination with the meta-learning algorithm.

[0061] Reward information refers to the immediate return fed back by the environment at each time step, but its uses are different. The Critic network uses reward information to update the value function, which helps the Critic network accurately evaluate the value of states and actions.

[0062] The Actor network uses reward information to calculate policy gradients, which helps the Actor network optimize the strategy so that it can obtain the Actor policy network in the future state: the input layer accepts the state of the environment, passes through two layers of 128-unit fully connected layers (each layer has ReLU activation function), and the output layer generates the probability distribution of the action through the Softmax activation function (for discrete action space) or the linear activation function to generate the specific action value (for continuous action space).

[0063] Critic value network: The input layer accepts the state of the environment, passes through two layers of 128 units of fully connected layers (each layer has ReLU activation function), and the output layer generates the value evaluation of the state through a linear activation function. (Get higher cumulative returns)

[0064] In a specific embodiment, meta-learning is a method for improving the adaptability and generalization ability of a model. The present invention uses a meta-learning algorithm in meta-feature extraction, meta-strategy adjustment, and meta-optimization, respectively. Obtaining meta-features specifically includes: the input feature is X, and its shape is N×d, where N is the number of samples and d is the feature dimension. The input feature is mapped to the feature space through a meta-feature extraction function g(·); Z=g(X;θg), where Z is the meta-feature and θg is the parameter of the meta-feature extraction function.

[0065] Meta-policy adjustment: In the Actor policy network and Critic value network, meta-learning methods are used to dynamically adjust the policy to quickly adapt to different network conditions.

[0066] Meta-optimization: During the training process, a meta-learning optimization algorithm is used to dynamically adjust the learning rate and parameter update strategy to improve the convergence speed and stability of the model.

[0067] In a specific embodiment, the temporal difference (TD) error is used to measure the difference between the actual reward and the predicted reward after the policy is executed. This error reflects the accuracy of the current policy's estimate of the value of the environment state. In PPO, the TD error is used to calculate the advantage function, which helps evaluate the relative advantage of taking a specific action in a specific state. Through the use of the advantage function, PPO can effectively update the strategy and promote the learning process to a better solution. The effect of the next action evaluated based on the advantage function and the target value specifically includes:

[0068] δ t =r t +γV(s t+1 )-V(s t );

[0069] Among them, δ t is the TD error at time t, r t is the reward, γ is the discount factor, V(s t ) and V(s t+1 ) are the value function predictions of the current state and the next state respectively;

[0070] A t =δ t +(γλ)δ t+1 +…+(γλ) T-t+1 δ T-1 ;

[0071] Among them, the advantage function A t represents the difference between the immediate reward obtained after taking an action and its corresponding state value function at time step t; λ is a parameter used for GAE to balance variance and bias, and T is the last time step in the decision sequence;

[0072] Use the collected data to calculate the advantage function for each step: A(s,a)=Q(s,a)-V(s t );

[0073] Among them, Q(s,a) is the state-action value function, that is, the expected total reward that can be obtained by taking action a in state s;

[0074] Calculate the target value Vtarget for each state: Vtarget = r t +γV(s t+1 ).

[0075] In a specific embodiment, the reward information is calculated based on a reward function, and the reward function is designed to improve the timeliness of information and optimize network performance. The reward information is calculated based on the reward function, and the reward function includes:

[0076]

[0077] Where n is the index of the sensor, k is the time step, delay is the delay of the current video block obtained from the environment, and sensor_selection is the currently selected sensor;

[0078] If the age of a sensor's information exceeds a set threshold, a violation is considered to have occurred. I is the indicator function, A_DIM is the number of sensors, returns I if the age exceeds the threshold, otherwise returns 0; tau[n] is the age threshold of the nth sensor;

[0079] Define a multi-objective optimization problem, the goal is to minimize the total delay of video transmission, minimize the information age of the sensor, minimize the number of violations, minimize the rebuffering events, minimize the sudden change of video quality and maximize the accuracy of the video. The optimization problem is divided into six sub-problems, which are defined by the following mathematical expression:

[0080] R total =R delay +R age +R quality +R violation +R rebuffer +R smooth ;

[0081] Among them, R delay is the delay penalty formula, R age is the information age penalty formula, R quality is the video quality reward formula, R violation is the penalty formula for violations, R rebuffer is the heavy buffer penalty formula, R smooth Penalty formula for smooth switching;

[0082] Delay penalty formula: Rdelay = -w1·delay; w1 is the weight coefficient of delay, and delay is the delay time of the video block obtained from the environment.

[0083] Information age penalty formula: R age =-w2·total_age; w2 is the weight coefficient of information age, and total_age is the sum of the current information ages of all sensors.

[0084] Video quality reward formula: R quality=w3·video_quality_reward; w3 is the weight coefficient of video quality, video_quality_reward is the reward value obtained from the HD_REWARD array according to the currently selected video bit rate. HD_REWARD = [1,2,3,12,15,20] # High-definition video reward.

[0085] Violation penalty formula: R violation = -λ·current_violation; λ is the penalty coefficient for violation, and current_violation represents the number of sensors that exceed the threshold in the current time step.

[0086] Rebuffer penalty formula: R rebuffer =-rebuf_penalty·sleep_time; rebuf_penalty is the penalty coefficient for rebuffering, and sleep_time is the delay time due to waiting for buffering.

[0087] Smooth switching penalty formula:

[0088]

[0089] smooth_penalty is the penalty coefficient for smooth switching, which is used to reduce sudden changes in video quality.

[0090] In multi-objective optimization, different objectives may conflict with each other. For example, improving video quality (increasing bit rate) may increase transmission delays and increase the possibility of rebuffering, while minimizing delays and rebuffering may limit the highest video quality you can choose. Scalarization techniques attempt to synthesize multiple optimization objectives into a single scalar objective function that can be solved using standard optimization methods. For applications with clear priorities and requiring fast responses, such as certain manufacturing and automation processes, scalarization techniques may be more applicable because they can quickly provide decision support and simplify control logic. It is reasonable to use information age as the most important objective in the application field of the present invention, while constraining the remaining objectives, that is, maximizing the weight of information age.

[0091] Each objective is multiplied by a weight, and then all weighted objectives are summed. The weights reflect the relative importance of different objectives. Weighted sum scalarization: Each objective is multiplied by a weight, and then all weighted objectives (six sub-problems) are summed:

[0092]

[0093] Among them, f i (x) is the i-th objective function, w iis the weight of the i-th objective function, and x is the decision variable.

[0094] In a specific embodiment, the shear probability ratio mechanism specifically includes:

[0095]

[0096] Among them, π θ and denote the current strategy and the old strategy respectively, a is the action selected in state s;

[0097] The clipping probability ratio mechanism is a technique used to avoid excessive policy updates during policy gradient optimization. Specifically, the mechanism clips the policy update ratio to limit the range of the ratio between the new policy and the old policy to avoid excessive adjustments during the policy update process, which can lead to model instability. This function avoids excessive adjustments during the policy update process by clipping the probability ratio mechanism, limiting the ratio to a range of [1-,1+], where ∈ is the clipping threshold and π θ is the probability of the current strategy, is the probability of the old policy. Use clipped probability ratios to avoid making too large adjustments during policy updates. The PPO algorithm can reduce policy update fluctuations caused by a single or a few samples by limiting the probability ratio within a certain range, because these extreme samples may have too large an impact on the gradient without clipping. By clipping the impact of these extreme probability ratios, PPO helps reduce the variance in the learning process and improves the stability of the overall policy.

[0098] In a specific embodiment, the loss calculation of the Critic value network and the Actor strategy network is also included, specifically including:

[0099] Actor Policy Network Loss: PPO uses a clipping probability ratio mechanism to avoid excessive adjustments during policy updates:

[0100]

[0101] Among them, π θ is the probability of the current strategy, is the probability of the old strategy, A t is the advantage function, ∈ is the clipping threshold;

[0102] Critic Value Network Loss: The goal of the value function is to estimate the value of the state, usually by minimizing the mean squared error between the predicted value and the actual return:

[0103]

[0104] Among them, V target=r+γV(s′),V φ (s t ) is the predicted value of the current Critic value network, r is the reward, γ is the discount factor, and V(s′) is the value estimate of the next state.

[0105] It also includes an entropy term (used to encourage strategy exploration). The entropy reward term is used to encourage strategy exploration and maintain strategy diversity. Specifically:

[0106] H(π θ )=E t [entropy(π θ (a t |s t ))];

[0107] Total loss function: L(θ,φ)=L CLIP (θ)+c1L VF (φ)-c2H(π θ ), where c1 and c2 are weight hyperparameters that control the impact of value function loss and entropy reward, respectively.

[0108] In a specific embodiment, the method for updating the parameters of the Critic value network and the Actor strategy network is a synchronous update method or an asynchronous update method.

[0109] Gradient descent is the main method for optimizing model parameters. During the learning process, the parameters of the Critic value network and the Actor policy network are optimized by calculating the gradient of the loss function and updating the parameters. These updates can be synchronous or asynchronous:

[0110] Synchronous Update: All parameters are updated together after each time step.

[0111] Asynchronous updates: Parameters of different parts can be updated independently at different times. Asynchronous updates are a way to improve training efficiency and stability. In asynchronous reinforcement learning algorithms, multiple agents learn in parallel, each interacting with the environment, independently updating their own policies and value networks, and then asynchronously applying these updates to the global model. This approach can speed up the learning process while reducing the risk of overfitting through diverse experiences.

[0112] The parameters of the Actor policy network and the Critic value network are updated in a single computing environment. In the traditional PPO algorithm, the parameters of the Actor and Critic networks are updated in a single computing environment. These network parameters are stored and updated to the same global network after each iteration, ensuring that the learning results obtained from the experimental data can be effectively applied to the iterative optimization of the strategy. The introduction of the meta-learning method can combine the grouped delayed update mechanism with meta-learning. Parameter updates are not performed immediately after each new data is collected, but after sufficient data is collected, and the sufficient data is divided into different groups for random sampling. This ensures that the update is based on reliable and stable data trends, while being conducive to exploration.

[0113] In meta-learning, adaptive learning rate management is an important part. The learning rate is dynamically adjusted according to the performance of the model during training: when the model performance improves, the learning rate is reduced to fine-tune the model parameters; when the model performance stagnates or deteriorates, the learning rate is increased to jump out of the possible local optimum.

[0114] Specific adaptive learning rate adjustment formula:

[0115] Among them, η t is the learning rate of the tth iteration, η0 is the initial learning rate, τ is the time constant, L i is the loss function for the i-th iteration. is the loss function L of the i-th iteration i The gradient of the parameter θ is is the loss function L of the i-1th iteration i-1 The gradient of the parameter θ.

[0116] In asynchronous updates, each parallel worker node autonomously trains its own Actor and Critic network, and asynchronously transmits the learned updates to the global network. The global network integrates the learning results of all nodes and updates the global model parameters. The asynchronous method allows multiple agents to execute in parallel in different copies of the environment. These agents do not need to wait for other agents to complete their episodes, and can independently update the global network based on their collected data, which improves the efficiency of sample utilization, reduces the variance of updates, and improves the stability of the learning process. In the asynchronous version of PPO, larger batches and longer update cycles are preferred, which helps to utilize more stable policy gradient estimates.

[0117] Forward propagation and back propagation are used to update the Actor policy network and Critic value network models, calculate losses, and optimize network parameters through the gradient descent algorithm to improve the accuracy of the model's policy evaluation and decision-making. The selection is based on actual conditions.

[0118] In a specific embodiment, the neural network input data is normalized by standard deviation Z-score. Ordinary normalization is not suitable for data with a large span or abnormal data. The purpose of normalization is to process state (s_batch) and reward (r_batch) data, as well as action probabilities. Before updating the parameters of the Critic value network and the Actor strategy network, normalization is also performed, specifically including:

[0119] Let the data be M, M={m1,m2,...,m n};

[0120] Calculate the mean of the data Among them, μ is the mean, m i is the i-th data point, and N is the total number of data points;

[0121] Calculate the standard deviation of the data

[0122] Let Y be less than μ-kσ or greater than μ+kσ. Let Y={y|y<μ-2σor y>μ+2σ}, where Y is the data in X that meets the conditions, Y={y1,y2,...,y n}, for each data point y i Normalize it to convert it into Among them, z i is the standard deviation of the ith data point. It can effectively reduce the impact of noise and outliers in the data while ensuring the consistency and standardization of the data, thereby improving the efficiency of data processing and the accuracy of the model.

[0123] In specific embodiment 1, it specifically includes:

[0124] Build a network simulation environment: For industrial IoT scenarios, use the ns3 simulation tool to build a multi-node video transmission model involving routers, servers, edge servers, and multiple network nodes, taking into account the complex transmission paths between nodes.

[0125] Real-time collection of network status data: In a simulated network environment, real-time monitoring of network performance, including end-to-end delay, throughput t, bandwidth B of each link, and packet loss rate P.

[0126] Customized link packet loss model: Design and implement a customized packet loss model to address network congestion and electromagnetic interference in industrial scenarios, and simulate data packet loss in a specific network environment.

[0127] Considering factors such as network congestion and electromagnetic interference in industrial scenarios, a customized link packet loss model is used to simulate packet loss in the network. This customized model will decide whether to lose packets based on the current time, simulated electromagnetic interference, and packet size (all packets larger than K bytes will be discarded, and the threshold K can be adjusted according to actual needs to simulate specific network environments and conditions).

[0128] Electromagnetic Interference (EMI):

[0129] Use a uniformly distributed random variable to simulate occasional electromagnetic interference, which can occasionally cause packet loss.

[0130] Assume that X is a uniformly distributed random variable, X~U(0,1), indicating whether electromagnetic interference will affect the transmission of data packets at any given time point. We define a threshold θ, assuming θ = 0.05, which means that there is a 5% probability that electromagnetic interference will significantly affect the network. When X≤θ, electromagnetic interference will cause the probability of packet loss to increase∈, otherwise, electromagnetic interference will not affect the packet loss rate, that is, the increased packet loss rate is 0.

[0131]

[0132] Network congestion:

[0133] Determine whether it is the network peak period based on the current time. If so, increase the packet loss probability to simulate the situation where the network is more likely to lose packets during high traffic periods.

[0134] Packet size:

[0135] If the packet size exceeds a certain threshold K, the possibility of packet loss increases, simulating that large packets are more susceptible to impact when the network environment is poor or the load is high. Assume that the threshold K = 1000

[0136] Logic: If the size of the packet C(p) exceeds 1000 bytes, then this function returns true, indicating that the packet should be discarded.

[0137]

[0138] Among them, C(p) is an indicator function, which takes the value of 1 (packet loss) when the size of p exceeds 1000 bytes, otherwise it takes the value of 0 (no packet loss).

[0139] P total =P base +P EMI +P otherfactors ;

[0140] The total packet loss probability P total is the basic packet loss probability P base(Pc) plus the increased packet loss probability P caused by electromagnetic interference EMI , and the packet loss probability P increased by network congestion otherfactors .

[0141] Introducing randomness to simulate network fluctuations: Use normally distributed random variables to simulate randomness in data transmission, simulate network fluctuations by dynamically adjusting the sending interval or size of data packets, and introduce randomness in the delay and data rate of data transmission to indirectly simulate network fluctuations.

[0142] normal distribution:

[0143] μ is the mean, which indicates the center of the distribution. σ is the standard deviation, which indicates the width of the distribution and the degree of dispersion of the data.

[0144] In the above network simulation environment, the time T required for download is calculated according to the current network status and video quality, and the network status is returned. If the packet load plus the data sent exceeds the size of the video segment, the time required to download the remaining part is calculated. The delay time, waiting time and video segment size of the downloaded video segment are returned.

[0145] Network information processing and conversion: Process input vectors and state data, apply fully connected layers, convolutional layers, and coordinate attention mechanisms, and optimize input data processing of the Actor strategy network and the Critic value network.

[0146] Convert the information form acceptable to the neural network model into a feature vector X that the neural network model can accept i (i=x1,x2,x3).

[0147] Network strategy and value assessment: Critic value network and Actor policy network are established to evaluate the value of strategy and make network scheduling decisions respectively. Critic value network evaluates the value of current strategy and helps predict future rewards. Actor policy network generates specific network scheduling decisions, such as priority allocation and bandwidth allocation of data packets. Critic value network is constructed to evaluate the value of current strategy. This network takes the input state vector, processes it with convolution and fully connected layers, and outputs the value estimate of the state.

[0148] The learning process of optimization strategy:

[0149] The PPO algorithm is used to optimize the learning process through the temporal difference error and clipping probability ratio mechanism. The meta-learning mechanism is introduced to train multiple sub-models in different simulation environments, and then the knowledge of these sub-models is extracted and integrated through the meta-learning framework to improve the accuracy and stability of strategy execution.

[0150] Reward function design: Design a reward function with the goal of improving information timeliness and optimizing network performance, and focus on dealing with complex network scheduling problems through multi-objective optimization.

[0151] Adjustment of learning rate and update mechanism: Implement adaptive learning rate management and grouped delayed update mechanism to optimize the update process of network parameters and improve the learning efficiency and stability of the model.

[0152] Data standardization: Implement Z-score normalization to reduce the impact of noise and outliers in the data and improve the efficiency of data processing and the accuracy of the model.

[0153] Asynchronous update mechanism: Asynchronous training is performed in multiple parallel working nodes to improve sample utilization efficiency and the stability of the learning process.

[0154] A network information age optimization system based on meta-deep reinforcement learning, such as Figure 2 As shown, including:

[0155] Model building module, building a video transmission model, collecting the current state data, initial reward information and action information of the video transmission model;

[0156] The evaluation module maps the current state data to the feature space through the meta-feature extraction function to obtain the meta-feature. The Actor strategy network combines the current strategy and the meta-feature to obtain the next action and sends the next action back to the video transmission model. The video transmission model gives the next state data and the next reward information, and evaluates the effect of the next action according to the advantage function and the target value.

[0157] The comprehensive update module evaluates the effect of the next action selected by the Actor policy network through the Critic value network and calculates the policy gradient of the Actor policy network, which is used to guide the Actor policy network to update information. The Critic value network updates the Critic value network value function based on the next reward information and the discount factor, and sets the clipping probability ratio mechanism to perform policy gradient clipping on the Critic value network and the Actor policy network; according to the Critic value network value function, the Actor policy network policy gradient and the next reward information, the parameters of the Critic value network and the Actor policy network are dynamically updated in combination with the meta-learning algorithm.

[0158] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0159] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A network information age optimization method based on meta-deep reinforcement learning, characterized in that: include: Establishing a video transmission model, and collecting current state data, initial reward information, and action information of the video transmission model; The current state data is mapped to the feature space through the meta-feature extraction function to obtain the meta-feature, and the Actor strategy network combines the current strategy with the meta-feature to obtain the next action, and the next action is sent back to the video transmission model. The video transmission model gives the next state data and the next reward information, and evaluates the effect of the next action according to the advantage function and the target value; The policy gradient of the Actor policy network is calculated by evaluating the effect of the next action selected by the Actor policy network through the Critic value network, which is used to guide the Actor policy network to update information. The Critic value network updates the Critic value network value function based on the next reward information and the discount factor, and sets a shearing probability ratio mechanism to perform policy gradient shearing on the Critic value network and the Actor policy network. According to the value function of the Critic value network, the policy gradient of the Actor policy network and the next step reward information, the parameters of the Critic value network and the Actor policy network are dynamically updated in combination with the meta-learning algorithm.

2. According to the method for optimizing network information age based on meta-deep reinforcement learning in claim 1, it is characterized in that: The method of obtaining meta-features specifically includes: inputting a feature X whose shape is N×d, where N is the number of samples and d is the feature dimension, and mapping the input feature to a feature space through a meta-feature extraction function g(·); Z=g(X;θg), where Z is a meta-feature and θg is a parameter of the meta-feature extraction function.

3. According to the method for optimizing network information age based on meta-deep reinforcement learning in claim 1, it is characterized in that: The effect of evaluating the next action according to the advantage function and the target value specifically includes: δ t =r t +γV(s t+1 )-V(s t ); Among them, δ t is the TD error at time t, r t is the reward, γ is the discount factor, V(s t ) and V(s t+1 ) are the value function predictions of the current state and the next state respectively; A t =d t +(gl)d t+1 +…+(cl) T-t+1 d T-1 ; Among them, the advantage function A t represents the difference between the immediate reward obtained after taking an action and its corresponding state value function at time step t; λ is a parameter used for GAE to balance variance and bias, and T is the last time step in the decision sequence; Use the collected data to calculate the advantage function for each step: A(s,a)=Q(s,a)-V(s t ); Among them, Q(s,a) is the state-action value function, that is, the expected total reward that can be obtained by taking action a in state s; Calculate the target value Vtarget for each state: Vtarget = r t +γV(s t+1 ).

4. According to the method for optimizing network information age based on meta-deep reinforcement learning in claim 1, it is characterized in that: The reward information is calculated based on a reward function, and the reward function includes: Where n is the index of the sensor, k is the time step, delay is the delay of the current video block obtained from the environment, and sensor_selection is the currently selected sensor; If the age of a sensor's information exceeds a set threshold, a violation is considered to have occurred. I is the indicator function, A_DIM is the number of sensors, returns I if the age exceeds the threshold, otherwise returns 0; tau[n] is the age threshold of the nth sensor; Define a multi-objective optimization problem, the goal is to minimize the total delay of video transmission, minimize the information age of the sensor, minimize the number of violations, minimize the rebuffering events, minimize the sudden change of video quality and maximize the accuracy of the video. The optimization problem is divided into six sub-problems, which are defined by the following mathematical expression: R total =R delay +R age +R quality +R violation +R rebuffer +R smooth ; Among them, R delay is the delay penalty formula, R age is the information age penalty formula, R quality is the video quality reward formula, R violation is the violation penalty formula, R rebuffer is the heavy buffer penalty formula, R smooth To smoothly switch the penalty formula; Weighted sum scalarization: Each objective is multiplied by a weight, and then all weighted objectives are summed: Among them, f i (x) is the i-th objective function, w i is the weight of the i-th objective function, and x is the decision variable.

5. According to the method of optimizing network information age based on meta-deep reinforcement learning in claim 1, it is characterized in that: The shear probability ratio mechanism specifically includes: Among them, π θ and denote the current strategy and the old strategy respectively, a is the action selected in state s; The clipping probability ratio mechanism is used to avoid excessive adjustments during the strategy update process and limit the ratio to the range of [1-,1+], where ∈ is the clipping threshold and π θ is the probability of the current strategy, is the probability of the old strategy.

6. The network information age optimization method based on meta-deep reinforcement learning according to claim 5 is characterized in that: It also includes loss calculation for the Critic value network and the Actor strategy network, specifically including: Actor strategy network loss: Among them, π θ is the probability of the current strategy, is the probability of the old strategy, A t is the advantage function, ∈ is the clipping threshold; Critic value network loss: Among them, V target =r+γV(s′),V φ (s t ) is the predicted value of the current Critic value network, r is the reward, γ is the discount factor, and V(s′) is the value estimate of the next state.

7. The network information age optimization method based on meta-deep reinforcement learning according to claim 1 is characterized in that: The method for updating the parameters of the Critic value network and the Actor strategy network is a synchronous update method or an asynchronous update method.

8. The network information age optimization method based on meta-deep reinforcement learning according to claim 7 is characterized in that: Before updating the parameters of the Critic value network and the Actor strategy network, a normalization process is also performed, specifically including: Let the data be M, M={m1,m2,...,m n }; Calculate the mean of the data Among them, μ is the mean, m i is the i-th data point, and N is the total number of data points; Calculate the standard deviation of the data Let Y be less than μ-kσ or greater than μ+kσ. Let Y={y|y<μ-2σor y>μ+2σ}, where Y is the data in X that meets the conditions, Y={y1,y2,...,y n }, for each data point y i Normalize it to convert it into Among them, z i is the standard deviation of the ith data point.

9. A network information age optimization system based on meta-deep reinforcement learning, applying a network information age optimization method based on meta-deep reinforcement learning as described in any one of claims 1-8, characterized in that: include: A model building module, which builds a video transmission model and collects current state data, initial reward information and action information of the video transmission model; An evaluation module maps the current state data to a feature space through a meta-feature extraction function to obtain meta-features. The Actor strategy network combines the current strategy with the meta-features to obtain a next action, and sends the next action back to the video transmission model. The video transmission model gives the next state data and the next reward information, and evaluates the effect of the next action according to the advantage function and the target value. The comprehensive update module evaluates the effect of the next action selected by the Actor policy network through the Critic value network and calculates the Actor policy network policy gradient, which is used to guide the Actor policy network to update information. The Critic value network updates the Critic value network value function based on the next reward information and the discount factor, and sets a shearing probability ratio mechanism to perform policy gradient shearing on the Critic value network and the Actor policy network; according to the Critic value network value function, the Actor policy network policy gradient and the next reward information, the parameters of the Critic value network and the Actor policy network are dynamically updated in combination with the meta-learning algorithm.

Images