An adaptive network offloading method and system based on deep reinforcement learning

Abstract

The present invention relates to the field of adaptive network traffic diversion technology, and discloses an adaptive network traffic diversion method and system based on deep reinforcement learning, comprising: obtaining traffic data of heterogeneous networks, establishing a protocol fingerprint library, and converting data of different protocols into a unified network state feature vector; constructing a multi-agent reinforcement learning framework, including defining a state space, an action space, and a reward function, and optimizing the strategy of each agent based on differential games; using a meta-learning model to adjust the weight of each agent's reward function, and using deep reinforcement learning to train the multi-agent reinforcement learning framework to learn the optimal diversion strategy under different heterogeneous network conditions; deploying the trained strategy to a heterogeneous network environment, and dynamically selecting the optimal network path based on the current network state. This significantly improves the heterogeneous data processing capability.

Description

Technical Field

[0001] The present invention relates to the field of adaptive network traffic diversion technology, and specifically to an adaptive network traffic diversion method and system based on deep reinforcement learning. Background Art

[0002] In recent years, multi-agent reinforcement learning (MARL) and meta-learning techniques have provided new approaches for solving complex network optimization problems, but practical deployment still faces significant challenges. While existing MARL solutions mitigate single-agent policy conflicts through distributed decision-making, they lack quantitative modeling of inter-agent game relationships, resulting in inefficient Nash equilibrium solutions. For example, in scenarios involving traffic control and security agent collaboration, traditional collaborative training methods fail to account for policy coupling effects, resulting in convergence times exceeding single-agent solutions by more than three times. At the protocol processing level, existing research attempts to extract cross-protocol commonalities (such as latency-sensitive service identifiers) through feature engineering, but manually designed features struggle to capture dynamic protocol interaction patterns. Furthermore, existing dynamic weight adjustment methods rely on manually set thresholds and struggle to adapt to nonlinear changes in the network environment. In edge computing scenarios, the high computational complexity of deep reinforcement learning models conflicts sharply with the resource constraints of edge devices, forcing existing solutions to compromise between model accuracy and deployment costs, resulting in reduced performance of traffic diversion strategies. At the same time, the dynamic topology of heterogeneous networks (such as the intermittent connectivity of low-orbit satellite nodes) places higher demands on the rapid migration of policies. Traditional methods require retraining models for each topology, which can take weeks. These technical bottlenecks severely restrict the practical application of adaptive traffic diversion systems. An innovative solution is urgently needed that can uniformly characterize multi-protocol features, dynamically coordinate conflicts among multiple objectives, and adapt to edge deployment. Summary of the Invention

[0003] In view of the above-mentioned problems, the present invention is proposed.

[0004] Therefore, the technical problem solved by the present invention is: the technical problems of low network diversion efficiency, large fluctuations in service quality and lagging security protection caused by multi-protocol differences, multi-target strategy conflicts and insufficient adaptability to dynamic environments in the existing heterogeneous network environment.

[0005] To solve the above technical problems, the present invention provides the following technical solutions: an adaptive network offloading method based on deep reinforcement learning, comprising:

[0006] Obtain traffic data from heterogeneous networks, build a protocol fingerprint library, and convert data from different protocols into a unified network status feature vector;

[0007] Build a multi-agent reinforcement learning framework, including defining the state space, action space, and reward function, and optimize the strategies of each agent based on differential games;

[0008] A meta-learning model is used to adjust the weights of each agent's reward function, and deep reinforcement learning is used to train a multi-agent reinforcement learning framework to learn the optimal diversion strategy under different heterogeneous network conditions;

[0009] The trained strategy is deployed to a heterogeneous network environment and the optimal network path is dynamically selected based on the current network status.

[0010] As a preferred solution of the adaptive network offloading method based on deep reinforcement learning described in the present invention, the traffic data of the heterogeneous network includes: protocol type identification, message header information, timestamp, timing mode, signal strength and quality indicators, bandwidth utilization, packet loss rate, data packet length, traffic size, traffic pattern, abnormal traffic pattern and authentication identification;

[0011] Establishing a protocol fingerprint library includes preprocessing the acquired traffic data, extracting key features from the preprocessed traffic data, establishing a binary coding template, encoding the key features, obtaining the protocol fingerprint, and storing it in the protocol fingerprint library in an index structure;

[0012] The key features include data features and multi-agent network state features; the multi-agent network state features include flow control agent state features, QoS management agent state features and security agent state features;

[0013] Converting data of different protocols into a unified network status feature vector includes: converting the protocol fingerprint into a numerical vector, normalizing the continuous values ​​in the numerical vector, and concatenating all normalized continuous values ​​into the network status feature vector.

[0014] As a preferred solution of the adaptive network offloading method based on deep reinforcement learning described in the present invention, wherein: the multi-agent reinforcement learning framework includes a traffic control agent, a QoS management agent and a security agent;

[0015] Defining the state space is to define the observable variables of each agent based on the state characteristics of the multi-agent network to obtain the state space;

[0016] The state space includes: the state space of the flow control agent, the state space of the QoS management agent and the state space of the security agent; the defined action space includes: the action space of the flow control agent, the action space of the QoS management agent and the action space of the security agent;

[0017] The reward functions defined include: reward function of traffic control agent, reward function of QoS management agent and reward function of security agent;

[0018] The reward function of the traffic control agent is to maximize network throughput and minimize load imbalance; the reward function of the QoS management agent is to ensure that the delay, packet loss rate and jitter of the service flow are maintained within the optimal range; the reward function of the security agent is to minimize the proportion of abnormal traffic while reducing false alarms.

[0019] As a preferred solution of the adaptive network offloading method based on deep reinforcement learning described in the present invention, wherein: optimizing the decision strategy of each agent based on differential game includes constructing the state equation of the differential game based on the state space of the multi-agent reinforcement learning framework to describe the dynamic process of the network state changing over time;

[0020] Solve the Nash equilibrium of the differential game to ensure that the overall strategy of the multi-agent reinforcement learning framework remains optimal when the network state is constantly changing.

[0021] As a preferred solution of the adaptive network diversion method based on deep reinforcement learning described in the present invention, the following steps are used: using a meta-learning model to adjust the weight of each agent's reward function, including analyzing the network state of the past N time steps based on the attention mechanism, calculating the weight of each agent's network state characteristics, and obtaining each agent's state characteristic weight vector; using a weighted summation method, and combining the state characteristic weight vector of each agent, dynamically adjusting the weight of each agent's reward function.

[0022] As a preferred solution of the adaptive network diversion method based on deep reinforcement learning described in the present invention, the method of training a multi-agent reinforcement learning framework using deep reinforcement learning includes constructing a deep reinforcement learning strategy network, setting an exploration and utilization mechanism, initializing an experience pool, and training a multi-agent reinforcement learning framework.

[0023] As a preferred solution of the adaptive network offloading method based on deep reinforcement learning described in the present invention, wherein: constructing a deep reinforcement learning strategy network includes: an input layer, a hidden layer and an output layer;

[0024] The input layer includes inputting the state space into the input layer to obtain an input vector;

[0025] The hidden layer includes a fully connected layer, a feature mapping layer and a strategy generation layer;

[0026] The feature mapping layer sets attention scores, calculates the weights of the internal features of each agent's state space, and performs weighted summation to obtain the final state feature vector of each agent; establishes differential game equations to describe the impact of different agent strategies on other agents; solves Nash equilibrium to ensure that multi-agent strategies remain optimal in a dynamic environment, and obtains the game impact between agents;

[0027] Combine the final state feature vector of each agent and the game influence between agents to generate the final strategy input vector of each agent;

[0028] The strategy generation layer includes calculating the Q value of each agent's different actions using the agent's final strategy input vector and action space, selecting the action with the largest Q value as the optimal strategy for each agent, and updating the Q value using the Q-Learning method.

[0029] The output layer includes outputting the optimal strategy of each agent.

[0030] An adaptive network traffic diversion system based on deep reinforcement learning, wherein:

[0031] The protocol fingerprint library module obtains traffic data of heterogeneous networks, establishes a protocol fingerprint library, and converts data from different protocols into a unified network status feature vector;

[0032] Multi-agent reinforcement learning module, which builds a multi-agent reinforcement learning framework, including defining the state space, action space, and reward function, and optimizing the strategies of each agent based on differential games;

[0033] The strategy module uses a meta-learning model to adjust the weights of each agent's reward function and uses deep reinforcement learning to train a multi-agent reinforcement learning framework to learn the optimal diversion strategy under different heterogeneous network conditions;

[0034] The deployment module deploys the trained strategy to a heterogeneous network environment and dynamically selects the optimal network path based on the current network status.

[0035] A computer device comprises: a memory and a processor; the memory stores a computer program, wherein the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0036] A computer-readable storage medium stores a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the methods of the present invention.

[0037] Beneficial effects of the present invention: The adaptive network diversion method based on deep reinforcement learning provided by the present invention realizes efficient parsing and dynamic adaptation across network protocols by constructing a protocol fingerprint library and a unified feature coding mechanism, significantly improving the heterogeneous data processing capabilities. Based on the multi-agent differential game framework, it coordinates the collaborative decision-making of flow control, service quality optimization and security protection, effectively resolves the traditional single-agent strategy conflicts, and simultaneously optimizes network throughput and transmission stability. The introduction of a meta-learning-driven dynamic weight adjustment mechanism enhances the system's adaptability to complex network environments and significantly reduces the risk of service quality violations. Combining deep reinforcement learning with lightweight deployment technology, it significantly compresses model resource consumption, realizes efficient reasoning and real-time response of edge devices, and breaks through the response delay bottleneck of traditional solutions in burst traffic processing and security defense. Ultimately, a set of intelligent diversion systems that support multi-protocol compatibility, multi-objective dynamic balance, and low resource occupancy are formed, providing flexible and reliable adaptive scheduling capabilities for heterogeneous network environments, and meeting the stringent requirements of new network applications for real-time and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 An overall flow chart of an adaptive network offloading method based on deep reinforcement learning provided in the first embodiment of the present invention. DETAILED DESCRIPTION

[0040] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0041] Example 1, reference Figure 1 , as an embodiment of the present invention, provides an adaptive network offloading method based on deep reinforcement learning, comprising:

[0042] S1: Obtain traffic data of heterogeneous networks, establish a protocol fingerprint library, and convert data from different protocols into a unified network status feature vector.

[0043] The traffic data of the heterogeneous network includes: protocol type identification, message header information, timestamp, timing mode, signal strength and quality indicators, bandwidth utilization, packet loss rate, data packet length, traffic size, traffic pattern, abnormal traffic pattern and authentication identification.

[0044] Establishing a protocol fingerprint library includes preprocessing the acquired traffic data, extracting key features from the preprocessed traffic data, establishing a binary coding template, encoding the key features, obtaining the protocol fingerprint, and storing it in the protocol fingerprint library in an index structure.

[0045] The key features include data features and multi-agent network state features. These multi-agent network state features include the state features of the flow control agent, the state features of the QoS management agent, and the state features of the security agent. Data features are used for traffic statistics, and multi-agent state features are used for reinforcement learning agent decision-making. Multi-agent state features include the state features of the flow control agent, the state features of the QoS management agent, and the state features of the security agent.

[0046] Converting data of different protocols into a unified network status feature vector includes: converting the protocol fingerprint into a numerical vector, normalizing the continuous values ​​in the numerical vector, and concatenating all normalized continuous values ​​into the network status feature vector.

[0047] Furthermore, by establishing a protocol fingerprint library and converting data from different protocols into a unified network state feature vector, traffic data in heterogeneous network environments can be standardized. By extracting key features such as protocol type, packet header information, bandwidth utilization, and packet loss rate, and using binary encoding to store protocol fingerprints, data retrieval efficiency and adaptability can be improved, providing a consistent input data format for subsequent multi-agent reinforcement learning. Furthermore, a normalization method is used to standardize values ​​across different ranges, resulting in a more balanced data distribution. This facilitates the stable training of reinforcement learning algorithms and improves the model's generalization capabilities in complex network environments.

[0048] Furthermore, data features are used for traffic statistics, while multi-agent state features are used for agent decision-making. This enables traffic control, QoS management, and security agents to optimize policies based on a unified state feature vector. By building a unified network state feature vector, agents can effectively perceive traffic characteristics under different network protocols, enabling intelligent cross-protocol traffic diversion, QoS optimization, and security protection, improving the overall network's adaptability and decision-making efficiency.

[0049] S2: Build a multi-agent reinforcement learning framework, including defining the state space, action space, and reward function, and optimize the strategy of each agent based on differential game.

[0050] The multi-agent reinforcement learning framework includes traffic control agent, QoS management agent and security agent.

[0051] The state space is defined by defining observable variables of each agent based on the state characteristics of the multi-agent network to obtain the state space.

[0052] The state space includes: the state space of the flow control agent, the state space of the QoS management agent and the state space of the security agent. The defined action space includes: the action space of the flow control agent, the action space of the QoS management agent and the action space of the security agent.

[0053] State space of the traffic control agent: network load, bandwidth utilization, and packet loss rate.

[0054] The state space of the QoS management agent: service type, QoS priority, and current network delay.

[0055] The state space of the security agent: abnormal traffic detection results, attack types, and historical security records.

[0056] Action space of the traffic control agent: Select the traffic distribution method (5G, WiFi6, satellite network).

[0057] Action space of the QoS management agent: adjusts the bandwidth priority of service flows such as video, voice, and data.

[0058] Action space of security agents: enabling or adjusting attack defense strategies (e.g., DDoS filtering, access control).

[0059] The reward functions defined include: reward function of traffic control agent, reward function of QoS management agent and reward function of security agent.

[0060] The reward function of the traffic control agent is to maximize the network throughput and minimize the load imbalance, which can be expressed as follows:

[0061] R T =α1B-α2L

[0062] Among them, R T represents the reward of the flow control agent, B represents the network throughput, L represents the degree of network load imbalance, and α1 and α2 represent the importance of determining each indicator.

[0063] The reward function of the QoS management agent is to ensure that the delay, packet loss rate and jitter of the service flow are maintained in the optimal range, which can be expressed as follows:

[0064] R Q =-β1D-β2P-β3J

[0065] Among them, R Q : represents the reward of the QoS management agent, D represents the delay, P represents the packet loss rate, J represents the jitter, and β1, β2, and β3 represent the weight coefficients.

[0066] The reward function of the security agent is to minimize the proportion of abnormal traffic while reducing false positives. The formula is expressed as:

[0067] R S =-γ1A+γ2(1-F)

[0068] Among them, R S represents the reward of the security agent, A represents the abnormal traffic ratio, F represents the false alarm rate, and the value range is 0 to 1. γ1 and γ2 represent the weight coefficients.

[0069] Optimizing the decision-making strategies of each agent based on differential games includes constructing the state equation of the differential game based on the state space of the multi-agent reinforcement learning framework to describe the dynamic process of the network state changing over time.

[0070] Solve the Nash equilibrium of the differential game to ensure that the overall strategy of the multi-agent reinforcement learning framework remains optimal when the network state is constantly changing. The formula is expressed as:

[0071]

[0072] Among them, R i represents the reward function of the ith agent, which represents the overall benefit in a given state, The optimal strategy of the i-th agent, The optimal strategy of all agents except the i-th agent is, represents the action set of the i-th agent, Represents all possible actions for the i-th agent.

[0073] Furthermore, by constructing a multi-agent reinforcement learning framework, the traffic control agent, QoS management agent, and security agent can adaptively optimize traffic diversion strategies in dynamic network environments. By properly defining the state space and action space, the agents can accurately perceive the network status and make autonomous decisions based on reinforcement learning, thereby more rationally allocating network resources. Specifically, the traffic control agent can dynamically adjust the distribution of data flows to improve network throughput and balance the load. The QoS management agent optimizes traffic scheduling based on service type and network conditions, ensuring that latency, packet loss rate, and jitter are within optimal ranges. The security agent effectively mitigates the impact of malicious attacks through anomaly detection and defense strategy adjustment, while also reducing false alarm rates and improving network security.

[0074] Furthermore, a differential game theory is used to optimize the decision-making strategies of each agent. By constructing a differential game state equation to describe network state changes and solving the Nash equilibrium, this approach ensures that each agent reaches the globally optimal decision-making outcome under the influence of each other. Compared to traditional reinforcement learning methods that rely on independent training, this method can dynamically adapt to the interactions between different agents in complex heterogeneous network environments, improving overall network performance, service quality, and security protection capabilities, and implementing an intelligent, autonomously optimized network diversion strategy.

[0075] S3: Use a meta-learning model to adjust the weights of each agent’s reward function, and use deep reinforcement learning to train a multi-agent reinforcement learning framework to learn the optimal diversion strategy under different heterogeneous network conditions.

[0076] The meta-learning model adjusts the weights of each agent's reward function by analyzing the network state over the past N time steps using an attention mechanism, calculating the weights of each agent's network state features, and obtaining each agent's state feature weight vector. Using a weighted summation method, combined with each agent's state feature weight vector, the weights of each agent's reward function are dynamically adjusted.

[0077] Using deep reinforcement learning to train a multi-agent reinforcement learning framework includes building a deep reinforcement learning policy network, setting up exploration and utilization mechanisms, initializing the experience pool, and training the multi-agent reinforcement learning framework.

[0078] Constructing a deep reinforcement learning policy network includes: input layer, hidden layer and output layer.

[0079] The input layer includes inputting the state space into the input layer to obtain an input vector.

[0080] The hidden layer includes a fully connected layer, a feature mapping layer and a strategy generation layer.

[0081] The feature mapping layer sets attention scores, calculates the weights of the internal features of each agent's state space, and performs a weighted sum to obtain the final state feature vector for each agent. A differential game equation is established to describe the impact of different agent strategies on other agents. Nash equilibrium is solved to ensure that multi-agent strategies remain optimal in a dynamic environment, and the game influence between agents is analyzed.

[0082] Combining the final state feature vector of each agent and the game influence between agents, the final strategy input vector of each agent is generated.

[0083] The strategy generation layer uses the final strategy input vector and action space of each agent to calculate the Q value of each agent's different actions, selects the action with the largest Q value as the optimal strategy for each agent, and uses the Q-Learning method to update the Q value.

[0084] The output layer includes outputting the optimal strategy of each agent.

[0085] Furthermore, by dynamically adjusting the reward function weights of each agent through a meta-learning model, the reinforcement learning system can adaptively optimize each agent's learning direction based on historical network states and environmental changes. By introducing an attention mechanism, the agent can focus on the most important historical state features, improving the accuracy and generalization of policy decisions. Furthermore, combined with a weighted summation method, this ensures that each agent can flexibly adjust its optimization objectives under varying network conditions, avoiding policy rigidity and improving adaptability to heterogeneous network environments.

[0086] Furthermore, a multi-agent reinforcement learning framework is trained using a deep reinforcement learning policy network. This layered processing of state information using fully connected layers, feature mapping layers, and policy generation layers enables efficient extraction of key features in complex network environments. Combined with differential game optimization, this ensures that each agent's strategy achieves global equilibrium, preventing system performance degradation caused by over-optimization of a single agent. Updating the Q-value through the Q-Learning algorithm continuously improves the agent's decision-making quality, enabling dynamic adjustment of the optimal traffic diversion strategy, increasing network throughput, optimizing the QoS experience, and enhancing the system's stability and reliability in complex environments.

[0087] S4: Deploy the trained strategy to a heterogeneous network environment and dynamically select the optimal network path based on the current network status.

[0088] By observing the reward curve, we can ensure that the agent's strategy converges and no longer fluctuates greatly.

[0089] The trained strategy is deployed to the network controller for actual traffic scheduling.

[0090] Real-time monitoring of the network environment allows for dynamic policy adjustments: When traffic spikes, the traffic control agent re-optimizes and distributes traffic. When business demands change, the QoS management agent adjusts priorities. When abnormal traffic is detected, the security agent proactively protects against it.

[0091] Furthermore, by deploying trained agent strategies in heterogeneous network environments, they can perceive changes in network status in real time and dynamically adjust network traffic diversion strategies. When traffic bursts increase, the traffic control agent automatically optimizes traffic scheduling to avoid network congestion. When service demands change, the QoS management agent promptly adjusts traffic priorities to ensure the quality of service for critical service flows. This adaptive optimization mechanism improves bandwidth utilization, reduces latency and packet loss, and thus enhances overall network performance.

[0092] Furthermore, through the agent's continuous monitoring and dynamic decision-making, it can quickly respond to abnormal traffic or malicious attacks. The security agent can proactively detect and intercept abnormal traffic, effectively reducing the impact of attacks on the network. This dynamic adjustment mechanism not only enhances network security defense capabilities, but also reduces false alarms and ensures that normal traffic is not disrupted. Furthermore, as the agent's strategy continuously learns and optimizes, the system can maintain a stable optimal strategy over the long term, reducing frequent manual intervention and lowering operation and maintenance costs.

[0093] Example 2, an embodiment of the present invention, provides an adaptive network offloading system based on deep reinforcement learning, including:

[0094] The protocol fingerprint library module obtains traffic data of heterogeneous networks, establishes a protocol fingerprint library, and converts data from different protocols into a unified network status feature vector.

[0095] The multi-agent reinforcement learning module builds a multi-agent reinforcement learning framework, including defining the state space, action space, and reward function, and optimizing the strategy of each agent based on differential game.

[0096] The strategy module uses a meta-learning model to adjust the weight of each agent's reward function, and uses deep reinforcement learning to train a multi-agent reinforcement learning framework to learn the optimal diversion strategy under different heterogeneous network conditions.

[0097] The deployment module deploys the trained strategy to a heterogeneous network environment and dynamically selects the optimal network path based on the current network status.

[0098] Embodiment 3, an embodiment of the present invention, is different from the first two embodiments in that:

[0099] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0100] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0101] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection portion having one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (ROM), and a memory card.

[0102] (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memories

[0103] (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting or processing in another suitable manner as necessary, and then stored in a computer memory.

[0104] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0105] Example 4 is an embodiment of the present invention, which provides an adaptive network offload method and system based on deep reinforcement learning. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.

[0106] To validate the applicability of an adaptive network offloading method based on deep reinforcement learning in complex heterogeneous network environments, experiments were conducted in a real-world network test environment consisting of a 5G base station, a WiFi6 access point, and a low-Earth orbit satellite link. The test network included a 5G base station supporting up to 1Gbps throughput, a WiFi6 access point supporting 600Mbps throughput, and a low-Earth orbit satellite link supporting 200Mbps throughput. Traffic types included HD video, VoIP calls, IoT data transmission, and web browsing, and simulated DDoS attack traffic was also included to test the security and adaptability of the network offloading method.

[0107] The experiment first collected data, capturing traffic data from heterogeneous networks. This data primarily includes key features such as protocol type, packet header information, timestamps, traffic patterns, bandwidth utilization, and packet loss rate. The collected data was preprocessed to extract key features and establish a protocol fingerprint library. All protocol data was converted into a unified network state feature vector, which was then fed into a reinforcement learning framework for intelligent decision-making.

[0108] The training of the multi-agent reinforcement learning framework is divided into three phases. The first phase is agent initialization, which constructs a flow control agent, a QoS management agent, and a security agent. The flow control agent is responsible for optimizing network traffic distribution, the QoS management agent ensures that the latency, packet loss rate, and jitter of different service flows remain within the optimal range, and the security agent detects and protects against abnormal traffic. The second phase conducts reinforcement learning training. The agents continuously optimize their decision-making strategies through interaction and optimize each agent's strategy based on differential games to achieve the optimal overall network traffic diversion strategy. During training, the agents learn under different network conditions and use meta-learning methods to dynamically adjust reward weights, enabling them to adapt to complex and changing network environments. In the third phase, after training is complete, the agent strategies are deployed to heterogeneous network environments, and real-time policy adjustments are performed to dynamically select the optimal network path.

[0109] The experiment set up various network scenarios, including high-load environments, sudden traffic fluctuations, and malicious attack traffic penetration, to test whether the method can balance traffic distribution, optimize QoS, and improve network security in these different scenarios. During the test, the intelligent agent monitored the network status in real time. The traffic control agent optimized network distribution when sudden traffic increased. The QoS management agent adjusted traffic priority according to business needs. The security agent proactively implemented defense strategies after detecting abnormal traffic. The entire experiment lasted several hours to ensure stability and optimization results in various scenarios.

[0110] In a 5G network environment, the throughput before optimization was 818.14 Mbps, which increased to 948.76 Mbps after optimization, a significant improvement. In a WiFi6 network environment, the throughput before optimization was 505.58 Mbps, which increased to 580.11 Mbps after optimization, showing a relatively stable increase in throughput. In a satellite network environment, the throughput before optimization was 163.41 Mbps, which increased to 197.11 Mbps after optimization, improving transmission efficiency compared to before optimization.

[0111] In terms of QoS metrics, the average latency before optimization was 51.81ms, which was reduced to 38.27ms after optimization. This significant reduction in latency improves network responsiveness. Regarding packet loss, the rate was 3.86% before optimization and 1.85% after optimization, reducing lost packets and improving transmission stability. Jitter was 10.64ms before optimization and decreased to 8.24ms after optimization, stabilizing data fluctuations and improving the experience for real-time services.

[0112] In terms of abnormal traffic detection and protection, the proportion of abnormal traffic before optimization dropped from 24.88% to 6.78% after optimization, demonstrating that the optimization method effectively reduces the impact of abnormal traffic. Furthermore, the false alarm rate dropped from 7.87% before optimization to 3.30% after optimization, demonstrating that this method effectively detects and protects against abnormal traffic while reducing false positives, ensuring system stability and security. Experimental data analysis demonstrates that the adaptive network offloading method based on deep reinforcement learning offers significant advantages over traditional methods across multiple key performance indicators. First, in terms of throughput optimization, this method adjusts traffic allocation strategies based on real-time network status, thereby improving link utilization. The throughput improvement is particularly significant in 5G networks, and throughput optimization for WiFi6 and satellite networks also shows a steady upward trend. Compared to traditional methods that use fixed traffic allocation strategies, this method dynamically adjusts network resource allocation, maintaining optimal throughput across different network types.

[0113] Secondly, in terms of QoS optimization, this method can dynamically adjust traffic priorities based on changing service demands, ensuring the quality of service for high-priority services. In high-definition video traffic scenarios, this method can effectively reduce latency, significantly improving video playback smoothness. Traditional methods, due to their lack of dynamic optimization capabilities, are prone to video freezes under high network loads. In voice call scenarios, this method can reduce packet loss, ensure stable voice data transmission, and improve call quality. Traditional methods are prone to degraded call quality when traffic bursts increase.

[0114] In terms of abnormal traffic suppression, this method uses a security agent trained through reinforcement learning to rapidly detect and isolate abnormal traffic. After a DDoS attack occurs, this method identifies attack traffic within a very short period of time and uses an intelligent traffic adjustment mechanism to mitigate its impact on normal business operations, thereby ensuring the stability of normal business traffic. Compared to traditional methods that rely on fixed thresholds for anomaly detection, this method dynamically optimizes detection strategies, improving detection accuracy and reducing false positives. Experimental data demonstrates that this method significantly improves the detection rate of abnormal traffic while significantly reducing the false positive rate compared to traditional methods.

[0115] Furthermore, in terms of computational efficiency and adaptability, this method significantly reduces the time cost of training agents compared to traditional methods. Furthermore, after deployment, it can quickly respond to network state changes, achieving real-time traffic diversion optimization. While traditional methods require manual intervention to adjust policies in dynamically changing network environments, this method automatically optimizes policies through reinforcement learning, reducing manual intervention and improving the intelligence of network management.

[0116] In summary, the adaptive network offloading method based on deep reinforcement learning outperforms traditional methods in terms of throughput optimization, QoS assurance, network security protection, and computational efficiency. It addresses the difficulty of existing offloading strategies in adapting to complex network environments and demonstrates significant technological innovation and practical application value. This method not only improves network resource utilization but also enables intelligent optimization and adjustment based on changes in network status, showing broad potential for application in heterogeneous network environments.

[0117] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An adaptive network offloading method based on deep reinforcement learning, characterized in that: include: Obtain traffic data from heterogeneous networks, build a protocol fingerprint library, and convert data from different protocols into a unified network status feature vector; Build a multi-agent reinforcement learning framework, including defining the state space, action space, and reward function, and optimize the strategies of each agent based on differential games; A meta-learning model is used to adjust the weights of each agent's reward function, and deep reinforcement learning is used to train a multi-agent reinforcement learning framework to learn the optimal diversion strategy under different heterogeneous network conditions; Deploy the trained strategy to a heterogeneous network environment and dynamically select the optimal network path based on the current network status; The multi-agent reinforcement learning framework includes traffic control agents, QoS management agents, and security agents; Defining the state space is to define the observable variables of each agent based on the state characteristics of the multi-agent network to obtain the state space; The state space includes: the state space of the flow control agent, the state space of the QoS management agent and the state space of the security agent; the defined action space includes: the action space of the flow control agent, the action space of the QoS management agent and the action space of the security agent; The reward functions defined include: reward function of traffic control agent, reward function of QoS management agent and reward function of security agent; The reward function of the traffic control agent is to maximize network throughput and minimize load imbalance; the reward function of the QoS management agent is to ensure that the latency, packet loss rate, and jitter of the service flow are maintained within the optimal range; the reward function of the security agent is to minimize the proportion of abnormal traffic while reducing false alarms; Optimizing the decision-making strategies of each agent based on differential games involves constructing the state equations of the differential game based on the state space of the multi-agent reinforcement learning framework to describe the dynamic process of network state changes over time; Solve the Nash equilibrium of the differential game to ensure that the overall strategy of the multi-agent reinforcement learning framework remains optimal when the network state is constantly changing; The meta-learning model is used to adjust the weight of each agent's reward function, including analyzing the past based on the attention mechanism. The network state of each time step is calculated, the weight of the network state feature of each agent is calculated, and the state feature weight vector of each agent is obtained; the weighted summation method is used, and the weight of the reward function of each agent is dynamically adjusted in combination with the state feature weight vector of each agent.

2. The adaptive network offloading method based on deep reinforcement learning according to claim 1, characterized in that: The traffic data of the heterogeneous network includes: protocol type identification, message header information, timestamp, timing mode, signal strength and quality indicators, bandwidth utilization, packet loss rate, data packet length, traffic size, traffic pattern, abnormal traffic pattern and authentication identification; Establishing a protocol fingerprint library includes preprocessing the acquired traffic data, extracting key features from the preprocessed traffic data, establishing a binary coding template, encoding the key features, obtaining the protocol fingerprint, and storing it in the protocol fingerprint library in an index structure; The key features include data features and multi-agent network state features; the multi-agent network state features include flow control agent state features, QoS management agent state features and security agent state features; Converting data of different protocols into a unified network status feature vector includes: converting the protocol fingerprint into a numerical vector, normalizing the continuous values ​​in the numerical vector, and concatenating all normalized continuous values ​​into the network status feature vector.

3. The adaptive network offloading method based on deep reinforcement learning according to claim 2, characterized in that: Using deep reinforcement learning to train a multi-agent reinforcement learning framework includes building a deep reinforcement learning policy network, setting up exploration and utilization mechanisms, initializing the experience pool, and training the multi-agent reinforcement learning framework.

4. The adaptive network offloading method based on deep reinforcement learning according to claim 3, characterized in that: Constructing a deep reinforcement learning strategy network includes: input layer, hidden layer and output layer; The input layer includes inputting the state space into the input layer to obtain an input vector; The hidden layer includes a fully connected layer, a feature mapping layer and a strategy generation layer; The feature mapping layer sets attention scores, calculates the weights of the internal features of each agent's state space, and performs weighted summation to obtain the final state feature vector of each agent; establishes differential game equations to describe the impact of different agent strategies on other agents; solves Nash equilibrium to ensure that multi-agent strategies remain optimal in a dynamic environment, and obtains the game impact between agents; Combine the final state feature vector of each agent and the game influence between agents to generate the final strategy input vector of each agent; The strategy generation layer includes calculating the Q value of each agent's different actions using the agent's final strategy input vector and action space, selecting the action with the largest Q value as the optimal strategy for each agent, and updating the Q value using the Q-Learning method. The output layer includes outputting the optimal strategy of each agent.

5. An adaptive network offloading system based on deep reinforcement learning using the method according to any one of claims 1 to 4, characterized in that: The protocol fingerprint library module obtains traffic data of heterogeneous networks, establishes a protocol fingerprint library, and converts data from different protocols into a unified network status feature vector; Multi-agent reinforcement learning module, which builds a multi-agent reinforcement learning framework, including defining the state space, action space, and reward function, and optimizing the strategies of each agent based on differential games; The strategy module uses a meta-learning model to adjust the weights of each agent's reward function and uses deep reinforcement learning to train a multi-agent reinforcement learning framework to learn the optimal diversion strategy under different heterogeneous network conditions; The deployment module deploys the trained strategy to a heterogeneous network environment and dynamically selects the optimal network path based on the current network status.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the adaptive network offloading method based on deep reinforcement learning according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the adaptive network offloading method based on deep reinforcement learning according to any one of claims 1 to 4 are implemented.

Images