Heterogeneous network backhaul link selection method, device and equipment based on reinforcement learning

Abstract

The invention provides a heterogeneous network backhaul link selection method, device and equipment based on reinforcement learning, and relates to the technical field of wireless communication. The method is applied to a heterogeneous convergence network, and the heterogeneous convergence network comprises a satellite network and a ground network. The method comprises the following steps: acquiring the real-time length of a service queue of the heterogeneous convergence network; inputting the real-time length into a trained link optimization model to obtain link resources allocated to the service queue; wherein the link optimization model is obtained by performing reinforcement learning training by taking service packet loss, queuing delay and network throughput as optimization targets; and scheduling the service queues of the heterogeneous convergence network based on the link resources allocated to the service queues. According to the method, multiple optimization targets of the service packet loss rate, the queuing delay and the network throughput can be comprehensively considered, link resources are allocated for the service queues according to different lengths of the service queues, and the method is suitable for backhaul selection of different types of service terminals in a heterogeneous fusion scene of a satellite network and a ground network.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology, and in particular to a method, apparatus, and device for selecting backhaul links in heterogeneous networks based on reinforcement learning. Background Technology

[0002] With the rapid evolution of 5G / 6G communication technologies, integrated space-ground information networks that deeply integrate terrestrial mobile communication networks with high, medium, and low orbit satellite networks have become an important direction for future communication development. This heterogeneous network environment can provide ubiquitous wide-area coverage, meeting the needs of various emerging applications. In this integrated network, the transmission characteristics of different links differ significantly: terrestrial base station links (such as 5G) typically have high capacity and low latency, but limited coverage; while satellite links (such as Tiantong and Iridium) have the advantage of wide coverage, but their propagation latency is higher and available bandwidth is relatively limited.

[0003] At the same time, the types of services carried in the network are becoming increasingly diversified, mainly divided into two categories: one is services that are extremely sensitive to latency (Delay-Sensitive, DS), such as industrial automation control signaling and real-time feedback in telemedicine. These services have strict upper limits on transmission latency, and once the timeout is exceeded, it is considered a transmission failure; the other is ordinary best-effort services (Best-Effort, BE), such as file downloads and non-real-time data backhaul. These services are not sensitive to latency, but pursue the highest possible throughput.

[0004] However, traditional methods often employ service scheduling strategies based on preset thresholds. For example, a fixed latency threshold is set for latency-sensitive services, and once this threshold is exceeded, a switch to a high-speed link is attempted. This approach lacks adaptability and cannot cope with dynamically changing network load and channel conditions. It is prone to causing significant timeouts for critical services during network congestion, or low resource utilization during network idle periods. This makes it difficult to fully leverage the advantages of heterogeneous communication networks, resulting in poor application performance of heterogeneous communication networks. Summary of the Invention

[0005] This invention provides a method, apparatus, and device for selecting backhaul links in heterogeneous networks based on reinforcement learning, in order to address the problem of improving the application effect of heterogeneous communication networks.

[0006] In a first aspect, embodiments of the present invention provide a backhaul link selection method for heterogeneous networks based on reinforcement learning, applied to heterogeneous fusion networks, which include satellite networks and terrestrial networks; the method includes: Obtain the real-time length of the service queue in the heterogeneous converged network; The real-time length is input into the trained link optimization model to obtain the link resources allocated to the service queue; the link optimization model is trained by reinforcement learning with service packet loss, queuing latency and network throughput as optimization objectives. The service queues of the heterogeneous converged network are scheduled based on the link resources allocated to the service queues.

[0007] In one possible implementation, before inputting the real-time length into the trained link optimization model to obtain the link resources allocated to the service queue, the following steps are also included: The real-time length of the service queue in the heterogeneous converged network is taken as the state, and the link resources allocated to the service queue are taken as the action. Different actions are executed for each state, forming multiple sets of state-action combinations. Record the next state corresponding to each state-action combination, and calculate the immediate reward value corresponding to the state-action combination based on a multi-objective reward function; wherein, the multi-objective reward function takes packet loss, queuing latency and network throughput as optimization objectives; Each state-action combination, along with the corresponding next state and immediate reward value, is combined into an experience tuple and stored in the experience replay buffer. Batch data is sampled from the experience replay buffer, and the initial link optimization model is trained by reinforcement learning based on the long-term optimization objective to obtain the trained link optimization model.

[0008] In one possible implementation, the multi-objective reward function is:

[0009] in, For instant reward value, The weight parameters are and satisfy the following conditions: , For the first The number of packet losses for time-slot latency-sensitive services. For the first There are still 10 days left until the timeout begins at the start of the time slot. The number of latency-sensitive service packets per time slot For the first Length of the time slot for normal services For the first Time slots are allocated to ground base station link resources for ordinary services. For the first Time slots are allocated to satellite link resources for regular services; The long-term optimization goal is:

[0010] in, To optimize the target value in the long term, For scheduling strategy, This is the discount factor.

[0011] In one possible implementation, the service queue information of the heterogeneous converged network is used as the state, and the link resources allocated to the service queue are used as actions. Different actions are executed for each state, forming multiple sets of state-action combinations, including: The service queue information of the heterogeneous converged network is used as the state, and the link resources allocated to the service queue are used as the action. Based on the available resource constraints, different actions are executed for each state, forming multiple sets of state-action combinations.

[0012] In one possible implementation, available resource constraints include: Link capacity constraints:

[0013]

[0014] Queue state constraints:

[0015]

[0016] Resource allocation non-negative integer constraint:

[0017] in, For ground base station link capacity, For satellite link capacity, For the first There are still 10 days left until the timeout begins at the start of the time slot. The number of latency-sensitive service packets per time slot , The maximum number of queuing slots. For the first Length of the time slot for normal services To be assigned to the Ground base station link resources in the tiered queue For the first Time slots are allocated to ground base station link resources for ordinary services. For the first Time slots are allocated to satellite link resources for regular services. It is the set of non-negative integers.

[0018] In one possible implementation, the service queues include latency-sensitive service queues; scheduling of the service queues in the heterogeneous converged network based on the link resources allocated to the service queues includes: The scheduling of latency-sensitive service queues is based on the link resources allocated to them and a hierarchical queue scheduling mechanism. The hierarchical queue scheduling mechanism is as follows:

[0019]

[0020]

[0021] in, Indicates the first At the start of the time slot The number of latency-sensitive service packets in the first-level queue, the number of packets in the second-level queue. The queue has a timeout period of [number] days remaining. A time-sensitive service packet in a time slot, , The maximum number of queuing slots. For the first The number of newly arriving latency-sensitive service packets in the time slot. To be assigned to the Ground base station link resources in the tiered queue.

[0022] In one possible implementation, the service queue includes ordinary service queues; scheduling of the service queues in the heterogeneous converged network based on the link resources allocated to the service queues includes: The scheduling of ordinary service queues is based on the link resources allocated to them and a single-queue management mechanism; the single-queue management mechanism is as follows:

[0023] in, For the first Length of the time slot for normal services For the first The number of new regular service packets arriving in the time slot, For the first Time slots are allocated to ground base station link resources for ordinary services. For the first Time slots are allocated to satellite link resources for regular services.

[0024] Secondly, embodiments of the present invention provide a backhaul link selection device for heterogeneous networks based on reinforcement learning, applied to heterogeneous fusion networks, the heterogeneous fusion networks including satellite networks and terrestrial networks; the device includes: The acquisition module is used to obtain the real-time length of the service queue in the heterogeneous converged network; The allocation module is used to input the real-time length into the trained link optimization model to obtain the link resources allocated to the service queue; the link optimization model is trained by reinforcement learning with service packet loss, queuing delay and network throughput as optimization objectives. The scheduling module is used to schedule service queues in the heterogeneous converged network based on the link resources allocated to the service queues.

[0025] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect or any possible implementation thereof.

[0026] The backhaul link selection method, apparatus, and device for heterogeneous networks based on reinforcement learning provided in this invention, through a link optimization model, comprehensively considers multiple optimization objectives such as service packet loss rate, queuing delay, and network throughput on the basis of heterogeneous converged networks. It allocates link resources to service queues of different lengths, and is applicable to backhaul selection of different types of service terminals in heterogeneous converged scenarios of satellite and terrestrial networks. Furthermore, it can further divide service queues into latency-sensitive services and ordinary services, thereby improving the network throughput of ordinary services while meeting the queuing delay requirements of latency-sensitive services. Attached Figure Description

[0027] Figure 1 This is an application scenario diagram of the backhaul link selection method for heterogeneous networks based on reinforcement learning provided in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the implementation of the reinforcement learning-based backhaul link selection method for heterogeneous networks provided in this embodiment of the invention. Figure 3 This is a performance comparison chart of the packet loss rate of the proposed method and the comparison method for latency-sensitive services under different network loads according to the embodiments of the present invention. Figure 4 This is a comparison chart of the ordinary service throughput performance of the proposed method and the comparative method under different network loads in the embodiments of the present invention. Figure 5 This is a comparison chart of the average latency performance of the proposed method and the comparative method for latency-sensitive services under different network loads according to the embodiments of the present invention. Figure 6 The convergence performance analysis diagram of the online backhaul link optimization method based on reinforcement learning in this invention; Figure 7 This is a schematic diagram of the structure of a heterogeneous network backhaul link selection device based on reinforcement learning provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] Figure 1 This is an application scenario diagram of the backhaul link selection method for heterogeneous networks based on reinforcement learning provided in this embodiment of the invention. For example... Figure 1 As shown, the heterogeneous converged network to which this invention is applied includes terrestrial networks and satellite networks, and the terminal types include latency-sensitive service terminals and ordinary service terminals. The backhaul link scheduling decision point is located in the air-ground converged communication device.

[0030] In a heterogeneous network environment integrating terrestrial and satellite networks, intelligently and efficiently allocating heterogeneous link resources to services with different Quality of Service (QoS) requirements has become a highly challenging technical problem. The applicant has identified the following deficiencies in existing technologies: 1. Static Threshold Scheduling Strategy: Traditional methods often employ scheduling strategies based on preset thresholds. For example, a fixed latency threshold is set for latency-sensitive services, and once this threshold is exceeded, a switch to a high-speed link is attempted. This method lacks adaptability and cannot cope with dynamically changing network load and channel conditions. It can easily lead to a large number of timeouts for critical services when the network is congested, or low resource utilization when the network is idle.

[0031] 2. Single Optimization Objective: Existing research often focuses on only a single optimization objective, such as maximizing total system throughput or minimizing average latency. Such a single objective is difficult to simultaneously meet the differentiated QoS requirements of different services. Sacrificing BE (Browser / Earnings) services to absolutely guarantee DS (Data Serving) services will waste network resources; conversely, it cannot guarantee the reliability of critical DS services.

[0032] 3. Lack of intelligent decision-making ability: Scheduling algorithms based on fixed rules cannot learn from historical experience and adapt to complex business models and network environments. Faced with high-dimensional state spaces (real-time lengths of each queue) and action spaces (discrete allocation of resources), traditional optimization algorithms (such as convex optimization and queuing theory models) are often difficult to apply in practice due to overly simplified models or excessively high computational complexity.

[0033] 4. Simple queue management mechanism: The traditional first-in-first-out (FIFO) queue management mechanism cannot effectively handle the tiered service requirements of latency-sensitive services. It treats all DS services with equal priority, and cannot distinguish between urgent services that are about to time out and ordinary DS services with a large latency margin, resulting in low scheduling efficiency and increasing the risk of timeout for urgent services.

[0034] Therefore, there is an urgent need for a scheduling method that can intelligently perceive network status, consider multi-dimensional optimization objectives, and perform fine-grained queue management to solve the problems in the existing technologies. To address these issues, this invention provides a reinforcement learning-based backhaul link selection method for heterogeneous networks. This method solves the problems of static scheduling, single optimization objectives, and lack of intelligent decision-making in existing technologies. It achieves intelligent backhaul scheduling for latency-sensitive and general services in a heterogeneous converged network environment of satellite and terrestrial networks, ensuring both the QoS requirements of latency-sensitive services and maximizing overall network resource utilization, while also possessing adaptive capabilities to dynamic network environments. The main improvements include the following: (1) Establish a multi-link backhaul system model of heterogeneous fusion network of satellite network and terrestrial network.

[0035] (2) Design a hierarchical queue management mechanism for latency-sensitive services, store latency-sensitive services in a hierarchical manner according to QoS remaining time limits, and establish a hierarchical queue scheduling mechanism; (3) Establish constraints on available resources in satellite and terrestrial networks to ensure that scheduling decisions meet link capacity limits and queue status constraints; (4) Construct a multi-objective optimization problem, taking into account service packet loss penalties, queuing delay and network throughput; (5) The backhaul link selection optimization problem is optimized online using deep reinforcement learning algorithms.

[0036] See Figure 2 The document illustrates a flowchart of the implementation of the reinforcement learning-based backhaul link selection method for heterogeneous networks provided in this embodiment of the invention, detailed below: This method is applied to heterogeneous converged networks, including satellite networks and terrestrial networks; the method includes: Step 201: Obtain the real-time length of the service queue of the heterogeneous converged network.

[0037] In this embodiment, a system model is established that includes two types of networks: a terrestrial network with a capacity of A packets / slot, characterized by low latency and high capacity; and a satellite network with a capacity of B packets / slot, characterized by high latency but wide-area coverage.

[0038] Two service types are defined: Delay-Sensitive (DS), where each data packet has a maximum tolerable number of queuing slots. Transmission failure occurs upon timeout, primarily affecting latency-sensitive services such as real-time voice and video calls, and industrial control. Latency-sensitive services will arrive at the following points: Best-effort (BE) services only concern throughput and have no hard deadlines. These primarily include file transfers, web browsing, email, and other services that can tolerate a certain level of latency. Best-effort services arrive as follows: .in, and These are the arrival rates for latency-sensitive services and regular services, respectively.

[0039] The backhaul link scheduling decision point is located at the air-to-ground converged communication device, at each time step. t Decisions are made on whether different terminal services use satellite network links or terrestrial network links for backhaul.

[0040] Step 202: Input the real-time length into the trained link optimization model to obtain the link resources allocated to the service queue; wherein, the link optimization model is trained by reinforcement learning with service packet loss, queuing delay and network throughput as optimization objectives.

[0041] In this embodiment, the complex multi-link, multi-service scheduling problem is formally modeled as a Markov decision process, and defined as follows: State: In any decision time slot The state of the system It is composed of the real-time queue leader (i.e., the number of data packets in the queue) of all DS service sub-queues and BE service queues.

[0042] Action: The system's actions Defined in this time slot The scheduler allocates a certain amount of transmission resources (such as the number of data packets or transmission bandwidth units) on available terrestrial and satellite links for each service queue.

[0043] Reward: Design a hierarchical instant reward function Used to evaluate in state Next action The reward function takes into account the timeout and packet loss penalties of the DS service, the queuing delay penalties of all services, and the throughput incentives of the BE service, and uses weighting factors to reflect the priority ranking of different optimization objectives.

[0044] Deep reinforcement learning (DRL) algorithms, such as those based on the actor-critic framework (e.g., PPO), are used to train the aforementioned link optimization model.

[0045] DRL agents learn through continuous interaction with the network environment (i.e., constantly observing the state, performing actions, and receiving rewards).

[0046] During the learning process, the agent optimizes its internal neural network parameters online to form a scheduling strategy that maximizes long-term cumulative discount rewards. .

[0047] At each decision slot, the scheduler uses the latest learned strategy to generate and execute the optimal resource allocation action based on the currently observed system state, while ensuring that the action meets the capacity limits of physical links and the non-empty constraints of queues.

[0048] Step 203: Schedule the service queues of the heterogeneous converged network based on the link resources allocated to the service queues.

[0049] In this embodiment, different scheduling rules can be adopted for different types of services. Following a hierarchical queue management mechanism for latency-sensitive services, these services are stored in tiers based on their remaining QoS time limits, and a hierarchical queue scheduling mechanism is implemented to meet their latency requirements. For ordinary services, a single queue management mechanism is used to improve network throughput.

[0050] This invention, through a link optimization model, comprehensively considers multiple optimization objectives such as service packet loss rate, queuing latency, and network throughput on the basis of heterogeneous converged networks. It allocates link resources to service queues of different lengths, which is applicable to backhaul selection of different types of service terminals in heterogeneous converged satellite and terrestrial network scenarios. Furthermore, it can divide service queues into latency-sensitive services and ordinary services, thereby improving the network throughput of ordinary services while meeting the queuing latency requirements of latency-sensitive services.

[0051] In one possible implementation, before inputting the real-time length into the trained link optimization model to obtain the link resources allocated to the service queue, the following steps are also included: The real-time length of the service queue in the heterogeneous converged network is taken as the state, and the link resources allocated to the service queue are taken as the action. Different actions are executed for each state, forming multiple sets of state-action combinations. Record the next state corresponding to each state-action combination, and calculate the immediate reward value corresponding to the state-action combination based on a multi-objective reward function; wherein, the multi-objective reward function takes packet loss, queuing latency and network throughput as optimization objectives; Each state-action combination, along with the corresponding next state and immediate reward value, is combined into an experience tuple and stored in the experience replay buffer. Batch data is sampled from the experience replay buffer, and the initial link optimization model is trained by reinforcement learning based on the long-term optimization objective to obtain the trained link optimization model.

[0052] In this embodiment, the meaning and values ​​of each mathematical symbol in the following text are shown in Table 1.

[0053] Table 1

[0054] The state space is defined as follows:

[0055] in, For decision-making gaps, For each latency-sensitive service queue ( ~ The real-time length of ) This represents the real-time length of the normal service queue. (In any decision time slot) The state of the system It is composed of the real-time queue leader (i.e., the number of data packets in the queue) of all DS service sub-queues and BE service queues.

[0056] Action space definition:

[0057] in, To be assigned Ground base station link resources of sub-queues The maximum number of queuing slots that DS services can tolerate. GB resources allocated to the BE queue, SAT resources allocated to the BE queue. For action space. During the decision-making time slot. The system's actions Defined in this time slot The scheduler allocates a certain amount of transmission resources (such as the number of data packets or transmission bandwidth units) on available terrestrial and satellite links for each service queue.

[0058] Multi-objective reward functions are used to evaluate the reward in state Next action The reward function takes into account the timeout and packet loss penalties of the DS service, the queuing delay penalties of all services, and the throughput incentives of the BE service, and uses weighting factors to reflect the priority ranking of different optimization objectives.

[0059] The PPO algorithm is used to train the link optimization model. The training process includes: (1) Initialize the parameters of the deep neural network as the initial link optimization model; the structure of the neural network can be set according to actual needs; (2) In each time slot According to the current state Select Action ; (3) Perform the action and observe the immediate reward. and the next state The state transition probability is jointly determined by the service arrival process and the scheduling decision. (4) Empirical tuples Store in the experience replay buffer; (5) Sample batch data from the experience replay buffer to update the neural network parameters; Repeat the above process until convergence, and you will get the trained link optimization model.

[0060] In one possible implementation, the multi-objective reward function is:

[0061] in, For instant reward value, The weight parameters are and satisfy the following conditions: , For the first The number of packet losses for time-slot latency-sensitive services. For the first There are still 10 days left until the timeout begins at the start of the time slot. The number of latency-sensitive service packets per time slot For the first Length of the time slot for normal services For the first Time slots are allocated to ground base station link resources for ordinary services. For the first Time slots are allocated to satellite link resources for regular services; The long-term optimization goal is:

[0062] in, To optimize the target value in the long term, For scheduling strategy, This is the discount factor.

[0063] In this embodiment, the multi-objective reward function combines both punitive and incentive components: (1) Severely punish those who exceed the time limit Ensure the strategy prioritizes compressing the DS packet loss rate; (2) Queueing penalty To approximately minimize the average latency and reduce the overall queue queuing latency; (3) Throughput reward, weight Mildly encourage Business Environment (BE) to seize idle capacity in order to improve overall resource utilization.

[0064] (4) Parameter relationship This ensures that the needs of latency-sensitive services, namely DS, are met first.

[0065] In the long-term discount target, strategy It is the decision rule of the RL agent, the discount factor. The closer it is to 1, the more it values ​​long-term returns; the closer it is to 0, the more short-sighted it is.

[0066] In one possible implementation, the service queue information of the heterogeneous converged network is used as the state, and the link resources allocated to the service queue are used as actions. Different actions are executed for each state, forming multiple sets of state-action combinations, including: The service queue information of the heterogeneous converged network is used as the state, and the link resources allocated to the service queue are used as the action. Based on the available resource constraints, different actions are executed for each state, forming multiple sets of state-action combinations.

[0067] In this embodiment, the available resource constraints may specifically include link capacity constraints, queue state constraints, and resource allocation non-negative integer constraints, which are used to ensure that scheduling decisions meet link capacity limits and queue state.

[0068] In one possible implementation, available resource constraints include: Link capacity constraints:

[0069]

[0070] Queue state constraints:

[0071]

[0072] Resource allocation non-negative integer constraint:

[0073] in, For ground base station link capacity, For satellite link capacity, For the first There are still 10 days left until the timeout begins at the start of the time slot. The number of latency-sensitive service packets per time slot , The maximum number of queuing slots. For the first Length of the time slot for normal services To be assigned to the Ground base station link resources in the tiered queue For the first Time slots are allocated to ground base station link resources for ordinary services. For the first Time slots are allocated to satellite link resources for regular services. It is the set of non-negative integers.

[0074] In this embodiment, the link capacity constraint is used to limit the amount of link resources allocated to not exceed the upper limit of physical link throughput, the queue state constraint is used to ensure "at most services available" - the queue virtual inventory cannot be overdrawn, and the resource allocation non-negative integer constraint is used to describe the granularity of discrete resources.

[0075] In one possible implementation, the service queues include latency-sensitive service queues; scheduling of the service queues in the heterogeneous converged network based on the link resources allocated to the service queues includes: The scheduling of latency-sensitive service queues is based on the link resources allocated to them and a hierarchical queue scheduling mechanism. The hierarchical queue scheduling mechanism is as follows:

[0076]

[0077]

[0078] in, Indicates the first At the start of the time slot The number of latency-sensitive service packets in the first-level queue, the number of packets in the second-level queue. The queue has a timeout period of [number] days remaining. A time-sensitive service packet in a time slot, , The maximum number of queuing slots. For the first The number of newly arriving latency-sensitive service packets in the time slot. To be assigned to the Ground base station link resources in the tiered queue.

[0079] In this embodiment, the traditional First-In-First-Out (FIFO) queue management mechanism treats all DS services with equal priority, failing to distinguish between urgent services about to time out and ordinary DS services with greater latency margins. This makes it ineffective in handling the tiered service requirements of latency-sensitive services, resulting in low scheduling efficiency and increased timeout risk for urgent services. This embodiment, however, tiers latency-sensitive service packets according to the remaining time slots before timeout and allocates link resources separately for different levels of latency-sensitive services. This prioritizes urgent services about to time out, avoiding the timeout risk for urgent services.

[0080] In one possible implementation, the service queue includes ordinary service queues; scheduling of the service queues in the heterogeneous converged network based on the link resources allocated to the service queues includes: The scheduling of ordinary service queues is based on the link resources allocated to them and a single-queue management mechanism; the single-queue management mechanism is as follows:

[0081] in, For the first Length of the time slot for normal services For the first The number of new regular service packets arriving in the time slot, For the first Time slots are allocated to ground base station link resources for ordinary services. For the first Time slots are allocated to satellite link resources for regular services.

[0082] In this embodiment, by scheduling and managing the ordinary service queue in this way, the packet loss rate of latency-sensitive services can be effectively reduced in high-load scenarios, while improving the throughput of ordinary services.

[0083] The method provided by the present invention will be verified through specific embodiments below. Figure 3 The chart shows a performance comparison of packet loss rate for latency-sensitive services obtained from simulations using the method of this invention. PPO-RL represents the method provided by this invention, FP is a fixed priority strategy, and WFQ is a weighted fair queuing strategy. In the low-load range (load factor 0.5-1.2), both PPO-RL and FP strategies maintain zero packet loss, while WFQ begins to experience packet loss from a load factor of 0.7, reaching a packet loss rate of 23.89% at a load factor of 1.0. This clearly demonstrates that a fixed allocation strategy that does not distinguish service priorities cannot meet the stringent QoS requirements of latency-sensitive services.

[0084] In the high-load range (load factor 1.3-2.0), the performance differences among the three strategies become more pronounced. When the load factor is 1.5, the packet loss rate of PPO-RL is 2.44%, the FP strategy is only 0.06%, while the WFQ strategy reaches a high of 45.27%. Although the FP strategy is slightly better than PPO-RL in terms of packet loss rate, this comes at the cost of BE service performance. PPO-RL achieves better overall performance while maintaining an acceptable low packet loss rate through intelligent scheduling.

[0085] Figure 4 This diagram illustrates the variation of BE service throughput with load, obtained through simulation of the method described in this invention. The WFQ strategy, by reserving fixed capacity for BE, maintains a stable throughput of approximately 4.0 packets / timeslot under various load conditions. The PPO-RL and FP strategies perform similarly under low load, but exhibit different trends as the load increases. Notably, at a load factor of 1.5, the BE throughput of PPO-RL (1.56 packets / timeslot) is 13.9% higher than that of the FP strategy (1.37 packets / timeslot). This indicates that PPO-RL has learned to more effectively utilize remaining resources to serve BE services while ensuring the QoS of DS services, demonstrating its advantage in intelligent resource allocation.

[0086] Figure 5 The simulation results show significant differences in latency performance among the three strategies obtained using the method of this invention. At a load factor of 1.0, the FP strategy achieves the lowest latency, PPO-RL maintains good low latency, while the WFQ strategy has a latency as high as 3 time slots, approaching the maximum tolerable latency limit. This result verifies the rationality of the design of this invention: the FP strategy obtains optimal latency through absolute priority but lacks flexibility; the WFQ strategy, due to its FIFO principle and lack of differentiation of urgency, leads to a backlog of a large number of urgent packets (d=0); while PPO-RL finds the optimal balance between latency and throughput through deep reinforcement learning.

[0087] Figure 6 The figure shows the convergence curve of the PPO-RL algorithm obtained from simulations of the method of this invention, with the vertical axis representing the cumulative reward for each training round. It can be seen that the cumulative reward steadily increases with training and stabilizes after approximately 50 rounds. This indicates that the algorithm used in this invention has good convergence and learning efficiency, and can learn an efficient scheduling strategy within a limited time.

[0088] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0089] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0090] Figure 7 The diagram illustrates the structure of a heterogeneous network backhaul link selection device based on reinforcement learning according to an embodiment of the present invention. For ease of explanation, only the parts relevant to the embodiment of the present invention are shown, and are described in detail below: like Figure 7 As shown, the reinforcement learning-based heterogeneous network backhaul link selection device 7 is applied to a heterogeneous fusion network, which includes satellite networks and terrestrial networks; the device includes: The acquisition module 71 is used to acquire the real-time length of the service queue in the heterogeneous converged network; The allocation module 72 is used to input the real-time length into the trained link optimization model to obtain the link resources allocated to the service queue; wherein, the link optimization model is trained by reinforcement learning with service packet loss, queuing delay and network throughput as optimization objectives. The scheduling module 73 is used to schedule the service queues of the heterogeneous converged network based on the link resources allocated to the service queues.

[0091] In one possible implementation, the allocation module 72 is also used for: Before inputting the real-time length into the trained link optimization model to obtain the link resources allocated to the service queue, the real-time length of the service queue in the heterogeneous fusion network is used as the state, and the link resources allocated to the service queue are used as the action. Different actions are executed for each state, forming multiple sets of state-action combinations. Record the next state corresponding to each state-action combination, and calculate the immediate reward value corresponding to the state-action combination based on a multi-objective reward function; wherein, the multi-objective reward function takes packet loss, queuing latency and network throughput as optimization objectives; Each state-action combination, along with the corresponding next state and immediate reward value, is combined into an experience tuple and stored in the experience replay buffer. Batch data is sampled from the experience replay buffer, and the initial link optimization model is trained by reinforcement learning based on the long-term optimization objective to obtain the trained link optimization model.

[0092] In one possible implementation, the multi-objective reward function is:

[0093] in, For instant reward value, The weight parameters are and satisfy the following conditions: , For the first The number of packet losses for time-slot latency-sensitive services. For the first There are still 10 days left until the timeout begins at the start of the time slot. The number of latency-sensitive service packets per time slot For the first Length of the time slot for normal services For the first Time slots are allocated to ground base station link resources for ordinary services. For the first Time slots are allocated to satellite link resources for regular services; The long-term optimization goal is:

[0094] in, To optimize the target value in the long term, For scheduling strategy, This is the discount factor.

[0095] In one possible implementation, the allocation module 72 is specifically used for: The service queue information of the heterogeneous converged network is used as the state, and the link resources allocated to the service queue are used as the action. Based on the available resource constraints, different actions are executed for each state, forming multiple sets of state-action combinations.

[0096] In one possible implementation, available resource constraints include: Link capacity constraints:

[0097]

[0098] Queue state constraints:

[0099]

[0100] Resource allocation non-negative integer constraint:

[0101] in, For ground base station link capacity, For satellite link capacity, For the first There are still 10 days left until the timeout begins at the start of the time slot. The number of latency-sensitive service packets per time slot , The maximum number of queuing slots. For the first Length of the time slot for normal services To be assigned to the Ground base station link resources in the tiered queue For the first Time slots are allocated to ground base station link resources for ordinary services. For the first Time slots are allocated to satellite link resources for regular services. It is the set of non-negative integers.

[0102] In one possible implementation, the service queue includes a latency-sensitive service queue; the scheduling module 73 is specifically used for: The scheduling of latency-sensitive service queues is based on the link resources allocated to them and a hierarchical queue scheduling mechanism. The hierarchical queue scheduling mechanism is as follows:

[0103]

[0104]

[0105] in, Indicates the first At the start of the time slot The number of latency-sensitive service packets in the first-level queue, the number of packets in the second-level queue. The queue has a timeout period of [number] days remaining. A time-sensitive service packet in a time slot, , The maximum number of queuing slots. For the first The number of newly arriving latency-sensitive service packets in the time slot. To be assigned to the Ground base station link resources in the tiered queue.

[0106] In one possible implementation, the service queue includes a regular service queue; the scheduling module 73 is specifically used for: The scheduling of ordinary service queues is based on the link resources allocated to them and a single-queue management mechanism; the single-queue management mechanism is as follows:

[0107] in, For the first Length of the time slot for normal services For the first The number of new regular service packets arriving in the time slot, For the first Time slots are allocated to ground base station link resources for ordinary services. For the first Time slots are allocated to satellite link resources for regular services.

[0108] This invention, through a link optimization model, comprehensively considers multiple optimization objectives such as service packet loss rate, queuing latency, and network throughput on the basis of heterogeneous converged networks. It allocates link resources to service queues of different lengths, which is applicable to backhaul selection of different types of service terminals in heterogeneous converged satellite and terrestrial network scenarios. Furthermore, it can divide service queues into latency-sensitive services and ordinary services, thereby improving the network throughput of ordinary services while meeting the queuing latency requirements of latency-sensitive services.

[0109] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. For example... Figure 8 As shown, the electronic device 8 of this embodiment includes a processor 80 and a memory 81. The memory 81 stores a computer program 82. When the processor 80 executes the computer program 82, it implements the steps in the various method embodiments described above. Alternatively, when the processor 80 executes the computer program 82, it implements the functions of each module / unit in the various device embodiments described above.

[0110] For example, computer program 82 may be divided into one or more modules / units, which are stored in memory 81 and executed by processor 80 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 82 in electronic device 8.

[0111] Electronic device 8 may include, but is not limited to, processor 80 and memory 81. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 8 may also include input / output devices, network access devices, buses, etc.

[0112] For the sake of simplicity and clarity, only the above-described functional modules / units are used as examples. In practical applications, the functions described above can be assigned to different functional modules / units as needed. These modules / units can be implemented in hardware, software, or a combination of both.

[0113] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0114] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A backhaul link selection method for heterogeneous networks based on reinforcement learning, characterized in that, Applied to heterogeneous converged networks, the heterogeneous converged networks including satellite networks and terrestrial networks; the method includes: Obtain the real-time length of the service queue in the heterogeneous converged network; The real-time length is input into the trained link optimization model to obtain the link resources allocated to the service queue; wherein, the link optimization model is obtained by reinforcement learning training with service packet loss, queuing delay and network throughput as optimization objectives; The service queues of the heterogeneous converged network are scheduled based on the link resources allocated to the service queues.

2. The backhaul link selection method for heterogeneous networks based on reinforcement learning according to claim 1, characterized in that, Before inputting the real-time length into the trained link optimization model to obtain the link resources allocated to the service queue, the method further includes: The real-time length of the service queue in the heterogeneous converged network is taken as the state, and the link resources allocated to the service queue are taken as the action. Different actions are executed for each state, forming multiple sets of state-action combinations. Record the next state corresponding to each state-action combination, and calculate the immediate reward value corresponding to the state-action combination based on a multi-objective reward function; wherein, the multi-objective reward function takes service packet loss, queuing delay and network throughput as optimization objectives; Each state-action combination, along with the corresponding next state and immediate reward value, is combined into an experience tuple and stored in the experience replay buffer. Batch data is sampled from the experience replay buffer, and the initial link optimization model is trained using reinforcement learning based on the long-term optimization objective to obtain the trained link optimization model.

3. The backhaul link selection method for heterogeneous networks based on reinforcement learning according to claim 2, characterized in that, The multi-objective reward function is: in, For instant reward value, The weight parameters are and satisfy the following conditions: , For the first The number of packet losses for time-slot latency-sensitive services. For the first There are still 10 days left until the timeout begins at the start of the time slot. The number of latency-sensitive service packets per time slot For the first Length of the time slot for normal services For the first Time slots are allocated to ground base station link resources for ordinary services. For the first Time slots are allocated to satellite link resources for regular services; The long-term optimization goal is: in, To optimize the target value in the long term, For scheduling strategy, This is the discount factor.

4. The backhaul link selection method for heterogeneous networks based on reinforcement learning according to claim 2, characterized in that, The process involves using the service queue information of the heterogeneous converged network as a state, and the link resources allocated to the service queue as actions. Different actions are executed for each state, forming multiple state-action combinations, including: The service queue information of the heterogeneous converged network is taken as the state, and the link resources allocated to the service queue are taken as the action. Based on the available resource constraints, different actions are executed for each state, forming multiple sets of state-action combinations.

5. The backhaul link selection method for heterogeneous networks based on reinforcement learning according to claim 4, characterized in that, The available resource constraints include: Link capacity constraints: Queue state constraints: Resource allocation non-negative integer constraint: in, For ground base station link capacity, For satellite link capacity, For the first There are still 10 days left until the timeout begins at the start of the time slot. The number of latency-sensitive service packets per time slot , The maximum number of queuing slots. For the first Length of the time slot for normal services To be assigned to the Ground base station link resources in the tiered queue For the first Time slots are allocated to ground base station link resources for ordinary services. For the first Time slots are allocated to satellite link resources for regular services. It is the set of non-negative integers.

6. The backhaul link selection method for heterogeneous networks based on reinforcement learning according to claim 1, characterized in that, The service queues include latency-sensitive service queues; the scheduling of service queues in the heterogeneous converged network based on the link resources allocated to the service queues includes: The latency-sensitive service queues are scheduled based on the link resources allocated to them and a hierarchical queue scheduling mechanism; the hierarchical queue scheduling mechanism is as follows: in, Indicates the first At the start of the time slot The number of latency-sensitive service packets in the first-level queue, the number of packets in the second-level queue. The queue has a timeout period of [number] days remaining. A time-sensitive service packet in a time slot, , The maximum number of queuing slots. For the first The number of newly arriving latency-sensitive service packets in the time slot. To be assigned to the Ground base station link resources in the tiered queue.

7. The backhaul link selection method for heterogeneous networks based on reinforcement learning according to claim 1, characterized in that, The service queues include ordinary service queues; the scheduling of service queues in the heterogeneous converged network based on the link resources allocated to the service queues includes: The scheduling of ordinary service queues is based on the link resources allocated to them and a single-queue management mechanism; the single-queue management mechanism is as follows: in, For the first Length of the time slot for normal services For the first The number of new regular service packets arriving in the time slot, For the first Time slots are allocated to ground base station link resources for ordinary services. For the first Time slots are allocated to satellite link resources for regular services.

8. A backhaul link selection device for heterogeneous networks based on reinforcement learning, characterized in that, Applied to heterogeneous converged networks, the heterogeneous converged networks including satellite networks and terrestrial networks; the device includes: The acquisition module is used to obtain the real-time length of the service queue in the heterogeneous converged network; The allocation module is used to input the real-time length into the trained link optimization model to obtain the link resources allocated to the service queue; wherein, the link optimization model is obtained by reinforcement learning training with service packet loss, queuing delay and network throughput as optimization objectives; The scheduling module is used to schedule the service queues of the heterogeneous converged network based on the link resources allocated to the service queues.

9. The backhaul link selection device for heterogeneous networks based on reinforcement learning according to claim 8, characterized in that, The allocation module is also used for: Before inputting the real-time length into the trained link optimization model to obtain the link resources allocated to the service queue, the real-time length of the service queue in the heterogeneous fusion network is taken as the state, and the link resources allocated to the service queue are taken as the action. Different actions are executed for each state, forming multiple sets of state-action combinations. Record the next state corresponding to each state-action combination, and calculate the immediate reward value corresponding to the state-action combination based on a multi-objective reward function; wherein, the multi-objective reward function takes service packet loss, queuing delay and network throughput as optimization objectives; Each state-action combination, along with the corresponding next state and immediate reward value, is combined into an experience tuple and stored in the experience replay buffer. Batch data is sampled from the experience replay buffer, and the initial link optimization model is trained using reinforcement learning based on the long-term optimization objective to obtain the trained link optimization model.

10. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

Images