Routing methods and devices

Abstract

This application provides a routing method and apparatus, relating to the field of communication technology, which can construct an efficient and reliable routing method to achieve reliable data transmission. The method includes: acquiring the load balancing rate of one or more neighboring nodes of a first node, and link state information between the first node and each neighboring node; determining a candidate node set including one or more second nodes based on the load balancing rate and link state information; and determining the next-hop node of the first node from the candidate node set by probing the candidate node set a preset number of times using a reinforcement learning model based on the link state information between the first node and each second node. The second node is a node among one or more neighboring nodes whose load balancing rate is greater than a first preset threshold and whose link state information meets the data transmission requirements.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to routing methods and apparatus. Background Technology

[0002] In a communication system, based on the traffic distribution of nodes, a traffic prediction model can be used to estimate the load of neighboring nodes of the current node, and the neighboring node with a lighter load can be selected as the next-hop node of the current node, thereby realizing signaling interaction.

[0003] However, due to the complexity and variability of the network environment, determining the next-hop node of the current node solely based on the load may result in low reliability of the determined next-hop node, affecting communication performance.

[0004] Therefore, how to construct an efficient and reliable routing method to achieve reliable data transmission has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a routing method and apparatus that can construct an efficient and reliable routing method to achieve reliable data transmission.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] Firstly, this application provides a routing method, which includes: obtaining the load balancing rate of one or more neighboring nodes of a first node and link state information between the first node and each neighboring node; determining a candidate node set including one or more second nodes based on the load balancing rate and link state information; and determining the next-hop node of the first node from the candidate node set by probing the candidate node set a preset number of times using a reinforcement learning model based on the link state information between the first node and each second node. The second node is a node among one or more neighboring nodes whose load balancing rate is less than or equal to a first preset threshold and whose link state information meets data transmission requirements.

[0008] In one possible implementation, the learning probability of the reinforcement learning model is determined based on the link state information; based on the learning probability, the reinforcement learning model probes the candidate node set a preset number of times.

[0009] In one possible implementation, the load balancing rate of adjacent nodes is determined by a traffic prediction model based on the historical traffic and current traffic of adjacent nodes.

[0010] In one possible implementation, an initial model is constructed; the initial model is used to process the traffic of adjacent nodes at N time points to obtain N first weights for each channel at N time points; based on the N first weights for each channel, a weighted average value for each channel is determined; based on the weighted average value, channels with a weighted average value less than a second preset threshold are deleted to obtain a traffic prediction model. The initial model includes multiple channels.

[0011] In one possible implementation, the initial model processes the traffic at time n to determine the second weight of each channel at time n; based on the traffic prediction accuracy of the initial model at time n and the average traffic prediction accuracy of the initial model over N times, the deviation value of the initial model at time n is determined; based on the N deviation values ​​over N times, the N second weights of each channel at N times are updated to obtain the N first weights of each channel at N times. Where 1 ≤ n ≤ N.

[0012] In one possible implementation, the second weight is determined based on the correlation of channel features; wherein the correlation includes one or more of the following: redundancy or complementarity; redundancy is used to indicate the degree of repetition of channel features of any two channels; complementarity is used to indicate the degree of complementarity of channel features of any two channels.

[0013] Secondly, this application provides a communication device that can be applied to a first node of the first aspect or any possible implementation of the first aspect to realize the functions performed by the first node. The communication device can be the first node, or it can be a chip, chip system, or system-on-a-chip of the first node, etc. The communication device can execute the functions performed by the first node through hardware, or it can execute corresponding software through hardware. The hardware or software includes one or more modules corresponding to the above functions, such as a transceiver module and a processing module.

[0014] The transceiver module is used to acquire the load balancing rate of one or more neighboring nodes of the first node and the link status information between the first node and each neighboring node; the processing module is used to determine a candidate node set based on the load balancing rate and the link status information; wherein, the candidate node set includes one or more second nodes, and the second node is a node among one or more neighboring nodes whose load balancing rate is less than or equal to a first preset threshold and whose link status information meets the data transmission requirements; the processing module is also used to determine the next-hop node of the first node from the candidate node set by probing the candidate node set a preset number of times through a reinforcement learning model based on the link status information between the first node and each second node.

[0015] In one possible implementation, the processing module is further configured to determine the learning probability of the reinforcement learning model based on the link state information; the processing module is further configured to probe the candidate node set a preset number of times based on the learning probability using the reinforcement learning model.

[0016] In one possible implementation, the processing module is also used to determine the load balancing rate of adjacent nodes based on the historical traffic and current traffic of adjacent nodes through a traffic prediction model.

[0017] In one possible implementation, the processing module is further configured to construct an initial model, wherein the initial model includes multiple channels; the processing module is further configured to process the traffic of the first node at N time points using the initial model to obtain N first weights for each channel at N time points; the processing module is further configured to determine the weighted average value of each channel based on the N first weights of each channel; the processing module is further configured to delete channels whose weighted average value is less than a second preset threshold based on the weighted average value to obtain the traffic prediction model.

[0018] In one possible implementation, the processing module is further configured to process the traffic at time n using the initial model to determine the second weight of each channel at time n; where 1 ≤ n ≤ N; the processing module is further configured to determine the deviation value of the initial model at time n based on the traffic prediction accuracy of the initial model at time n and the average traffic prediction accuracy of the initial model over N times; the processing module is further configured to update the N second weights of each channel at N times based on the N deviation values ​​over N times, thereby obtaining the N first weights of each channel at N times.

[0019] In one possible implementation, the second weight is determined based on the correlation of channel features; wherein the correlation includes one or more of the following: redundancy or complementarity; redundancy is used to indicate the degree of repetition of channel features of any two channels; complementarity is used to indicate the degree of complementarity of channel features of any two channels.

[0020] Thirdly, this application provides a communication device comprising: a processor and a communication interface; the communication interface and the processor are coupled, the processor being configured to execute computer programs or instructions to implement the routing method described in the first aspect or any possible implementation thereof.

[0021] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the routing method described in the first aspect or any possible implementation thereof.

[0022] Fifthly, this application provides a communication system including the communication apparatus as described in the second aspect or any possible implementation thereof.

[0023] In this application, the load status of adjacent nodes can be determined by obtaining their load balancing rates, and the impact of external factors on the link status can be determined by obtaining the link status information between the first node and its adjacent nodes. By selecting nodes whose load balancing rates are less than or equal to a first preset threshold and whose link status information meets the data transmission requirements, a candidate node set can be determined, which can narrow down the range of next-hop nodes, reduce resource consumption, and effectively improve the efficiency of next-hop node selection. By considering the link status information between the first node and the second node, the impact of external factors on the link status can be taken into account during the detection process, improving the reliability of the detection results. At the same time, by using a reinforcement learning model to detect the candidate node set a preset number of times, the computational complexity can be reduced and the speed of next-hop node selection can be improved. Attached Figure Description

[0024] Figure 1 A schematic diagram of a communication system provided in an embodiment of this application;

[0025] Figure 2 A schematic diagram of a communication device provided in an embodiment of this application;

[0026] Figure 3 A flowchart illustrating a routing method provided in an embodiment of this application;

[0027] Figure 4 A schematic diagram illustrating the calculation of the second weight, provided as an embodiment of this application;

[0028] Figure 5 A schematic diagram of a process for calculating a first weight provided in an embodiment of this application;

[0029] Figure 6 A schematic diagram illustrating the calculation of a first weight, provided as an embodiment of this application;

[0030] Figure 7 A schematic diagram of a communication device provided in an embodiment of this application;

[0031] Figure 8 This is a schematic diagram of a communication device provided in an embodiment of this application. Detailed Implementation

[0032] Before describing the embodiments of this application, the technical terms involved in the embodiments of this application will be described.

[0033] 1) Routing strategy: A flexible routing and forwarding mechanism based on the access network.

[0034] 2) Reinforcement learning model: It can consist of learner or agent, environment, and action.

[0035] In this context, the strategist is the main agent in reinforcement learning. In this application, the strategist can be one or more static agents.

[0036] Among them, the environment refers to the load balancing status of adjacent nodes that can be perceived (load balancing status can be link utilization, packet delivery rate, and / or remaining bandwidth, etc.) and the quality of service (QoS) of user services. QoS can be latency, bandwidth, and / or packet loss rate, etc.

[0037] In this application, the selection strategy can be the next-hop node selection strategy; it can also refer to the system's strategy set at time t, which can be represented as: No restrictions will be imposed.

[0038] Optionally, reinforcement learning models may also include a reward and a state.

[0039] The reward can be the reward obtained for successfully accessing the next-hop node and meeting the QoS requirements of the basic user services. In this application, the reward can be the average throughput of the next-hop node selected by the policymaker when meeting the QoS of the current user services, which can be expressed as: .

[0040] Wherein, state represents the state of the communication system at the current time t. In this application, state... It can be represented as: .in, Let M represent the k-th strategist, and M be the set of candidate nodes. This represents the latency, bandwidth, or packet loss rate of the service forwarded by the m-th node in the candidate node set. The probability of assigning available nodes.

[0041] For example, when the first node has three second nodes that can forward data, The value can be 1 / 3.

[0042] With the deepening research on integrated space-air-ground networks, the communication of low Earth orbit (LEO) satellite networks has attracted many researchers to study it due to its advantages such as wide coverage, flexible networking, no geographical restrictions (not limited by wireless coverage in environments such as oceans and forests), simple access (direct access by satellite terminals, access by mobile terminals through corresponding satellites, or access by gateway stations), and low link loss.

[0043] However, LEO satellite networks have disadvantages such as short orbital periods, frequent link switching, and highly dynamic topology changes. Furthermore, satellite links are susceptible to external factors (or external environment, such as rain attenuation), which can easily cause link degradation.

[0044] Given the above, LEO satellite networks are prone to packet loss or congestion. Even with pre-built reliable routing strategies, data cannot be reliably forwarded due to the deterioration of current links. Furthermore, QoS metrics for user services (such as latency, bandwidth, and packet loss rate) and node load balancing status (such as link utilization, packet delivery rate, and remaining bandwidth) must be considered to ensure that routing strategies meet the needs of user services.

[0045] Therefore, to ensure the reliability of communication in a communication system, traffic prediction models can be used to predict traffic based on the distribution of traffic data, and reliable routing strategies can be selected based on the traffic predictions.

[0046] For example, traffic prediction can be performed using regression analysis models, Markov models, or deep learning models.

[0047] However, regression analysis cannot effectively reflect the correlation and self-similarity characteristics of traffic flow data distribution, nor can it effectively handle abrupt changes in traffic flow data distribution. Markov models assume that the distribution of traffic flow data is only related to the current traffic flow data and is unrelated to the traffic flow data at historical moments. This stochastic process without aftereffects fails to adequately characterize the complex self-similarity and nonlinear characteristics of traffic flow data distribution. While deep learning models can characterize the self-similarity and nonlinear characteristics of traffic flow data distribution to some extent, their computational complexity is high due to the need for a large amount of historical traffic flow data for training, making them unsuitable for realizing dynamic changes in traffic flow prediction models.

[0048] For example, when the traffic distribution in a certain region has a certain degree of randomness and suddenness, if the model remains unchanged, updates very slowly, or has high computational complexity, it will not be able to meet the needs of limited computational resources.

[0049] In addition, most routing strategies only consider the load balancing status and the QoS of user services. Once a node is affected by external factors, it cannot more accurately determine the link status based on the load balancing status and the QoS of user services.

[0050] In conclusion, how to construct an efficient and reliable routing method to achieve reliable data transmission has become an urgent technical problem to be solved.

[0051] To address the aforementioned issues, this application provides a routing method comprising: acquiring the load balancing rate of one or more neighboring nodes of a first node and link state information between the first node and each neighboring node; determining a candidate node set based on the load balancing rate and link state information; and probing the candidate node set a preset number of times using a reinforcement learning model based on the link state information between the first node and each second node, thereby determining the next-hop node of the first node from the candidate node set. The candidate node set includes one or more second nodes; each second node is a node among the one or more neighboring nodes whose load balancing rate is less than or equal to a first preset threshold and whose link state information meets data transmission requirements.

[0052] In this embodiment, by obtaining the load balancing rate of adjacent nodes, the load status of adjacent nodes can be determined. By obtaining the link status information between the first node and adjacent nodes, the impact of external factors on the link status can be determined. By selecting nodes whose load balancing rate is less than or equal to a first preset threshold and whose link status information meets the data transmission requirements, a candidate node set can be determined, which can narrow the range of next-hop nodes, reduce resource consumption, and effectively improve the efficiency of next-hop node selection. By considering the link status information between the first node and the second node, the impact of external factors on the link status can be taken into account during the detection process, improving the reliability of the detection results. At the same time, by using a reinforcement learning model to detect the candidate node set a preset number of times, the computational complexity can be reduced and the speed of next-hop node selection can be improved.

[0053] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0054] by Figure 1 Taking an example, the communication system provided in the embodiments of this application will be described.

[0055] A communication system may include one or more transmitting devices, one or more node devices, and one or more receiving devices, without limitation.

[0056] For example, as follows Figure 1 As shown, taking communication between the sending device and the receiving device as an example, when the sending device and the receiving device communicate, the data can be forwarded through one or more node devices.

[0057] Optionally, the sending device can be a terminal device or a network device, without restriction.

[0058] Optionally, the receiving device can be a terminal device or a network device, without restriction.

[0059] Optionally, the node device can be a terminal device or a network device, without restriction.

[0060] In this application embodiment, the terminal device may be a device with wireless transceiver function or a chip or chip system that can be set in the device, which can allow users to access the network and is a device used to provide voice and / or data connectivity to users. The terminal device may also be referred to as user equipment (UE), user unit (subscriber unit), terminal, mobile station (MS), or mobile terminal (MT), etc.

[0061] Optionally, the terminal device in this application embodiment can be a user-side device used to implement wireless communication functions, such as a terminal or a chip that can be used in the terminal. The terminal device can be user equipment (UE), access terminal, terminal unit, terminal station, mobile station, mobile station, remote station, remote terminal, mobile device, wireless communication device, terminal agent, or terminal apparatus in a fifth-generation mobile communication network or a public land mobile network (PLMN) evolved after 5G. Access terminals can be cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, drones, robots, point-of-sale (POS) machines, customer-premises equipment (CPE) or wearable devices, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical care, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. Alternatively, a terminal can be a communication-enabled terminal in the Internet of Things (IoT), such as a terminal in vehicle-to-everything (V2X) communication (e.g., vehicle-to-everything devices), a terminal in device-to-device (D2D) communication, or a terminal in machine-to-machine (M2M) communication. Terminals can be mobile or fixed.

[0062] In this application embodiment, the network device can be any device deployed in the access network capable of wireless communication with terminal devices. It can also be a chip or chip system that can be configured in the aforementioned device, a logical node or logical module, or a function implemented in software. It can be used to implement functions such as wireless physical control, resource scheduling and wireless resource management, wireless access control, and mobility management. Specifically, the network device can be a device that supports wired access or a device that supports wireless access.

[0063] Optionally, the network device in this application embodiment is a device that connects a terminal device to a wireless network. The network device may be a node in a radio access network (RAN), or it may be a base station, and may be referred to as a radio access network node (or device).

[0064] For example, network equipment may include evolved base stations (NodeBs, eNBs, or e-NodeBs) in long-term evolution technology (LTE) systems or enhanced LTE (LTE-Advanced, LTE-A) systems, such as traditional macro base stations (eNBs) and micro base stations (eNBs) in heterogeneous network scenarios. Alternatively, it may include next-generation node Bs (gNBs) in new radio (NR) systems. Alternatively, it may include transmission reception points (TRPs), home base stations (e.g., home evolved NodeBs, or home Node Bs, HNBs), base band units (BBUs), base band pools (BBU pools), or wireless fidelity (Wi-Fi) access points (APs), etc. Alternatively, it can include base stations in non-terrestrial networks (NTNs), which can be deployed on flying platforms or satellites. In NTNs, network devices can act as Layer 1 (L1) relays, base stations, or integrated access and backhaul (IAB) nodes. Alternatively, network devices can be devices that implement base station functions in IoT, such as those in drone communication, V2X, D2D, or M2M.

[0065] For example, the network device can be a satellite deployed in the LEO satellite network. When the transmitting end device and the receiving end device communicate, they can select one or more satellites for data forwarding. During data transmission, when each device needs to determine the next hop node, it can refer to the routing method provided in this application to determine the next hop node.

[0066] Network equipment can also be modules or units capable of performing some of the functions of a base station. For example, network equipment can be a central unit (CU), a distributed unit (DU), a CU-control plane (CP), a CU-user plane (UP), or a radio unit (RU). CU and DU can be set up separately or included in the same network element, such as in a baseband unit (BBU). RU can be included in radio frequency equipment or radio frequency units, such as in a remote radio unit (RRU), an active antenna unit (AAU), or a remote radio head (RRH).

[0067] In different systems, CU (or CU-CP and CU-UP), DU, or RU may have different names, but those skilled in the art will understand their meaning. For example, a network device can be a network device or a module of a network device in an Open Radio Access Network (ORAN) system. In an ORAN system, CU can also be called open (O)-CU, DU can also be called O-DU, CU-CP can also be called O-CU-CP, CU-UP can also be called O-CU-UP, and RU can also be called O-RU. Any of the units among CU (or CU-CP, CU-UP), DU, and RU in this application can be implemented through a software module, a hardware module, or a combination of software and hardware modules.

[0068] Optionally, the base station in this application embodiment may include various forms of base stations, such as: macro base station, micro base station (also known as small station), relay station, access point, home base station, TRP, transmitting point (TP), or mobile switching center, etc. This application embodiment does not specifically limit these.

[0069] In practical implementation, Figure 1 As shown: the transmitting device, node device, and receiving device can adopt... Figure 2 The shown composition structure, or including Figure 2The components shown. Figure 2 This application provides a schematic diagram of the composition of a communication device 20, which can be object storage or a chip or system-on-a-chip within object storage; it can also be a chip or system-on-a-chip of a transceiver module or a processing module. For example... Figure 2 As shown, the communication device 20 includes a processor 201, a transceiver 202, and a communication line 203.

[0070] Furthermore, the communication device 20 may also include a memory 204. The processor 201, memory 204, and transceiver 202 can be connected via a communication line 203.

[0071] The processor 201 can be a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 201 can also be other devices with processing capabilities, such as circuits, devices, or software modules, without limitation.

[0072] Transceiver 202 is used to communicate with other devices or other communication networks. These other communication networks can be Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc. Transceiver 202 can be a module, circuit, transceiver, or any device capable of enabling communication.

[0073] Communication line 203 is used to transmit information between the components included in communication device 20.

[0074] Memory 204 is used to store instructions. These instructions can be computer programs.

[0075] The memory 204 can be a read-only memory (ROM) or other type of static storage device that can store static information and / or instructions; it can also be a random access memory (RAM) or other type of dynamic storage device that can store information and / or instructions; it can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, etc., without limitation.

[0076] It should be noted that the memory 204 can exist independently of the processor 201 or can be integrated with the processor 201. The memory 204 can be used to store instructions, program code, or some data, etc. The memory 204 can be located inside or outside the communication device 20, without limitation. The processor 201 is used to execute the instructions stored in the memory 204 to implement the routing method provided in the following embodiments of this application.

[0077] In one example, processor 201 may include one or more CPUs, for example Figure 2 CPU0 and CPU1 in the CPU.

[0078] As an optional implementation, the communication device 20 includes multiple processors, for example, besides Figure 2 In addition to processor 201, it may also include processor 207.

[0079] As an optional implementation, the communication device 20 also includes an output device 205 and an input device 206. For example, the input device 206 is a device such as a keyboard, mouse, microphone, or joystick, and the output device 205 is a device such as a display screen or speaker.

[0080] It should be noted that the communication device 20 can be a desktop computer, laptop computer, network server, mobile phone, tablet computer, wireless terminal, embedded device, chip system, or other device. Figure 2 Equipment with a similar structure. Furthermore... Figure 2 The structural composition shown does not constitute a limitation on the communication device, except... Figure 2 In addition to the components shown, the communication device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0081] In this embodiment of the application, the chip system may be composed of chips or may include chips and other discrete devices.

[0082] Furthermore, the actions, terms, etc., involved in the various embodiments of this application can be referenced interchangeably without limitation. The message names or parameter names in the messages exchanged between the various devices in the embodiments of this application are merely examples, and other names may be used in specific implementations without limitation.

[0083] based on Figure 1 The communication system shown is referenced. Figure 3 This application describes a routing method provided in an embodiment. The processing performed by a single execution entity (sending device, node device, and receiving device) shown in the embodiment can also be divided into multiple execution entities, which can be logically and / or physically separated without limitation.

[0084] Figure 3 This is a schematic diagram of a routing method provided in an embodiment of this application, such as... Figure 3 As shown, the method may include:

[0085] S301, The first node obtains the load balancing rate of one or more adjacent nodes of the first node, and the link status information between the first node and the adjacent nodes.

[0086] The first node can be a sending device or a node device, without restriction.

[0087] Optionally, the load balancing rate of one or more neighboring nodes of the first node can be determined by using a traffic prediction model based on the historical traffic and current traffic of neighboring nodes.

[0088] Optionally, link state information may include one or more of the following: latency, bandwidth, or packet loss rate, without limitation.

[0089] S302. The first node determines the set of candidate nodes based on the load balancing rate and link status information.

[0090] The candidate node set may include one or more second nodes.

[0091] The second node can be one or more adjacent nodes whose load balancing rate is less than or equal to the first preset threshold and whose link status information meets the data transmission requirements.

[0092] Optionally, a dynamic proxy based on a traffic prediction model can be used to obtain the load balancing rate of one or more adjacent nodes of the first node.

[0093] Optionally, when the load balancing rate of a neighboring node of the first node is less than or equal to a first preset threshold, and the link state information of the first node and the neighboring node meets the data transmission requirements, the neighboring node can be included in the candidate node set.

[0094] The value of the first preset threshold can be determined based on the actual communication scenario or actual communication needs.

[0095] For example, the first preset threshold can be set between 70% and 80%.

[0096] For example, taking a first preset threshold of 75% as an example, when the load balancing rate of a neighboring node of the first node is less than or equal to 75%, and the link status information of the neighboring node meets the data transmission requirements, the neighboring node is included in the candidate node set; or, when the load balancing rate of a neighboring node of the first node is greater than 75%, the neighboring node is not included in the candidate node set.

[0097] Optionally, data transmission requirements can be based on the QoS requirements of the user's services.

[0098] Based on the above description of constructing the candidate node set, this application proposes several possible implementations:

[0099] In a first possible embodiment, when the load balancing rate of the neighboring nodes of the first node is greater than a first preset threshold, the neighboring node is not included in the candidate node set.

[0100] In a second possible embodiment, when the load balancing rate of the neighboring nodes of the first node is less than or equal to the first preset threshold, the neighboring node can interact with the first node through signaling to obtain the link status information between the neighboring node and the first node. If the link status information meets the data transmission requirements, the neighboring node is included in the candidate node set.

[0101] In a third possible embodiment, when the load balancing rate of the neighboring nodes of the first node is less than or equal to the first preset threshold, the neighboring node can interact with the first node through signaling to obtain the link status information between the neighboring node and the first node. If the link status information does not meet the data transmission requirements, the neighboring node will not be included in the candidate node set.

[0102] S303. Based on the link status information between the first node and the second node, the first node uses a reinforcement learning model to probe the candidate node set a preset number of times, and determines the next hop node of the first node from the candidate node set.

[0103] Optionally, the learning probability of the reinforcement learning model can be determined based on the link state information between the first node and the second node.

[0104] The learning probability represents the degree of influence of external factors and takes a value between 0 and 1.

[0105] For example, the learning probability can be set between 0.01 and 0.1.

[0106] Optionally, the candidate node set can be probed a preset number of times using a reinforcement learning model based on the learning probability.

[0107] The preset number of times can be determined based on the actual communication scenario or actual communication needs.

[0108] Alternatively, reinforcement learning algorithms mainly include two types: policy space search and value function estimation.

[0109] Among them, policy space search is suitable for situations where environmental changes are relatively small, while the core of value function estimation is to estimate the environmental state.

[0110] Since the load balancing state of the nodes in this application is a dynamic variable, the value function estimation method can be used to select the next-hop node in this application.

[0111] Based on the above description of selecting the next-hop node from the candidate node set and the reinforcement learning algorithm, this application provides specific details by proposing the following possible embodiments:

[0112] In one possible implementation, based on value function estimation, the detection process of the static agent k in the reinforcement learning model can be represented as: .

[0113] Among them, S k S represents the state of static agent (a static agent can also be a strategist) k at time t. k . represents the state of static agent k at time t+1. This represents the strategy of static agent k at time t (the strategy can be viewed as the selection of the second node). Let M represent the strategy of the static proxy at time t+1, and M represent the number of second nodes in the candidate node set. Let β represent the learning probability, and let β represent the influence coefficient of the policy choice at the previous time step on the policy choice at the next time step.

[0114] in, This can be represented as predicting the strategy choice and state in the next time step from the previous time step. This is a prediction of the Q value from the previous time step to the next time step.

[0115] Optionally, based on the above detection process, each static agent in state Choose strategy At that time, the Q value can be obtained: (This can also be called Q-estimation, because the Q-value obtained during this probing process may be correct or incorrect.) A reward (also known as throughput) can be obtained for each probe: Correspondingly, when multiple static proxies perform a single probe, the average throughput can be obtained. .

[0116] In the exploration process, the larger the Q value, the stronger the correlation with the reward, and the better it can illustrate the benefits of choosing a certain strategy in a certain state; correspondingly, the larger the Q value, the greater the reward obtained.

[0117] Optionally, the initial selection strategy can be random assignment. During the detection process, detection is performed based on the target with the largest Q value. The second node with the largest Q value is obtained by iteratively combining the learning probability, and this second node is used as the next hop node.

[0118] For example, based on a preset number of probes, multiple static proxies probe the second node in the candidate node set. During the probe process, the learning probability is determined based on the link state information. After each probe, the average throughput is obtained, and the Q value is updated. The next probe can be performed based on the average throughput. After a preset number of probes, the second node with the highest Q value can be obtained, and this second node can be used as the next hop node.

[0119] In another possible embodiment, the static proxy can also probe the set of candidate nodes and select the next-hop node without being limited by a preset number of times.

[0120] For example, the average network throughput at time t (which can be from time K to time K+N) can be expressed as: .

[0121] in, The number of probes is the number of times the static agent of the reinforcement learning model probes neighboring node j. The second probe (the probe ends when the predicted state matches the current state); The average throughput obtained when the static agent of the reinforcement learning model probes neighboring node j.

[0122] Optional, The static proxy of the reinforcement learning model can obtain the load balancing status of adjacent nodes and the QoS of user services.

[0123] Optionally, the network throughput maximization target can be obtained based on the network's average throughput.

[0124] For example, the throughput maximization objective can be expressed as: .

[0125] Where M represents the number of second nodes in the candidate node set.

[0126] Optionally, based on the goal of maximizing throughput, a next-hop node can be selected to enable data transmission.

[0127] Based on the above Figure 3 The method described herein determines the load status of adjacent nodes by obtaining their load balancing rates. It also determines the impact of external factors on the link status by obtaining the link status information between the first node and its adjacent nodes. By selecting nodes whose load balancing rates are less than or equal to a first preset threshold and whose link status information meets data transmission requirements, a candidate node set is determined, narrowing the range of next-hop nodes, reducing resource consumption, and effectively improving the efficiency of next-hop node selection. Furthermore, by considering the link status information between the first and second nodes, the impact of external factors on the link status can be taken into account during the detection process, improving the reliability of the detection results. Simultaneously, by using a reinforcement learning model to probe the candidate node set a preset number of times, computational complexity can be reduced, increasing the speed of next-hop node selection.

[0128] Optionally, based on the above description of the method for selecting the next-hop node, the routing method may further include: the first node constructs an initial model; the initial model is used to process the traffic of adjacent nodes at N time points to obtain N first weights for each channel at N time points; based on the N first weights for each channel, the weighted average value of each channel is determined; based on the weighted average value, channels with a weighted average value less than a second preset threshold are deleted to obtain a first predicted traffic model.

[0129] The initial model may include multiple channels.

[0130] Optionally, an initial model can be built based on ensemble learning methods.

[0131] Optionally, traffic data from adjacent nodes can be input into the initial model.

[0132] For example, as follows Figure 4 As shown, the flow data from time K to time K+N is input into the initial model.

[0133] For example, taking the traffic data at time K as an example, the traffic data can be divided into N convolutional layers of features through convolution, and the features of the N convolutional layers can be pooled to obtain channel features.

[0134] In this system, channel features are connected to neurons in the hidden layer via channels.

[0135] Optionally, based on the processing of traffic data at N time points using the initial model, N first weights for each channel at N time points can be obtained.

[0136] Optionally, the process of obtaining the first weight can be as follows: Figure 5 As shown, the steps may include:

[0137] S501. The first node processes the traffic at time n using the initial model to determine the second weight of each channel at time n.

[0138] Where n is greater than or equal to 1 and less than or equal to N.

[0139] Optionally, the second weight can be determined based on the correlation of the channel features.

[0140] The correlation includes one or more of the following: redundancy or complementarity.

[0141] Redundancy is used to indicate the degree of repetition of channel features between two channels; complementarity is used to indicate the degree of complementarity of channel features between two channels.

[0142] For example, when the redundancy between two channels is high, one of the two channels can be deleted at will; when the complementarity between two channels is high, neither channel can be deleted.

[0143] Optionally, the redundancy between two channels can be obtained through the mutual information of the features of the two channels. When the redundancy is greater than the preset redundancy threshold, it can be said that there are many duplicate channel features between the two channels, and one of the two channels can be deleted at will.

[0144] The preset redundancy threshold can be determined by... express, The value can be determined based on the actual communication scenario or actual communication needs.

[0145] For example, the redundancy between channel i and channel k can be expressed as: .when Greater than When there are many overlapping channel features between channel i and channel k, you can either keep channel i and delete channel k, or keep channel k and delete channel i; Less than or equal to At this time, there are fewer overlapping channel features between channel i and channel k, so neither channel i nor channel k can be deleted.

[0146] Where H represents the channel set of the convolutional layer after pooling in the l-th layer. The value represents the channel feature of the convolutional layer after pooling at layer l, where i and k represent channel i and channel k, respectively.

[0147] For example, The value can be 80%, when When the value is greater than 80%, there are many overlapping channel features between channel i and channel k. Channel i can be retained and channel k can be deleted.

[0148] To further measure To determine whether traffic forecasting is truly helpful, the complementarity between the two channels can also be considered.

[0149] Optionally, complementarity can be obtained through the mutual information of the channel features of the two channels and the mutual information between the channel features and the neurons in the hidden layer. When the complementarity is greater than a preset complementarity threshold, the channel features of the two channels are complementary; or, when the complementarity is less than or equal to the preset complementarity threshold, the channel features of the two channels are not complementary.

[0150] The preset complementarity threshold can be represented by θ, and the value of θ can be determined according to the actual communication scenario or actual communication needs.

[0151] For example, the complementarity between channel i and channel k can be specifically expressed as follows: .when When the value is greater than θ, the two channels are complementary, with fewer overlapping channel features between channel i and channel k, and neither channel i nor channel k can be deleted; when When the value of θ is less than or equal to θ, the two channels are not complementary. There are many overlapping channel features between channel i and channel k. You can keep channel i and delete channel k, or keep channel k and delete channel i.

[0152] Where Y represents a neuron in the hidden layer.

[0153] For example, the value of θ can be between 20% and 40%. Taking a value of θ of 30% as an example... When the percentage is less than 30%, there are many overlapping features between channel i and channel k, so channel i should be retained and channel k should be deleted. When the percentage is greater than 30%, retain channel i and channel k.

[0154] Optionally, you can The value is greater than as well as The second weight of channels whose values ​​are less than or equal to θ is set to 0, and the second weight of the remaining channels is set to 1 / P; or, the second weight of the channels can be set to 1 / P. The value is greater than The second weight of the channel is set to 0, and the second weight of the remaining channels is set to 1 / P; or, the second weight of the channel can be set to 1 / P. The second weight of channels whose values ​​are less than or equal to θ is set to 0, and the second weight of the remaining channels is set to 1 / P without restriction.

[0155] Where P represents the total number of channels in the convolutional layer after pooling in the l-th layer.

[0156] Based on the above description of the correlation between channels, this application proposes several possible embodiments:

[0157] In the first possible embodiment, when the redundancy between channel i and channel k is greater than Furthermore, when the complementarity between channel i and channel j is less than θ, the second weight of channel j is set to 0, and the second weight of channel i is set to 1 / P; or, when the redundancy between channel i and channel j is less than θ... Furthermore, when the complementarity between channel i and channel j is greater than θ, the second weights of channel i and channel j are both set to 1 / P.

[0158] In the second possible embodiment, when the redundancy between channel i and channel k is greater than Set the second weight of channel j to 0 and the second weight of channel i to 1 / P; or, when the redundancy between channel i and channel j is less than 0. Set the second weights of channel i and channel j to 1 / P.

[0159] In a third possible embodiment, when the complementarity between channel i and channel k is less than θ, the second weight of channel j is set to 0, and the second weight of channel i is set to 1 / P; or, when the complementarity between channel i and channel j is greater than θ, the second weights of channel i and channel j are both set to 1 / P.

[0160] Optionally, since the traffic data of the channels is constantly changing, the second weight of each channel is not static and can be updated as the traffic data of the channels changes.

[0161] S502. The first node determines the deviation value of the initial model at time n based on the traffic prediction accuracy of the initial model at time n and the average traffic prediction accuracy of the initial model over N times.

[0162] Where 1≤n≤N.

[0163] Optionally, the accuracy of traffic prediction can be calculated using "1 - error".

[0164] The error is the ratio of the number of data transmission errors to the total number of data transmissions during the communication process.

[0165] For example, as follows Figure 6 As shown, the deviation values ​​at N time points can be obtained.

[0166] For example, you can use Let represent the deviation value. Then, the deviation value at time n can be expressed as: .

[0167] in, , The accuracy of traffic flow prediction at time n can be used. It can predict the accuracy of average flow over N time points.

[0168] S503. The first node updates the N second weights of each channel at N time points based on the N deviation values ​​at N time points, thereby obtaining the N first weights of each channel at N time points.

[0169] For example, the first weight of channel i at time n can be expressed as: ,but, .

[0170] in, This represents the second weight of channel i at time n.

[0171] Based on the above description of obtaining the first weight, the weighted average of channel i can be determined by calculating the first weight of channel i at all times; correspondingly, the weighted average of all channels can be obtained using the same method.

[0172] For example, the following Figure 6 For example, the second weight of channel i at time K+1 is Then the first weight of channel i at time K+1 is The second weight of channel i at time k+2 is Then the first weight of channel i at time k+2 is Similarly, the second weight of channel i at time K+N is... Then the first weight of channel i at time K+N is Based on the first weight of channel i at all times, the weighted average of the i-th channel can be calculated: .

[0173] Optionally, channels with a weighted average value less than a second preset threshold can be deleted based on the weighted average value to obtain a traffic prediction model.

[0174] For example, with the second preset threshold set to 0.5, when the first weight of channel i is less than 0.5, channel i is deleted; when the first weight of channel i is greater than or equal to 0.5, channel i is retained.

[0175] It should be noted that the various methods provided in the embodiments of this application can be implemented individually or in combination, and there is no limitation.

[0176] This application embodiment can divide each device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0177] When dividing each function into modules according to its corresponding function. Figure 7 A communication device is shown that can perform actions performed by a transmitting device, or by a receiving device, or by a node device.

[0178] The communication device 70 may include a transceiver module 701 and a processing module 702. Exemplarily, the communication device 70 may be a communication equipment, or a chip or other combination device or component having the aforementioned communication device functions applied within a communication equipment. When the communication device 70 is a communication equipment, the transceiver module 701 may be a transceiver, which may include an antenna and radio frequency circuitry; the processing module 702 may be a processor (or processing circuitry), such as a baseband processor, which may include one or more CPUs. When the communication device 70 is a component having the aforementioned communication device functions, the transceiver module 701 may be a radio frequency unit; the processing module 702 may be a processor (or processing circuitry), such as a baseband processor. When the communication device 70 is a chip system, the transceiver module 701 may be an input / output interface of a chip (e.g., a baseband chip); the processing module 702 may be a processor (or processing circuitry) of the chip system, and may include one or more central processing units. It should be understood that the transceiver module 701 in the embodiments of this application can be implemented by a transceiver or transceiver-related circuit components; the processing module 702 can be implemented by a processor or processor-related circuit components (or, referred to as processing circuit).

[0179] For example, the transceiver module 701 can be used to perform... Figures 3 to 6 In the illustrated embodiment, all transmit and receive operations performed by the communication device, and / or other processes used to support the technology described herein; the processing module 702 can be used to perform Figures 3 to 6 The embodiments shown include all operations performed by the communication device other than the transmit and receive operations, and / or other processes used to support the techniques described herein.

[0180] As another feasible approach Figure 7 The transceiver module 701 can be replaced by a transceiver unit, which can integrate the functions of the transceiver module 701; the processing module 702 can be replaced by a processor, which can integrate the functions of the processing module 702. Furthermore, Figure 7 The communication device 70 shown may also include a memory.

[0181] Alternatively, when the processing module 702 is replaced by a processor and the transceiver module 701 is replaced by a transceiver, the communication device 70 involved in the embodiments of this application can also be... Figure 8 The communication device 80 shown includes a processor, which can be logic circuit 801, and a transceiver, which can be interface circuit 802. Further, Figure 8 The communication device 80 shown may also include a memory 803.

[0182] This application also provides a computer program product that, when executed by a computer, can implement the functions of any of the above method embodiments.

[0183] This application also provides a computer program that, when executed by a computer, can implement the functions of any of the above method embodiments.

[0184] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be implemented by a computer program instructing related hardware. This program can be stored in the computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be an internal storage unit of the terminal (including a data sending end and / or a data receiving end) of any of the foregoing embodiments, such as the terminal's hard disk or memory. The computer-readable storage medium can also be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal. Further, the computer-readable storage medium can include both the terminal's internal storage unit and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0185] It should be noted that the terms "first" and "second" in the description, claims and drawings of this application are used to distinguish different objects, rather than to describe a specific order. "First" and "second" are only used for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of this embodiment, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0186] In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that comprises a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0187] It should be understood that in this application, "at least one (item)" means one or more. "A plurality of" means two or more. "At least two (items)" means two or three or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist simultaneously. Here, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (item) below" or its similar expressions refer to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b or c, or at least one (item) of a, b and c, can mean: a, b, c, "a and b", "a and c", "b and c" or "a and b and c", where a, b and c can be single or multiple. "When..." and "if" both refer to corresponding processing under certain objective circumstances, do not limit time, do not require a judgment action during implementation, and do not imply other limitations.

[0188] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or implementation described as "exemplary" or "for example" in the embodiments of this application should not be construed as more preferred or more advantageous than other embodiments or implementations. Rather, using words such as "exemplary" or "for example" aims to present relevant concepts in a specific way for easy understanding.

[0189] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0191] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0192] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0193] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A routing method characterized by, The method comprises: obtaining a load balancing rate of one or more adjacent nodes of a first node, and link state information between the first node and each adjacent node; determining a candidate node set according to the load balancing rate and the link state information; wherein the candidate node set comprises one or more second nodes; the second nodes are nodes in the one or more adjacent nodes, the load balancing rate of which is less than or equal to a first preset threshold, and the link state information of which meets data transmission requirements; determining a next hop node of the first node from the candidate node set by performing a preset number of detections on the candidate node set through a reinforcement learning model according to link state information between the first node and each second node.

2. The method of claim 1, wherein, The method further comprises: determining a learning probability of the reinforcement learning model according to the link state information; performing a preset number of detections on the candidate node set through the reinforcement learning model based on the learning probability.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: constructing an initial model; wherein the initial model comprises a plurality of channels; 4. The method of claim 3, wherein, processing the traffic of N time points through the initial model to obtain N first weights of each channel at N time points; determining a weighted average value of each channel according to the N first weights of each channel; deleting a channel with a weighted average value less than a second preset threshold according to the weighted average value to obtain the traffic prediction model. The method further comprises: processing the traffic of the nth time point through the initial model to determine a second weight of each channel at the nth time point; wherein 1≤n≤N; 5. The method of claim 4, wherein, determining a deviation value of the initial model at the nth time point according to a traffic prediction accuracy of the initial model at the nth time point and an average traffic prediction accuracy of the initial model at N time points; updating N second weights of each channel at N time points according to N deviation values at N time points to obtain N first weights of each channel at N time points.

6. The method of claim 5, wherein: the second weight is determined according to the correlation of the channel features; wherein the correlation comprises one or more of the following: redundancy or complementarity; the redundancy is used to indicate the degree of repetition of the channel features of any two channels; and the complementarity is used to indicate the degree of complementarity of the channel features of any two channels. The method comprises: a transceiver module configured to obtain a load balancing rate of one or more adjacent nodes of a first node, and link state information between the first node and each adjacent node; 7. A communication device, characterized by ​ ​ determine a candidate node set according to the load balancing rate and the link state information; wherein the candidate node set comprises one or more second nodes, and the second nodes are nodes in the one or more adjacent nodes, for which the load balancing rate is less than or equal to a first preset threshold and the link state information meets a data transmission requirement; The processing module is further configured to determine a next hop node of the first node from the candidate node set by performing a preset number of detections on the candidate node set according to link state information between the first node and each of the second nodes through a reinforcement learning model.

8. The apparatus of claim 7, wherein The processing module is further configured to determine a learning probability of the reinforcement learning model according to the link state information. The processing module is further configured to perform a preset number of detections on the candidate node set through the reinforcement learning model based on the learning probability.

9. A communication device, characterized by A processor and a communication interface; the communication interface and the processor are coupled, and the processor is configured to run a computer program or instructions to implement the routing method as claimed in any one of claims 1-6. When a computer executes the instructions, the computer executes the routing method as claimed in any one of claims 1-6.

10. A computer-readable storage medium having stored therein instructions, the computer-readable storage medium comprising: ​

Images