Training method of satellite routing prediction model and low-orbit satellite routing method

Abstract

The invention provides a training method of a satellite routing prediction model and a low-orbit satellite routing method. The method comprises the following steps: obtaining observation information of each satellite node in a current satellite network environment state and a weight combination of each target randomly extracted based on a multi-target optimization model; inputting the observation information of each satellite node and the randomly selected weight combination of each target into a respective satellite routing prediction model to obtain Q values of a plurality of actions; and inputting Q values output by each satellite route prediction model into a Mixing network for joint training, calculating a loss value based on the main joint Q value, the auxiliary joint Q value and a preset loss function, adjusting parameters of the satellite route prediction model based on the loss value, and repeating the training process until a preset iteration condition is reached. According to the method, the weights of different routing indexes of the multi-objective optimization model are used as observations to be input into the satellite routing prediction model for training, so that diversified strategies meeting different service preferences can be generated.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a training method for a satellite routing prediction model and a low-orbit satellite routing method. Background Art

[0002] With the advancement of technologies such as 5G communications, artificial intelligence, big data, and the Internet of Things, user needs and the scope of communication services are constantly growing, and are developing in the direction of diversification and complexity. Business transmission requires high-performance indicators such as higher bandwidth and lower latency to support user needs. Traditional ground networks cannot meet the above needs due to limitations such as limited coverage and expensive equipment. However, the rapid development of low-orbit satellite networks in recent years has provided new ideas for solving the above problems. Low-orbit satellite networks are usually deployed at an altitude of 400km-2000km above the ground. Therefore, compared with geosynchronous orbit satellites and high-orbit satellites, routing based on low-orbit satellites has lower latency and better communication quality.

[0003] Low-orbit satellite networks are usually composed of dozens or hundreds of periodically moving satellites. The load size of satellite nodes changes rapidly, the state of inter-satellite links is unstable, and the satellite network topology changes periodically. Traditional routing algorithms, such as the shortest path algorithm or link state algorithm, are usually difficult to adapt to highly dynamic inter-satellite link changes. In addition, these methods usually only optimize for a single goal (such as minimizing delay or maximizing throughput). For example, they cannot balance delay and load balancing, resulting in the inability to meet differentiated business needs. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a low-orbit satellite routing method based on multi-objective reinforcement learning to improve the efficiency of the routing algorithm and balance multiple routing indicators to meet diversified business needs.

[0005] In a first aspect, a method for training a satellite routing prediction model is provided, which is applied to a low-orbit satellite routing scenario, and each satellite node corresponds to a satellite routing prediction model. The method includes: Obtain observation information of each satellite node under the current satellite network environment state and the weight combination of each target randomly selected based on the multi-objective optimization model; the observation information includes the current data queue length of each satellite node and the current data queue length of its neighboring node, and the distance from the neighboring node to the target satellite node; the multi-objective optimization model is a target optimization model pre-built based on various routing indicators, and the routing indicators at least include routing transmission delay and routing load balancing; The observation information of each satellite node and the weight combination of each randomly selected target are input into the respective satellite routing prediction model to obtain the Q value of multiple actions; the action is to select the next hop of the data packet in the neighboring node under the current input, and the Q value is the expected reward value after selecting the action; The Q values ​​output by each satellite route prediction model are input into the Mixing network for joint training, wherein the Mixing network is composed of a main network and a sub-network; the main network outputs a main joint Q value, and the sub-network outputs a sub-joint Q value; Calculate the loss value based on the primary joint Q value and the secondary joint Q value and the preset loss function; The parameters of the satellite routing prediction model are adjusted based on the loss value, and the above training process is repeated until the preset iteration conditions are met.

[0006] Optionally, the method further comprises: The action corresponding to the maximum Q value among the Q values ​​output by each satellite routing prediction model interacts with the current satellite network environment to generate the next satellite network environment state, and obtain the next observation information under the next satellite network environment state; Calculate the reward value for executing the action with the maximum Q value based on the reward function; The current satellite network environment state, current observation information, reward value, next satellite network environment state, next observation information and weight are combined to construct a set of experience data; Store the experience data into a pre-built experience replay pool; When the amount of data in the experience replay pool reaches a preset threshold, several experience samples are randomly selected from the experience replay pool to train the Mixing network.

[0007] Optionally, the method further comprises: Monotonic constraints are established for the Mixing network and each satellite route prediction model so that the action corresponding to the maximum Q value of each route prediction model is consistent with the action corresponding to the maximum joint Q value output by the Mixing network. The monotonic constraints are:

[0008] The left side shows the best action selection for the Mixing network as a whole; the right side shows the best action selection for each routing prediction model; Indicates that in a given state , and randomly selected weight combinations Next, select Action Expected value; Indicates the number of route prediction models.

[0009] Optionally, the method further comprises: The preset number of training times is spaced to increase the probability of weight combinations not being drawn.

[0010] Optionally, the multi-objective optimization model is:

[0011] in, , Both represent routing delay indicators; , Both represent routing load balancing indicators; Indicates the total number of data packets; Represents the local area formed by the current satellite node and its neighboring nodes; represents a satellite node; Indicates the local area The current queue length of satellite nodes; Represents the average queue length of all satellite nodes in the local area.

[0012] Optionally, the method further comprises: During the training process, the global status information of all satellite nodes is obtained at each preset interval; The global state information is input into the hypernetwork to generate the parameters of the Mixing network.

[0013] In a second aspect, a low-orbit satellite routing method is provided, which is applied to the training method of any satellite routing prediction model in the first aspect, and the method includes: Obtain observation information of the current satellite node; the current satellite node includes an initial satellite node and an intermediate satellite node, the initial satellite node is a satellite node that generates a data packet; the intermediate satellite node is a node that forwards a data packet; the observation information includes the current data queue length of the current satellite node and the current data queue length of its neighboring node, and the distance from the neighboring node to the target satellite node; The observation information of the current satellite node is input into the satellite routing prediction model deployed in the current satellite node to obtain the Q values ​​of multiple actions; the action is to select the next hop of the data packet in the neighboring node under the current observation information; Select the next satellite node for data packet transmission based on the action corresponding to the maximum Q value; The above routing process is repeated for the next satellite node until the target satellite node is reached.

[0014] In a third aspect, a low-orbit satellite routing device is provided, which is applied to the training method of any satellite routing prediction model in the first aspect, and the device includes: The acquisition unit is used to acquire the observation information of the current satellite node; the current satellite node includes an initial satellite node and an intermediate satellite node, the initial satellite node is a satellite node that generates a data packet; the intermediate satellite node is a node that forwards a data packet; the observation information includes the current data queue length of the current satellite node and the current data queue length of its neighbor node, and the distance from the neighbor node to the target satellite node; A prediction unit is used to input the observation information of the current satellite node into the satellite routing prediction model deployed in the current satellite node to obtain Q values ​​of multiple actions; the action is to select the next hop of the data packet in the neighboring node under the current observation information; The selection unit is used to select the next satellite node for data packet transmission based on the action corresponding to the maximum Q value; and repeatedly execute the above routing process for the next satellite node until the target satellite node is reached.

[0015] In a fourth aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the method steps described in any one of the first aspect or the second aspect when executing the program stored in the memory.

[0016] According to a fifth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method steps described in any one of the first aspect or the second aspect are implemented.

[0017] The present invention provides a training method for a satellite routing prediction model and a low-orbit satellite routing method. The method obtains observation information of each satellite node under the current satellite network environment state and a weight combination of each target randomly extracted based on a multi-objective optimization model; inputs the observation information of each satellite node and the weight combination of each randomly extracted target into their respective satellite routing prediction models to obtain Q values ​​of multiple actions; the Q values ​​output by each satellite routing prediction model are jointly input into a Mixing network for joint training, and the loss value is calculated based on the main joint Q and the secondary joint Q value and a preset loss function, and the parameters of the satellite routing prediction model are adjusted based on the loss value, and the above training process is repeated until the preset iteration condition is reached. The present invention trains by inputting the weights of different routing indicators of the multi-objective optimization model as observations into the satellite routing prediction model, so that the weight sampling is uniform, and a variety of strategies that meet different business preferences can be generated, and centralized training is performed based on the Q value of the entire satellite network through the Mixing network, rather than being limited to local information, to meet highly dynamic intersatellite link changes, so that routing decisions are more intelligent and flexible, and can quickly respond to changes in network status, and improve overall network performance and reliability.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A flow chart of a method for training a satellite route prediction model provided by an embodiment of the present invention is shown; Figure 2 A schematic diagram of the structure of a low-orbit satellite network and its training and execution provided by an embodiment of the present invention is shown; Figure 3 A flow chart of a low-orbit satellite routing method provided by an embodiment of the present invention is shown; Figure 4 A schematic structural diagram of a low-orbit satellite routing device provided by an embodiment of the present invention is shown; Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0022] Considering that low-orbit satellite networks are usually composed of dozens or hundreds of periodically moving satellites, the satellite node load size changes rapidly, the inter-satellite link state is unstable, and the satellite network topology changes periodically, traditional routing algorithms, such as the shortest path algorithm or link state algorithm, are usually difficult to adapt to the highly dynamic changes in the inter-satellite links. In addition, these methods usually only optimize for a single goal (such as minimizing delay or maximizing throughput), for example, they cannot balance delay and load balancing, resulting in the inability to meet differentiated business needs.

[0023] Based on this, the embodiments of the present invention provide a training method for a satellite routing prediction model and a low-orbit satellite routing method and device based on multi-objective reinforcement learning, which are described below through embodiments.

[0024] The embodiment of the present invention provides a training method for a satellite routing prediction model, which is applied to a low-orbit satellite routing scenario. Each satellite node corresponds to a satellite routing prediction model. Figure 1 As shown, the method includes: Step S101: Obtain observation information of each satellite node under the current satellite network environment state and a weight combination of each objective randomly selected based on a multi-objective optimization model.

[0025] Among them, the observation information includes the current data queue length of each satellite node and the current data queue length of its neighboring nodes, as well as the distance from the neighboring node to the target satellite node; the multi-objective optimization model is a target optimization model pre-built based on various routing indicators, and the routing indicators at least include routing transmission delay and routing load balancing.

[0026] like Figure 2 As shown in the figure, a low-orbit satellite network system is constructed. The low-orbit satellite network system divides the low-orbit satellite constellation into There are evenly distributed satellites, the angle between adjacent orbital planes is , and the angle between two adjacent satellites in the same orbit is In polar regions, the relative motion of satellites in different orbits is fast, and in polar regions, it is difficult for satellite antenna systems to track the positions of neighboring satellites. and The area is defined as the polar region. Considering the communication cost and technical factors, no inter-satellite links are established in the polar region.

[0027] In this low-orbit satellite network system, each satellite node can generate and forward data packets. Low-orbit satellites have limited computing and storage resources. During peak traffic periods, data packets need to be stored in a local cache queue waiting for processing. Data packets follow the first-in-first-out principle. Similarly, the cache queue processes data packets on a first-come-first-served basis. The cache queue reduces certain data packet losses, but introduces queuing delays. In data packet transmission, a waiting queue is required for queuing, so the embodiment of the present invention establishes Packet queuing model, where the first Represents the arrival process of data packets; it follows Poisson distribution, the arrival interval follows exponential distribution, and the average arrival rate is ; The second It represents the service time required for packet forwarding, which follows an exponential distribution and has an average forwarding rate of ;1 means; is the maximum capacity of the packet queue; Indicates that the number of packet sources is infinite. At a certain moment, there are N When there are packets, the newly arrived packets will be rejected from entering the queue.

[0028] In the existing technology, in data packet transmission, how to select the optimal next-hop satellite node generally only considers a single indicator, such as minimizing data transmission delay or maximizing throughput, which limits its ability to meet the diverse quality service needs of future satellite Internet.

[0029] Therefore, the embodiment of the present invention constructs a data packet delay model and a load balancing model; First, let’s talk about the construction process of the data packet delay model: Consider that the delay of a data packet is composed of propagation delay and queuing delay. Therefore, the total delay of a data packet from one satellite node to another is: (1); Among them, the propagation delay for: (2); in, Indicates the size of the forwarded data packet; Satellite node The data packet transmission rate between ; can be expressed as: (3); in, is the satellite node in time slot t The bandwidth of the intersatellite link; The signal-to-noise ratio of the intersatellite link at this time can be expressed as: (4); In the above formula, For satellite The transmission power of For satellite The transmit antenna gain. For satellite The receiving antenna gain; is the noise power of the intersatellite link, and (5). is the Boltzmann constant, For satellite Noise temperature at the receiver; It is the loss caused by distance attenuation during signal propagation in free space, which can be expressed as follows: (6); in, Satellite node The distance between is the carrier frequency of the signal.

[0030] In another feasible implementation manner, the queuing delay can be expressed as: (7); in, It represents the effective arrival rate of data packets arriving at the satellite node per unit time; Indicates the average forwarding rate; Satellite node The average number of queued packets can be expressed as: (8); in, , is a constant; Indicates the maximum capacity of the packet queue.

[0031] In a low-orbit satellite network, the source node Generate a data packet and transmit it to the destination satellite node d through continuous forwarding of the intermediate satellite nodes. Therefore, the data packet needs to be transmitted through multiple hops through the relay nodes. Then the path of data packet k from the source node satellite s to the destination satellite d can be expressed as , then the total inter-satellite routing delay of data packet pkt is: (9).

[0032] Secondly, considering the traffic balance problem in the network, when forwarding data packets, satellite nodes need to consider the load of neighboring nodes and try to avoid passing through high-load nodes to alleviate traffic congestion in the network. Therefore, the load balancing model can be expressed as: (10); Among them, zone is the local area composed of the current satellite node and neighboring nodes; represents a satellite node; Indicates the local area The current queue length of satellite nodes; Represents the average queue length of all satellite nodes in the local area. Finally, the multi-objective optimization model is constructed based on the above two models: (11); in, , Both represent routing delay indicators; , Both represent routing load balancing indicators; Indicates the total number of data packets; Represents the local area formed by the current satellite node and its neighboring nodes; represents a satellite node; Indicates the local area The current queue length of satellite nodes; Represents the average queue length of all satellite nodes in the local area.

[0033] The embodiment of the present invention can meet the needs of different services by optimizing the two indicators of delay indicator and load balancing.

[0034] Step S102: inputting the observation information of each satellite node and the weight combination of each randomly selected target into the respective satellite route prediction model to obtain the Q value of multiple actions.

[0035] The action is to select the next hop of the data packet in the neighboring node under the current input, and the Q value is the expected reward value after selecting the action.

[0036] During training, satellite nodes are regarded as intelligent agents, and each intelligent agent has an Agent network, which is the satellite routing prediction model.

[0037] Specifically, each agent passes through the state-action value function conduct decision making. The decision can be expressed as follows: (12); In the above formula, choose an initial A value of 0.1 is a common starting point, meaning that about 90% of the time the action with the maximum Q value will be chosen, while 10% of the time random exploration will be done in the action space. This ensures that the algorithm has enough exploration opportunities in the early stages, and relies more on the knowledge already acquired in the later stages. This mechanism enables the agent to not only continue to explore unknown areas and find potential better solutions, but also effectively use known information to obtain immediate benefits, thereby continuously improving its behavior strategy during the learning process.

[0038] The randomly selected weight combination of each target is the weight combination of the two indicators of routing delay and load balancing, which can quantify the importance of different targets according to different tasks or service levels. By randomly selecting this weight combination as the input of the Agent network, these weights can be used to adjust the priority of the agent when making decisions.

[0039] Step S103: inputting the Q values ​​output by each satellite route prediction model into the Mixing network for joint training.

[0040] Among them, the Mixing network consists of a main network and a sub-network; the main network outputs the main joint Q value, and the sub-network outputs the sub-joint Q value.

[0041] Exemplarily, the main network and the secondary network include two groups of networks, each group of networks corresponds to an optimization objective, specifically, one group of networks corresponds to routing delay, and one group of networks corresponds to load balancing; the input is the Q vector of n Agent networks, that is , the Q values ​​corresponding to the same target are combined and input into a corresponding set of networks, and finally the two sets of networks are integrated to output the multi-target joint Q value, that is, the Q values ​​of m targets are combined into a single Q value. Vector In the embodiment of the present invention, m is 2.

[0042] Step S104: Calculate the loss value based on the primary joint Q value, the secondary joint Q value and the preset loss function; adjust the parameters of the satellite routing prediction model based on the loss value, and repeat the above training process until the preset iteration condition is reached.

[0043] In one example, the iteration condition is preset, such as reaching a preset number of iterations or the loss value gradually converges to reach a local minimum.

[0044] In the embodiment of the present invention, since the various agents are homogeneous, in order to promote cooperation between the agents and reduce the overhead of the neural network model, a parameter sharing mechanism is adopted, that is, all Agent networks share a set of parameters.

[0045] In an example, the loss function looks like this: (13); in, represents the output of the secondary network, represents the main joint Q value output by the main network; are the parameters of the Mixing network; are the parameters of the Agent network; The weight vector space representing the optimization objective; Represents the experience pool data; the goal of the loss function is to minimize the difference between the output of the main network and the output of the target network to update the Agent network parameters.

[0046] Specifically, It can be expressed as follows: (14); in, It’s an instant reward; is the discount factor; represents the action space; The weight vector space representing the optimization objective; represents the parameters of the secondary network; Represents the joint Q value of the secondary network output.

[0047] The embodiment of the present invention trains all Agent networks at the same time, using centralized training. During centralized training, satellite nodes are used as virtual intelligent agents to obtain observation information, and global states are used as auxiliary information for training, thereby obtaining the optimal routing strategy. This can not only reduce training time and improve training efficiency, but also take global information into consideration to obtain a better routing strategy.

[0048] In the above network parameter update process, if the parameter update frequency is too frequent, the model will be unstable and difficult to converge. Therefore, the secondary network parameters are not updated every time training, but are updated regularly after a certain number of training rounds, so as to ensure the stability of the training process.

[0049] The present invention uses the weights of different routing indicators of the multi-objective optimization model as observation inputs to the satellite routing prediction model for training, so that the weight sampling is uniform, and diversified strategies that meet different business preferences can be generated. The Mixing network is used to perform centralized training based on the Q value of the entire satellite network, rather than being limited to local information, to meet the highly dynamic changes in intersatellite links, making routing decisions more intelligent and flexible, and being able to quickly respond to changes in network status, thereby improving overall network performance and reliability.

[0050] Based on the above embodiment, the method further includes: The action corresponding to the maximum Q value among the Q values ​​output by each satellite routing prediction model interacts with the current satellite network environment to generate the next satellite network environment state, and obtain the next observation information under the next satellite network environment state; Calculate the reward value for executing the action with the maximum Q value based on the reward function; The current satellite network environment state, current observation information, reward value, next satellite network environment state, next observation information and weight are combined to construct a set of experience data; Store the experience data into a pre-built experience replay pool; When the amount of data in the experience replay pool reaches a preset threshold, several experience samples are randomly selected from the experience replay pool to train the Mixing network.

[0051] After the agent interacts with the environment, the state, observation, reward, action, and weight of each step are recorded, that is, When the number of experience items is greater than the batch, random sampling training is performed, and the number of experience items extracted each time is a batch. When the capacity of the experience replay pool is greater than the threshold, the old experience data is deleted to ensure the validity of the experience data.

[0052] Based on the above embodiment, the method further includes: Monotonic constraints are established for the Mixing network and each satellite route prediction model so that the action corresponding to the maximum Q value of each route prediction model is consistent with the action corresponding to the maximum joint Q value output by the Mixing network. The monotonic constraints are: (15); The left side shows the best action selection for the Mixing network as a whole; the right side shows the best action selection for each routing prediction model; Indicates that in a given state , and randomly selected weight combinations Next, select Action Expected value; Indicates the number of route prediction models.

[0053] Based on the above embodiment, the method further includes: The preset number of training times is spaced to increase the probability of weight combinations not being drawn.

[0054] During training, randomly extracting weights may lead to uneven solution sets. Therefore, at fixed intervals, the density and uniformity of the solution set are checked to increase the probability of weights that are not selected, so that the weights that are not selected are emphasized in the subsequent training process. For example, in a multi-objective optimization model, the weight of routing delay is 0.1, and the weight of load balancing is 0.9. If the weight combination is that the weight of delay is less than the weight of load balancing, it will cause uneven selection of actions. Then the probability of weights that are not selected can be increased, and the weight of routing delay will most likely become 0.6, and the weight of load balancing will be 0.4. Finally, a set of dense and uniform disposable solution sets is obtained to meet the preferences of different businesses for weights.

[0055] Based on the above embodiment, the method further includes: During the training process, the global status information of all satellite nodes is obtained at each preset interval; The global state information is input into the hypernetwork to generate the parameters of the Mixing network.

[0056] During centralized training, satellite nodes act as virtual agents to obtain observation information, and the global state is used as auxiliary information for training to obtain the optimal routing strategy.

[0057] After completing the centralized training, the Agent network is deployed on the satellite. The satellite can obtain its own state-action function by inputting local observations. , and select The action with the largest value is used as the next hop. This distributed decision-making is real-time and avoids unnecessary communication overhead.

[0058] Based on the above embodiment, the embodiment of the present invention provides a low-orbit satellite routing method, which is applied to the training method of the above satellite routing prediction model, such as Figure 3 As shown, the method includes: Step S301: Obtain observation information of the current satellite node.

[0059] The current satellite nodes include the initial satellite node and the intermediate satellite node. The initial satellite node is the satellite node that generates the data packet; the intermediate satellite node is the node that forwards the data packet; the observation information includes the current data queue length of the current satellite node and the current data queue length of its neighbor node, as well as the distance from the neighbor node to the target satellite node.

[0060] Step S302: inputting the observation information of the current satellite node into the satellite route prediction model deployed in the current satellite node to obtain Q values ​​of multiple actions.

[0061] The action is to select the next hop of the data packet among the neighbor nodes under the current observation information.

[0062] Step S303: Select the next satellite node for data packet transmission based on the action corresponding to the maximum Q value; repeat the above routing process for the next satellite node until the target satellite node is reached.

[0063] The embodiment of the present invention adopts a centralized training and distributed execution framework. During the distributed execution, the satellite nodes only make independent routing decisions based on their own observation information, thus saving communication overhead between satellites.

[0064] Based on the same inventive concept, a low-orbit satellite routing device is provided, which is applied to the training method of the above-mentioned satellite routing prediction model, such as Figure 4 As shown, the device comprises: The acquisition unit 401 is used to acquire the observation information of the current satellite node; the current satellite node includes an initial satellite node and an intermediate satellite node, the initial satellite node is a satellite node that generates a data packet; the intermediate satellite node is a node that forwards a data packet; the observation information includes the current data queue length of the current satellite node and the current data queue length of its neighbor node, and the distance from the neighbor node to the target satellite node; The prediction unit 402 is used to input the observation information of the current satellite node into the satellite routing prediction model deployed in the current satellite node to obtain Q values ​​of multiple actions; the action is to select the next hop of the data packet in the neighboring node under the current observation information; The selection unit 403 is used to select the next satellite node for data packet transmission based on the action corresponding to the maximum Q value; and repeatedly execute the above routing process for the next satellite node until the target satellite node is reached.

[0065] Based on the same technical concept, an embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504; Memory 503, used for storing computer programs; The processor 501 is used to implement the training method of the satellite routing prediction model and the steps of the low-orbit satellite routing method when executing the program stored in the memory 503.

[0066] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0067] The communication interface is used for communication between the above electronic device and other devices.

[0068] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0069] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0070] The computer program product of the satellite routing prediction model training method and the low-orbit satellite routing method provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0071] The device of the low-orbit satellite routing method provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0072] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0073] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0075] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0076] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0077] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or perform equivalent replacements on some of the technical features thereof; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for training a satellite route prediction model, characterized in that: Applied to low-orbit satellite routing scenarios, each satellite node corresponds to a satellite routing prediction model, and the method includes: Obtain observation information of each satellite node under the current satellite network environment state and a weight combination of each objective randomly selected based on a multi-objective optimization model; the observation information includes the current data queue length of each satellite node and the current data queue length of its neighboring node, and the distance from the neighboring node to the target satellite node; the multi-objective optimization model is a target optimization model pre-constructed based on various routing indicators, and the routing indicators at least include routing transmission delay and routing load balancing; The observation information of each satellite node and the weight combination of each randomly selected target are input into the respective satellite route prediction models to obtain the Q values ​​of multiple actions; the action is to select the next hop of the data packet in the neighboring node under the current input, and the Q value is the expected reward value obtained after selecting the action; The Q values ​​output by each satellite route prediction model are input into a Mixing network for joint training, wherein the Mixing network is composed of a main network and a secondary network; the main network outputs a main joint Q value, and the secondary network outputs a secondary joint Q value; Calculating a loss value based on the primary joint Q value and the secondary joint Q value and a preset loss function; The parameters of the satellite route prediction model are adjusted based on the loss value, and the above training process is repeated until a preset iteration condition is reached.

2. The method according to claim 1, characterized in that The method further comprises: The action corresponding to the maximum Q value among the Q values ​​output by each satellite routing prediction model interacts with the current satellite network environment to generate the next satellite network environment state, and obtain the next observation information under the next satellite network environment state; Calculate the reward value for executing the action with the maximum Q value based on the reward function; The current satellite network environment state, current observation information, reward value, next satellite network environment state, next observation information and weight are combined to construct a set of experience data; storing the experience data in a pre-built experience replay pool; When the amount of data in the experience replay pool reaches a preset threshold, a number of experience samples are randomly selected from the experience replay pool to train the Mixing network.

3. The method according to claim 1, characterized in that The method further comprises: A monotonic constraint condition is established for the Mixing network and each satellite route prediction model so that the action corresponding to the maximum Q value of each route prediction model is consistent with the action corresponding to the maximum joint Q value output by the Mixing network, wherein the monotonic constraint condition is: The left side shows the best action selection for the Mixing network as a whole; the right side shows the best action selection for each routing prediction model; Indicates that in a given state , and randomly selected weight combinations Next, select Action Expected value; Indicates the number of route prediction models.

4. The method according to claim 1, characterized in that: The method further comprises: The preset number of training times is spaced to increase the probability of weight combinations not being drawn.

5. The method according to claim 1, characterized in that The multi-objective optimization model is: in, , Both represent routing delay indicators; , Both represent routing load balancing indicators; Indicates the total number of data packets; Represents the local area formed by the current satellite node and its neighboring nodes; represents a satellite node; Indicates the local area The current queue length of satellite nodes; Represents the average queue length of all satellite nodes in the local area.

6. The method according to claim 1, characterized in that The method further comprises: During the training process, the global status information of all satellite nodes is obtained at each preset interval; The global state information is input into the super network to generate the parameters of the Mixing network.

7. A low-orbit satellite routing method, characterized in that: A training method for a satellite route prediction model applied to any one of claims 1 to 6, the method comprising: Acquire observation information of a current satellite node; the current satellite node includes an initial satellite node and an intermediate satellite node, the initial satellite node is a satellite node that generates a data packet; the intermediate satellite node is a node that forwards a data packet; the observation information includes a current data queue length of the current satellite node and a current data queue length of its neighboring node, and a distance from the neighboring node to the target satellite node; Inputting the observation information of the current satellite node into the satellite routing prediction model deployed in the current satellite node to obtain Q values ​​of multiple actions; the action is to select the next hop of the data packet in the neighboring node under the current observation information; Select the next satellite node for data packet transmission based on the action corresponding to the maximum Q value; The above routing process is repeated for the next satellite node until the target satellite node is reached.

8. A low-orbit satellite routing device, characterized in that: A training method for a satellite route prediction model applied to any one of claims 1 to 6, the device comprising: An acquisition unit is used to acquire observation information of a current satellite node; the current satellite node includes an initial satellite node and an intermediate satellite node, the initial satellite node is a satellite node that generates a data packet; the intermediate satellite node is a node that forwards a data packet; the observation information includes a current data queue length of the current satellite node and a current data queue length of its neighboring node, and a distance from the neighboring node to the target satellite node; A prediction unit, used to input the observation information of the current satellite node into a satellite routing prediction model deployed in the current satellite node to obtain Q values ​​of multiple actions; the action is to select the next hop of the data packet in the neighboring node under the current observation information; The selection unit is used to select the next satellite node for data packet transmission based on the action corresponding to the maximum Q value; and repeatedly execute the above routing process for the next satellite node until the target satellite node is reached.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images