Satellite networking routing method based on deep reinforcement learning

Abstract

The invention relates to the technical field of communication, in particular to a satellite networking routing method based on deep reinforcement learning, which comprises the following steps: generating a first action value function corresponding to each decision action based on a feature vector of a current network state, determining a decision action selection probability in the current network state based on a current target model, selecting and outputting a corresponding second action value function for each decision action by utilizing a preset exploration and utilization strategy, a decision action selection probability and the first action value function; and calculating a target action value function corresponding to each decision action in the initialized target network and a difference value between the target action value function and the second action value function, calculating a minimum loss function of the current target model according to the difference value, and optimizing a satellite networking routing algorithm. Therefore, the problems of low data transmission efficiency, packet loss and the like caused by the fact that a routing algorithm is difficult to adapt to a frequently changing network environment due to high delay and high bit error rate of satellite networking are solved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a satellite networking routing method based on deep reinforcement learning. Background Art

[0002] Satellite networks, especially LEO (Low Earth Orbit) satellite networks, have high latency, high bit error rate and frequently changing dynamic topology. Routing algorithms need to quickly adapt to this highly dynamic network environment, which becomes a major challenge in network design and optimization.

[0003] In related technologies, the high latency and high bit error rate of satellite networking increase the uncertainty and complexity of data transmission. Its dynamic topology requires the routing algorithm to have high flexibility and rapid response capabilities to adapt to the frequently changing link status.

[0004] However, the routing algorithms in related technologies perform poorly when faced with the above situation and are difficult to adapt to the frequently changing network environment, resulting in inefficient data transmission and even serious packet loss. In addition, routing decisions in satellite networks need to comprehensively consider multiple constraints, such as bandwidth, latency, number of hops, load balancing, etc., which further increases the complexity of routing decisions and urgently needs to be resolved. Summary of the invention

[0005] The present invention provides a satellite networking routing method based on deep reinforcement learning to solve the problems that the high latency and high bit error rate of satellite networking make it difficult for the routing algorithm to adapt to the frequently changing network environment, resulting in low data transmission efficiency and even serious packet loss.

[0006] The first aspect of the present invention provides a satellite networking routing method based on deep reinforcement learning, comprising the following steps: Acquire the current network state, and generate a first action value function corresponding to each decision action in the current network state based on a feature vector of the current network state; Based on the current target model, determine the probability of selecting a decision action under the current network state, use a preset exploration and utilization strategy, and select each decision action based on the decision action selection probability and the first action value function, and output a corresponding second action value function according to the selected decision action; Calculate the target action value function corresponding to each decision action in the initialized target network, calculate the minimum loss function of the current target model according to the difference between the target action value function and the second action value function, and determine the best optimization strategy according to the minimum loss function to optimize the satellite networking routing algorithm according to the best optimization strategy.

[0007] According to an embodiment of the present invention, the method of using a preset exploration and utilization strategy, selecting each decision action based on the decision action selection probability and the first action value function, and outputting a corresponding second action value function according to the selected decision action includes: Using the preset exploration and exploitation strategy, randomly selecting from each of the decision actions according to the first decision action selection probability and the first action value function, and outputting a corresponding second action value function according to the selected decision action; Alternatively, the preset exploration and utilization strategy is used to determine the highest action value function from each decision action according to the second decision action selection probability and the first action value function, and the corresponding second action value function is output according to the selected decision action.

[0008] According to one embodiment of the present invention, after outputting the corresponding second action value function according to the selected decision action, the method further includes: Determine a reward function of the current target model based on the second action value function, generate a new network state based on the selected decision action, and generate an experience sample of the current target model according to the reward function, the current network state, the selected decision action and the new network state; Determine a target task of the current target model, set an action experience replay buffer of the current target model according to the target task, and store experience samples of the current target model in the action experience replay buffer; Determining whether the action experience playback buffer is full; If the action experience playback buffer is full, the latest stored experience sample replaces the earliest stored experience sample.

[0009] According to one embodiment of the present invention, determining the reward function of the current target model based on the second action value function includes: Defining a positive reward and a negative reward of an initial reward function based on the feedback result of the target task and the current network state; A weighted sum reward function of the target task is defined according to the target task, and the positive reward and the negative reward are adjusted using the weight coefficient of the weighted sum reward function to optimize the initial reward function to obtain the reward function of the current target model.

[0010] According to one embodiment of the present invention, after storing the experience sample of the current target model in the action experience playback buffer, the method further includes: A preset number of experience samples in the current target model are randomly extracted from the action experience playback buffer according to a preset period, and the current target model is updated based on the extracted experience samples.

[0011] According to one embodiment of the present invention, before acquiring the current network status, the method further includes: Constructing the initial target model; The initial target model is initialized, and the initialized initial target model is trained to obtain a current target model, and the current network state is obtained based on the current target model.

[0012] According to a satellite networking routing method based on deep reinforcement learning according to an embodiment of the present invention, a first action value function corresponding to each decision action is generated based on the feature vector of the current network state, the decision action selection probability under the current network state is determined based on the current target model, and the preset exploration and utilization strategy, decision action selection probability and the first action value function are used to select and output the corresponding second action value function for each decision action, and the target action value function corresponding to each decision action in the initialized target network is calculated, and the difference between the target action value function and the second action value function is calculated, so as to calculate the minimum loss function of the current target model according to the difference value, so as to optimize the satellite networking routing algorithm. Thus, the problem that the high delay and high bit error rate of the satellite networking make it difficult for the routing algorithm to adapt to the frequently changing network environment, resulting in low data transmission efficiency and serious packet loss, is solved.

[0013] A second aspect of the present invention provides a satellite networking routing device based on deep reinforcement learning, including: A generation module, used to obtain a current network state, and generate a first action value function corresponding to each decision action under the current network state based on a feature vector of the current network state; A determination module, used to determine the probability of selecting a decision action under the current network state based on the current target model, use a preset exploration and utilization strategy, and select each decision action based on the decision action selection probability and the first action value function, and output a corresponding second action value function according to the selected decision action; The optimization module is used to calculate the target action value function corresponding to each decision action in the initialized target network, calculate the minimum loss function of the current target model according to the difference between the target action value function and the second action value function, and determine the best optimization strategy according to the minimum loss function, so as to optimize the satellite networking routing algorithm according to the best optimization strategy.

[0014] According to one embodiment of the present invention, the determining module includes: A selection unit, configured to randomly select from each of the decision actions according to the first decision action selection probability and the first action value function using the preset exploration and utilization strategy, and output a corresponding second action value function according to the selected decision action; Alternatively, the preset exploration and utilization strategy is used to determine the highest action value function from each decision action according to the second decision action selection probability and the first action value function, and the corresponding second action value function is output according to the selected decision action.

[0015] According to an embodiment of the present invention, after outputting the corresponding second action value function according to the selected decision action, the determination module further includes: a generating unit, configured to determine a reward function of the current target model based on the second action-value function, generate a new network state based on the selected decision action, and generate an experience sample of the current target model according to the reward function, the current network state, the selected decision action and the new network state; A storage unit, used to determine a target task of the current target model, set an action experience replay buffer of the current target model according to the target task, and store experience samples of the current target model in the action experience replay buffer; A judging unit, used to judge whether the action experience playback buffer is full; The replacement unit is used to replace the earliest stored experience sample with the latest stored experience sample if the action experience playback buffer is full.

[0016] According to one embodiment of the present invention, the generating unit includes: A definition subunit, used to define a positive reward and a negative reward of an initial reward function based on the feedback result of the target task and the current network state; An optimization subunit is used to define a weighted sum reward function of the target task according to the target task, and adjust the positive reward and the negative reward using the weight coefficient of the weighted sum reward function, optimize the initial reward function, and obtain the reward function of the current target model.

[0017] According to an embodiment of the present invention, after storing the experience sample of the current target model in the action experience playback buffer, the storage unit further includes: The updating subunit is used to randomly extract a preset number of experience samples in the current target model from the action experience playback buffer according to a preset period, and update the current target model based on the extracted experience samples.

[0018] According to an embodiment of the present invention, before acquiring the current network status, the generating module further includes: A construction unit, used to construct an initial target model; The acquisition unit is used to initialize the initial target model, train the initialized initial target model, obtain the current target model, and acquire the current network state based on the current target model.

[0019] According to a satellite networking routing device based on deep reinforcement learning according to an embodiment of the present invention, a first action value function corresponding to each decision action is generated based on the feature vector of the current network state, the probability of selecting the decision action under the current network state is determined based on the current target model, and the preset exploration and utilization strategy, the probability of selecting the decision action and the first action value function are used to select and output the corresponding second action value function for each decision action, and the target action value function corresponding to each decision action in the initialized target network is calculated, and the difference between the target action value function and the second action value function is calculated, so as to calculate the minimum loss function of the current target model according to the difference value, so as to optimize the satellite networking routing algorithm. Thus, the problem that the high delay and high bit error rate of the satellite networking make it difficult for the routing algorithm to adapt to the frequently changing network environment, resulting in low data transmission efficiency and serious packet loss, is solved.

[0020] An embodiment of the third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a satellite networking routing method based on deep reinforcement learning as described in the above embodiment.

[0021] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute a satellite networking routing method based on deep reinforcement learning as described in the above embodiment.

[0022] A fifth aspect of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements a satellite networking routing method based on deep reinforcement learning as described in the above embodiment.

[0023] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of a satellite networking routing method based on deep reinforcement learning provided according to an embodiment of the present invention; Figure 2 An overall flow chart of a routing algorithm for optimizing satellite networking according to an embodiment of the present invention; Figure 3 This is an example diagram of a satellite networking routing device based on deep reinforcement learning according to an embodiment of the present invention; Figure 4 FIG. 4 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0026] A satellite networking routing method based on deep reinforcement learning according to an embodiment of the present invention is described below with reference to the accompanying drawings. In view of the problem mentioned in the above background technology that the high delay and high bit error rate of satellite networking make it difficult for the routing algorithm to adapt to the frequently changing network environment, resulting in low data transmission efficiency and even serious packet loss, the present invention provides a satellite networking routing method based on deep reinforcement learning, in which a first action value function corresponding to each decision action is generated based on the feature vector of the current network state, the decision action selection probability under the current network state is determined based on the current target model, and the preset exploration and utilization strategy, decision action selection probability and the first action value function are used to select and output the corresponding second action value function for each decision action, calculate the target action value function corresponding to each decision action in the initialized target network, and calculate the difference between the target action value function and the second action value function, so as to calculate the minimum loss function of the current target model according to the difference value, so as to optimize the satellite networking routing algorithm. Thus, the problem that the high delay and high bit error rate of satellite networking make it difficult for the routing algorithm to adapt to the frequently changing network environment, resulting in low data transmission efficiency, serious packet loss and the like is solved.

[0027] Specifically, before introducing the embodiments of the present invention, the defects of satellite networking routing algorithms in related technologies are first introduced. Satellite communication links usually have high delays and bit error rates due to long distances and special transmission media, which poses a huge challenge to traditional routing algorithms. Traditional algorithms are often unable to operate efficiently in such high-delay, high-bit error rate environments, which can lead to inefficient data transmission and even serious packet loss.

[0028] The topology of satellite networks changes frequently. For example, low-Earth orbit satellites move rapidly in orbit, causing the link connections between satellites to change frequently. This dynamic nature requires routing algorithms to be able to quickly adapt to new network topologies, while traditional static or semi-static routing algorithms are difficult to work effectively in such environments.

[0029] In addition, routing decisions in satellite networks need to consider multiple constraints, such as bandwidth, latency, number of hops, load balancing, etc. Since these factors influence each other, routing decisions will become very complicated. Traditional routing algorithms can usually only consider a small number of constraints and it is difficult to find the optimal routing path under multiple constraints. Therefore, based on the above-mentioned problems, an embodiment of the present invention proposes a solution that combines deep reinforcement learning with GPT-4.0 to optimize the satellite networking routing algorithm, thereby improving the efficiency of routing decisions. The specific implementation plan will be described in the following embodiments.

[0030] Specifically, Figure 1 A schematic diagram of a flow chart of a satellite networking routing method based on deep reinforcement learning provided in an embodiment of the present invention.

[0031] like Figure 1 As shown, the satellite networking routing method based on deep reinforcement learning includes the following steps: In step S101, the current network state is obtained, and a first action-value function corresponding to each decision action in the current network state is generated based on a feature vector of the current network state.

[0032] According to one embodiment of the present invention, before obtaining the current network status, it also includes: constructing an initial target model; initializing the initial target model, and training the initialized initial target model to obtain the current target model, and obtaining the current network status based on the current target model.

[0033] Specifically, an embodiment of the present invention first constructs an initial target model, such as a DQN (Deep Q-Network) model. At this time, the DQN model is an untrained model. The DQN model combines the traditional Q-learning algorithm with deep learning and approximates the Q-value function by using a convolutional neural network; secondly, the DQN model is initialized, and the first training is performed based on the initialized DQN model to obtain the current target model, that is, the trained DQN model, and the current network state is obtained based on the current target model; finally, a feature vector of the current network state is received, and a first action value function, that is, a Q value, is generated for each possible output decision action in the current network state based on the received feature vector.

[0034] It should be noted that the input feature vector represents a numerical array of the current network state, including node connection status, link bandwidth, link delay, packet loss rate and traffic load, etc., and the output Q value represents a vector, each element of which represents the value of performing a specific decision action under the current network state. In this process, the embodiment of the present invention can use GPT-4.0 technology to describe and explain the network state and decision-making process in natural language to improve the interpretability and transparency of the DQN model.

[0035] In step S102, based on the current target model, the probability of decision action selection under the current network state is determined, and the preset exploration and utilization strategy is used to select each decision action based on the decision action selection probability and the first action value function, and the corresponding second action value function is output according to the selected decision action.

[0036] According to one embodiment of the present invention, a preset exploration and utilization strategy is utilized, and each decision action is selected based on the decision action selection probability and the first action value function, and a corresponding second action value function is output according to the selected decision action, including: utilizing the preset exploration and utilization strategy to randomly select from each decision action according to the first decision action selection probability and the first action value function, and outputting the corresponding second action value function according to the selected decision action; or utilizing the preset exploration and utilization strategy to determine the highest action value function from each decision action according to the second decision action selection probability and the first action value function, and outputting the corresponding second action value function according to the selected decision action.

[0037] Among them, the preset exploration and utilization strategies can be selected by technical personnel in this field according to actual experimental requirements and are not specifically limited here.

[0038] Specifically, Figure 2 As shown in the figure, at each time step, the agent (such as the DQN model) observes the current network state based on the current target model after this update, for example, observes the dynamic changes of the current network link, traffic fluctuations, etc. to determine the probability of decision action selection under the current network state, and uses a preset exploration and utilization strategy, such as the ε-greedy strategy, to randomly select from each decision action according to the first decision action selection probability (such as ε) and the first action value function. By randomly selecting actions, the agent can explore new state-action pairs, which helps to find action choices that are better than the current strategy, thereby improving the effectiveness of the overall strategy. At the same time, exploration can prevent the agent from falling into a local optimal solution to ensure that the agent can globally optimize its strategy.

[0039] Optionally, an embodiment of the present invention may also utilize the ε-greedy strategy to determine the decision action corresponding to the highest action value function from each decision action according to the second decision action selection probability (for example, 1-ε) and the first action value function, and output the corresponding second action value function, that is, the highest Q value, based on the selected decision action. By selecting the decision action corresponding to the current highest Q value, the intelligent agent can utilize known strategies to obtain the maximized cumulative reward. Using existing experience, the intelligent agent can achieve the optimal decision in a short period of time to improve the performance of the system.

[0040] Among them, the two action selection formulas are as follows:

[0041] It should be noted that the purpose of using the ε-greedy strategy in the embodiment of the present invention is to balance the exploration of new strategies and the use of existing strategies when training the intelligent agent. For the selection of ε, the embodiment of the present invention uses GPT-4.0 to generate natural language prompts for attenuation. In the early stage of training, a higher ε value can be selected to encourage exploration to avoid falling into a local optimal solution too early; as the training proceeds, the embodiment of the present invention can make more detailed adjustments to the ε value according to the prompts. If the performance of the DQN model improves during the training process, the ε value can be reduced according to the linear attenuation strategy to increase the use of existing strategies; if the performance of the DQN model tends to be stable during the training process, the ε value used remains stable, and more strategies that have been learned are used at this time to improve the stability and performance of the model.

[0042] Among them, the linear attenuation strategy formula can be expressed as:

[0043] in, are the start and end time steps of the initial stage, are the start and end time steps of the mid-term phase, is the exploration probability in the initial stage, is the exploration probability in the mid-term stage.

[0044] According to one embodiment of the present invention, after outputting the corresponding second action value function according to the selected decision action, it also includes: determining the reward function of the current target model based on the second action value function, and generating a new network state based on the selected decision action, generating experience samples of the current target model according to the reward function, the current network state, the selected decision action and the new network state; determining the target task of the current target model, setting the action experience playback buffer of the current target model according to the target task, and storing the experience samples of the current target model in the action experience playback buffer; judging whether the action experience playback buffer is full; if the action experience playback buffer is full, replacing the earliest stored experience samples with the latest stored experience samples.

[0045] According to one embodiment of the present invention, the reward function of the current target model is determined based on the second action value function, including: defining the positive reward and negative reward of the initial reward function based on the feedback result of the target task and the current network state; defining the weighted sum reward function of the target task according to the target task, and adjusting the positive reward and negative reward using the weight coefficient of the weighted sum reward function, optimizing the initial reward function, and obtaining the reward function of the current target model.

[0046] Specifically, Figure 2 As shown, in an embodiment of the present invention, based on the target task of the DQN model, the positive reward and negative reward of the initial reward function are defined according to the feedback result of the target task and the current network state. For example, when the delay experienced by the data packet increases, the packet loss rate increases, the routing decision cost increases, the DQN model detects abnormal traffic or potential network attacks, etc., a negative reward is given; when the network throughput increases, the network load distribution is more balanced, the network resources are more efficiently utilized, the stability of the network state is enhanced, and the intelligent agent can better adapt to changes in the network state, etc., a positive reward is given.

[0047] Taking the above factors into consideration, the embodiment of the present invention can design a weighted sum reward function, and use the weight coefficient of the weighted sum reward function to adjust the positive reward and negative reward to balance multiple target tasks, thereby optimizing the initial reward function and obtaining the reward function of the current target model, that is, the reward function of the DQN model. Taking the four targets of delay change, packet loss rate change, network throughput change and network load distribution as examples:

[0048] in, For throughput rewards, For load balancing rewards, To delay the punishment, is the packet loss rate penalty, is the throughput reward weight coefficient, is the weight coefficient of the load balancing reward, is the weight coefficient of delay penalty, is the weight coefficient of the packet loss rate penalty, so that the importance of different factors can be adjusted through the weight coefficient.

[0049] Furthermore, when the DQN model is initialized, the initial value of the weight coefficient is set. During the training process, the weight coefficient is determined by GPT-4.0 according to the requirements of the specific situation. If the current network delay is high, the delay penalty can be increased at this time. The weight coefficient is set to encourage the model to pay more attention to reducing latency. By dynamically adjusting and interpreting the reward function, the adaptability of the model in complex network environments is improved.

[0050] It should be noted that the reward function provides a clear learning goal for the agent, which is to maximize the cumulative reward. By quantifying the agent's strategy into numerical values, the algorithm can evaluate and optimize these performance indicators, thereby helping the agent to optimize the routing algorithm.

[0051] Furthermore, if Figure 2 As shown, in an embodiment of the present invention, after the agent performs an action, it is necessary to store the experience samples of the DQN model, and update the DQN model by periodically implementing experience playback. First, the reward function of the current target model is determined based on the second action value function, and a new network state after the action is executed is generated based on the selected decision action. The experience samples of the current target model are generated according to the reward function, the current network state, the selected decision action and the new network state; secondly, the action experience playback buffer of the DQN model is set according to the specific target task to be performed, each experience sample is converted into a four-tuple (such as state, execution action, reward, new state), and stored in the action experience playback buffer; finally, it is determined whether the action experience playback buffer is full; if the action experience playback buffer is full, the latest stored experience sample replaces the earliest stored experience sample to keep the experience playback buffer updated.

[0052] According to one embodiment of the present invention, after storing the experience samples of the current target model in the action experience replay buffer, it also includes: randomly extracting a preset number of experience samples in the current target model from the action experience replay buffer according to a preset period, and updating the current target model based on the extracted experience samples.

[0053] The preset period and the preset number can be selected by technicians in this field according to actual test requirements and are not specifically limited here.

[0054] Specifically, during the experience replay training process, a certain number of experience samples (forming a small batch) can be randomly extracted from the experience replay buffer according to a preset period, and the current target model can be updated based on the extracted experience samples, thereby reducing the correlation between the data and improving the stability of the training process. Then, according to the iteratively updated current target model, the decision action corresponding to the highest Q value is continuously obtained, thereby obtaining the maximum cumulative reward, and using the existing experience to enable the intelligent agent to achieve the optimal decision in a short period of time to improve the performance of the system.

[0055] In step S103, the target action value function corresponding to each decision action in the initialized target network is calculated, the minimum loss function of the current target model is calculated according to the difference between the target action value function and the second action value function, and the optimal optimization strategy is determined according to the minimum loss function to optimize the satellite networking routing algorithm according to the optimal optimization strategy.

[0056] Specifically, in an embodiment of the present invention, an introduced target network is used to calculate a stable target action value function, i.e., a target Q value, to reduce fluctuations in the training process. The initial target network is initialized synchronously when the DQN model is initialized, and the parameters of the subsequent target network are copied from the DQN model at fixed steps, thereby realizing the update of the target network.

[0057] Furthermore, in an embodiment of the present invention, MSE (Mean Squared Error) can be used to calculate the difference between the second Q value (i.e., the second action value function) under the current online network state and the target Q value under the target network. By measuring the difference between the second Q value and the target Q value and guiding the DQN model training, the loss function value is minimized to obtain the minimum loss function of the current target model. Then, through the minimum loss function, the DQN model can learn a strategy that can select the best action under a given state to maximize the expected cumulative reward, thereby optimizing the satellite networking routing algorithm. Therefore, the DQN training method based on the target network not only improves the stability of the learning process, but also helps to improve the performance of the final strategy.

[0058] Among them, MSE can be expressed as:

[0059] Where N is the number of empirical samples, Predict Q values ​​for the DQN model, is the target Q value.

[0060] The target Q value can be expressed as:

[0061] in, For the reward, is the discount factor, are the parameters of the target network, The determination of needs to consider many factors comprehensively. Since this task requires long-term planning, a higher discount factor can be initially selected without detailed explanation.

[0062] In summary, through the specific description of the above embodiments, the embodiments of the present invention can achieve the following beneficial effects: (1) Strong dynamic adaptability: The deep reinforcement learning model can quickly adapt to new topologies and network states in a dynamically changing satellite network environment through continuous learning and optimization. The intelligent agent can adjust routing strategies based on real-time data to ensure optimal network performance. By combining the natural language descriptions and explanations generated by GPT-4.0, operators can better understand and manage the network decision-making process, improving the interpretability and transparency of the model.

[0063] (2) Capable of processing multiple constraints: By designing a reasonable reward function, the DRL model can simultaneously consider multiple constraints, such as bandwidth, latency, number of hops, load balancing, etc., to achieve multi-objective optimization. GPT-4.0 can dynamically adjust and interpret these weight coefficients, generate adjustment suggestions based on actual network conditions, and improve the adaptability of the model in complex network environments. This multi-constraint processing capability is significantly better than traditional routing algorithms, allowing the optimal routing path to be found in complex network environments.

[0064] According to a satellite networking routing method based on deep reinforcement learning according to an embodiment of the present invention, a first action value function corresponding to each decision action is generated based on the feature vector of the current network state, the decision action selection probability under the current network state is determined based on the current target model, and the preset exploration and utilization strategy, decision action selection probability and the first action value function are used to select and output the corresponding second action value function for each decision action, and the target action value function corresponding to each decision action in the initialized target network is calculated, and the difference between the target action value function and the second action value function is calculated, so as to calculate the minimum loss function of the current target model according to the difference value, so as to optimize the satellite networking routing algorithm. Thus, the problem that the high delay and high bit error rate of the satellite networking make it difficult for the routing algorithm to adapt to the frequently changing network environment, resulting in low data transmission efficiency and serious packet loss, is solved.

[0065] Next, a satellite networking routing device based on deep reinforcement learning proposed according to an embodiment of the present invention is described with reference to the accompanying drawings.

[0066] Figure 3 It is a block diagram of a satellite networking routing device based on deep reinforcement learning according to an embodiment of the present invention.

[0067] like Figure 3 As shown, the satellite networking routing device 10 based on deep reinforcement learning includes: a generation module 100, a determination module 200 and an optimization module 300.

[0068] The generating module 100 is used to obtain the current network state and generate a first action value function corresponding to each decision action in the current network state based on the feature vector of the current network state; The determination module 200 is used to determine the probability of selecting a decision action under the current network state based on the current target model, use the preset exploration and utilization strategy, and select each decision action based on the probability of selecting the decision action and the first action value function, and output the corresponding second action value function according to the selected decision action; The optimization module 300 is used to calculate the target action value function corresponding to each decision action in the initialized target network, calculate the minimum loss function of the current target model according to the difference between the target action value function and the second action value function, and determine the best optimization strategy according to the minimum loss function, so as to optimize the satellite networking routing algorithm according to the best optimization strategy.

[0069] According to one embodiment of the present invention, the determination module 200 includes: A selection unit, configured to randomly select from each decision action according to the first decision action selection probability and the first action value function using a preset exploration and utilization strategy, and output a corresponding second action value function according to the selected decision action; Alternatively, a preset exploration and utilization strategy is used to determine the highest action value function from each decision action according to the second decision action selection probability and the first action value function, and the corresponding second action value function is output according to the selected decision action.

[0070] According to one embodiment of the present invention, after outputting the corresponding second action value function according to the selected decision action, the determination module 200 further includes: A generating unit, configured to determine a reward function of a current target model based on a second action-value function, generate a new network state based on a selected decision action, and generate an experience sample of the current target model according to the reward function, the current network state, the selected decision action, and the new network state; A storage unit, used to determine a target task of the current target model, set an action experience replay buffer of the current target model according to the target task, and store the experience samples of the current target model in the action experience replay buffer; A judgment unit, used to judge whether the action experience playback buffer is full; The replacement unit is used to replace the earliest stored experience sample with the latest stored experience sample if the action experience playback buffer is full.

[0071] According to one embodiment of the present invention, after outputting the corresponding second action value function according to the selected decision action, the determination module 200 includes: According to one embodiment of the present invention, a generating unit includes: a positive reward and a negative reward for defining an initial reward function based on a feedback result of a target task and a current network state; The optimization subunit is used to define the weighted sum reward function of the target task according to the target task, and use the weight coefficient of the weighted sum reward function to adjust the positive reward and negative reward, optimize the initial reward function, and obtain the reward function of the current target model.

[0072] According to one embodiment of the present invention, after storing the experience sample of the current target model in the action experience playback buffer, the storage unit further includes: The updating subunit is used to randomly extract a preset number of experience samples in the current target model from the action experience playback buffer according to a preset period, and update the current target model based on the extracted experience samples.

[0073] According to an embodiment of the present invention, before acquiring the current network status, the generating module 100 further includes: A construction unit, used to construct an initial target model; The acquisition unit is used to initialize the initial target model, train the initialized initial target model, obtain the current target model, and obtain the current network state based on the current target model.

[0074] According to a satellite networking routing device based on deep reinforcement learning according to an embodiment of the present invention, a first action value function corresponding to each decision action is generated based on the feature vector of the current network state, the probability of selecting the decision action under the current network state is determined based on the current target model, and the preset exploration and utilization strategy, the probability of selecting the decision action and the first action value function are used to select and output the corresponding second action value function for each decision action, and the target action value function corresponding to each decision action in the initialized target network is calculated, and the difference between the target action value function and the second action value function is calculated, so as to calculate the minimum loss function of the current target model according to the difference value, so as to optimize the satellite networking routing algorithm. Thus, the problem that the high delay and high bit error rate of the satellite networking make it difficult for the routing algorithm to adapt to the frequently changing network environment, resulting in low data transmission efficiency and serious packet loss, is solved.

[0075] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include: Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0076] When the processor 402 executes the program, a satellite networking routing method based on deep reinforcement learning provided in the above embodiment is implemented.

[0077] Furthermore, the electronic device further comprises: The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0078] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0079] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0080] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0081] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0082] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0083] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the satellite networking routing method based on deep reinforcement learning is implemented as described above.

[0084] An embodiment of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements a satellite networking routing method based on deep reinforcement learning as in the above embodiment.

[0085] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0086] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0087] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0088] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0089] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0090] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0091] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0092] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A satellite networking routing method based on deep reinforcement learning, characterized in that: The following steps are involved: Acquire the current network state, and generate a first action value function corresponding to each decision action in the current network state based on a feature vector of the current network state; Based on the current target model, determine the probability of selecting a decision action under the current network state, use a preset exploration and utilization strategy, and select each decision action based on the decision action selection probability and the first action value function, and output a corresponding second action value function according to the selected decision action; Calculate the target action value function corresponding to each decision action in the initialized target network, calculate the minimum loss function of the current target model according to the difference between the target action value function and the second action value function, and determine the best optimization strategy according to the minimum loss function to optimize the satellite networking routing algorithm according to the best optimization strategy.

2. According to claim 1, a satellite networking routing method based on deep reinforcement learning is characterized in that: The using of a preset exploration and utilization strategy, and selecting each decision action based on the decision action selection probability and the first action value function, and outputting a corresponding second action value function according to the selected decision action, includes: Using the preset exploration and exploitation strategy, randomly selecting from each of the decision actions according to the first decision action selection probability and the first action value function, and outputting a corresponding second action value function according to the selected decision action; Alternatively, the preset exploration and utilization strategy is used to determine the highest action value function from each decision action according to the second decision action selection probability and the first action value function, and the corresponding second action value function is output according to the selected decision action.

3. According to claim 1, a satellite networking routing method based on deep reinforcement learning is characterized in that: After outputting the corresponding second action value function according to the selected decision action, the method further includes: Determine a reward function of the current target model based on the second action value function, generate a new network state based on the selected decision action, and generate an experience sample of the current target model according to the reward function, the current network state, the selected decision action and the new network state; Determine a target task of the current target model, set an action experience replay buffer of the current target model according to the target task, and store experience samples of the current target model in the action experience replay buffer; Determining whether the action experience playback buffer is full; If the action experience playback buffer is full, the latest stored experience sample replaces the earliest stored experience sample.

4. According to claim 3, a satellite networking routing method based on deep reinforcement learning is characterized in that: The determining the reward function of the current target model based on the second action-value function includes: Defining a positive reward and a negative reward of an initial reward function based on the feedback result of the target task and the current network state; A weighted sum reward function of the target task is defined according to the target task, and the positive reward and the negative reward are adjusted using the weight coefficient of the weighted sum reward function to optimize the initial reward function to obtain the reward function of the current target model.

5. According to claim 3, a satellite networking routing method based on deep reinforcement learning is characterized in that: After storing the experience samples of the current target model in the action experience playback buffer, the method further includes: A preset number of experience samples in the current target model are randomly extracted from the action experience playback buffer according to a preset period, and the current target model is updated based on the extracted experience samples.

6. The satellite networking routing method based on deep reinforcement learning according to claim 1, characterized in that: Before obtaining the current network status, the method further includes: Constructing the initial target model; The initial target model is initialized, and the initialized initial target model is trained to obtain a current target model, and the current network state is obtained based on the current target model.

7. A satellite networking routing device based on deep reinforcement learning, characterized in that: include: A generation module, used to obtain a current network state, and generate a first action value function corresponding to each decision action under the current network state based on a feature vector of the current network state; A determination module, used to determine the probability of selecting a decision action under the current network state based on the current target model, use a preset exploration and utilization strategy, and select each decision action based on the decision action selection probability and the first action value function, and output a corresponding second action value function according to the selected decision action; The optimization module is used to calculate the target action value function corresponding to each decision action in the initialized target network, calculate the minimum loss function of the current target model according to the difference between the target action value function and the second action value function, and determine the best optimization strategy according to the minimum loss function, so as to optimize the satellite networking routing algorithm according to the best optimization strategy.

8. A satellite networking routing device based on deep reinforcement learning according to claim 7, characterized in that: The determining module comprises: A selection unit, configured to randomly select from each of the decision actions according to the first decision action selection probability and the first action value function using the preset exploration and utilization strategy, and output a corresponding second action value function according to the selected decision action; Alternatively, the preset exploration and utilization strategy is used to determine the highest action value function from each decision action according to the second decision action selection probability and the first action value function, and the corresponding second action value function is output according to the selected decision action.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a satellite networking routing method based on deep reinforcement learning as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement a satellite networking routing method based on deep reinforcement learning as described in any one of claims 1-6.

Images