Network planning method and device based on hybrid reinforcement learning strategy

Abstract

The invention discloses a network planning method and device based on a hybrid reinforcement learning strategy, and the method comprises the steps: firstly, providing a basic network structure and an adjustment strategy through building a data link network topological structure model and a strategy support library; and then, an optimization strategy model is constructed based on multi-relation constraints, and efficient network planning decision optimization is performed through a hybrid reinforcement learning strategy so as to cope with a complex network environment. And then, generating an initial network planning structure through small sample data input, and finally, performing security review through the trusted network planning decision model to obtain a target network planning structure conforming to the standard. According to the method, a mixed strategy reinforcement learning method is adopted, high-dimensional target optimization and agile and efficient decision making under the condition of dynamic environment change are achieved, an intelligent planning technology based on a small data set and empirical knowledge is introduced, and the credibility and safety of network planning decision making are enhanced.

Description

Technical Field

[0001] This application relates to the field of network planning technology, and more specifically, to a network planning method and apparatus based on a hybrid reinforcement learning strategy. Background Technology

[0002] Currently, data link networks play a crucial role in modern communications, transportation, and logistics, and their efficient planning and optimization have a significant impact on system stability and resource utilization. However, with the expansion of network scale and increasing complexity, traditional planning methods face challenges and struggle to adapt to changing network environments and resource demands in real time. Existing technical literature includes some research based on network topology and control strategy modeling, but these methods mostly rely on large amounts of historical data or global optimization algorithms, making them ill-suited for rapidly changing dynamic scenarios. Furthermore, the application of traditional reinforcement learning algorithms in data link networks is often limited by the large-scale data requirements and low sample efficiency, particularly in network planning.

[0003] One improvement is to model the data link network topology and adjustment strategies, enabling the system to quickly adapt to different link or node failures. However, this approach often faces challenges in practical deployments, such as large data requirements and difficulty in effective generalization. To address these issues, some research has begun exploring small-sample learning strategies to improve adaptability to scarce data. However, these methods still have significant room for improvement in inference speed and accuracy, particularly in decision-making agility and real-time planning.

[0004] Furthermore, some studies combining the "expert in-the-loop" strategy have shown that, with the real-time participation of human experts, systems can achieve higher decision-making reliability and efficiency in complex network planning scenarios. These methods leverage experts' domain knowledge and introduce feedback mechanisms into reinforcement learning frameworks to enhance system credibility. However, due to the lack of automation and real-time guarantees, existing solutions still face significant challenges in achieving a balance between credibility and decision-making efficiency in highly complex data link networks.

[0005] Existing technologies have certain advantages in planning and reasoning in data link networks, such as efficient planning capabilities applicable to specific scenarios, but they generally lack effective methods to adapt to small sample data in rapidly changing network environments while taking into account decision agility and reliability. Summary of the Invention

[0006] To address at least one deficiency or improvement need in the prior art, this invention provides a network planning method and apparatus based on a hybrid reinforcement learning strategy, which has the advantages of being able to quickly adapt to dynamically changing network environments, meet real-time decision-making requirements, achieve efficient real-time planning, and balance reliability and decision-making efficiency in complex network environments.

[0007] To achieve the above objectives, according to a first aspect of the present invention, a network planning method based on a hybrid reinforcement learning strategy is provided, comprising: A data link network topology model is established, and based on the data link network topology model, a data link network structure adjustment and optimization strategy is pre-determined according to different business requirements to obtain a strategy support library. Based on the aforementioned strategy support library, construct an optimization strategy model for a data link network structure with multiple relational constraints. The network planning and decision optimization model is obtained by using a hybrid strategy agile and efficient reinforcement learning decision-making method for the optimization strategy model. Using small sample data from real-world scenarios as input to the efficient decision-making model, the initial network planning structure output by the efficient decision-making model is obtained; The initial network planning structure is used as input to a trusted network planning decision model to obtain a target network planning structure that meets security review standards. The trusted network planning decision model is a decision model built by experts in a loop based on a preset action library, including: constructing a preset action library based on security review standards; obtaining the actions of the initial network planning structure; determining whether the action is a dangerous action before execution; and replacing the dangerous action with a safe action if the action is dangerous, thereby enhancing the credibility of the decision result and obtaining the trusted network planning decision model.

[0008] Furthermore, based on the aforementioned data link network topology model, data link network structure adjustment and optimization strategies are pre-established according to different business requirements, resulting in a strategy support library, including: Configure node functional attributes and node location characteristics for network nodes according to business requirements. The node functional attributes include command nodes, relay nodes and ordinary nodes, and the location characteristics include air, land and sea. Perlin noise simulation was used, and a coastline was generated by rotation and movement to divide the action area and obtain a set of position coordinates. Based on the node functional attributes, location characteristics, and location coordinate set, a data link network structure adjustment and optimization strategy is determined, resulting in a strategy support library.

[0009] Furthermore, the step of constructing an optimization strategy model based on a data link network structure with multiple relational constraints based on the strategy support library includes: A deep deterministic strategy gradient algorithm is used to process high-dimensional continuous action spaces; A multi-agent deep deterministic policy gradient algorithm is used to handle the cooperation and competition relationships among multiple agents and generate an optimized policy model.

[0010] Furthermore, a hybrid strategy-based agile and efficient reinforcement learning decision-making method is applied to the optimized strategy model to optimize network planning and decision-making, resulting in an efficient decision-making model, including: For network node planning, power adjustment planning, frequency adjustment planning, and temporary node planning, respectively establish policy constraints and implementation methods; Based on the established policy constraints and implementation methods, the actions of relay nodes and ordinary nodes are controlled and scored and adjusted using the MADDPG model to optimize network planning and obtain an efficient decision-making model.

[0011] Furthermore, the implementation of the network node planning includes: A multi-agent deep deterministic policy gradient algorithm is used to control and score the actions of relay nodes. Each relay node not in the maximum relay node group is identified as an independent agent. After executing an action, the closer the agent is to the maximum relay node group, the higher the score; otherwise, a penalty is imposed. Once a relay node enters the maximum relay node group, it is no longer controlled as an independent agent. The MADDPG algorithm is used to control and adjust the actions and scores of ordinary nodes. Each ordinary node is defined as an independent agent. A circle with the nearest relay node as the center and the maximum communication distance as the radius is used as a reference. The closer the ordinary node is to this circle, the higher the score. If it is outside the circle, it is penalized. When it is inside the circle, the score remains unchanged and is adjusted according to the distance between the ordinary node and other ordinary nodes to avoid the overlap of ordinary nodes. The implementation methods of the power adjustment plan include: The maximum transmission distance of a normal node is determined to be its nearest relay node, and its own transmission power is calculated based on the maximum transmission distance; A deep deterministic policy gradient algorithm is used to control the transmission power of relay nodes, and all relay nodes are identified as an agent. The agent is scored only if the constraints are met, otherwise the score is zero. If the constraints cannot be met within a set step size, the agent is penalized.

[0012] Furthermore, the implementation method of the frequency adjustment plan includes: When changing channels, the input power, received power, and maximum transmission distance of each node are adjusted according to the Friis propagation loss model. The MADDPG algorithm is used to control and score the actions of relay nodes. Each relay node that is not in the largest relay node group is identified as an independent agent. After executing an action, the closer it is to the largest relay node group, the higher the score; otherwise, a penalty is imposed. Once a relay node enters the largest relay node group, it is no longer controlled as an independent agent. The MADDPG algorithm is used to control and adjust the actions of ordinary nodes. Each ordinary node is defined as an independent agent. A circle with the nearest relay node as the center and the maximum communication distance as the radius is used as a reference. The closer the ordinary node is to this circle, the higher the score. If it is outside the circle, it is penalized. When it is inside the circle, the score remains unchanged and is adjusted according to the distance between the ordinary node and other ordinary nodes to avoid the overlap of ordinary nodes.

[0013] Furthermore, the implementation method of the temporary node planning includes: Each temporary relay node is defined as an independent intelligent agent, and its switching state and position are controlled by the MADDPG algorithm; Each connected relay node is defined as a relay node group; the node information of the largest relay node group and its neighboring relay node groups is taken as input, and the output includes whether the node is started, the effective location after starting, and the moving distance; if starting enables the largest relay node group to connect with the neighboring relay node groups, a reward is given, and the shorter the moving distance, the higher the reward; if the connection is not established, a penalty is imposed. After all relay nodes are merged into one group, isolated ordinary nodes are located to obtain isolated nodes; all relay nodes include ephemeral relay nodes that have been started. The program takes information about isolated nodes and the nearest relay node as input and outputs whether to start the node, the start location, and the movement distance. If starting the node enables the isolated ordinary node to connect with the nearby relay node group, a reward is given, and the shorter the movement distance, the higher the reward. If the connection is not established, a penalty is imposed.

[0014] Furthermore, the step of using small sample data from real-world scenarios as input to the efficient decision-making model to obtain the initial network planning structure output by the efficient decision-making model includes: Extract the corresponding state-action pairs from the decision data available in empirical knowledge to construct a new decision set; The different situational information in the state is decoupled into evidence entities, the action information in the joint action is decoupled into action entities, and the development relationship from evidence entities to action entities is established based on empirical knowledge. A relational graph is constructed using evidence entities, action entities, and relationships. The learning of the network planning model is guided by an adversarial imitation learning method to generate an initial network planning structure suitable for small sample conditions.

[0015] According to a second aspect of the present invention, a network planning apparatus based on a hybrid reinforcement learning strategy is also provided, comprising: a data link network topology model building module, used to build a data link network topology model and, based on the data link network topology model, pre-determine data link network structure adjustment and optimization strategies according to different business requirements to obtain a strategy support library; a target optimization decision module, used to construct an optimization strategy model of a data link network structure based on multiple relational constraints based on the strategy support library; a reinforcement learning decision module, used to perform network planning decision optimization on the optimization strategy model using a hybrid strategy agile and efficient reinforcement learning decision method to obtain an efficient decision model; and a small sample planning module, used to plan small samples in real-world scenarios. This data serves as input to the efficient decision-making model, obtaining the initial network planning structure output by the efficient decision-making model. The expert-in-the-loop trusted planning module uses the initial network planning structure as input to the trusted network planning decision model to obtain a target network planning structure that meets security review standards. The trusted network planning decision model is a decision model built using an expert-in-the-loop approach based on a preset action library, including: constructing a preset action library based on security review standards; obtaining actions from the initial network planning structure; determining whether an action is a dangerous action before execution; and replacing a dangerous action with a safe action if the action is dangerous, thereby enhancing the credibility of the decision result and obtaining the trusted network planning decision model.

[0016] Thirdly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the network planning method based on any of the above-described hybrid reinforcement learning strategies.

[0017] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: By establishing a data link network topology model and policy support library, a basic network structure and adjustment strategies are provided. Next, an optimization strategy model is constructed based on multi-relationship constraints, and a hybrid reinforcement learning strategy is used for efficient network planning decision optimization to cope with complex network environments. Then, an initial network planning structure is generated using small sample data input. Finally, a security review is performed using a trusted network planning decision model to obtain a target network planning structure that conforms to standards.

[0018] This method enables data link networks to rapidly adapt to different link or node failures, solving the problems of large data requirements and poor generalization in traditional methods. Combined with a few-shot learning strategy, it improves adaptability to scarce data and significantly enhances inference speed and accuracy. The "expert in-the-loop" strategy enhances the credibility and security of decisions, enabling efficient decision-making in complex network planning scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating an optional network planning method based on a hybrid reinforcement learning strategy provided in this application embodiment; Figure 2 A flowchart illustrating an optional node movement strategy implementation method provided in an embodiment of this application; Figure 3 A flowchart illustrating an optional node mobility power implementation method provided in this application embodiment; Figure 4 A flowchart illustrating an optional node frequency strategy implementation method provided in an embodiment of this application; Figure 5 A flowchart illustrating an optional node as a relay node strategy implementation method provided in this application embodiment; Figure 6 A flowchart illustrating an optional small dataset data link network planning and reasoning method based on adversarial imitation learning, provided for an embodiment of this application; Figure 7 A flowchart illustrating an optional small sample library relay node provided in an embodiment of this application; Figure 8 A flowchart illustrating another optional "expert in-the-loop" trusted data link network planning technique provided for embodiments of this application; Figure 9 A schematic diagram of the experimental results of an optional "expert in loop" application provided in this embodiment of the application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0022] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0023] According to one aspect of the embodiments of this application, a network planning method based on a hybrid reinforcement learning strategy is provided. The following is in conjunction with... Figure 1-9 This application describes a network planning method based on a hybrid reinforcement learning strategy provided in its embodiments.

[0024] Figure 1 This is a flowchart illustrating an optional network planning method based on a hybrid reinforcement learning strategy provided in an embodiment of this application, as shown below. Figure 1 As shown, the process of this method may include the following steps: S102, Establish a data link network topology model, and based on the data link network topology model, predetermine data link network structure adjustment and optimization strategies according to different business needs to obtain a strategy support library; S104, Construct an optimization strategy model based on the strategy support library and a data link network structure with multiple relational constraints; S106, The network planning and decision optimization is performed on the optimization strategy model using a hybrid strategy agile and efficient reinforcement learning decision-making method to obtain an efficient decision model; S108, Using small sample data from real-world scenarios as input to the efficient decision-making model, the initial network planning structure output by the efficient decision-making model is obtained; S110, the initial network planning structure is used as input to the trusted network planning decision model to obtain the target network planning structure that meets the security review criteria.

[0025] The trusted network planning decision model is a decision model established by experts in the loop based on a preset action library, including: constructing a preset action library based on security review standards; obtaining the actions of the initial network planning structure; determining whether the action is a dangerous action before execution; and replacing the dangerous action with a safe action if the action is a dangerous action, so as to enhance the credibility of the decision result and obtain the trusted network planning decision model.

[0026] This application proposes a hybrid reinforcement learning strategy for application in network planning and reasoning, aiming to solve problems such as insufficient real-time adaptability, large data requirements, low inference efficiency, and lack of credibility guarantees in data link network planning.

[0027] By establishing a data link network topology model and pre-determining data link network structure adjustment and optimization strategies based on different business needs, a strategy support library is obtained. Based on the strategy support library, an optimization strategy model for the data link network structure based on multi-relationship constraints is constructed. An agile and efficient reinforcement learning decision-making method using hybrid strategies is applied to the optimization strategy model for network planning decision optimization, resulting in an efficient decision model. Small sample data from real-world scenarios are used as input to the efficient decision model to obtain the initial network planning structure output by the efficient decision model. The initial network planning structure is then used as input to a trusted network planning decision model to obtain a target network planning structure that meets security review standards. The trusted network planning decision model is a decision model established by experts in a loop based on a pre-defined action library.

[0028] Through steps S102 to S108 described above, the technical solution of this application solves the problem of efficient planning and optimization of data link networks under different business requirements. Specifically, firstly, a basic network structure and adjustment strategy are provided by establishing a data link network topology model and a strategy support library. Next, an optimization strategy model is constructed based on multi-relationship constraints, and efficient network planning decision optimization is performed through a hybrid reinforcement learning strategy to cope with complex network environments. Then, an initial network planning structure is generated through small sample data input, and finally, a security review is performed through a trusted network planning decision model to obtain a target network planning structure that conforms to standards.

[0029] In this embodiment, a data link network topology model is first established, and optimization strategies for adjusting the data link network structure are pre-determined based on different business requirements, resulting in a strategy support library. Further, an optimization strategy model for the data link network structure based on multi-relationship constraints is constructed based on the strategy support library. Then, an agile and efficient reinforcement learning decision-making method using hybrid strategies is applied to the optimization strategy model for network planning decision optimization, resulting in an efficient decision model. Next, small sample data from real-world scenarios is used as input to the efficient decision model to obtain the initial network planning structure output by the efficient decision model. Finally, the initial network planning structure is used as input to a trusted network planning decision model to obtain a target network planning structure that meets security review standards.

[0030] This method enables data link networks to rapidly adapt to different link or node failures, solving the problems of large data requirements and poor generalization in traditional methods. Combined with a few-shot learning strategy, it improves adaptability to scarce data and significantly enhances inference speed and accuracy. Furthermore, the "expert in-the-loop" strategy enhances the credibility and security of decision-making, enabling efficient decision-making in complex network planning scenarios.

[0031] In an exemplary embodiment, based on the data link network topology model, a data link network structure adjustment and optimization strategy is pre-established according to different business requirements, resulting in a strategy support library, including: Configure node functional attributes and node location characteristics for network nodes according to business requirements. The node functional attributes include command nodes, relay nodes and ordinary nodes, and the location characteristics include air, land and sea. Perlin noise simulation was used, and a coastline was generated by rotation and movement to divide the action area and obtain a set of position coordinates. Based on the node functional attributes, location characteristics, and location coordinate set, a data link network structure adjustment and optimization strategy is determined, resulting in a strategy support library.

[0032] In this embodiment, network nodes are configured with node functional attributes and location characteristics according to business requirements. Node functional attributes include command nodes, relay nodes, and ordinary nodes. For example, command nodes are assigned command functions, relay nodes are assigned communication functions, and ordinary nodes are randomly assigned four different functional attributes. Location characteristics include air, land, and sea. Node functional attributes and location characteristics can be configured through a user interface or a preset configuration file. For example, nodes can be dragged and dropped to the corresponding geographical location through a graphical interface, and their functional attributes can be selected.

[0033] Perlin noise is used to simulate and generate coastlines through rotation and movement, dividing the action area into two parts: ocean and land. For example, network nodes with a location type of "ocean" are distributed in the ocean, nodes with a location type of "land" are distributed on land, and nodes with a location type of "air" are randomly distributed throughout the area. Perlin noise is a commonly used algorithm in computer graphics that generates smooth, random noise to simulate natural terrain. By rotating and moving the generated noise, coastlines that conform to actual geographical features can be generated, and different action areas can be defined.

[0034] The data link network structure adjustment and optimization strategy is determined based on the node's functional attributes, location characteristics, and location coordinates. Functional attributes are assigned to each node according to actual needs. For example, command nodes are assigned command functions, relay nodes are assigned communication functions, and ordinary nodes are randomly assigned four different functional attributes.

[0035] Furthermore, a strategy support library based on prior experience is established according to the functional attributes, location characteristics, and positions of nodes. Thus, algorithms can analyze the functional attributes and location characteristics of different nodes, and combined with a set of location coordinates, generate an optimized network structure adjustment strategy. These strategies can be stored in the strategy support library for use in subsequent network planning and adjustments.

[0036] For example, for nodes with insufficient coverage, it is generally recommended to add repeater nodes or adjust power; for nearby nodes that are far away or have weak connections, it is recommended to move repeater nodes; and in cases of significant interference, it is recommended to adjust the frequency to reduce interference.

[0037] The recommended strategy follows three principles: "adding nodes," "optimizing connections," and "reducing interference" to ensure network stability and coverage, and to ensure that the optimal solution can be adopted in different scenarios, thereby improving the network's responsiveness.

[0038] The above solution addresses the problem of how to configure functional attributes and location characteristics of network nodes to establish data link network structure adjustment and optimization strategies based on business needs. Specifically, configuring node functional attributes and location characteristics can meet different business requirements; using Perlin noise simulation to generate coastlines and divide action areas provides a more realistic geographical environment simulation; and combining node functional attributes, location characteristics, and location coordinate sets determines specific optimization strategies, thus establishing a flexible and efficient strategy support library. This not only supports network structure adjustment and optimization under different business needs but also enhances the accuracy of cross-domain communication and the adaptive capability of information hinges, making the data link network more flexible and efficient in diverse task environments.

[0039] In an exemplary embodiment, the step of constructing an optimization strategy model based on a data link network structure with multiple relational constraints based on the strategy support library includes: A deep deterministic strategy gradient algorithm is used to process high-dimensional continuous action spaces; A multi-agent deep deterministic policy gradient algorithm is used to handle the cooperation and competition relationships among multiple agents and generate an optimized policy model.

[0040] In this embodiment, firstly, the parameters of the policy network and the value network are initialized; then, state-action pairs are sampled and generated in the environment, and the value of the actions is evaluated through the value network; next, the parameters of the policy network are updated based on the evaluation results; finally, the above process is repeated until the policy network converges. The implementation of the multi-agent deep deterministic policy gradient algorithm is similar, except that the above process is performed among multiple agents, and the policy is optimized through shared experience and collaborative learning.

[0041] By employing deep deterministic policy gradient (DPR) and multi-agent DPR algorithms, we can effectively handle high-dimensional continuous action spaces and the cooperative and competitive relationships among multiple agents, thereby constructing an optimized policy model. Deterministic actions are generated through a DPR network, and the quality of the policy is evaluated through a value network to update the network parameters. The multi-agent DPR algorithm is an extension of the DPR algorithm, applicable to multi-agent systems. Through collaborative learning among multiple agents, it can handle the cooperative and competitive relationships between agents, thus generating a more optimized policy model. This optimized policy model can better adapt to complex network environments, improving the efficiency of network planning and the quality of decision-making.

[0042] In an exemplary embodiment, a hybrid strategy-based agile and efficient reinforcement learning decision-making method is used to optimize the network planning and decision-making of the optimization strategy model, resulting in an efficient decision-making model, including: For network node planning, power adjustment planning, frequency adjustment planning, and temporary node planning, respectively establish policy constraints and implementation methods; Based on the established policy constraints and implementation methods, the actions of relay nodes and ordinary nodes are controlled and scored and adjusted using the MADDPG model to optimize network planning and obtain an efficient decision-making model.

[0043] In this embodiment, policy constraints and implementation methods are established for network node planning, power adjustment planning, frequency adjustment planning, and temporary node planning, respectively. Based on the established policy constraints and implementation methods, the actions of relay nodes and ordinary nodes are controlled and scored and adjusted through the MADDPG model to optimize network planning and obtain an efficient decision model.

[0044] Specifically, in network node planning, a multi-agent deep deterministic policy gradient algorithm is used to control and adjust the actions of relay nodes. Each relay node not in the maximum relay node group is defined as an independent agent. After executing an action, the closer the agent is to the maximum relay node group, the higher the score; otherwise, a penalty is imposed. Once a relay node enters the maximum relay node group, it is no longer controlled as an independent agent. The MADDPG algorithm is used to control and adjust the actions of ordinary nodes. Each ordinary node is defined as an independent agent. A circle with the nearest relay node as the center and the maximum communication distance as the radius is used as a reference. The closer an ordinary node is to this circle, the higher the score; if it is outside the circle, a penalty is imposed. When inside the circle, the score remains unchanged and is adjusted according to the distance between the ordinary node and other ordinary nodes to avoid overlap of ordinary nodes.

[0045] For example, a. Policy constraints: 1. Ensure all nodes are connected to each other under the condition that only the node positions are changed; 2. Ordinary nodes that are far apart can communicate with each other through relay nodes; 3. Try to avoid placing ordinary nodes too close together.

[0046] b. Implementation method (e.g.) Figure 2 ) The MADDPG model is used to control the actions of relay nodes, perform scoring and adjustments, and ensure that each relay node not in the largest relay node group is... Figure 1 Each agent is an independent intelligent agent. After performing an action, the closer the agent is to the largest relay node group, the higher its score; if it moves away from the largest relay node group, it is penalized. Once a relay node enters the largest relay node group, it is no longer considered an intelligent agent.

[0047] The MADDPG model is used to control the actions of ordinary nodes, and to score and adjust them. Each ordinary node is regarded as an agent. The closer it is to the nearest relay node, the higher its score. If it moves away from the circle, it is penalized. If it is inside the circle, the score remains unchanged.

[0048] Adjustments are made based on the distance between ordinary nodes and other ordinary nodes to avoid overlapping ordinary nodes while keeping the score unchanged.

[0049] In the power adjustment planning, the maximum transmission distance of a normal node is determined to be its nearest relay node, and its own transmission power is calculated based on the maximum transmission distance. The transmission power of the relay nodes is controlled by a deep deterministic policy gradient algorithm, and all relay nodes are identified as an agent. The agent is scored only if the constraints are met, otherwise the score is zero. If the constraints cannot be met within the set step size, the agent is penalized.

[0050] For example, a. policy constraints 1. Ensure all nodes are connected to each other while only changing the power. 2. Ordinary nodes that are far apart can communicate with each other through relay nodes; 3. To minimize the total power used by all nodes.

[0051] b. Implementation method (e.g.) Figure 3 ) The maximum transmission distance for all ordinary nodes is the nearest relay node, and their own transmission power is calculated.

[0052] The transmit power of relay nodes is controlled by the DDPG model, and all relay nodes are treated as an agent. Scores are only awarded under constraints 1 and 2; otherwise, the score is 0. A penalty is applied if the constraints are not met within a set timeframe.

[0053] Under the condition that constraints 1 and 2 are satisfied, the smaller the maximum total power of all relay nodes, the higher the score.

[0054] In frequency adjustment planning, when changing channels, the input power, received power, and maximum transmission distance of each node are adjusted according to the Friis propagation loss model. The MADDPG algorithm is used to control and score the actions of relay nodes. Each relay node outside the maximum relay node group is identified as an independent agent. After executing an action, the closer it is to the maximum relay node group, the higher its score; otherwise, a penalty is imposed. Once a relay node enters the maximum relay node group, it is no longer controlled as an independent agent. The MADDPG algorithm is also used to control and score the actions of ordinary nodes. Each ordinary node is identified as an independent agent. A circle with the nearest relay node as the center and the maximum communication distance as the radius is used as a reference. The closer an ordinary node is to this circle, the higher its score; if it is outside the circle, a penalty is imposed. When inside the circle, the score remains unchanged and is adjusted according to the distance between the ordinary node and other ordinary nodes to avoid overlap of ordinary nodes.

[0055] For example, a. policy constraints 1. In the event of a channel attack, modify the frequency and change the channel, and accordingly modify the network topology so that the network remains connected under the new channel; 2. Ordinary nodes that are far apart can communicate with each other through relay nodes; 3. Try to avoid overlapping of ordinary nodes.

[0056] b. Implementation method (e.g.) Figure 4 ) When changing channels, the input power, received power, and maximum transmission distance of each node are adjusted according to the Friis propagation loss model.

[0057] Based on the adjusted input power, received power, and maximum transmission distance, the MADDPG model controls the actions and scores of relay nodes. Each relay node outside the maximum group is treated as an independent agent; the closer it is to the maximum group after performing an action, the higher its score, and vice versa. Relay nodes entering the maximum group are no longer considered agents.

[0058] The MADDPG model controls the actions of ordinary nodes, assigning scores and making adjustments. Each ordinary node is considered an agent, and its score is higher the closer it is to a circle centered on its nearest relay node, with a maximum communication distance as its radius. Moving further away from this circle incurs a penalty, while the score remains unchanged when within the circle. Nodes can communicate with each other through relay nodes; 3. The total power used by all nodes is minimized.

[0059] In the planning of temporary relay nodes, each temporary relay node is defined as an independent intelligent agent, and its on / off state and position are controlled using the MADDPG algorithm. Each connected relay node is defined as a relay node group. The node information of the largest relay node group and its neighboring relay node groups are used as input, and the output includes whether the node is started, the effective position after starting, and the moving distance. If starting enables the largest relay node group to connect with the neighboring relay node groups, a reward is given, and the shorter the moving distance, the higher the reward; if the connection is not established, a penalty is imposed. After all relay nodes are merged into one group, isolated ordinary nodes are located to obtain isolated nodes; all relay nodes include started temporary relay nodes. The information of isolated nodes and the nearest relay node is used as input, and the output includes whether the node is started, the start position, and the moving distance. If starting enables the isolated ordinary node to connect with the neighboring relay node groups, a reward is given, and the shorter the moving distance, the higher the reward; if the connection is not established, a penalty is imposed.

[0060] For example, a. policy constraints 1. Ensure all nodes are connected to each other under the condition that only the temporary relay node is started and moved; 2. Ordinary nodes that are far apart can communicate with each other through relay nodes or activated temporary relay nodes; 3. Minimize the number of temporary relay nodes that are activated and reduce the travel distance of each temporary relay node.

[0061] b. Implementation method (e.g.) Figure 5 ) Each temporary relay node is considered an agent. The MADDPG model is used to control the switching on and off and the location of the temporary relay nodes, but only a small number of agents are activated in each step.

[0062] Ignoring all ordinary nodes, each connected relay node is considered a relay node group. The input consists of the node information of the largest relay node group and its neighboring relay node groups. The output includes whether the node is activated, its effective location after activation, and the distance it travels. If activation connects the largest relay node group and its neighboring relay node groups, a reward is given, with higher rewards for shorter travel distances. Activated temporary relay nodes are considered relay nodes. If connectivity is lost, a penalty is imposed.

[0063] After merging all relay nodes (including activated ephemeral relay nodes) into a group, locate the isolated ordinary node. In each step, the isolated node and its nearest relay node are used as input, and the output is whether to activate the node, the activation location, and the movement distance. If activation connects the isolated ordinary node to the nearest relay node group, a reward is given, with higher rewards for shorter movement distances; activated ephemeral nodes are considered relay nodes. If no connection is established, a penalty is imposed.

[0064] This application employs a hybrid strategy-based agile and efficient reinforcement learning decision-making method, enabling rapid adaptation to changes in the network environment. It optimizes network planning by controlling and adjusting node actions through the MADDPG model. High-dimensional objective optimization strategy modeling based on multi-relationship constraints further enhances the system's adaptability under cooperative and competitive relationships, ensuring high efficiency and accuracy in dynamic environments. This scheme optimizes network planning and obtains an efficient decision-making model by establishing policy constraints and implementation methods, and using the MADDPG model to control and adjust node actions. This improves the agility and efficiency of network planning, enabling better adaptation to dynamically changing network environments.

[0065] In an exemplary embodiment, the step of using small sample data from real-world scenarios as input to the efficient decision-making model to obtain the initial network planning structure output by the efficient decision-making model includes: Extract the corresponding state-action pairs from the decision data available in empirical knowledge to construct a new decision set; The different situational information in the state is decoupled into evidence entities, the action information in the joint action is decoupled into action entities, and the development relationship from evidence entities to action entities is established based on empirical knowledge. A relational graph is constructed using evidence entities, action entities, and relationships. The learning of the network planning model is guided by an adversarial imitation learning method to generate an initial network planning structure suitable for small sample conditions.

[0066] In this embodiment, data suitable for decision-making is first extracted from empirical knowledge to form state-action pairs, and a new decision set is constructed. Then, different situational information within the states is decoupled into evidence entities, and action information in joint actions is decoupled into action entities. Based on empirical knowledge, a developmental relationship from evidence entities to action entities is established. Next, a relationship graph is constructed using evidence entities, action entities, and the relationships between them. Finally, an adversarial imitation learning method is used to guide the learning of the network planning model, generating an initial network planning structure suitable for small sample conditions.

[0067] Specifically, in real-world network planning applications, network conditions are presented in the form of small samples, which affects the generalization of decision-making models and makes it difficult to discover the optimal planning strategy. However, there is a wealth of empirical knowledge available in reality that can provide guidance for decision-making models.

[0068] First, utilize the decision data available from empirical knowledge. Extract the corresponding state-action pairs and construct a new decision set. Then, the different situational information in the state is decoupled into evidence entities, the action information in the joint action is decoupled into action entities, and the development relationship from evidence entities to action entities is established based on empirical knowledge. Then, a relationship graph is constructed using evidence entities, action entities, and relationships.

[0069] This method transforms the decision-making process from experiential knowledge into a reasoning process within a graph, and guides the learning of a network planning model through adversarial imitation learning. The training objective of the network planning model is to match the state-action trajectory distribution generated by the model with the guided trajectory distribution. A small dataset data link network planning and reasoning method based on adversarial imitation learning is proposed, such as... Figure 6 As shown.

[0070] Taking the adjustment strategy of moving relay nodes as an example, this invention constructs a small sample library for network planning and reasoning based on the adjustment strategy modeling rules described above. This invention selects three information points from the small sample library to illustrate the positions of relay nodes before and after the execution of their actions, as shown in Table 1. "1:" and "2:" represent the relay node numbers, and the content within parentheses indicates the coordinates of the relay node before and after the algorithm action is executed. Taking relay node 1 in the first row as an example, before executing the moving strategy, node 1's position is (249.74, 246.85). After executing the adjustment strategy of moving relay nodes, its position becomes (252.65, 255.92). This small sample data can alleviate the problem of data sparsity and help the algorithm model improve its adaptability to various real-world scenarios. Based on the small sample library constructed above, this invention selects an example of an algorithm model's moving strategy execution to illustrate the role of small sample information. For example... Figure 7 As shown, green dots represent the positions of each node, solid red lines represent the connections between command nodes and relay nodes, and dashed black lines represent the connections between ordinary nodes and relay (command) nodes.

[0071] This application constructs a relational graph and guides the learning of the network planning model using adversarial imitation learning, enabling the effective generation of initial network planning structures under conditions of small sample data, ensuring the accuracy and effectiveness of the planning. Addressing the small sample characteristics of data chain networks in real-world scenarios, it applies intelligent planning techniques based on small datasets and empirical knowledge, achieving reasonable generation and optimization of network structures under limited data conditions. This reduces dependence on large-scale training data and meets planning needs under resource-scarce circumstances.

[0072] In one exemplary embodiment, an expert-built trusted network planning and decision-making model is established based on a preset action library, including: A library of preset actions was built based on security review standards. The actions of obtaining the initial network planning structure are determined. Before the action is executed, it is determined whether the action is a dangerous action. If the action is a dangerous action, it is replaced with a safe action to enhance the credibility of the decision result and obtain a credible network planning decision model.

[0073] During network planning and reasoning, a situation may arise where a large number of nodes lose contact due to the inability of relay nodes and command nodes to communicate. This situation significantly impacts training efficiency. Therefore, this invention proposes a "reliable planning" approach based on "experts in the loop".

[0074] First, a pre-defined action library needs to be built based on security review standards. This library contains various possible network planning actions and their security assessment criteria. After obtaining the actions for the initial network planning structure, a judgment needs to be made before execution to determine whether the action is dangerous. Specifically, the risk level of the action can be assessed using pre-defined security standards and an expert knowledge base. If an action is determined to be dangerous, it needs to be replaced with a corresponding safe action. In this process, the expert-in-the-loop mechanism plays a crucial role, enabling real-time monitoring and feedback of the security assessment results to ensure that the replaced action complies with the security standards.

[0075] Before performing an action, determine whether it is a dangerous action and replace it with a safe action. For example... Figure 8 As shown, the expert-in-the-loop mechanism acts as a "safety filter," reviewing the actions generated by the system for security. Only actions that pass the review are executed. This design improves the reliability and security of the system's decisions in complex scenarios.

[0076] An action library is a module that stores executable actions, including safe actions, unsafe actions, and safe actions that are re-evaluated. For example... Figure 9 The left and right sides represent the experimental conditions in the loop with and without the use of experts, respectively.

[0077] Observing (a) and (b), we can see that the cumulative reward change in (a) without using the expert-in-the-loop model is very irregular during training, sometimes increasing and sometimes decreasing. This leads to instability in the training process and affects the model's performance. In contrast, the cumulative reward growth in (b) with the expert-in-the-loop model is more stable during training, showing a monotonically increasing trend overall, and exhibiting more reliable performance. Furthermore, observing the experimental results in (c) and (d), we can see that there are still detached nodes in the lower right corner of (c), which means that the model without the expert-in-the-loop model does not guarantee the global connectivity of network nodes after executing the action. In contrast, (d) successfully re-enters the network from the detached nodes, resulting in better performance.

[0078] According to another aspect of the embodiments of this application, a network planning apparatus based on a hybrid reinforcement learning strategy is also provided, comprising: The data link network topology model building module is used to build a data link network topology model and, based on the data link network topology model, predetermine data link network structure adjustment and optimization strategies according to different business needs to obtain a strategy support library. The objective optimization decision module is used to construct an optimization strategy model based on a data link network structure with multiple relational constraints, based on the strategy support library. The reinforcement learning decision module is used to optimize the network planning and decision-making of the optimization strategy model by employing a hybrid strategy agile and efficient reinforcement learning decision-making method to obtain an efficient decision model. The small sample planning module is used to take small sample data from real-world scenarios as input to the efficient decision-making model and obtain the initial network planning structure output by the efficient decision-making model. The expert-in-the-loop trusted planning module is used to take the initial network planning structure as input to the trusted network planning decision model to obtain a target network planning structure that meets the security review standards. The trusted network planning decision model is a decision model established by experts in the loop based on a preset action library.

[0079] Through the modules described above, a data link network topology model and optimization strategy library suitable for large-scale joint operations were established, improving the accuracy of cross-domain interoperability and network adaptive capabilities. A hybrid policy reinforcement learning method based on MADDPG was adopted to achieve high-dimensional objective optimization and agile, efficient decision-making under dynamically changing environments. Intelligent planning techniques based on small datasets and empirical knowledge were introduced to address the problem of small sample characteristics of network situations in real-world scenarios. A reliable planning technique based on "experts in the loop" was developed, enhancing the credibility and security of network planning decisions.

[0080] In one exemplary embodiment, the data link network topology model building module includes: The node partitioning unit is used to divide network nodes into command nodes, relay nodes, and ordinary nodes according to actual needs. The node location feature assignment unit is used to assign location features to all network nodes according to actual needs. The coastline generation unit is used to simulate Perlin noise and generate coastlines by rotation and movement, dividing the action area into two parts: ocean and land. The node functional attribute assignment unit is used to assign functional attributes to each node according to actual needs. The strategy support library building unit is used to build a strategy support library based on prior experience, according to the node's functional attributes, node location characteristics, and the node's location.

[0081] In one exemplary embodiment, the expert in-loop reliable planning module includes: The processing unit is used to build a library of preset actions based on security review standards; In the loop unit, the expert obtains the actions of the initial network planning structure. Before the action is executed, it is determined whether the action is a dangerous action. If the action is a dangerous action, it is replaced with a safe action to enhance the credibility of the decision result and obtain a credible network planning decision model.

[0082] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0083] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the network planning method based on the hybrid reinforcement learning strategy provided in the above embodiments. The method includes: establishing a data link network topology model, and pre-determining a data link network structure adjustment and optimization strategy according to different business requirements based on the data link network topology model to obtain a strategy support library. Based on the aforementioned strategy support library, construct an optimization strategy model for a data link network structure with multiple relational constraints. The network planning and decision optimization model is obtained by using a hybrid strategy agile and efficient reinforcement learning decision-making method for the optimization strategy model. Using small sample data from real-world scenarios as input to the efficient decision-making model, the initial network planning structure output by the efficient decision-making model is obtained; The initial network planning structure is used as input to a trusted network planning decision model to obtain a target network planning structure that meets security review standards. The trusted network planning decision model is a decision model built by experts in a loop based on a preset action library, including: constructing a preset action library based on security review standards; obtaining the actions of the initial network planning structure; determining whether the action is a dangerous action before execution; and replacing the dangerous action with a safe action if the action is dangerous, thereby enhancing the credibility of the decision result and obtaining the trusted network planning decision model.

[0084] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0085] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0087] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0089] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0090] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0091] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are intended to be exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A network planning method based on a hybrid reinforcement learning strategy, characterized in that, include: A data link network topology model is established, and based on the data link network topology model, a data link network structure adjustment and optimization strategy is pre-determined according to different business requirements to obtain a strategy support library. Based on the aforementioned strategy support library, construct an optimization strategy model for a data link network structure with multiple relational constraints. The network planning and decision optimization model is obtained by using a hybrid strategy agile and efficient reinforcement learning decision-making method for the optimization strategy model. Using small sample data from real-world scenarios as input to the efficient decision-making model, the initial network planning structure output by the efficient decision-making model is obtained; The initial network planning structure is used as input to a trusted network planning decision model to obtain a target network planning structure that meets security review standards. The trusted network planning decision model is a decision model built by experts in a loop based on a preset action library, including: constructing a preset action library based on security review standards; obtaining the actions of the initial network planning structure; determining whether the action is a dangerous action before execution; and replacing the dangerous action with a safe action if the action is dangerous, thereby enhancing the credibility of the decision result and obtaining the trusted network planning decision model.

2. The network planning method based on a hybrid reinforcement learning strategy as described in claim 1, characterized in that, Based on the aforementioned data link network topology model, data link network structure adjustment and optimization strategies are pre-established according to different business requirements, resulting in a strategy support library, including: Configure node functional attributes and node location characteristics for network nodes according to business requirements. The node functional attributes include command nodes, relay nodes and ordinary nodes, and the location characteristics include air, land and sea. Perlin noise simulation was used, and a coastline was generated by rotation and movement to divide the action area and obtain a set of position coordinates. Based on the node functional attributes, location characteristics, and location coordinate set, a data link network structure adjustment and optimization strategy is determined, resulting in a strategy support library.

3. The network planning method based on a hybrid reinforcement learning strategy as described in claim 1, characterized in that, The optimization strategy model for constructing a data link network structure based on multiple relation constraints based on the strategy support library includes: A deep deterministic strategy gradient algorithm is used to process high-dimensional continuous action spaces; A multi-agent deep deterministic policy gradient algorithm is used to handle the cooperation and competition relationships among multiple agents and generate an optimized policy model.

4. The network planning method based on a hybrid reinforcement learning strategy as described in claim 1, characterized in that, The network planning and decision optimization model is obtained by using a hybrid strategy agile and efficient reinforcement learning decision-making method to optimize the optimization strategy model, including: For network node planning, power adjustment planning, frequency adjustment planning, and temporary node planning, respectively establish policy constraints and implementation methods; Based on the established policy constraints and implementation methods, the actions of relay nodes and ordinary nodes are controlled and scored and adjusted using the MADDPG model to optimize network planning and obtain an efficient decision-making model.

5. The network planning method based on a hybrid reinforcement learning strategy as described in claim 4, characterized in that, The implementation methods of the network node planning include: A multi-agent deep deterministic policy gradient algorithm is used to control and score the actions of relay nodes. Each relay node not in the maximum relay node group is identified as an independent agent. After executing an action, the closer the agent is to the maximum relay node group, the higher the score; otherwise, a penalty is imposed. Once a relay node enters the maximum relay node group, it is no longer controlled as an independent agent. The MADDPG algorithm is used to control and adjust the actions and scores of ordinary nodes. Each ordinary node is defined as an independent agent. A circle with the nearest relay node as the center and the maximum communication distance as the radius is used as a reference. The closer the ordinary node is to this circle, the higher the score. If it is outside the circle, it is penalized. When it is inside the circle, the score remains unchanged and is adjusted according to the distance between the ordinary node and other ordinary nodes to avoid the overlap of ordinary nodes. The implementation methods of the power adjustment plan include: The maximum transmission distance of a normal node is determined to be its nearest relay node, and its own transmission power is calculated based on the maximum transmission distance; A deep deterministic policy gradient algorithm is used to control the transmission power of relay nodes, and all relay nodes are identified as an agent. The agent is scored only if the constraints are met, otherwise the score is zero. If the constraints cannot be met within a set step size, the agent is penalized.

6. The network planning method based on a hybrid reinforcement learning strategy as described in claim 4, characterized in that, The implementation methods of the frequency adjustment plan include: When changing channels, the input power, received power, and maximum transmission distance of each node are adjusted according to the Friis propagation loss model. The MADDPG algorithm is used to control and score the actions of relay nodes. Each relay node that is not in the largest relay node group is identified as an independent agent. After executing an action, the closer it is to the largest relay node group, the higher the score; otherwise, a penalty is imposed. Once a relay node enters the largest relay node group, it is no longer controlled as an independent agent. The MADDPG algorithm is used to control and adjust the actions of ordinary nodes. Each ordinary node is defined as an independent agent. A circle with the nearest relay node as the center and the maximum communication distance as the radius is used as a reference. The closer the ordinary node is to this circle, the higher the score. If it is outside the circle, it is penalized. When it is inside the circle, the score remains unchanged and is adjusted according to the distance between the ordinary node and other ordinary nodes to avoid the overlap of ordinary nodes.

7. The network planning method based on a hybrid reinforcement learning strategy as described in claim 4, characterized in that, The implementation methods of the temporary node planning include: Each temporary relay node is defined as an independent intelligent agent, and its switching state and position are controlled by the MADDPG algorithm; Each connected relay node is defined as a relay node group; the node information of the largest relay node group and its neighboring relay node groups is taken as input, and the output includes whether the node is started, the effective location after starting, and the moving distance; if starting enables the largest relay node group to connect with the neighboring relay node groups, a reward is given, and the shorter the moving distance, the higher the reward; if the connection is not established, a penalty is imposed. After all relay nodes are merged into one group, isolated ordinary nodes are located to obtain isolated nodes; all relay nodes include ephemeral relay nodes that have been started. The program takes information about isolated nodes and the nearest relay node as input and outputs whether to start the node, the start location, and the movement distance. If starting the node enables the isolated ordinary node to connect with the nearby relay node group, a reward is given, and the shorter the movement distance, the higher the reward. If the connection is not established, a penalty is imposed.

8. The network planning method based on a hybrid reinforcement learning strategy as described in claim 1, characterized in that, The step of using small sample data from real-world scenarios as input to the efficient decision-making model to obtain the initial network planning structure output by the efficient decision-making model includes: Extract the corresponding state-action pairs from the decision data available in empirical knowledge to construct a new decision set; The different situational information in the state is decoupled into evidence entities, the action information in the joint action is decoupled into action entities, and the development relationship from evidence entities to action entities is established based on empirical knowledge. A relational graph is constructed using evidence entities, action entities, and relationships. The learning of the network planning model is guided by an adversarial imitation learning method to generate an initial network planning structure suitable for small sample conditions.

9. A network planning device based on a hybrid reinforcement learning strategy, characterized in that, include: The data link network topology model building module is used to build a data link network topology model and, based on the data link network topology model, predetermine data link network structure adjustment and optimization strategies according to different business needs to obtain a strategy support library. The objective optimization decision module is used to construct an optimization strategy model based on a data link network structure with multiple relational constraints, based on the strategy support library. The reinforcement learning decision module is used to optimize the network planning and decision-making of the optimization strategy model by employing a hybrid strategy agile and efficient reinforcement learning decision-making method to obtain an efficient decision model. The small sample planning module is used to take small sample data from real-world scenarios as input to the efficient decision-making model and obtain the initial network planning structure output by the efficient decision-making model. The expert-in-the-loop trusted planning module is used to take the initial network planning structure as input to the trusted network planning decision model to obtain a target network planning structure that meets security review standards. The trusted network planning decision model is a decision model built by experts in the loop based on a preset action library, including: constructing a preset action library based on security review standards; obtaining the actions of the initial network planning structure; determining whether the action is a dangerous action before execution; and replacing the dangerous action with a safe action if the action is dangerous, so as to enhance the credibility of the decision result and obtain the trusted network planning decision model.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the network planning method based on the hybrid reinforcement learning strategy as described in any one of claims 1 to 8.

Images