Maritime communication network routing method and network system based on multi-agent cooperation

Abstract

The invention discloses an offshore communication network routing method and network system based on multi-agent cooperation, and the method comprises the steps: taking a source node of a data packet as a target node, and obtaining the local observation information and the current network state of the target node; and determining a next-hop node based on the local observation information and the current network state through an intelligent agent deployed by the target node, and transmitting the data packet to the next-hop node. According to the invention, maritime communication routing is carried out through multi-agent cooperation, the neighbor node information is used as the local observation information of the target node when the target node determines the next hop node, and then the target node is promoted to determine the next hop node based on the local observation information and the current network state of the target node. Therefore, not only can the communication requirement of a large-scale dynamic network be met, but also the coupling between the individual intelligent agent and the global data can be relieved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a maritime communication network routing method and network system based on multi-agent collaboration. Background Art

[0002] As the navigation area continues to expand, the derivative risks caused by maritime activities are increasing, and maritime rescue missions have become particularly arduous. Although the maritime communication network has initially achieved transmission service quality assurance, it is still insufficient in terms of coverage and communication quality compared with the land communication network.

[0003] Efficient routing algorithms are essential to ensure the completion of various maritime rescue missions. However, the access of a large number of terminal devices and the high mobility of ships pose severe challenges to existing routing strategies. First, as the scale of the network expands, the use of a central controller will bring huge overhead, resulting in the inability of centralized scheduling mechanisms to meet the communication needs of large-scale dynamic networks.

[0004] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0005] The technical problem to be solved by the present application is to provide a maritime communication network routing method and network system based on multi-agent collaboration in view of the deficiencies of the existing technology.

[0006] In order to solve the above technical problems, the first aspect of the present application provides a maritime communication network routing method based on multi-agent collaboration, which is applied to a maritime dynamic network, wherein the maritime dynamic network includes a plurality of network nodes deployed with agents; the maritime communication network routing method based on multi-agent collaboration specifically includes:

[0007] The source node of the data packet is used as the target node, and the local observation information and the current network status of the target node are obtained, wherein the local observation information includes the neighbor node information and the destination node;

[0008] The intelligent agent deployed by the target node determines the next hop node based on the local observation information and the current network state, and transmits the data packet to the next hop node, wherein the next hop node is a neighbor node of the target node;

[0009] The next hop node is taken as the target node, and the step of acquiring the local observation information and the current network status of the target node is performed until the destination node is reached, so as to transmit the data packet to the destination node.

[0010] The maritime communication network routing method based on multi-agent collaboration, wherein the neighbor nodes of the target node are network nodes that are determined based on the network graph corresponding to the maritime dynamic network and have a link connection with the target node, wherein the network graph includes several ships deployed with agents as nodes and the connection links between the nodes as edges.

[0011] The maritime communication network routing method based on multi-agent collaboration, wherein the neighbor node information includes the target Q value of the neighbor node, link status information and cache queue status.

[0012] The method for routing a maritime communication network based on multi-agent collaboration, wherein the method further comprises:

[0013] When a mobile network node appears in the marine dynamic network, determining a local network area where the mobile network node is located, wherein the local network area includes network nodes in the marine dynamic network whose links change due to the movement of the mobile network node;

[0014] Online learning is performed on the intelligent agents deployed at each network node in the local network area.

[0015] In the marine communication network routing method based on multi-agent collaboration, the reward function used in the training process of the agent is:

[0016] When the next hop node is the destination node, a positive reward is given;

[0017] When the next hop node is not the target node and the cache is in full load state, a negative reward is given;

[0018] When the next hop node is not the target node and the cache state is not full, a set reward is given.

[0019] The method for routing a maritime communication network based on multi-agent collaboration, wherein the next hop node is taken as a target node, and the step of acquiring local observation information and current network status of the target node is performed until the destination node is reached, so as to transmit the data packet to the destination node, the method comprises:

[0020] Obtaining a routing path corresponding to the data packet;

[0021] The agent is fine-tuned based on the routing path to update the policy network of the agent.

[0022] The maritime communication network routing method based on multi-agent collaboration, wherein the current network status includes buffer status, data packet queuing time and data packet transmission time.

[0023] In a second aspect, the present application provides a maritime communication network routing device based on multi-agent collaboration, which is applied to a maritime dynamic network, wherein the maritime dynamic network includes a plurality of network nodes deployed with agents; the maritime emergency communication device based on multi-agent distributed collaboration specifically includes:

[0024] An acquisition module, used to take the source node of the data packet as the target node, and acquire the local observation information and current network status of the target node, wherein the local observation information includes neighbor node information and the destination node;

[0025] A determination module, configured to determine a next-hop node based on the local observation information and the current network state through an agent deployed at the target node, and transmit the data packet to the next-hop node, wherein the next-hop node is a neighbor node of the target node;

[0026] The execution module is used to take the next hop node as the target node, and execute the step of acquiring the local observation information and the current network status of the target node until the destination node is reached, so as to transmit the data packet to the destination node.

[0027] A third aspect of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any of the above-described methods for maritime communication network routing based on multi-agent collaboration.

[0028] The fourth aspect of the present application provides a marine dynamic network system, which includes a number of network nodes, each of which is deployed with an intelligent agent; data packets are transmitted between the network nodes in the marine dynamic network system by adopting the marine communication network routing method based on multi-agent collaboration as described above.

[0029] Beneficial effects:

[0030] 1. This application uses multi-agent collaboration to perform maritime communication routing, and uses neighbor node information as local observation information of the target node when the target node determines the next hop node. Then, based on the local observation information and the current network status of the target node, the target node is prompted to determine the next hop node. This not only meets the communication needs of large-scale dynamic networks, but also alleviates the coupling between individual agents and global data.

[0031] 2. When mobile nodes appear in the marine dynamic network, the present application utilizes an online learning mechanism to train the local network area in the marine dynamic network, which not only effectively reduces the frequent signaling forwarding between nodes, but also significantly reduces the retraining time and communication and computing costs caused by dynamic changes in the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A flowchart of a maritime communication network routing method based on multi-agent collaboration provided in an embodiment of the present application.

[0034] Figure 2 This is the principle flow chart of the intelligent agent.

[0035] Figure 3 Schematic diagram of data packet routing.

[0036] Figure 4 Comparison chart of average delivery rates for fixed large-scale topologies.

[0037] Figure 5 Comparison chart of average delivery time for fixed large-scale topologies.

[0038] Figure 6 A comparison chart of the number of packet losses caused by congestion for a fixed large-scale topology.

[0039] Figure 7 Comparison chart of retransmission rates for fixed large-scale topologies.

[0040] Figure 8 Comparison chart of the average delivery rate, average delivery time, number of packet losses due to congestion, and retransmission rate of the four algorithms for dynamically changing large-scale topologies.

[0041] Fig. 9 A comparison chart of training time.

[0042] Fig.10 A functional block diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The embodiment of the present application provides a method and a network system for routing a maritime communication network based on multi-agent collaboration. To make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0045] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0046] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0047] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0048] This embodiment provides a method for routing a maritime communication network based on multi-agent collaboration, and the method is applied to a maritime dynamic network, wherein the maritime dynamic network includes a plurality of network nodes, each of the plurality of network nodes is deployed with an agent, the agent is trained using a deep reinforcement learning method, and is deployed with an optimal decision strategy after training. The agents deployed in each network node in the maritime dynamic network form a multi-intelligent distributed structure, and a maritime communication network routing is constructed for the network nodes in the maritime dynamic network through multi-agent collaboration.

[0049] Specifically, the marine dynamic network can be a large-scale dynamic network composed of a cluster of unmanned ships. For example, the marine dynamic network is a large-scale dynamic network formed at sea by USVs (unmanned surface vessels) in a marine application communication scenario. The marine dynamic network can be modeled as a graph G (N, E), where N represents a finite set of nodes in the network, and E represents a link set consisting of connected node pairs. The establishment of a link between two nodes refers to a wireless communication link established between the two nodes when the distance between the two nodes is less than a preset distance threshold, for example, a wireless communication link established through an Ad hoc network or a satellite relay. In addition, in order to facilitate the determination of the distance between the two nodes, each node is configured with location information, where the location information can be a two-dimensional coordinate determined based on longitude and latitude. For example, the marine dynamic network is a large-scale dynamic network formed at sea by USVs (unmanned surface vessels) in a marine application communication scenario.

[0050] like Figure 1 As shown, the present application provides a method for routing a maritime communication network based on multi-agent collaboration, which specifically includes:

[0051] S10. Taking the source node of the data packet as the target node, and obtaining the local observation information and the current network status of the target node, wherein the local observation information includes the neighbor node information and the destination node.

[0052] Specifically, the data packet can be generated by any network node in the offshore dynamic network, and the network node that generates the data packet can be used as the source node, and the destination node is the network node to receive the data packet, wherein the source node and the network node can both be any network node in the offshore dynamic network, that is, each network node in the offshore dynamic network can be used as a source node or a destination node.

[0053] The local observation information and the current network state are taken as the current state of the target node. The agent determines the next hop node based on the local observation information and the current network state. The next hop node is the neighbor node of the target node. In other words, the state space of the agent includes the local observation information and the current network state, and the action space includes all neighbor nodes. The agent determines the current action based on the current state, that is, determines the next hop node based on the local observation information and the current network state.

[0054] The local observation information is used to reflect the network status of the target node's neighboring nodes, and the current network status is used to reflect the network status of the target node at the current moment. The local observation information may include neighbor node information and the destination node. The neighbor node information is used to reflect the network status of the neighbor node, which may include one or more of the neighbor node's link status, buffer queue status, and target Q value of the neighbor node.

[0055] In a typical implementation of an embodiment of the present application, the neighbor node information includes a target Q value of the neighbor node, link status information, and a cache queue status, wherein the target Q value of the neighbor node is the expected benefit of the neighbor node as a next-hop node, which may be pre-configured based on the network status of the neighbor node, or calculated through a target network in the neighbor node. The link status information is used to reflect the link status of the neighbor node, wherein the link status may include the link delay and link availability status of the link connecting the neighbor node, and the link availability status includes link availability and link unavailability. The cache queue status is used to reflect the usage of the cache queue of the neighbor node, and when the cache queue status reaches or approaches a full load state, the neighbor node may be avoided from being selected as the next-hop node to prevent packet loss or network congestion due to buffer overload.

[0056] Furthermore, the current network status may include cache queue status, data packet queuing time, and data packet transmission time, wherein the data packet queuing time reflects the node processing speed and current load, and when the data packet queuing time reaches a preset time threshold, it will prompt that the node is heavily loaded or has insufficient resources. The data packet transmission time is the link transmission time from the target node to the neighboring node, which is usually determined by the physical distance or link quality. The data packet transmission time affects the total transmission delay of the data packet, and the shorter the data packet transmission time, the more helpful it is to optimize the transmission efficiency. To this end, the intelligent experience tends to use neighboring nodes with small data packet queuing time and short data packet transmission time as the next hop node, thereby improving the transmission speed.

[0057] S20. The intelligent agent deployed through the target node determines the next hop node based on the local observation information and the current network status, and transmits the data packet to the next hop node.

[0058] Specifically, the next hop node is a neighbor node of the target node, wherein the neighbor node of the target node is a network node determined based on the network graph corresponding to the marine dynamic network and having a link connection with the target node, wherein the network graph includes a number of ships deployed with intelligent agents as nodes and the connection links between the nodes as edges. In other words, the marine dynamic network can be modeled as a network graph in advance, and then each target node determines its corresponding neighbor node based on the network graph. Of course, in practical applications, when mobile nodes appear in the marine dynamic network, the network graph of the marine dynamic network will also be dynamically updated based on the movement of the mobile nodes. For example, when the mobile node disconnects the link connection with some network nodes due to movement, the node pair with the disconnected link connection will be removed from the link set of the network graph, or, when the mobile node establishes a link with some network nodes that have not established a link due to movement, the node pair with the newly established link connection will be added to the link set of the network graph.

[0059] S30, taking the next hop node as the target node, and executing the step of acquiring the local observation information and the current network status of the target node until the destination node is reached, so as to transmit the data packet to the destination node.

[0060] Specifically, after obtaining the next hop node, the next hop node is used as the target node. The intelligent agent configured by the target node continues to determine the next hop node based on the local observation information of the target node and the current network status until the next hop node is the destination node. In this way, a route can be established between the source node and the destination node, and the data packet can be transmitted from the source node to the destination node through the route.

[0061] The embodiment of the present application performs maritime communication routing through multi-agent collaboration, and uses neighbor node information as local observation information of the target node when the target node determines the next hop node, and then prompts the target node to determine the next hop node based on the local observation information and the current network status of the target node. This not only meets the communication needs of large-scale dynamic networks, but also alleviates the coupling between individual agents and global data.

[0062] Further, in one implementation, the marine dynamic network may include several local area networks, and the data packet may be transmitted within the local area network or across the local area network. The local area network has the characteristics of high connection quality and low latency, while the cross-local area network may have the problem of low connection quality or high latency. Therefore, when constructing a route between the source node and the destination node, the local area network state between the source node and the destination node may be determined first, and then the determined local area network state may be used to determine the construction method of constructing the route between the source node and the destination node. Specifically, when the local area network state is the same local area network, the shortest path method may be directly used to construct the route for the source node and the destination node, for example, by minimizing the path Q value (the neighbor node with the best target Q value is selected) to select the next hop node for the target node, etc.; when the local area network state is across local area networks, the source node of the data packet is used as the target node, and the local observation information and current network state of the target node are obtained, that is, the route is constructed for the source node and the destination node through the process of steps S10-S30. In this way, the construction method of constructing routes for the source node and the destination node is selected based on the local area network status between the source node and the destination node, so that the routes for the source node and the destination node can be constructed more quickly, thereby improving the transmission speed of the data packet.

[0063] Furthermore, in one implementation, since the network nodes in the marine dynamic network may move, the movement of the network nodes may cause some links in the marine dynamic network to change. Therefore, when the network nodes in the marine dynamic network move, an online learning mechanism may be used to perform local learning on the agents configured by the mobile nodes and their related network nodes, so as to improve the routing effect of constructing routing through distributed multi-agents.

[0064] Based on this, the maritime communication network routing method based on multi-agent collaboration also includes:

[0065] When a mobile network node appears in the marine dynamic network, determining the local network area where the mobile network node is located;

[0066] Online learning is performed on the intelligent agents deployed at each network node in the local network area.

[0067] Specifically, the appearance of a mobile network node in a dynamic offshore network refers to a change in the location information of a network node in the dynamic offshore network, and the change in the location information of the network node causes a change in the link status between the network node and some other network nodes in the dynamic offshore network, for example, the link with some other network nodes is disconnected, or a link is established with some other nodes, or the link with some other network nodes is disconnected and a link is established with some other nodes, etc.

[0068] To this end, when a mobile network node appears in a dynamic network at sea, the local network area where the mobile network node is located can be determined based on the link changes in the dynamic network at sea, wherein the local network area includes network nodes in the dynamic network at sea whose links change due to the movement of the mobile network node, that is, the local network area includes the mobile network node, the network node that disconnects the link with the mobile node due to the movement of the mobile network node, and the network node that establishes a link with the mobile node due to the movement of the mobile network node, and the network node that disconnects the link is different from the network node that establishes the link. For example, during the movement of the mobile network node, network node A first disconnects the link with the mobile network node, and then establishes a link with the mobile network node. Then, when the local network area where the mobile network node is located is locally learned, the local network area does not include the network node A. Of course, in actual applications, the local network area can also include network node A in the local network area according to user settings.

[0069] After obtaining the local network area, the agents deployed in the network nodes included in the local network area are subjected to online learning, wherein the online learning process may use the training data used in training the agents, or the routing data accumulated in the multi-agent collaboration process. In addition, when the agents are subjected to local learning and training, a reward function is formed when the agents determine each next-hop node, and the decision strategy is optimized based on the reward function. Specifically, Figure 2 As shown, after the Q network PolicyNet determines the next hop node a based on the current state S, a reward function r and a new current state S′ are formed, and the Q value in the Q network PolicyNet is updated based on (S, a, r, S′) so that the agent can gradually optimize its strategy, where (S, a, r, S′) is stored in the experience pool Replay Buffer. At the same time, (S, a, r, S′) stored in the experience pool Replay Buffer will be used to train the target network TargetNe, calculate a stable target Q value through the target network TargetNe, and update the network parameters in the Q network PolicyNet based on the target Q value, where the network parameters of the target network TargetNe are the same as those in the Q network PolicyNet, and after a certain number of steps, the target network TargetNe will synchronize the network parameters from the Q network PolicyNet to avoid instability during training.

[0070] Furthermore, in the training process of the agent and the local training of the network nodes in the local network area, the reward function used is:

[0071] When the next hop node is the destination node, a positive reward is given;

[0072] When the next hop node is not the target node and the cache is in full load state, a negative reward is given;

[0073] When the next hop node is not the target node and the cache state is not full, a set reward is given.

[0074] Specifically, a positive reward is used to indicate that a data packet successfully reaches the destination node, a negative reward is used to reflect data packet loss or network congestion, and a set reward is used to reflect that a data packet is in transmission. Among them, the positive reward and the negative reward can be preset, and the set reward can be determined based on the buffer queue state and the link delay. For example, after obtaining the buffer queue state and the link delay, the set reward is calculated based on the preset correspondence between the buffer queue state and the link delay and the set reward. In other words, the positive reward, the negative reward, and the calculation method of the set reward based on the buffer queue state and the link delay are preset, and then after the next hop node is determined, the corresponding reward can be selected according to the preset conditions satisfied by the next hop node, wherein the preset conditions include being the destination node, not being the destination node and the cache state is in a full load state, and not being the destination node and the cache state is not in a full load state.

[0075] For example: Figure 3 As shown in the figure, the routing process of data packet p from source node S1 to intermediate node J, the destination node of data packet p is D1, then when the next hop node is determined to be intermediate node J, the following operations will be performed:

[0076] Determine whether the intermediate node J is the destination node D1;

[0077] If J is equal to D1, the environment will give a positive reward r a , indicating that data packet p successfully reaches the destination;

[0078] If J is not equal to D1, check the cache status of J. If the cache status is full, retransmit. If the packet p has been retransmitted multiple times, the network is considered congested, and the packet p will be discarded. The environment will give node J a negative reward r. c ; If the buffer state is not full, the packet p will be forwarded to J's buffer queue, and a set reward will be given based on J's buffer queue state and link delay.

[0079] In one implementation, in the process of constructing routes for maritime communications using multi-agent distributed collaboration, each time a routing path is constructed, the routing path is stored in an experience pool, and the agent is updated based on the routing path so that a better routing path can be constructed through multi-agent distributed collaboration. Based on this, the method takes the next hop node as the target node, and performs the steps of obtaining the local observation information and the current network state of the target node until the destination node is reached to transmit the data packet to the destination node, and the method includes: obtaining the routing path corresponding to the data packet; and fine-tuning the agent based on the routing path to update the policy network of the agent.

[0080] It should be noted that after fine-tuning the agent online, fine-tuning can be performed once for each routing path built, or after a preset number of routing paths are built. In other words, the frequency of fine-tuning can be set according to actual needs, or according to user needs, etc.

[0081] In summary, this embodiment provides a method for routing a maritime communication network based on multi-agent collaboration, which is applied to a dynamic maritime network, wherein the dynamic maritime network includes several network nodes with agents deployed thereon; the method for routing a maritime communication network based on multi-agent collaboration specifically includes: taking the source node of the data packet as the target node, and obtaining the local observation information and current network status of the target node; through the agent deployed at the target node, determining the next hop node based on the local observation information and the current network status, and transmitting the data packet to the next hop node; taking the next hop node as the target node, and executing the step of obtaining the local observation information and the current network status of the target node until the destination node, so as to transmit the data packet to the destination node. The present application performs maritime communication routing through multi-agent collaboration, and when the target node determines the next hop node, the neighbor node information is used as the local observation information of the target node, and then the target node is prompted to determine the next hop node based on the local observation information and the current network status of the target node, which can not only meet the communication needs of large-scale dynamic networks, but also alleviate the coupling between individual agents and global data. At the same time, when mobile nodes appear in the marine dynamic network, the present application uses an online learning mechanism to train the local network area in the marine dynamic network, which not only effectively reduces the frequent signaling forwarding between nodes, but also significantly reduces the retraining time and communication and computing costs caused by dynamic changes in the network.

[0082] In order to further illustrate the maritime communication network routing method based on multi-agent collaboration provided by the embodiment of the present application, a simulation experiment was conducted. In this simulation experiment, a grid topology containing 49 network nodes was constructed, and each network node had a fixed sending queue and buffer size. Python 3.6 was used to build an event-driven simulator, and Pytorch was used as a deep reinforcement learning framework to implement routing based on reinforcement learning RL. The following are the simulation results of routing schemes in fixed large-scale topologies and dynamic large-scale topologies.

[0083] A. Fixed large-scale topology

[0084] In the scenario where the node positions are fixed, the performance of the maritime communication network routing method based on multi-agent collaboration (denoted as the collaborative RL-Route algorithm) and the traditional shortest path algorithm (this protocol is the best choice for static routing methods under low load) provided in the embodiment of the present application are compared, mainly measured by the following four evaluation indicators: average delivery rate, average delivery time, number of packet losses due to congestion, and retransmission rate. The results of the four evaluation indicators are compared as follows: Figure 4-Figure 7 shown.

[0085] Depend on Figure 4-Figure 7 As shown in the figure, when the network load is less than 800 packets (i.e., the initial number of packets in the network), RL-Route and the shortest path algorithm perform well. However, as the network load increases, the performance of the collaborative RL-Route algorithm is much better than the shortest path algorithm. Specifically, the average delivery rate of the shortest path algorithm is 30% lower than that of the collaborative RL-Route algorithm. In addition, the collaborative RL-Route algorithm also reduces the retransmission rate by 30%.

[0086] B. Dynamically changing large-scale topology

[0087] In the scenario where the node position changes, the performance of the shortest path algorithm, the collaborative RL-Route algorithm (no retraining), the collaborative RL-Route algorithm (global training), and the collaborative RL-Route algorithm (local training) are compared. The comparison results are as follows: Figure 8 As shown. Figure 8 It can be seen that under low network load, the shortest path algorithm, cooperative RL-Route algorithm (no retraining), cooperative RL-Route algorithm (global training), and cooperative RL-Route algorithm (local training) perform the same. However, when the network load exceeds 1000 packets, the performance of the shortest path algorithm and the cooperative RL-Route algorithm without retraining deteriorates significantly. In contrast, when the load is large enough, the algorithms using global training or local training can still show good performance.

[0088] Specifically, the algorithm using global training or local training can keep the average packet delivery rate at around 90%, which is almost unaffected by the network load, and the average delivery time remains relatively constant. Global training or local training can also effectively reduce the number of congestion occurrences by more than 500 times. The collaborative RL-Route algorithm (global training) has significant advantages over the collaborative RL-Route algorithm (local training).

[0089] In addition, although the performance of local training on the four evaluation metrics is similar to that of global training, local training takes significantly less time than global training. Fig. 9As shown in Figure 2, the time spent on local training and global training is recorded. At the same time, data analysis further illustrates the outstanding advantages of our proposed local training method.

[0090] Based on the above-mentioned maritime communication network routing method based on multi-agent collaboration, this embodiment provides a maritime communication network routing device based on multi-agent collaboration, which is applied to a maritime dynamic network, wherein the maritime dynamic network includes a plurality of network nodes deployed with agents; Fig.10 As shown, the marine emergency communication device based on multi-agent distributed collaboration specifically includes:

[0091] The acquisition module 100 is used to take the source node of the data packet as the target node and acquire the local observation information and the current network status of the target node, wherein the local observation information includes the neighbor node information and the destination node;

[0092] A determination module 200, configured to determine a next hop node based on the local observation information and the current network state through an agent deployed at the target node, and transmit the data packet to the next hop node, wherein the next hop node is a neighbor node of the target node;

[0093] The execution module 300 is used to take the next hop node as the target node, and execute the step of acquiring the local observation information and the current network status of the target node until the destination node is reached, so as to transmit the data packet to the destination node.

[0094] Based on the above-mentioned maritime communication network routing method based on multi-agent collaboration, this embodiment provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the maritime communication network routing method based on multi-agent collaboration as described in the above-mentioned embodiment.

[0095] Based on the above-mentioned maritime communication network routing method based on multi-agent collaboration, this embodiment provides a maritime dynamic network system, which includes a number of network nodes, each of which is deployed with an agent, and the network nodes in the maritime dynamic network system transmit data packets by adopting the maritime communication network routing method based on multi-agent collaboration as described in the above-mentioned embodiment.

[0096] In addition, the specific process of loading and executing the plurality of instruction processors in the above storage medium and the network node has been described in detail in the above method, and will not be described one by one here.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for routing a maritime communication network based on multi-agent collaboration, characterized in that: Applied to a dynamic network at sea, the dynamic network at sea includes a number of network nodes deployed with intelligent agents; the routing method for a marine communication network based on multi-agent collaboration specifically includes: The source node of the data packet is used as the target node, and the local observation information and the current network status of the target node are obtained, wherein the local observation information includes the neighbor node information and the destination node; The intelligent agent deployed by the target node determines the next hop node based on the local observation information and the current network state, and transmits the data packet to the next hop node, wherein the next hop node is a neighbor node of the target node; The next hop node is taken as the target node, and the step of acquiring the local observation information and the current network status of the target node is performed until the destination node is reached, so as to transmit the data packet to the destination node.

2. The method for maritime communication network routing based on multi-agent collaboration according to claim 1, characterized in that: The neighbor nodes of the target node are network nodes that are determined based on the network graph corresponding to the marine dynamic network and are connected to the target node by links, wherein the network graph includes several ships deployed with intelligent agents as nodes and the connecting links between the nodes as edges.

3. The method for maritime communication network routing based on multi-agent collaboration according to claim 1, characterized in that: The neighbor node information includes the target Q value of the neighbor node, link state information and cache queue state.

4. The method for maritime communication network routing based on multi-agent collaboration according to claim 1, characterized in that: The method further comprises: When a mobile network node appears in the marine dynamic network, determining a local network area where the mobile network node is located, wherein the local network area includes network nodes in the marine dynamic network whose links change due to the movement of the mobile network node; Online learning is performed on the intelligent agents deployed at each network node in the local network area.

5. The method for maritime communication network routing based on multi-agent collaboration according to claim 1, characterized in that: The reward function used in the training process of the agent is: When the next hop node is the destination node, a positive reward is given; When the next hop node is not the target node and the cache is in full load state, a negative reward is given; When the next hop node is not the target node and the cache state is not full, a set reward is given.

6. The method for maritime communication network routing based on multi-agent collaboration according to claim 1, characterized in that: After taking the next hop node as the target node and executing the step of acquiring the local observation information and the current network status of the target node until the destination node is reached, so as to transmit the data packet to the destination node, the method comprises: Obtaining a routing path corresponding to the data packet; The agent is fine-tuned based on the routing path to update the policy network of the agent.

7. The method for maritime communication network routing based on multi-agent collaboration according to claim 1, characterized in that: The current network status includes buffer status, data packet queuing time and data packet transmission time.

8. A maritime communication network routing device based on multi-agent collaboration, characterized in that: Applied to a dynamic network at sea, the dynamic network at sea includes a number of network nodes deployed with intelligent agents; the emergency communication device at sea based on multi-agent distributed collaboration specifically includes: An acquisition module, used to take the source node of the data packet as the target node, and acquire the local observation information and current network status of the target node, wherein the local observation information includes neighbor node information and the destination node; A determination module, configured to determine a next-hop node based on the local observation information and the current network state through an agent deployed at the target node, and transmit the data packet to the next-hop node, wherein the next-hop node is a neighbor node of the target node; The execution module is used to take the next hop node as the target node and execute the step of acquiring the local observation information and the current network status of the target node until the destination node is reached, so as to transmit the data packet to the destination node.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the maritime communication network routing method based on multi-agent collaboration as described in any one of claims 1-7.

10. A marine dynamic network system, characterized in that: The offshore dynamic network system includes a number of network nodes, each of which is deployed with an intelligent agent; the network nodes in the offshore dynamic network system transmit data packets by adopting the offshore communication network routing method based on multi-agent collaboration as described in any one of claims 1-7.

Images