Marine monitoring network communication system and method based on reinforcement learning

Abstract

The invention discloses an ocean monitoring network communication system and method based on reinforcement learning, and the system comprises an underwater subnet which is composed of a plurality of underwater nodes, and the nodes are arranged according to a cellular grid topological structure, and achieve the coverage of an ocean region; the overwater subnet is composed of satellite nodes and water surface buoy nodes, and the water surface buoy nodes carry out relay data transmission through the satellite nodes; wherein the satellite node realizes data transmission with the water surface buoy node through radio waves; the water surface buoy nodes have wireless electromagnetic wave communication and sound wave communication capabilities and are responsible for realizing cross-medium information transmission in the ocean heterogeneous network; the underwater nodes communicate with each other through sound waves; in a link layer, nodes communicate through a spatial multiplexing time division multiple access mechanism, and a time slot alignment mechanism based on reinforcement learning is constructed, so that local time slot alignment and concurrent scheduling optimization are realized. According to the invention, the communication time delay is effectively reduced; dynamic time slot alignment is realized, and dependence on global time synchronization is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of wireless communications and underwater monitoring networks, and in particular to a marine monitoring network communication system and method based on reinforcement learning. Background Art

[0002] As the world pays more and more attention to the development of marine resources and marine environmental monitoring, building an efficient, stable and wide-coverage marine communication network has become an important basic support for many fields such as marine scientific research, military applications, energy development, and environmental protection.

[0003] Currently, marine communication networks primarily consist of two components: surface and underwater. Surface networks are typically based on radio communication technology, including ships, buoys, and satellite nodes, offering advantages such as high transmission rates and low latency. Underwater networks, on the other hand, primarily rely on underwater acoustic communication, which, due to its strong penetration, is well-suited for information transmission in deep-sea environments. The distributed deployment of underwater sensor nodes enables real-time perception and data collection of marine environmental parameters.

[0004] However, due to the complexity of the marine environment, particularly the slow propagation speeds, narrow bandwidth, and severe multipath effects of underwater channels, underwater communication networks face numerous challenges in practical applications, such as large fluctuations in communication quality, extended data latency, and uneven node energy consumption. Furthermore, the need for coverage across large ocean areas places higher demands on the network's scalable deployment, stability, and adaptability.

[0005] In this context, how to design a marine communication solution that is suitable for large-scale deployment while taking into account communication efficiency, energy balance and system flexibility has become an important research direction in the current field of marine information networks. Summary of the Invention

[0006] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide an ocean monitoring network communication system and method based on reinforcement learning.

[0007] The first technical solution adopted by the present invention is:

[0008] A marine monitoring network communication system based on reinforcement learning, comprising:

[0009] The underwater subnet consists of multiple underwater nodes arranged in a cellular grid topology to achieve coverage of large ocean areas;

[0010] The water subnet is composed of satellite nodes and surface buoy nodes. The surface buoy nodes relay data transmission through satellite nodes.

[0011] Among them, the satellite node is equipped with radio frequency wireless communication equipment, which realizes data transmission with the surface buoy node through radio waves; the surface buoy node is also equipped with radio communication equipment and underwater acoustic communication equipment, and has radio electromagnetic wave communication and acoustic wave communication capabilities, and is responsible for realizing cross-media information transmission in marine heterogeneous networks; the underwater node is equipped with underwater acoustic communication equipment, and the nodes communicate with each other through sound waves; at the link layer, the nodes communicate through an improved spatial multiplexing time division multiple access (TDMA) mechanism, and a time slot alignment mechanism based on reinforcement learning is constructed to realize local time slot alignment and concurrent scheduling optimization.

[0012] Furthermore, each node is provided with and maintains a neighbor table and a link layer load table. The neighbor table records the IP address, node type, location information, remaining energy and classification identifier of the adjacent nodes. The link layer load table records the cache status of the link between the node and each neighbor node.

[0013] When sending a data packet, a node adds its own remaining energy and link cache information to the packet header. Neighboring nodes passively monitor the received packet header information and dynamically update the energy status and load information of neighboring nodes in the neighbor table and link layer load table without introducing additional control signaling.

[0014] Furthermore, the ocean monitoring network communication system preferentially uses the water subnet link when forwarding data packets;

[0015] When a source node generates and sends a data packet, or a relay node participates in routing forwarding, the following steps are performed:

[0016] First, determine whether the destination node exists in its neighbor table. If so, forward the data packet directly to the destination node.

[0017] If there is no destination node in the neighbor table, it is further determined whether there is a neighbor node belonging to the water subnet, including a surface buoy node or a satellite node. If so, the data packet is forwarded to this type of node first;

[0018] If there is neither the destination node nor the surface subnet node in the neighbor table, the data packet will continue to be forwarded in the underwater subnet.

[0019] Furthermore, the underwater subnet adopts a cellular grid topology to classify the nodes in the underwater subnet. Every seven nodes are divided into a cellular unit. The nodes in each cell are divided into different scheduling categories according to their spatial position in the unit. This makes the categories of nodes in the same unit different, thus realizing spatial reuse scheduling.

[0020] At the link layer, a time division multiple access (TDMA) method with spatial multiplexing is used to schedule the transmission time slots of nodes. Based on the divided scheduling categories, 21 different pairs of communication nodes are constructed as vertices in the graph, and a conflict-free time slot scheduling path is generated through a graph traversal algorithm.

[0021] Furthermore, based on the gap filling technology, multiple data transmission tasks are inserted during the propagation delay, and the time slot length is set to the sum of the data packet transmission delay and the protection interval, that is, T s =T t +T g , where T s is the time slot length, T t is the data packet transmission delay, T g For protection interval.

[0022] Furthermore, the time slot alignment mechanism includes:

[0023] Without the need for global time synchronization, a node calculates its own transmission start time by monitoring the reception time and propagation delay of neighboring node data packets. The time offset range is discretized into N time buckets, each bucket corresponds to a delay adjustment action. The node uses the Q-learning algorithm to select the optimal delay adjustment value so that as many one-hop neighbors as possible meet the time slot alignment condition, thereby improving the efficiency of concurrent communication.

[0024] Furthermore, the node selects an optimal delay adjustment value using a Q-learning algorithm, including:

[0025] The node uses the Q-learning learning strategy to select an optimal delay adjustment action a according to the current state (neighborhood offset distribution) after each round of communication. t , and give an immediate reward r according to the number of neighbors that are successfully aligned t ; This reward is used to update the Q value table:

[0026]

[0027] Where α is the learning rate, γ is the discount factor, and s t is the current state, a t is the current action, r t For immediate rewards;

[0028] Through the above learning process, the node can gradually adjust its own transmission timing to align the transmission time slots of more one-hop neighbors, thereby reducing the probability of collision and improving the concurrent transmission capability of the link layer and overall communication efficiency.

[0029] The second technical solution adopted by the present invention is:

[0030] A method for ocean monitoring network communication based on reinforcement learning, applied to the ocean monitoring network communication system as described above, comprises the following steps:

[0031] When the source node generates a data packet and prepares to send it, it checks whether the destination node exists in the neighbor table; if so, it directly forwards the data packet to the destination node to complete the data transmission;

[0032] If the destination node does not exist in the neighbor table, check whether there is a neighbor node belonging to the water subnet (such as a surface buoy node or a satellite node); if so, the data packet is forwarded to the water subnet node first;

[0033] If there is neither the destination node nor the water subnet node in the neighbor table, the node calculates the forwarding factor based on the geographical distance between the neighbor node and the destination node, the link load factor, and the remaining energy;

[0034] Sort neighbor nodes according to forwarding factors, and select several neighbors with better forwarding factor values ​​to form a candidate set;

[0035] Perform energy screening on neighbor nodes in the candidate set and eliminate candidates whose remaining energy is significantly lower than that of other nodes;

[0036] From the filtered candidate set, the neighbor node with the best forwarding factor and satisfying energy conditions is selected as the next hop node to forward the data packet.

[0037] Furthermore, the forwarding factor is calculated as follows:

[0038] F=α·d+(1-α)·Q

[0039] Where d represents the normalized vertical distance of the neighbor node relative to the line connecting the current node and the destination node, Q represents the link load factor, and α is the weight coefficient.

[0040] The beneficial effects of the present invention are as follows: the present invention adopts a grid deployment structure, divides the monitoring area into multiple cells, and deploys a number of underwater nodes in each cell. The nodes communicate with each other through an improved spatial multiplexing TDMA mechanism, combined with reinforcement learning technology, optimizes the time slot alignment strategy based on historical interaction information, and implements cross-layer congestion perception and adaptive control in the routing strategy. In addition, the present invention also introduces gap filling scheduling, neighbor table maintenance and forwarding factor dynamic calculation mechanism to improve link scheduling flexibility and routing robustness. Experimental results show that the present invention can significantly reduce communication delay, improve system throughput and balance node energy consumption, and is suitable for large-scale underwater monitoring and perception applications in marine environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 1 is a schematic diagram of the system structure of a large-scale ocean monitoring network based on reinforcement learning in an embodiment of the present invention;

[0043] Figure 2 is a flow chart of a routing mechanism in a method for a large-scale ocean monitoring network based on reinforcement learning in an embodiment of the present invention;

[0044] Figure 3 Schematic diagram of the structure of the neighbor table and link layer load table maintained by a node in an embodiment of the present invention;

[0045] Figure 4 This is a diagram of cellular grid topology and scheduling category division in an embodiment of the present invention;

[0046] Figure 5 Schematic diagram of spatial multiplexing TDMA communication node pair patterning and time slot scheduling paths in an embodiment of the present invention;

[0047] Figure 6 2 is a schematic diagram comparing time slot arrangement and delay under the gap filling mechanism in an embodiment of the present invention;

[0048] Figure 7 2 is a schematic diagram of the passive Q-learning time slot alignment process in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application. For the step numbers in the following embodiments, they are provided only for the convenience of explanation and are not intended to limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0050] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. The singular forms of "a", "said", and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise clearly defined, words such as setting, installing, and connecting should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

[0051] In the description of this application, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on this application.

[0052] In the description of this application, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly specifying the number or order of the technical features indicated.

[0053] In the description of this application, "and / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0054] Explanation of terms:

[0055] Q-learning: A reinforcement learning algorithm that is a model-free prediction algorithm.

[0056] In order to solve the problems of low data delivery rate, high communication delay, low channel utilization and uneven distribution of node energy consumption in the existing marine information network, the present invention provides a large-scale marine monitoring network communication system and method based on reinforcement learning, which is suitable for complex and dynamic underwater environments. Among them, the system adopts a grid deployment structure to divide the monitoring area into multiple cells, and deploys several underwater nodes in each cell. The nodes communicate through an improved spatial multiplexing TDMA mechanism, combined with reinforcement learning technology, optimizes the time slot alignment strategy based on historical interaction information, and implements cross-layer congestion perception and adaptive control in the routing strategy. The method also introduces gap filling scheduling, neighbor table maintenance and forwarding factor dynamic calculation mechanism to improve link scheduling flexibility and routing robustness. Experimental results show that the present invention can significantly reduce communication delay, improve system throughput and balance node energy consumption, and is suitable for large-scale underwater monitoring and perception applications in marine environments.

[0057] like Figure 1 As shown, this embodiment provides a large-scale ocean monitoring network communication system based on reinforcement learning, including three types of nodes: satellite nodes, surface buoy nodes and underwater nodes, forming a cross-domain communication network including underwater subnets and surface subnets.

[0058] Among them, the underwater subnet is composed of a large number of underwater nodes, which are arranged in a cellular grid topology to achieve coverage of a vast ocean area.

[0059] The water subnet consists of surface buoy nodes and satellite nodes. The buoy nodes communicate with the satellite nodes via radio frequency to achieve data relay.

[0060] The hardware configuration of each node is as follows:

[0061] 1) Satellite node: equipped with radio frequency wireless communication equipment, responsible for radio wave data transmission between the satellite node and the surface buoy node;

[0062] 2) Surface buoy nodes: Equipped with radio communication equipment and underwater acoustic communication equipment, supporting cross-media information transmission of radio electromagnetic waves and underwater acoustic signals;

[0063] 3) Underwater nodes: Equipped with underwater acoustic communication equipment, which communicates with underwater nodes or buoy nodes through sound waves.

[0064] In some embodiments, see Figure 3 Each node maintains a neighbor table and a link layer load. The neighbor table stores information such as neighbor node type, location, remaining energy, and scheduling category. The link layer load table dynamically records the number of packets queued for transmission by the node and its neighbor nodes at the link layer, reflecting the current communication pressure. Both tables are updated through passive monitoring without the need for additional control signaling.

[0065] As an implementation method, the underwater subnet adopts a cellular grid topology, such as Figure 4 As shown in the figure, the entire monitoring area is divided into several regular cellular units. Nodes within each cell are classified into different scheduling categories based on their spatial location within the cell. Nodes of each category are evenly distributed within the cell, and adjacent cells are deployed in a translationally symmetrical manner. This allows for spatial dislocation of nodes of the same scheduling category across different cells, reducing interference between similar nodes and supporting spatial multiplexing scheduling.

[0066] In one embodiment, the system prioritizes the use of overwater subnet links when forwarding data packets. When a source node generates and sends a data packet, or a relay node participates in routing forwarding, it first determines whether the destination node exists in its neighbor table. If so, it directly forwards the data packet to the destination node.

[0067] If there is no destination node in the neighbor table, it is further determined whether there is a neighbor node belonging to the water subnet, including a surface buoy node or a satellite node. If so, the data packet is forwarded to this type of node first;

[0068] If there is neither the destination node nor the surface subnet node in the neighbor table, the data packet will continue to be forwarded in the underwater subnet until it reaches the destination node or an surface link is found.

[0069] Based on the above system, such as Figure 2 As shown, this embodiment also provides a large-scale ocean monitoring network communication method based on reinforcement learning. In this method, after a node receives a link layer data packet, it first determines whether the data packet is sent to this node. If not, extract the energy and load information in the packet header, update the neighbor table, link layer load table and Q value table; then discard the data packet. If so, after updating the above table items, determine whether it is the destination node. If so, receive the data packet and end the processing, otherwise enter the routing decision process. The routing decision gives priority to reaching the destination node directly, and then forwards it to the water node. Finally, the candidate nodes are screened based on the forwarding factor, and the nodes with lower energy are eliminated. The optimal node is selected as the next hop to complete the forwarding. This process realizes data perception, dynamic update of neighbor information and Q value and intelligent routing forwarding, which improves the network transmission efficiency and stability.

[0070] In some embodiments, at the link layer, the method of this embodiment adopts a time division multiple access (TDMA) mechanism of spatial multiplexing. Figure 5As shown in the figure, based on the defined scheduling categories, 21 different pairs of communication nodes are constructed as vertices in the graph. Considering that nodes are half-duplex, only transmit or receive operations can be performed in any time slot, and bidirectional communication is not possible simultaneously. If two communication pairs contain nodes of the same category, there is a scheduling conflict, and time slots cannot be allocated in sequence. However, if any two communication pairs do not contain nodes of the same scheduling category, the two groups of communications will not conflict and can be arranged in a sequential order on the scheduling path. Through graph modeling, all non-conflicting communication pairs are connected, and a graph traversal algorithm is used to determine a path that passes through all vertices without repeated visits, which serves as the conflict-free time slot scheduling sequence. The communication sequence is arranged according to this path, realizing TDMA time slot allocation under spatial multiplexing.

[0071] In some embodiments, to address the problem of long propagation delay and low data transmission rate in underwater acoustic channels, this embodiment introduces gap-filling technology to improve channel utilization and reduce the total delay of the communication cycle. In specific implementation, the time slot length in traditional TDMA scheduling is changed from propagation delay T p Adjust to data transmission delay T t Add guard interval T g The sum of T s =T t +T g This adjustment makes each time slot more compact, fully utilizing the gap time within the propagation delay and avoiding a large amount of time slot resource waste.

[0072] In the specific operation, the calculation is performed in a propagation delay period T p The maximum number of communication node pairs that can be arranged within According to this number, all 21 pairs of communication nodes are divided into The rounds are scheduled in sequence to complete all communication tasks. The total delay is:

[0073]

[0074] Compared to traditional solutions The method of this embodiment significantly reduces the total transmission delay, thereby improving the overall scheduling efficiency and system throughput. Figure 6 , which compares the scheduling methods of traditional TDMA and the gap filling mechanism of the present invention.

[0075] As an implementation method, the method of this embodiment adopts a time division multiple access (TDMA) mechanism with spatial multiplexing at the link layer. In view of the instability of underwater network propagation media and the difficulty in achieving a global unified clock, this embodiment introduces a time slot alignment mechanism based on reinforcement learning (Q-learning). In the specific implementation, nodes do not need explicit time synchronization, but dynamically adjust their own transmission timing by locally monitoring the transmission behavior of neighboring nodes, thereby achieving effective alignment of local time slots and concurrent scheduling optimization. Figure 7 As shown in Figure 1, the basic process of passive Q-learning time slot alignment is illustrated.

[0076] The node first monitors the data packets sent by the neighbor node and records the receiving time T recv , combined with the estimated propagation delay T p And the packet sending time T t , calculate the actual sending start time of the neighbor node:

[0077]

[0078] Then compare this time with your preset sending time Compare and get the relative clock offset:

[0079]

[0080] In the case where each node has multiple neighbors, this embodiment sets the time offset range [-T cycle ,T cycle ] is discretely divided into N equally spaced time buckets, each of which corresponds to an executable time slot delay adjustment action:

[0081]

[0082] in, Indicates the delay step represented by a single time bucket.

[0083] After the node performs the selected delay adjustment action and completes the transmission, it counts the neighbors that meet |Δ i -T delay |≤T g The number of nodes, which is used as the immediate reward r t , used to evaluate the effect of the current alignment strategy. Here, T delay is the delay value adjusted this time, T g is the maximum allowed alignment error. By controlling the sending time difference of one-hop neighbors to T g Within the range, it effectively reduces time slot overlap and reception conflicts, and improves the system's concurrent communication efficiency.

[0084] Reinforcement learning is used to achieve alignment optimization. The core process includes:

[0085] A1: Set state s t is the current neighbor time offset distribution (expressed as a vector of the number of neighbors in the time bucket);

[0086] A2: Select action a t , corresponding to the delay adjustment bucket number of the current round;

[0087] A3: Calculate the instant reward a t , that is, the number of neighbor nodes that meet the alignment conditions;

[0088] A2: According to the next state s t+1 Maximum Q value Estimating future rewards to facilitate long-term optimal slot alignment.

[0089] The state-action value function Q value update follows the Bellman equation:

[0090]

[0091] Among them, α is the learning rate and γ is the discount factor.

[0092] In order to improve the accuracy of adjustment, a neighbor weight mechanism is introduced. The weight w is assigned according to the neighbor node type. i , giving priority to aligning with ground nodes that have stable time synchronization capabilities. When receiving data packets from ground nodes, they are given higher weights to guide learning to prioritize synchronization with them. The immediate reward is defined as the weighted sum:

[0093]

[0094] Where 1(·) is an indicator function that reflects whether the corresponding neighbors meet the alignment condition.

[0095] As an implementation method, the method of this embodiment adopts an intelligent routing mechanism based on neighbor information and link status to achieve efficient and balanced data packet forwarding. Each node in the network maintains a neighbor table and a link layer load table to support routing and scheduling decisions. The neighbor table records static information such as the IP address, node type, geographic location, remaining energy and scheduling category of the neighbor node; the link layer load table is used to dynamically record the number of data packets that have not yet been sent by this node and its neighbors at the link layer. Since the link layer adopts the time division multiplexing (TDMA) mechanism, the node can only transmit in the allocated time slot, resulting in some data packets needing to be queued in the link layer cache, so it is necessary to maintain the load information to reflect the real-time communication pressure of each node.

[0096] When sending a data packet, a node writes its remaining energy and link-layer cache status into the packet header. Neighboring nodes passively monitor received packets, analyze the energy and load information contained in them, and dynamically update their local neighbor table and link-layer load table accordingly. This enables continuous awareness of network status without the need for additional control signaling, significantly reducing protocol overhead.

[0097] The routing decision-making mechanism of the method in this embodiment includes the following steps:

[0098] S1: When the source node generates a data packet and prepares to send it, it checks whether the destination node exists in the neighbor table. If so, it directly forwards the data packet to the destination node, completing the data transmission;

[0099] S2. If the destination node does not exist in the neighbor table, check whether there is a neighbor node belonging to the water subnet (such as a surface buoy node or a satellite node). If so, the data packet is forwarded to the water subnet node first;

[0100] S3. If there is neither the destination node nor the water subnet node in the neighbor table, the node calculates the forwarding factor based on the geographical distance between the neighbor node and the destination node, the link load factor, and the remaining energy;

[0101] S4. Sort neighbor nodes according to forwarding factors, and select several neighbors with better forwarding factor values ​​to form a candidate set;

[0102] S5. Perform energy screening on neighbor nodes in the candidate set and eliminate candidates whose remaining energy is significantly lower than that of other nodes.

[0103] S6. Finally, the neighbor node with the best forwarding factor and satisfying energy requirements is selected from the filtered candidate set as the next hop node, and the data packet is forwarded.

[0104] S7. Repeat the above process until the data packet reaches the destination node.

[0105] In step S3, the node calculates the forwarding factor F for the candidate neighbor. The specific calculation formula is as follows:

[0106] F=α·d+(1-α)·Q

[0107] Where d represents the normalized vertical distance of the neighbor node relative to the line connecting the current node and the destination node, reflecting the geometric rationality of the path; Q represents the link load factor, which is the larger value of the number of packets queued from the current node to the neighbor node and the number of packets queued to the next hop node from the neighbor node, reflecting the link congestion; α∈[0,1] is the weight coefficient, which is used to balance the priority of the shortest path and the lightest load.

[0108] In summary, the present invention has the following advantages and beneficial effects compared to the prior art:

[0109] (1) In terms of time slot scheduling, the present invention adopts a spatially multiplexed TDMA mechanism, combined with a passive time slot alignment strategy and a gap filling mechanism based on reinforcement learning. Without the need for global clock synchronization, the transmission timing is dynamically adjusted through local node monitoring and learning, achieving local alignment and concurrent scheduling of multiple nodes, significantly improving channel utilization and transmission efficiency while reducing communication delay.

[0110] (2) In terms of routing design, the present invention constructs an intelligent routing mechanism that integrates neighbor information and link load perception, dynamically calculates the forwarding factor by considering multiple factors such as node energy consumption, link congestion and geographical location, and assists nodes in selecting the optimal path, thereby achieving energy consumption balance and load sharing, extending network life, improving overall robustness, and adapting to the communication needs in large-scale complex ocean monitoring environments.

[0111] (3) The present invention improves channel utilization and data transmission efficiency; effectively reduces communication delay; realizes dynamic time slot alignment, reducing dependence on global time synchronization; achieves energy consumption balance and load distribution through intelligent routing mechanism, and extends network life; adapts to large-scale and complex marine environments, and has good robustness and scalability.

[0112] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0113] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made based on the essence of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A marine monitoring network communication system based on reinforcement learning, characterized in that: include: The underwater subnet consists of multiple underwater nodes arranged in a cellular grid topology to achieve coverage of the ocean area; The water subnet is composed of satellite nodes and surface buoy nodes. The surface buoy nodes relay data transmission through satellite nodes. The satellite nodes transmit data to and from the surface buoy nodes via radio waves. The surface buoy nodes are capable of both radio and acoustic wave communication, and are responsible for implementing cross-media information transmission in the marine heterogeneous network. The underwater nodes communicate with each other via acoustic waves. At the link layer, nodes communicate through a time-division multiple access mechanism with spatial multiplexing, and a time slot alignment mechanism based on reinforcement learning is constructed to achieve local time slot alignment and concurrent scheduling optimization.

2. The ocean monitoring network communication system based on reinforcement learning according to claim 1, characterized in that: Each node has a neighbor table and a link layer load table. The neighbor table records the IP address, node type, location information, remaining energy and classification identification of adjacent nodes. The link layer load table records the cache status of the link between the node and each neighbor node. When sending a data packet, a node adds its own remaining energy and link cache information to the packet header. Neighboring nodes passively monitor the received packet header information and dynamically update the energy status and load information of neighboring nodes in the neighbor table and link layer load table without introducing additional control signaling.

3. The ocean monitoring network communication system based on reinforcement learning according to claim 1, characterized in that: The ocean monitoring network communication system preferentially uses the water subnet link when forwarding data packets; When a source node generates and sends a data packet, or a relay node participates in routing forwarding, the following steps are performed: First, determine whether the destination node exists in its neighbor table. If so, forward the data packet directly to the destination node. If there is no destination node in the neighbor table, it is further determined whether there is a neighbor node belonging to the water subnet, including a surface buoy node or a satellite node. If so, the data packet is forwarded to this type of node first; If there is neither the destination node nor the surface subnet node in the neighbor table, the data packet will continue to be forwarded in the underwater subnet.

4. The ocean monitoring network communication system based on reinforcement learning according to claim 1, characterized in that: The underwater subnet adopts a cellular grid topology. The nodes in the underwater subnet are classified into seven cells, each comprising a single cell. Nodes within each cell are assigned different scheduling categories based on their spatial location within the cell. This allows for different node categories within the same cell, thus achieving spatial reuse scheduling. At the link layer, a time division multiple access method with spatial multiplexing is used to schedule the transmission time slots of nodes. Based on the divided scheduling categories, 21 different pairs of communication nodes are constructed as vertices in the graph, and a conflict-free time slot scheduling path is generated through a graph traversal algorithm.

5. The ocean monitoring network communication system based on reinforcement learning according to claim 1, characterized in that: Based on the gap filling technology, multiple data transmission tasks are inserted during the propagation delay, and the time slot length is set to the sum of the data packet transmission delay and the protection interval, that is, T s =T t +T g , where T s is the time slot length, T t is the data packet transmission delay, T g For protection interval.

6. The ocean monitoring network communication system based on reinforcement learning according to claim 1, characterized in that: The time slot alignment mechanism includes: A node calculates its own transmission start time by monitoring the reception time and propagation delay of neighboring node data packets. The time offset range is discretized into N time buckets, each bucket corresponding to a delay adjustment action. The node uses the Q-learning algorithm to select the optimal delay adjustment value so that as many one-hop neighbors as possible meet the time slot alignment condition, thereby improving the efficiency of concurrent communication.

7. The ocean monitoring network communication system based on reinforcement learning according to claim 6, characterized in that: The node selects an optimal delay adjustment value using a Q-learning algorithm, including: The node uses the Q-learning learning strategy to select an optimal delay adjustment action a according to the current state after each round of communication. t , and give an immediate reward r according to the number of neighbors that are successfully aligned t ; This reward is used to update the Q value table: In the formula, α is the learning rate, γ is the discount factor, and s t is the current state, a t is the current action, r t For immediate rewards; Through the above learning process, the node can gradually adjust its own transmission timing to align the transmission time slots of more one-hop neighbors, thereby reducing the probability of collision and improving the concurrent transmission capability of the link layer and overall communication efficiency.

8. The ocean monitoring network communication system based on reinforcement learning according to claim 7, characterized in that: In the reinforcement learning algorithm, in order to accelerate the dynamic alignment efficiency and effect of the network, the network regards the surface buoy node as a synchronized node and gives it a higher weight priority, so the immediate reward r t The definition is as follows: Where 1(·) is the indicator function, Δ i is the time offset of neighbor node i, T delay is the current delay adjustment action, T g is the guard interval, which is also the maximum alignment error.

9. A method for ocean monitoring network communication based on reinforcement learning, applied to the ocean monitoring network communication system according to any one of claims 1 to 8, characterized in that: The following steps are involved: When the source node generates a data packet and prepares to send it, it checks whether the destination node exists in the neighbor table; If it exists, the data packet is directly forwarded to the destination node to complete the data transmission; If the destination node does not exist in the neighbor table, check whether there is a neighbor node belonging to the water subnet; If it exists, the data packet will be forwarded to the water subnet node first; If there is neither the destination node nor the water subnet node in the neighbor table, the node calculates the forwarding factor based on the geographical distance between the neighbor node and the destination node, the link load factor, and the remaining energy; Sort neighbor nodes according to forwarding factors, and select several neighbors with better forwarding factor values ​​to form a candidate set; Perform energy screening on neighbor nodes in the candidate set and eliminate candidates whose remaining energy is significantly lower than that of other nodes; From the filtered candidate set, the neighbor node with the best forwarding factor and satisfying energy conditions is selected as the next hop node to forward the data packet.

10. The ocean monitoring network communication method based on reinforcement learning according to claim 9, characterized in that: The forwarding factor is calculated as follows: F=α·d+(1-α)·Q Where d represents the normalized vertical distance of the neighbor node relative to the line connecting the current node and the destination node, Q represents the link load factor, and α is the weight coefficient.

Images